[BUG] v76.1.0 broke pywin32 builds

[Avasam]

setuptools version

setuptools>=76.1.0

Python version

All

OS

Windows (x86, x64 and cross-compiled arm64)

Additional environment information

I'd love to provide a proper MRE, but this only happens on GitHub's CI... building works fine on any recent setuptools version locally... Which also makes it quite the puzzle to bissect.

I made sure to clear pip's cache, and delete any build, dist or pywin32.egg-info folder I had in my local repo.

As additional information, @mhammond mentioned that: mhammond/pywin32#2493 (comment)

[...] this is generated from the .mc file - iirc, distutils etc handled this, although there is an ominous comment here

    # Work around bpo-36302/bpo-42009 - it sorts sources but this breaks
    # support for building .mc files etc :(
    def compile(self, sources, **kwargs):
        # re-sort the list of source files but ensure all .mc files come first.
        def key_reverse_mc(a):
            b, e = os.path.splitext(a)
            e = "" if e == ".mc" else e
            return (e, b)

        sources = sorted(sources, key=key_reverse_mc)
        return MSVCCompiler.compile(self, sources, **kwargs)

(as a sidenote, if whatever hack pywin32 has to do here was handled by setuptools, that'd get it much closer to stop using distutils and MSVCCompiler directly)

Description

Since setuptools v76.1, pywin32's CI builds fail.

I tried specifically with v76.1, v77.0.3 and v88. All the same error.

Expected behavior

A successful build.

How to Reproduce

1. Clone https://github.com/mhammond/pywin32 or start from this PR: Fix setuptools 76.1 issue (and bump to >=77.0.3) mhammond/pywin32#2587
2. Make sure to set setuptools>=76.1 in pyproject.toml:

   [build-system]
   # setuptools==75.4 dropped support for Python 3.8
   # setuptools==76.1 breaks building of .mc files
   # win32/src/PythonService.cpp(49): fatal error C1083: Cannot open include file: 'PythonServiceMessages.h': No such file or directory
   requires = [
     "setuptools >=77.0.3; python_version >='3.9'", 
     "setuptools <76.1; python_version <'3.9'",
   ]
   build-backend = "setuptools.build_meta"

   or (and ignore Python 3.8 failures)

   [build-system]
   # setuptools==76.1 breaks building of .mc files
   # win32/src/PythonService.cpp(49): fatal error C1083: Cannot open include file: 'PythonServiceMessages.h': No such file or directory
   requires = ["setuptools >=77.0.3"]
   build-backend = "setuptools.build_meta"

3. See the CI fail.

Output

https://github.com/mhammond/pywin32/actions/runs/14719773942/job/41311171599?pr=2587#step:5:799

  win32/src/PythonService.cpp(49): fatal error C1083: Cannot open include file: 'PythonServiceMessages.h': No such file or directory
  -- distutils hack to expose all include & lib dirs
  -- orig compiler.include_dirs: ['C:\\hostedtoolcache\\windows\\Python\\3.9.13\\x64\\include', 'C:\\hostedtoolcache\\windows\\Python\\3.9.13\\x64\\Include']
  -- orig compiler.library_dirs: ['C:\\hostedtoolcache\\windows\\Python\\3.9.13\\x64\\libs', 'C:\\hostedtoolcache\\windows\\Python\\3.9.13\\x64', 'C:\\hostedtoolcache\\windows\\Python\\3.9.13\\x64\\PCbuild\\amd64', 'build\\temp.win-amd64-cpython-39\\Release']
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
  error: subprocess-exited-with-error
  
  Building wheel for pywin32 (pyproject.toml) did not run successfully.
  exit code: 1
  
  See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: 'C:\hostedtoolcache\windows\Python\3.9.13\x64\python.exe' 'C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py' build_wheel 'C:\Users\RUNNER~1\AppData\Local\Temp\tmp7gl02_fu'
  cwd: D:\a\pywin32\pywin32
  Building wheel for pywin32 (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for pywin32
Failed to build pywin32

Notice:  A new release of pip is available: 25.0.1 -> 25.1
Notice:  To update, run: python.exe -m pip install --upgrade pip
ERROR: Failed to build installable wheels for some pyproject.toml based projects (pywin32)

[Avasam]

One immediately noticeable difference between my local environment and the CI is that I'm using VS Build Tools 2022 on Windows 10, and the CI is using whatever comes with the windows-2019 agent.

[Avasam]

Progress: I found why it wasn't failing locally: the generated .h and .cpp files are not cleaned after a build. So PythonServiceMessages.h wasn't missing. Running git clean win32 -f -X cleaned up enough I can start testing.

I also find that in setuptools>=76.1, pywin32's my_compiler.compile is never run (easy to do by ensuring it always raises). So the re-ordering it depends on, is also never done.

By removing pyproject definitions and using pip install . --no-build-isolation, I'm able to test specific setuptools commits by installing them with pip uninstall -y setuptools && pip install git+https://github.com/pypa/setuptools.git@<commit_number>

[Avasam]

I've bisected the failing commit to be aeefe34

• aeefe34 is the first that has the behaviour of not calling my_compiler.compile

• b6a539a, 408274b, & 6c0427b fails with (see error below)

  ERROR: Exception:
  Traceback (most recent call last):
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\cli\base_command.py", line 105, in _run_wrapper
      status = _inner_run()
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\cli\base_command.py", line 96, in _inner_run
      return self.run(options, args)
             ~~~~~~~~^^^^^^^^^^^^^^^
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\cli\req_command.py", line 68, in wrapper
      return func(self, options, args)
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\commands\install.py", line 387, in run
      requirement_set = resolver.resolve(
          reqs, check_supported_wheels=not options.target_dir
      )
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\resolution\resolvelib\resolver.py", line 77, in resolve
      collected = self.factory.collect_root_requirements(root_reqs)
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 545, in collect_root_requirements
      reqs = list(
          self._make_requirements_from_install_req(
      ...<2 lines>...
          )
      )
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 501, in _make_requirements_from_install_req
      cand = self._make_base_candidate_from_link(
          ireq.link,
      ...<2 lines>...
          version=None,
      )
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 233, in _make_base_candidate_from_link
      self._link_candidate_cache[link] = LinkCandidate(
                                         ~~~~~~~~~~~~~^
          link,
          ^^^^^
      ...<3 lines>...
          version=version,
          ^^^^^^^^^^^^^^^^
      )
      ^
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 306, in __init__
      super().__init__(
      ~~~~~~~~~~~~~~~~^
          link=link,
          ^^^^^^^^^^
      ...<4 lines>...
          version=version,
          ^^^^^^^^^^^^^^^^
      )
      ^
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 159, in __init__
      self.dist = self._prepare()
                  ~~~~~~~~~~~~~^^
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 236, in _prepare
      dist = self._prepare_distribution()
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 317, in _prepare_distribution
      return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\operations\prepare.py", line 532, in prepare_linked_requirement
      return self._prepare_linked_requirement(req, parallel_builds)
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\operations\prepare.py", line 647, in _prepare_linked_requirement
      dist = _get_prepared_distribution(
          req,
      ...<3 lines>...
          self.check_build_deps,
      )
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\operations\prepare.py", line 71, in _get_prepared_distribution
      abstract_dist.prepare_distribution_metadata(
      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
          finder, build_isolation, check_build_deps
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      )
      ^
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\distributions\sdist.py", line 69, in prepare_distribution_metadata
      self.req.prepare_metadata()
      ~~~~~~~~~~~~~~~~~~~~~~~~~^^
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\req\req_install.py", line 575, in prepare_metadata
      self.metadata_directory = generate_metadata(
                                ~~~~~~~~~~~~~~~~~^
          build_env=self.build_env,
          ^^^^^^^^^^^^^^^^^^^^^^^^^
          backend=self.pep517_backend,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          details=details,
          ^^^^^^^^^^^^^^^^
      )
      ^
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\operations\build\metadata.py", line 34, in generate_metadata
      distinfo_dir = backend.prepare_metadata_for_build_wheel(metadata_dir)
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\utils\misc.py", line 723, in prepare_metadata_for_build_wheel
      return super().prepare_metadata_for_build_wheel(
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
          metadata_directory=metadata_directory,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          config_settings=cs,
          ^^^^^^^^^^^^^^^^^^^
          _allow_fallback=_allow_fallback,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      )
      ^
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_vendor\pyproject_hooks\_impl.py", line 224, in prepare_metadata_for_build_wheel
      return self._call_hook(
             ~~~~~~~~~~~~~~~^
          "prepare_metadata_for_build_wheel",
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      ...<4 lines>...
          },
          ^^
      )
      ^
    File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_vendor\pyproject_hooks\_impl.py", line 402, in _call_hook
      raise BackendUnavailable(
      ...<4 lines>...
      )
  pip._vendor.pyproject_hooks._impl.BackendUnavailable: Cannot import 'setuptools.build_meta'

• a36b7ba works fine

So it sounds like it would be a distutils change ?

Oh and to help debugging, here's the traceback to the "breakpoint Exception" I put in pywin32, to try and see why that code is no longer called in >=76

Traceback (most recent call last):
  File "<string>", line 2, in <module>
    exec(compile('''
    ~~~~^^^^^^^^^^^^
    # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<32 lines>...
    exec(compile(setup_py_code, filename, "exec"))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ''' % ('E:\\Users\\Avasam\\Documents\\Git\\pywin32\\setup.py',), "<pip-setuptools-caller>", "exec"))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<pip-setuptools-caller>", line 35, in <module>
  File "E:\Users\Avasam\Documents\Git\pywin32\setup.py", line 2038, in <module>
    dist = setup(
        name="pywin32",
    ...<114 lines>...
        ],
    )
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\setuptools\__init__.py", line 117, in setup
    return distutils.core.setup(**attrs)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\setuptools\_distutils\core.py", line 186, in setup
    return run_commands(dist)
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\setuptools\_distutils\core.py", line 202, in run_commands
    dist.run_commands()
    ~~~~~~~~~~~~~~~~~^^
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\setuptools\_distutils\dist.py", line 1002, in run_commands
    self.run_command(cmd)
    ~~~~~~~~~~~~~~~~^^^^^
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\setuptools\dist.py", line 999, in run_command
    super().run_command(command)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\setuptools\_distutils\dist.py", line 1021, in run_command
    cmd_obj.run()
    ~~~~~~~~~~~^^
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\setuptools\command\bdist_wheel.py", line 369, in run
    self.run_command("build")
    ~~~~~~~~~~~~~~~~^^^^^^^^^
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\setuptools\_distutils\cmd.py", line 357, in run_command
    self.distribution.run_command(command)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\setuptools\dist.py", line 999, in run_command
    super().run_command(command)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\setuptools\_distutils\dist.py", line 1021, in run_command
    cmd_obj.run()
    ~~~~~~~~~~~^^
  File "E:\Users\Avasam\Documents\Git\pywin32\setup.py", line 354, in run
    build.run(self)
    ~~~~~~~~~^^^^^^
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\setuptools\_distutils\command\build.py", line 140, in run
    self.run_command(cmd_name)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\setuptools\_distutils\cmd.py", line 357, in run_command
    self.distribution.run_command(command)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\setuptools\dist.py", line 999, in run_command
    super().run_command(command)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\setuptools\_distutils\dist.py", line 1021, in run_command
    cmd_obj.run()
    ~~~~~~~~~~~^^
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\setuptools\command\build_ext.py", line 99, in run
    _build_ext.run(self)
    ~~~~~~~~~~~~~~^^^^^^
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 375, in run
    self.build_extensions()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "E:\Users\Avasam\Documents\Git\pywin32\setup.py", line 565, in build_extensions
    self.build_extension(ext)
    ~~~~~~~~~~~~~~~~~~~~^^^^^
  File "E:\Users\Avasam\Documents\Git\pywin32\setup.py", line 711, in build_extension
    build_ext.build_extension(self, ext)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\setuptools\command\build_ext.py", line 264, in build_extension
    _build_ext.build_extension(self, ext)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python313\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 572, in build_extension
    objects = self.compiler.compile(
        sources,
    ...<5 lines>...
        depends=ext.depends,
    )
  File "E:\Users\Avasam\Documents\Git\pywin32\setup.py", line 926, in compile
    raise Exception("breakpoint")
Exception: breakpoint

[Avasam]

Oh, even better, I found this code is never called. Which makes a lot more sense:

from distutils import ccompiler

def my_new_compiler(**kw):
    raise Exception("Is this ever called?")  # This is never hit in setuptools>=76.1
    if "compiler" in kw and kw["compiler"] in (None, "msvc"):
        return my_compiler()
    return orig_new_compiler(**kw)

# No way to cleanly wedge our compiler sub-class in.
orig_new_compiler = ccompiler.new_compiler
ccompiler.new_compiler = my_new_compiler  # type: ignore[assignment] # Assuming the caller will always use only kwargs

[Avasam]

I ended up being able to fix my issue by delaying an import to make sure the function override happens before it's imported/used by setuptools. mhammond/pywin32#2587

I assume the change in distutils that broke that was actually the move of compilers.

Anyway, this isn't for setuptools to "fix". Other than by considering the feature request of #2806