[BUG] `setuptools==77.0.1` fails to import if `packaging<24.2` is installed

[jakelishman]

setuptools version

77.0.1

Python version

Tested on 3.9 and 3.13

OS

macOS

Additional environment information

No response

Description

In an environment which has a manually installed version of packaging less than the implicit requirement in setuptools of 24.2, import setuptools fails while attempting to import packaging.licenses.

setuptools vendors packaging at 24.2 and does not declare a separate dependency on it with no optionals specified. If the environment already has packaging<24 installed in it, pip install -U setuptools 'packaging<24.2' (or the like) will produce what pip considers to be a valid environment, but setuptools will import a version of packaging it is incompatible with.

This can have a knock-on effect causing some packages (I haven't tracked down the full set of criteria needed) to fail with an incorrect error message, such as (pyperclip==1.9.0 is the example I saw in our CI):

 Collecting pyperclip>=1.8 (from cmd2>=1.0.0->cliff>=2.8.0->stestr>=3.2.0->qiskit-neko==0.0.1)
  Downloading pyperclip-1.9.0.tar.gz (20 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
      [end of output]

Actually, setuptools==77.0.1 is installed, it's packaging.licenses that fails to import (since it's pulling from the out-of-date version of packaging, not the vendored one), but the raised ModuleNotFoundError is treated by pip as being that setuptools is absent.

Expected behavior

The import should succeed.

How to Reproduce

1. Create a new virtual environment, seeded with pip install -U setuptools 'packaging<24.2'. No error is thrown during package resolution because setuptools without an optional does not depend on packaging (due to the vendoring).
2. Attempt to import setuptools

Output

>>> import setuptools
Traceback (most recent call last):
  File "<python-input-0>", line 1, in <module>
    import setuptools
  File "/Users/jake/tmp/setuptools-repro/venv313/lib/python3.13/site-packages/setuptools/__init__.py", line 27, in <module>
    from .dist import Distribution
  File "/Users/jake/tmp/setuptools-repro/venv313/lib/python3.13/site-packages/setuptools/dist.py", line 15, in <module>
    from packaging.licenses import canonicalize_license_expression
ModuleNotFoundError: No module named 'packaging.licenses'

[jakelishman]

By the way, I believe this to be the root cause of the off-topic discussion in #4892, such as #4892 (comment).

[abravalheri]

Thanks @jakelishman.

This is a limitation of the current setuptools vendoring system. I don't think we can do much about it other than detecting the import error and raising an exception asking the users to install a more up-to-date version of packaging.

[jakelishman]

Thanks for the quick reply. I think the main way users will see this error is through the pip failure mode that says:

ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.

so I guess potentially a patch to pip that causes it to show the full exception message from the failed import setuptools would be the first step necessary?

As a couple of heuristic measures, are either of these suitable?

1. make the packaging.licences handling fallible, so building packages that do not need to touch the new licensing handling to build can succeed, even in the awkward build environment
2. modify the vendoring system to use custom metapath loaders that first query the available version of a given package, and reject it from the import path if its version is too low? That could be a way of letting setuptools skip over a bad packaging version during its import.

I think there's an awful lot of mess in option 2 (if not only: there's a bootstrapping problem with detecting the package versions), though I think some variant on that is potentially workable. I suspect we'd end up needing some logic such that if any vendored package isn't imported from the "natural" location, it's instead stored in something like setuptools._imports.packaging and not added to sys.modules under the packaging name, to avoid other packages in the environment getting unexpected dependencies.

That's not pretty at all, so if there's a better solution, I'd be interested.

[abravalheri]

Thank you very much @jakelishman, I am working on #4898 for option 1.
There have been many attempts of dealing with the challenging vendoring process, and usually if you fix one thing you end up breaking another... So for now I think I will stay away from that one 😅.

[KayTeeFive]

Oh god,
@jakelishman thanks for amazing information.
I've also opened bug at the same time: [BUG] python setup.py egg_info did not run successfully #4895
That WA works:

pip install packaging==24.2

[jakelishman]

No worries. If it helps anybody else, I made some full repro steps for an environment that produces this failure mode over on pip, where we're discussing passing through the exception raised by setuptools. See pypa/pip#13290 (comment) for that.