[BUG] Some packages no longer install their requirements.

[dwoz]

setuptools version

setuptools==68.2.0

Python version

>= 3.8

OS

All OSes

Additional environment information

No response

Description

Packages that change Distribution.install_requires after Distribution._finalize_options is called fail to install requirements.

The Salt codebase has, for a long time, modified Distribution.install_requires in an overridden Distribution.parse_command_line method in it's setup.py.

The changes in #4029 specifically 3c9d6ac96b breaks installing of requirements on current and previous Salt releases (see saltstack/salt#64149).

This change will also affect any other packages that make the assumption install_requires can be modified after the point which _finalize_options get's called in parse_config_files.

Expected behavior

Ideally, preserve previous behavior on order to not break existing package installations.

We, Salt, can fix this issue for future versions but it's a bummer all of our existing versions now fail to install properly.

How to Reproduce

Create a virtual environment. Pip install salt.

Output

This behavior can be observed by pip installing the latest Salt release.

$ pip3 install salt==3006.3
Collecting salt==3006.3
  Using cached salt-3006.3-py3-none-any.whl
Installing collected packages: salt
Successfully installed salt-3006.3

[ilya-lavrenov]

We have similar issue:

• setuptools earlier 68.2.0

Installation log

  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Requirement already satisfied: networkx<=3.1 in /agent/_work/_tool/Python/3.11.2/x64/lib/python3.11/site-packages (from openvino-dev) (3.1)
Requirement already satisfied: numpy>=1.16.6 in /agent/_work/_tool/Python/3.11.2/x64/lib/python3.11/site-packages (from openvino-dev) (1.23.5)
Collecting opencv-python (from openvino-dev)
  Obtaining dependency information for opencv-python from https://files.pythonhosted.org/packages/f5/d0/2e455d894ec0d6527e662ad55e70c04f421ad83a6fd0a54c3dd73c411282/opencv_python-4.8.0.76-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Downloading opencv_python-4.8.0.76-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB)
Requirement already satisfied: openvino-telemetry>=2022.1.0 in /agent/_work/_tool/Python/3.11.2/x64/lib/python3.11/site-packages (from openvino-dev) (2023.1.1)
Requirement already satisfied: pillow>=8.1.2 in /agent/_work/_tool/Python/3.11.2/x64/lib/python3.11/site-packages (from openvino-dev) (10.0.0)
Collecting pyyaml>=5.4.1 (from openvino-dev)
  Obtaining dependency information for pyyaml>=5.4.1 from https://files.pythonhosted.org/packages/7b/5e/efd033ab7199a0b2044dab3b9f7a4f6670e6a52c089de572e928d2873b06/PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Downloading PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
Requirement already satisfied: requests>=2.25.1 in /agent/_work/_tool/Python/3.11.2/x64/lib/python3.11/site-packages (from openvino-dev) (2.31.0)
Collecting texttable>=1.6.3 (from openvino-dev)
  Downloading texttable-1.6.7-py2.py3-none-any.whl (10 kB)
Collecting tqdm>=4.54.1 (from openvino-dev)
  Obtaining dependency information for tqdm>=4.54.1 from https://files.pythonhosted.org/packages/00/e5/f12a80907d0884e6dff9c16d0c0114d81b8cd07dc3ae54c5e962cc83037e/tqdm-4.66.1-py3-none-any.whl.metadata
  Downloading tqdm-4.66.1-py3-none-any.whl.metadata (57 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.6/57.6 kB 10.1 MB/s eta 0:00:00
Processing /agent/_work/1/_w/install_pkg/tools/openvino-2023.2.0-12330-cp311-cp311-manylinux_2_31_x86_64.whl (from openvino-dev)
Collecting scipy<1.11,>=1.8 (from openvino-dev)
  Downloading scipy-1.10.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.1/34.1 MB 77.0 MB/s eta 0:00:00
Requirement already satisfied: charset-normalizer<4,>=2 in /agent/_work/_tool/Python/3.11.2/x64/lib/python3.11/site-packages (from requests>=2.25.1->openvino-dev) (3.2.0)
Requirement already satisfied: idna<4,>=2.5 in /agent/_work/_tool/Python/3.11.2/x64/lib/python3.11/site-packages (from requests>=2.25.1->openvino-dev) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /agent/_work/_tool/Python/3.11.2/x64/lib/python3.11/site-packages (from requests>=2.25.1->openvino-dev) (1.26.16)
Requirement already satisfied: certifi>=2017.4.17 in /agent/_work/_tool/Python/3.11.2/x64/lib/python3.11/site-packages (from requests>=2.25.1->openvino-dev) (2023.7.22)
Downloading PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (757 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 757.7/757.7 kB 94.4 MB/s eta 0:00:00
Downloading tqdm-4.66.1-py3-none-any.whl (78 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.3/78.3 kB 38.0 MB/s eta 0:00:00
Downloading opencv_python-4.8.0.76-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.7/61.7 MB 52.8 MB/s eta 0:00:00
Building wheels for collected packages: jstyleson
  Building wheel for jstyleson (setup.py): started
  Building wheel for jstyleson (setup.py): finished with status 'done'
  Created wheel for jstyleson: filename=jstyleson-0.0.2-py3-none-any.whl size=2384 sha256=96a786fcd8366c20a88055535b1be1462018b70bd7a0ec90ae3ba2025fc57b4e
  Stored in directory: /home/AzDevOps/.cache/pip/wheels/ad/63/0e/50090147fb424100f7d9078b71c21b9e7468b6f643515a60d6
Successfully built jstyleson
Installing collected packages: texttable, jstyleson, addict, tqdm, scipy, pyyaml, openvino, opencv-python, openvino-dev
  Attempting uninstall: scipy
    Found existing installation: scipy 1.11.2
    Uninstalling scipy-1.11.2:
      Successfully uninstalled scipy-1.11.2
Successfully installed addict-2.4.0 jstyleson-0.0.2 opencv-python-4.8.0.76 openvino-2023.2.0 openvino-dev-2023.2.0 pyyaml-6.0.1 scipy-1.10.1 texttable-1.6.7 tqdm-4.66.1

• with setuptools 68.2.0:

Installation log

Processing /agent/_work/1/_w/install_pkg/tools/openvino_dev-2023.2.0-12331-py3-none-any.whl
Installing collected packages: openvino-dev
Successfully installed openvino-dev-2023.2.0

[abravalheri]

Hi @dwoz, @ilya-lavrenov, could you please try #4043 and see if that solves your problem?

You can test that by installing the version of setuptools given in the PR + wheel in a virtual environment and then running python -m build –no-isolation (from that environment, bonus points for testing pip install –-no-build-isolation .). Something like the following (please revise, I wrote from memory):

python3.11 -m venv .venv
.venv/bin/python -m pip install -U 'pip>=23.2' 'wheel>=0.41.2' 'build>=1.0.3'
.venv/bin/python -m pip install https://github.com/abravalheri/setuptools/archive/refs/heads/issue-4042.zip
.venv/bin/python -m build --no-isolation .
.venv/bin/python -m pip install dist/*.whl
.venv/bin/python -m pip install –-no-build-isolation .

---

This change will also affect any other packages that make the assumption install_requires can be modified after the point which _finalize_options get's called in parse_config_files.

This assumption was never fully safe, even before setuptools<=68.1.0. Maybe the assumption was working for a subset of specific situations and values of install_requires, but not in general.

The reason being that setuptools was already using _finalize_requires to post-process install_requires, extras_require and provides_extra metadata in order to produce .egg-info directories and "ok-ish" PKG-INFO files to be used in the sdists. Note that the files in the .egg-info directory predate wheels and core metadata standardisation, so the file format is different from the ones you find in a .whl file.

If a customisation modifies these attributes after _finalize_requires has been called, the necessary post-processing is not done. Chances are the files in .egg-info and PKG-INFO may deviate from the file format expected by setuptools, being prone to potential parsing errors.

That may or not interfere with to the way pip parses/uses those files too.
I say "may or not", because for the wheel files, bdist_wheel (which is implemented in pypa/wheel, not in pypa/setuptools) regenerates the METADATA file and discards .egg-info/requires.txt. That may "mascarade" eventual problems in PKG-INFO and .egg-info/requires.txt.

[ilya-lavrenov]

.venv/bin/python -m pip install https://github.com/abravalheri/setuptools/archive/refs/heads/issue-4042.zip

Local experiment with this package showed that issue is resolved.
We are using python3 -m pip wheel . in our scripts.

@abravalheri
Could you please mark package on pypi as yanked? In this case it will not be downloaded by users.

[dwoz]

Hi @dwoz, @ilya-lavrenov, could you please try #4043 and see if that solves your problem?

You can test that by installing the version of setuptools given in the PR + wheel in a virtual environment and then running python -m build –no-isolation (from that environment, bonus points for testing pip install –-no-build-isolation .). Something like the following (please revise, I wrote from memory):

python3.11 -m venv .venv
.venv/bin/python -m pip install -U 'pip>=23.2' 'wheel>=0.41.2' 'build>=1.0.3'
.venv/bin/python -m pip install https://github.com/abravalheri/setuptools/archive/refs/heads/issue-4042.zip
.venv/bin/python -m build --no-isolation .
.venv/bin/python -m pip install dist/*.whl
.venv/bin/python -m pip install –-no-build-isolation .

This does indeed resolve the issue for Salt. I validated with the following:

testit/bin/pip3 install 'pip>=23.2' 'wheel>=0.41.2' 'build>=1.0.3'
testit/bin/python -m pip install https://github.com/abravalheri/setuptools/archive/refs/heads/issue-4042.zip
./testit/bin/pip install --no-build-isolation --no-cache salt -v

This assumption was never fully safe, even before setuptools<=68.1.0. Maybe the assumption was working for a subset of specific situations and values of install_requires, but not in general.

The reason being that setuptools was already using _finalize_requires to post-process install_requires, extras_require and provides_extra metadata in order to produce .egg-info directories and "ok-ish" PKG-INFO files to be used in the sdists. Note that the files in the .egg-info directory predate wheels and core metadata standardisation, so the file format is different from the ones you find in a .whl file.

This makes perfect sense and is clear based on how _finalize_requires works. We've had plans to fix the way we handle this in salt and hope to address it in the next major release. Having said that, if we can keep older versions working as expected with this proposed fix it will really help.

If a customisation modifies these attributes after _finalize_requires has been called, the necessary post-processing is not done. Chances are the files in .egg-info and PKG-INFO may deviate from the file format expected by setuptools, being prone to potential parsing errors.

That may or not interfere with to the way pip parses/uses those files too. I say "may or not", because for the wheel files, bdist_wheel (which is implemented in pypa/wheel, not in pypa/setuptools) regenerates the METADATA file and discards .egg-info/requires.txt. That may "mascarade" eventual problems in PKG-INFO and .egg-info/requires.txt.

Thanks, this all makes sense as well. For us, I think a better way to handle this would have been to make our modifications in Distribution.__init__ long before _finalize_options is called. However, we're looking at moving away from setup.py completely in favor of pyproject.toml.

[abravalheri]

Thank you very much for trying out. A new version should be out soon (once the CI finishes running).

[ilya-lavrenov]

Thanks @abravalheri

Could you please mark package on pypi as yanked? In this case it will not be downloaded by users.

Can we do it?
I suppose in this case even explicit python -m pip install setuptools==68.2.0 will not work?

[abravalheri]

Now that the new package is in PyPI, I don't see a strong reason for yanking¹ (also because this was not a generalised bug, just an oversight in the way the customisation was implemented for some specific packages²).

Footnotes

1. Anyone installing the package without a version range will simply get the latest version, which now is not subject to the problem described here. ↩

2. See explanation why the customisation approach was never fully safe in https://github.com/pypa/setuptools/issues/4042#issuecomment-1712053117. ↩

[dwoz]

@abravalheri Immense thanks for tackling this so quickly~!