[BUG] Setuptools 69.0.3 `setup.py develop` generates a .egg-link file with underscores in the name component

[ichard26]

setuptools version

69.0.3

Python version

CPython 3.11.7

OS

Ubuntu 22.04.03 LTS

Additional environment information

No response

Description

Happy New Year! 🎉

I was debugging pip's CI which has been red ever since the release of Setuptools 69.0.3. Most of them are easily fixed by removing the underscore -> dash normalization assumption, however, there is one place where leaving the underscores intact causes issues: legacy editable installs.

As documented in the Setuptools documentation, the distribution name part of egg filenames should be normalized by safe_name():

The “name” and “version” should be escaped using the to_filename() function provided by pkg_resources, after first processing them with safe_name() and safe_version() respectively. These latter two functions can also be used to later “unescape” these parts of the filename. (For a detailed description of these transformations, please see the “Parsing Utilities” section of the pkg_resources manual.)

Setuptools does not honour this.¹ This is actually fine in most situations as far as I can tell since there are modern ways for pip to discover installed distributions that don't rely on eggs, but setup.py develop does not generate this modern metadata. Thus, pip falls back to searching sys.path for a .egg-link file to determine whether a distribution is editably installed. Pip assumes the egg link name will be normalized by safe_name() so this logic returns a false negative despite version_pkg being editably installed in fact.

If I'm being honest, I have no idea whose problem this is, but this does mean for projects that do not implement PEP 518 and have underscores in their name will not be recognized as editably installed. If this is better transferred to the pip repository, please let me know!

Expected behavior

See above.

How to Reproduce

0. Create an environment with pip<24, setuptools==69.0.3 and wheel.
1. Create a new directory for testing and create a setup.py representing a version_pkg:

# setup.py
import setuptools; setuptools.setup(name="version_pkg")

2. Run pip install -e . in the directory
3. Run pip freeze or pip list and observe that pip doesn't realize it's an editable install
4. Observe that the egg-link file has a underscore
5. Rerun this with Setuptools 69.0.2 and observe that pip can recognize that version_pkg is installed as an editable

Output

ichard26@asus-ubuntu:~/dev/oss/pip/temp/pip-test-package$ pip install setuptools==69.0.3 -q
ichard26@asus-ubuntu:~/dev/oss/pip/temp/pip-test-package$ python ../generate.py 
ichard26@asus-ubuntu:~/dev/oss/pip/temp/pip-test-package$ pip install -e .
Obtaining file:///home/ichard26/dev/oss/pip/temp/pip-test-package
  Preparing metadata (setup.py) ... done
Installing collected packages: version_pkg
  Running setup.py develop for version_pkg
Successfully installed version_pkg-0.1
ichard26@asus-ubuntu:~/dev/oss/pip/temp/pip-test-package$ pip freeze
version_pkg==0.1
ichard26@asus-ubuntu:~/dev/oss/pip/temp/pip-test-package$ pip list
Package     Version
----------- -------
pip         23.3.2
setuptools  69.0.3
version_pkg 0.1
wheel       0.42.0
ichard26@asus-ubuntu:~/dev/oss/pip/temp/pip-test-package$ ls -gha ~/dev/oss/pip/venv/lib/python3.11/site-packages/ | grep "version"
-rw-rw-r--  1 ichard26   50 Jan  1 16:56 version_pkg.egg-link

Footnotes

1. Well, if you read the documentation closely, apparently you're supposed to pass the made safe name to pkg_resources.to_filename() which would turn the dash back into an underscore but setuptools does not seem to do this anyway /shrug ↩

[matthuisman]

there is some issue with 69.0.3 and pip and editable installs

Python 3.10.9
Pip 23.3.2

Directory name: sys_testcases
Package name in setup.py: sys_testcases
pip install -e .

pip list

(its not shown as editable)

If I change the package name to sys-testcases - it works as expected

[abravalheri]

Hi @ichard26, thank you very much for reporting the issue.

If I had to guess, I would say this is probably related to the change in #4159, which was motivated by #2522.

Probably a previous implementation was optimised to not need a second pass of to_filename, but once safe_name was modified, the circumstances changed and the observed behaviour deviated from the docs (an oversight that never got flagged out and fixed).

We could do an extra pass of to_filename after safe_name for the .egg-link file to be compliant with the doc, but as you pointed out yourself it would not change anything and the filename would still have an underscore in the test case you are analysing... Would that be the acceptable "new normal" for pip? Should we favour instead "behaviour from 69.0.2" over "documentation"?

@jaraco what is your opinion on what is the best approach here?

[ichard26]

We essentially have three options:

1. Update setuptools to maintain pre v69.0.3 naming behaviour -> pip remains unchanged
2. Update setuptools to honour its documentation -> pip still needs an update
3. Only update pip to look for both pre-v69.0.3 and post-v69.0.3 egg-link files -> setuptools remains unchanged

No. 1 would lead to the least fallout and is my preferred option, but I have zero stake in this so I'll defer to setuptools and pip developers to decide on the right path forward.

[sbidoul]

@jaraco @abravalheri is there any chance you can revert to the previous way of naming .egg-link files (assuming this is technically feasible)? This broke pip freeze but also pip uninstall of legacy editable installs. I'm painfully coming up with a pip update but this change is nevertheless quite disruptive and in a legacy area that I'd prefer to leave alone.

[sbidoul]

The pip change is in pypa/pip#12477. The modified egg-link file detection is in src/pip/_internal/utils/egg_link.py.

[ablatner]

My project's constraint autoupdater is affected by this. After bumping setuptools to 69.0.3, subsequent updates with pip list -e don't recognize editable installs with underscores. For now, we can pin setuptools to 69.0.2, but I'm curious what we should do going forwards.

@sbidoul, will the next release of pip include pypa/pip#12477, and will that provide compatibility with setuptools 69.0.3?

[sbidoul]

will the next release of pip include pypa/pip#12477, and will that provide compatibility with setuptools 69.0.3?

@ablatner yes, it is merged so it will be in the next pip release.

[jaraco]

Apologies for being late to the conversation. It's been a rough couple of months for me personally. I got almost no open source work done during the winter break (where I usually have my most productive period).

Here is the design principles I'd like to advocate for:

1. Most importantly, valid characters in the user's indicated package name should be retained in the metadata.
2. Any conversion (normalization/mangling) that occurs should be based on a standard and implemented in a re-usable function (like safe_name).
3. The naming on disk should be relatively stable and predictable. Unit tests should protect this expectation.

is there any chance you can revert to the previous way of naming .egg-link files (assuming this is technically feasible)?

Yes, probably, and that would honor (3). I think that's what @abravalheri was suggesting.

It was unintentional that the name of the metadata path on disk changed. I'm slightly surprised that the test suite didn't manifest failures for this missed expectation.

In #4159, I did propose that the behavior could be rolled back if it caused unexpected disruption, which it did, to be followed by a more involved approach. That's still an option.

@sbidoul Given the work on pip, what would you like to see from setuptools at this point?