New resolver reports version conflict on github repo URL variations

[fiendish]

The resolver appears to treat projectname @ git+https://git@github.com/org/repo.git as different from projectname @ git+https://github.com/org/repo.git even though they're the same thing and the only difference is the "git@" in the url. If something is required directly and indirectly (via another requirement) using slightly different but totally valid URL variations, pip fails with a version conflict error.

pip install -r requirements.txt on a requirements.txt containing:

project-abc @ git+https://github.com/org/project-abc.git
project-def @ git+https://github.com/org/project-def.git

Where project-def has a requirements.txt containing the "git@" URL variant:

project-abc @ git+https://git@github.com/org/project-abc.git

Produces

ERROR: Cannot install -r requirements.txt (line 2) ... because these package
versions have conflicting dependencies.

The conflict is caused by:
    The user requested project-abc 0.1.0 (from git+https://github.com/org/project-abc.git)
    project-def 0.1.2 depends on project-abc 0.1.0 (from git+https://****@github.com/org/project-abc.git)

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

Removing the "user@" from the second requirements.txt avoids this, but it worked before the resolver upgrade and may require changes to packages upstream from the user.

Expected Behavior

Working URL variations for the same package should not affect version resolution.

Environment details

pip 20.3.1
macOS Mojave 10.14.6
Python 3.8.6

[fiendish]

Simplified demo using a random real python repo and no dependency chaining:

$ pip install git+https://github.com/fiendish/xlrd git+https://git@github.com/fiendish/xlrd

Collecting git+https://github.com/fiendish/xlrd
  Cloning https://github.com/fiendish/xlrd to /private/var/folders/7r/zzgsf_917vn98xy97_274d24z3zxz5/T/pip-req-build-z3za5_mk
Collecting git+https://****@github.com/fiendish/xlrd
  Cloning https://****@github.com/fiendish/xlrd to /private/var/folders/7r/zzgsf_917vn98xy97_274d24z3zxz5/T/pip-req-build-6x7m6gbw
ERROR: Cannot install xlrd 1.2.0 (from git+https://****@github.com/fiendish/xlrd) and xlrd 1.2.0 (from git+https://github.com/fiendish/xlrd) because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested xlrd 1.2.0 (from git+https://github.com/fiendish/xlrd)
    The user requested xlrd 1.2.0 (from git+https://****@github.com/fiendish/xlrd)

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

[uranusjr]

I believe this is expected; two URLs with different log in information are treated as different sources since they are technically different URLs, and can return entirely different contents in theory. pip did not actually check this previously (even if the URLs entirely differ), and simply picked the first URL it seems and ignore the second one. The new resolver is stricter on this and require the URLs to be equivalent.

[fiendish]

can return entirely different contents in theory

Except that they didn't. Pip should know this because the conflict message appears after both are cloned.

two URLs with different log in information are treated as different sources since they are technically different URLs

Also I'm pretty sure that's not how github works?

[uranusjr]

A clone is not sufficient, since Git’s clone is not reproducible. pip would need to actually build the package to tell whether two package contents are indeed the same, and even that has problems due to technicaly details, e.g. #5648. So pip opts the strictest route, and reject different URLs (ideally pip should not even clone, but that’s an implementation quirk that should be improved).

Also I'm pretty sure that's not how github works?

It is how GitHub works for humans now, but pip does not know that. pip does not contain information specific to a given index (and should not IMO); so the user needs to help it fill these gaps.

[fiendish]

A clone is not sufficient, since Git’s clone is not reproducible.

Can you explain this part? If I clone both versions to adjacent locations, and then diff shows the cloned contents are identical, which is what's happening here, then the clones are the same even if by some quirk of the phase of the moon or neutrino interference they would result in different builds. What scenario is this protecting?

so the user needs to help it fill these gaps.

This puts a weird burden on unrelated distributed projects to agree to all coordinate to use the same URL format, which may not even be possible, in order to prevent something that seems in my ignorance to be slightly far fetched.

[uranusjr]

Can you explain this part? If I clone both versions to adjacent locations, and then diff shows the cloned contents are identical, which is what's happening here, then they're the same.

IIUC the content of .git is subject to change and not reliable. That is usually not a part of a package not thus not relevant, but again pip does not know that, which is why a build step would be needed.

This puts a weird burden on unrelated distributed projects to agree to all coordinate to use the same URL format.

And on the flip side, not having that burden on you means more burden to us, because we get to maintain the functionality that satisfies your convenience 🙂

[fiendish]

the content of .git is subject to change

Modification times or something else? Do you have a resource where I can learn about what kinds of changes we're talking about? I'm only able to find references for modification times or changes between different computers (seemingly not relevant here).

[fiendish]

(I feel like I should also add that the formulation where local and remote requirements.txt files used different URL format for the same project leads to the seemingly infinite install looping behavior already reported elsewhere. I didn't originally want to put it in this report because it's already on your radar, but now I'm second guessing that.)

[blaiseli]

I think I just experienced a variation on this issue while re-building a singularity container of mine that starts by updating pip, then git clones a repo that contains a requirements.txt file and tries to use it:

ERROR: Cannot install -r requirements.txt (line 52) and libworkflows 0.3 (from git+https://gitlab.pasteur.fr/bli/libworkflows.git@2b10cc3e5ab61284853463d2950a157295ad7f16) because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested libworkflows 0.3 (from git+https://gitlab.pasteur.fr/bli/libworkflows.git@2b10cc3e5ab61284853463d2950a157295ad7f16)
    libhts 0.3 depends on libworkflows 0.3 (from git+https://gitlab.pasteur.fr/bli/libworkflows.git)

My requirements.txt pulls libhts, which has libworkflows as a dependency, and also libworkflows directly.

The indirect dependency is specified in the install_requires section of the setup.py of libhts, without a version, just the git repo address.

The direct dependency is specified with a version constraint in requirements.txt.
The specified version is the last commit in master, so it is actually the same as the indirect dependency. In any case, it would make sense that the same git repo URL, except with no commit hash specified, should be considered as compatible with the repo URL containing the extra commit hash.

[blaiseli]

I'm trying to figure out how I could solve my issue with spurious incompatibilities between the same package installed via git but one via requirements.txt and the other via the install_requires section of a setup.py.

I was wondering whether I would have to remove the git commit hash from the requirements.txt file or add it in the install_requires section of the setup.py of the package that depends on it. However, after reading https://packaging.python.org/discussions/install-requires-vs-requirements/, it seems that the recommended good practice is to pin specific versions in requirements.txt but not in the install_requires section of a setup.py.

@uranusjr Is there a way I can follow the above recommendations, while being able to use the new resolver?

[uranusjr]

From pip's perspective, any URL spec is pinning the requirement. So it does not really matter; by using a URL is setup.py, you're already pinning.

[blaiseli]

Wouldn't it possible to modify pip so that it at least considers that when the urls just differ by presence / absence of a commit hash, the one without the commit hash should be considered as satisfied by the one with the commit hash?