What does a URL constraint mean?

[pradyunsg]

Carrying forward discussion from #8210

---

To discuss about a constraints.txt file with the following content:

packaging @ git+https://github.com/pypa/packaging@20.1

0. Should this be allowed?
1. What functionality does this provide?
2. Is this the best way to provide it?

[uranusjr]

To me, this means whenever packaging is requested, use this link I provided. Further version specifiers should still apply (and fail resolution if they are not satisfied by the link target). The behaviour would be similar to

pip install <packages-i-want> "packaging @ git+https://github.com/pypa/packaging@20.1"

but without packaging being marked as user_requested (unless it is listed in <packages-i-want>, of course).

[dhellmann]

I think of it as constraining the location from which distributions can be downloaded, just as a version expression like >=20.1 would constrain the version(s) that could be downloaded.

[pfmoore]

OK. I have done a more detailed investigation into the way that we'd implement NAME @ URL style constraints. The idea in principle is straightforward, but it has a number of interactions with other features of pip that make the overall behaviour far more complex.

The current implementation will have behaviours for all of these points that arise "by accident" as a result of the implementation choice to treat constraints as "just requirements that we don't install". The new resolver doesn't have that option, because the underlying model is completely different.

It's also true that there are likely reasonable choices to be made as to "how should this interaction work". It's not necessarily hard to choose a "correct" behaviour for any of the following points. But the implementation is decidedly non-trivial in the new resolver's existing model, so the cost of these features is disproportionately high, and we don't really have any idea at the moment if there's any actual use cases for these sorts of interactions (that's actual use cases as opposed to "yes I can see how this might be useful" 🙂)

• Multiple URLs - should they be allowed? What would they mean? After all we can only get a project from one place. But consider this in relation to the next points.
• Hash checking. If someone were to have a constraint file that said foo @ https://some/url and a requirements file that said foo --hash=xxx, what should happen? Should we hash-check the URL? Is it of any practical use? Should we select one of multiple URL constraints based on the hash?
• Wheel compatibility tags. Should URLs for constraints be required to match the target wheel compatibility rules? Should we allow multiple URLs and pick the one that matches the target system? Have we just implemented a package index via constraint URLs?
• Should we reject the edge cases that we don't support, or are we OK with just leaving them as "undefined behaviour - do not use"? In particular, even detecting some of these cases is costly.

There are quite likely other questions. These are basically just some that came up in the process of trying to see how to implement this and saying "ouch, that would be a problem" a few times 🙂

As I said, it's almost certainly possible to come up with a well defined spec for this feature. And maybe that spec would even match the current behaviour. But we don't have that spec right now, and no-one has really done any of the work needed to come up with one.

So I propose that we consider URLs as constraints as "deprecated pending work being done to properly define the feature"¹, and assume that they won't be supported in at least the initial version of the new resolver. People currently using URLs in constraint files will need to either stick to the old resolver, or find a workaround.

The only actual user of URLs as constraints that I am aware of is @sbidoul, who explained his use here. Maybe there are (more or less clumsy) workarounds that could be used instead? @dhellmann seemed OK here with not including this in the new resolver, at least initially.

To reiterate, this is not a blanket refusal to accept this feature. It's simply noting that we need to clarify the design in order to rewrite it for the new resolver, and that we're not blocking the release of the new resolver on this feature.

¹ This has horrible echoes of the dependency links mess. I really want to not have that happen again, so I want to be very clear that the plan this time is to drop the feature without any replacement if necessary, and not to leave it hanging around forever "until we have the replacement defined".

[sbidoul]

@pfmoore don't the edge case you mention also arise when direct URLs are used as regular install-requires?

[pfmoore]

@sbidoul I have no idea to be honest. I've not spotted any test failures that do that, but if there are any, then yes, that's something we'd need to address (and having addressed it, we may well then have a better chance of implementing URLs as constraints).

[uranusjr]

OTOH I very much doubt there are robust tests or user experience on this. The legacy resolver handles non-user-specified direct URLs very poorly, so there are likely many edge cases currently precluded by other restrictions. I’m +1 on delaying thoughts on this until we are able to make the new resolver widely available for a while, to let the users come up with crazy cool use cases that expose considerations we missed.

[pfmoore]

Agreed, this whole area is really emergent behaviour from a set of features that were implemented independently and never really considered in combination.

This is probably something that needs to be covered under #6536 (new resolver rollout) which is something that needs to be prioritised at some point. There will be areas where the new resolver works differently, that either aren't covered by tests, or where we've made a decision to keep a difference. and we need to look at how we get feedback on that.