Consider creating "exclusion list" to fall-back to the old resolver automatically

[potiuk]

What did you want to do?

I would love that PIP maintainers consider quickly adding an exclusion list to the released PIP version, to fall-back to old resolver for some of the projects for which it is known that the new resolver causes problems, or when particular features are used that the new resolver is not ready for yet.

Such a list could be rather small. From the comment here It seems that for vast majority of the project have no problems with the new resolver, but there are some - for example Apache Airflow, but likely quite a few other projects in the category of platforms/tools that have problems with the installation during the new resolver that seem to be difficult to solve quickly. Some of the issues are described in those: #9187, #9223, #9215, #9214, #9203, #9186 . Using any of the problematic projects as a direct installation target or (if possible and straightforward) would automatically trigger the legacy mode for PIP (which could be overwritten for testing fixes with another flag). This could be accompanied by a warning message that explains why the new resolver is used and that there is an on-going effort to fix it.

Also, there are some features that are broken in the new resolver: #9229, #9219, #9217, #9216, #9190, #9188 , #9182, #9180. and using any of the features (maybe impossible for all those, but maybe straightforward for some, like using proxy) could automatically trigger legacy resolver, again including a warning message stating that that problem will get fixed in the upcoming version but for now, PIP is falling back to the old resolver).

Additional information

The current state of resolver for some projects is seriously problematic (in short - new resolver does not work for them), but there seems to be a rather small list of such projects and features that are broken, and it could be rather easy to keep such a list updated. It could for example be kept remotely at the PIP GitHub site as part of the PIP repository, and it could be (for a time being) remotely refreshed by PIP from time to time to be able to keep adding new projects to the list, if/when new problems are reported. PIP - by its nature requires the network to function, so refreshing such a list from a well-known, PIP-maintainer controlled location should be possible and could be attempted (with quick falling back to the latest cached version of the list) every time a pip install/update etc. command is run.

There was a proposal raised by my in this comment to withdraw the new resolver until the problem is solved, but after the comment here it is rather clear that despite the severity of the problems, they seem to be isolated to a rather small group of projects and features. And there is a big value in as many projects trying the new PIP resolver by default as possible. On the other hand, some of the projects are suffering because their users are unaware of the PIP problems, and no matter how hard those projects try (for example here apache/airflow#12838) their users will get a perception, that those projects are broken, rather than PIP.

Having such an exclusion list might be the best of both worlds. On one hand, it might give the projects so needed stability of installation, on the other hand, this might give the PIP team a necessary testing ground for the new resolver - taking into account that it works in the majority of cases and there are only a handful of exceptions.

It would also give the PIP team the necessary breathing space to fix all such issues properly and without having maintainers of such projects expressing their frustration and feelings - this would give an easy to maintain workaround for anyone whose problems are severe enough that makes it impossible to workaround in a different way.

Another good side of that - having such list of "problematic" problems, could give the PIP team to build relationships with those "heavy users" of PIP that could be helpful in testing any future features of PIP. This would be a great opportunity to turn the "problem" into an "opportunity" of building better cooperation between some open-souirce projects.

Please kindly consider my proposal.

Also - if you would consider that, I am happy to help with the implementation. I do not know PIP internals. but maybe it ain't so difficult. Happy to spend quite some time on it if you want to work on more important stuff.

[uranusjr]

I understand the rationale of excluding cases because they are known to not work well, but wouldn’t think proposal only kick the can down the road until it becomes a problem again? I don’t imagine Airflow can drop dependencies since they are needed for its functionalities, but pip also cannot keep maintaining the old resolver indefinitely so that some packages can stay out of the migration. If there’s one thing the beta rollout phase shows to me, in Airflow’s case, is that Airflow users do not use the new resolver unless we force them to, and disabling the new resolver on Airflow would only keep them in the pre-20.3 era, and come back when things inevitably break again. How can this be prevented?

The tension between package maintainers and pip maintainers is mutual. We regularly need to deal with “pip does not work on package X” reports. The problem here is not the new resolver needs to backtrack, and the legacy resolver does not make the root cause go away, but only make it pop up in some other places, that users would blame pip for, instead of the package they want to install. This is the main reason pip maintainers want to avoid going back to the legacy resolver. The problem has always been there, but 20.3 finally surfaces it to the package maintainers, instead of putting it on pip’s issue tracker.

Summarising to make my position explicit. I do not object the proposal to disable the new resolver temporarily on Airflow, but the projects being disabled must provide a concrete plan to improve things. The issues reported against the new resolver are the best source the pip maintainers have to identify issues in the new resolver (and to isolate things that are not resolver issues; many references you listed above are exactly that), and a proposal to disable the new resolver must provide an alternative method to coming up with examples like this, so the new resolver can really be improved while it’s disabled.

[potiuk]

I understand the rationale of excluding cases because they are known to not work well, but wouldn’t think proposal only kick the can down the road until it becomes a problem again? I don’t imagine Airflow can drop dependencies since they are needed for its functionalities, but pip also cannot keep maintaining the old resolver indefinitely so that some packages can stay out of the migration. If there’s one thing the beta rollout phase shows to me, in Airflow’s case, is that Airflow users do not use the new resolver unless we force them to, and disabling the new resolver on Airflow would only keep them in the pre-20.3 era, and come back when things inevitably break again. How can this be prevented?

As I wrote above - it is just a matter of establishing close collaboration with the maintainers of the "excluded" packages. We can help you with testing the fixes and even maybe contribute some !

Airflow maintainers are already deeply vested into making PIP 20.3 works for us. The way our test harness works is that when we make it works for us - it will start working for our users, so no user involvement is needed.

As an example: We already have the issue opened apache/airflow#12838 that will remain opened until all problems are fixed. The first one identified in #9203 has already turned into PR: #9226 but already some of our maintainers (@kaxil) tested it and i turns out that there are more problems (which other reported as well so they will be eventually fixed).

If we have a new flag (--force-new-resolver or something) we can even make a scheduled run on our CI that will continue running and failing until the problems gets fixed - and then, when PIP with fixes gets released, we will ask you to remove the exclusion. Seems like a workable plan to me.

[potiuk]

And undoubtedly this will be the same for other projects - in order to make it to the list - the maintainers of such package would have to come to you and ask for it - at the same time pledging their efforts to test and disable when the problem is fixed. That sounds like a win-win situation. Also such exclusion could have an expire date and it would require and active "please we need more time" with good justification from the maintainers to extend it.

[pfmoore]

One concern I have about this proposal is that I'm not actually sure how it would work in practice. Pip can't use different resolvers for different "parts" of an install, so we need to work out how to choose the resolver for any given install. And we can't know what packages are used in an install until we run the resolver (because of dependencies, some of which might not be possible to calculate until install time). So there's a chicken-and-egg issue here - if the user issues the command pip install XXX, should pip use the new resolver? If XXX changes to depend on airflow, should the answer to that question change? How do we know? If the user has to tell us, how is that different from the user disabling the new resolver manually, which they can do right now?

I'm not against a mechanism like this in principle, but in practice I have no idea how to design such a thing, and the level of detail in the proposal here leaves far too many questions unanswered. I'm concerned that if we try to work through those questions (like I've just done in the previous paragraph) you might see that as "pip's developers are continually blocking proposals to solve the issue" - it's not the intention, but it would be easy to get that feeling.

[uranusjr]

As I wrote above - it is just a matter of establishing close collaboration with the maintainers of the "excluded" packages. We can help you with testing the fixes and even maybe contribute some!

That’s what I’m trying to ask. We can only fix things if we know what to fix, and we need user reports to know what are not working well. “The resolver takes too long” is not directly actionable because it’s not caused by a logical error, but a lack of optimisation. We need to look into different issues and find a pattern, so we can optimise the algorithm to avoid/reduce the phenomenon. Once we create the exclusion rule, the reports would stop coming in, and you will need to find another way to generate test cases for the fix. Do you have a solution?

[potiuk]

One concern I have about this proposal is that I'm not actually sure how it would work in practice. Pip can't use different resolvers for different "parts" of an install, so we need to work out how to choose the resolver for any given install. And we can't know what packages are used in an install until we run the resolver (because of dependencies, some of which might not be possible to calculate until install time). So there's a chicken-and-egg issue here - if the user issues the command pip install XXX, should pip use the new resolver? If XXX changes to depend on airflow, should the answer to that question change? How do we know? If the user has to tell us, how is that different from the user disabling the new resolver manually, which they can do right now?

Let's make it simple and straightforward, and limit it only to packages that are directly invoked by the user in PIP command. No dependencies. That would solve probably all issues of our users which otherwise would land in the "cannot install airflow" camp. Otherwise - if airflow is used as a dependency, it would still be installed with the new resolver, and you will get issues reported by people using airflow as dependency. Which is what you want I understand. Win-win again.

That’s what I’m trying to ask. We can only fix things if we know what to fix, and we need user reports to know what are not working well. “The resolver takes too long” is not directly actionable because it’s not caused by a logical error, but a lack of optimisation. We need to look into different issues and find a pattern, so we can optimise the algorithm to avoid/reduce the phenomenon. Once we create the exclusion rule, the reports would stop coming in, and you will need to find another way to generate test cases for the fix. Do you have a solution?

Yes. Our CI system. When you exclude airflow, and add capability to force the resolver, we pledge to run the tests continuously and let you know about the problems. Our CI is done in the way that it install the most complex combination of extras you can imagine and keeps the constraints consistent automatically. Also We can produce a number of "typical" combinations and run our CI with those. And we can have timeout for the test to test the "too long" case. And you could either subscribe to the result yourself or we will observe it and let you know when the problems are fixed.

And anyone else who you add to the list will have to do similar thing and make sure the tests are running for their packages.

[potiuk]

And BTW - this is not a new concept by any means. Microsoft does it for years and provides different rules for applications installed on Windows OS. Linux kernel has numerous exclusions for different kinds of drivers and has some special tweaks. And here - you have maintainers of projects, ready to help you and cooperate. Let's break the wall "pip maintainers vs. projects that use PIP" and let's cooperate on this, empathetically understanding the other part needs and expectations, especially that it can cost very little if we limit the mechanism as described above.

[pradyunsg]

At a high level: I'm a strong -1 to pip maintaining a "list of packages that need special casing". Such a list will never cover all the "special cases" out in the wild (eg: we're not going to hold a list of packages in $corp's internal index that can't be used with a certain feature in pip, same applies to packages on PyPI as well) [note 1]. pip doesn't handle numpy differently from $corp-cricital-package and I'd much prefer that we maintain that consistency.

From #9187 (comment):

I'm saying users need to change their workflow to adapt to a change made upstream toward being stricter. We'll work to address the slowness/takes-forever behavior here, but that doesn't mean it'll magically solve the underlying issue that's causing the problems for these users: that there are dependency conflicts in their dependency graph that are non-trivial to resolve.

Again, the underlying issue here is NOT that pip's using the new resolver -- that's either surfacing the fact that airflow has dependency conflicts in its dependency graph that are making the resolution process take longer than ideal, or that there's a bug in the resolver. If you use a lockfile to deterministically generate the dependency graph or provide a dependency specification that limits the extent of the resolution to something more reasonable, that'll address the underlying issue.

[note 1]: "make that list configurable" -- no, because then you might as well configure pip to use the behavior you want instead. :)

Microsoft does it for years and provides different rules for applications installed on Windows OS.

Microsoft is one of the largest corporations in the world, and likely hires more people for maintaining their HQ building, than we have devs volunteering to maintain pip. Similar resource level differences apply for the Linux Kernel analog. :)

---

So, I just went ahead and tried to understand what the issue is for the Airflow project. IIUC, you are already using constraint files to limit which versions are used with your application:

pip install "apache-airflow[postgres,google]==1.10.13" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.13/constraints-3.7.txt" 

This fails. I think I see the underlying bug you're hitting that's causing issues for you:

Collecting google-cloud-bigquery[bqstorage,pandas]<3.0.0dev,>=1.11.1
  Using cached google_cloud_bigquery-2.5.0-py2.py3-none-any.whl (210 kB)
  Using cached google_cloud_bigquery-2.4.0-py2.py3-none-any.whl (210 kB)
  Using cached google_cloud_bigquery-2.3.1-py2.py3-none-any.whl (208 kB)
  Using cached google_cloud_bigquery-2.2.0-py2.py3-none-any.whl (200 kB)
  Using cached google_cloud_bigquery-2.1.0-py2.py3-none-any.whl (190 kB)
  Using cached google_cloud_bigquery-2.0.0-py2.py3-none-any.whl (193 kB)

Despite the constraint file having google-cloud-bigquery==2.3.1. The extra-based requirement is not being properly constrainted by the constraint file.

PS: TIL pip allows remote URLs are requirements and constraints?! WTF.

[pradyunsg]

The extra-based requirement is not being properly constrainted by the constraint file.

This is a bug in the new resolver, and I'd rather we fix that instead of adding mechanisms (as proposed in OP) to work around the issue.

[potiuk]

Yes. Thanks for looking at it @pradyunsg.This was our original request in #9203. If you wait long enough the installation will crash - this is yet another problem we faced (PR to that is already created #9226 ).

If you can solve those before our planned releases - this is fantastic.

But we could not get information (@kaxil asked for it when we can expect this to be solved). Can you - at least provisionally give us indication if this is days or weeks, or months if you cannot tell a concrete date? My proposal is only valid, if you are not able to solve those issues in days to come. So please, any indication of how long it will take will be helpful.

BTW. I am happy to test any pre-release versions directly from github to test if the problem is solved. So let me know as soon as possible. I will even be happy to open another issue for that if you think it is necessary to split thsoe two isses (hopefully there are no more issues like that).

BTW. 2. We are using fully consistent constraints in 1.10.14rc (I worked on it specifically to test the new resolver before it was out) - the 1.10.13 had still dependency conflicts which I resolved while trying out the new resolver, as I suspected the conflicts could have been caused by that. This command tests all dependencies with fully consistent set of constraints:

UPDATE: I created the issue for the unnecessary downloading of the dependencies additionally to #9203 to keep those issues separated: Issue in #9232 .

Do you think the problems can be solved next week please?

[potiuk]

This is a bug in the new resolver, and I'd rather we fix that instead of adding mechanisms (as proposed in OP) to work around the issue.

Please @pradyunsg let us know your feelings when we can expect fix to this error raised in #9233