New resolver: pip seems to not do anything and stuck during resolution

[noamkush]

Environment

• pip 20.2 (also tried master)
• Python 3.6

On a clean environment, running pip install --use-feature=2020-resolver -r requirements.txt

boto3>=1.14.33
elasticsearch-dsl>=7.1,<7.2
elasticsearch>=7.0,<8.0
user-agents>=1.1.0
requests_mock>=1.6
awsebcli>=3.18

The installation is stuck indefinitely, repeating the last output, see log. I kept it running for 5+ minutes but it keeps going (unreasonable for such a short list of requirements). I cut the log short since it repeats itself.

Collecting boto3>=1.14.33
  Using cached boto3-1.14.33-py2.py3-none-any.whl (129 kB)
Collecting elasticsearch-dsl<7.2,>=7.1
  Using cached elasticsearch_dsl-7.1.0-py2.py3-none-any.whl (53 kB)
Collecting botocore<1.18.0,>=1.17.33
  Using cached botocore-1.17.33-py2.py3-none-any.whl (6.5 MB)
Processing /home/noam/.cache/pip/wheels/99/f3/df/12f8953cef0e59c251c6e9cf7816d202e8e778a3a928a645a6/user_agents-2.1-cp36-none-any.whl
Collecting ua-parser>=0.9.0
  Using cached ua_parser-0.10.0-py2.py3-none-any.whl (35 kB)
Collecting requests_mock>=1.6
  Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Processing /home/noam/.cache/pip/wheels/c2/3a/2f/3c725041250659af00a69f4fd3e211f69a1638e4ad79701fc2/awsebcli-3.18.2-py3-none-any.whl
Requirement already satisfied: setuptools>=20.0 in /home/noam/.virtualenvs/tmp-a6fdd377fc9345b0/lib/python3.6/site-packages (from awsebcli>=3.18->-r requirements.txt (line 6)) (49.2.0)
Processing /home/noam/.cache/pip/wheels/22/23/ff/a2333c06c7cee314937b0e24e46386d070082213ca940e71c6/awsebcli-3.18.1-cp36-none-any.whl
Requirement already satisfied: setuptools>=20.0 in /home/noam/.virtualenvs/tmp-a6fdd377fc9345b0/lib/python3.6/site-packages (from awsebcli>=3.18->-r requirements.txt (line 6)) (49.2.0)
Collecting awsebcli>=3.18
  Using cached awsebcli-3.18.0.tar.gz (249 kB)
Collecting requests_mock>=1.6
  Using cached requests_mock-1.7.0-py2.py3-none-any.whl (23 kB)
Requirement already satisfied: setuptools>=20.0 in /home/noam/.virtualenvs/tmp-a6fdd377fc9345b0/lib/python3.6/site-packages (from awsebcli>=3.18->-r requirements.txt (line 6)) (49.2.0)
Requirement already satisfied: setuptools>=20.0 in /home/noam/.virtualenvs/tmp-a6fdd377fc9345b0/lib/python3.6/site-packages (from awsebcli>=3.18->-r requirements.txt (line 6)) (49.2.0)
Collecting requests_mock>=1.6
  Using cached requests_mock-1.6.0-py2.py3-none-any.whl (22 kB)
Requirement already satisfied: setuptools>=20.0 in /home/noam/.virtualenvs/tmp-a6fdd377fc9345b0/lib/python3.6/site-packages (from awsebcli>=3.18->-r requirements.txt (line 6)) (49.2.0)
Requirement already satisfied: setuptools>=20.0 in /home/noam/.virtualenvs/tmp-a6fdd377fc9345b0/lib/python3.6/site-packages (from awsebcli>=3.18->-r requirements.txt (line 6)) (49.2.0)
Requirement already satisfied: setuptools>=20.0 in /home/noam/.virtualenvs/tmp-a6fdd377fc9345b0/lib/python3.6/site-packages (from awsebcli>=3.18->-r requirements.txt (line 6)) (49.2.0)
Requirement already satisfied: setuptools>=20.0 in /home/noam/.virtualenvs/tmp-a6fdd377fc9345b0/lib/python3.6/site-packages (from awsebcli>=3.18->-r requirements.txt (line 6)) (49.2.0)
Requirement already satisfied: setuptools>=20.0 in /home/noam/.virtualenvs/tmp-a6fdd377fc9345b0/lib/python3.6/site-packages (from awsebcli>=3.18->-r requirements.txt (line 6)) (49.2.0)
Requirement already satisfied: setuptools>=20.0 in /home/noam/.virtualenvs/tmp-a6fdd377fc9345b0/lib/python3.6/site-packages (from awsebcli>=3.18->-r requirements.txt (line 6)) (49.2.0)
Requirement already satisfied: setuptools>=20.0 in /home/noam/.virtualenvs/tmp-a6fdd377fc9345b0/lib/python3.6/site-packages (from awsebcli>=3.18->-r requirements.txt (line 6)) (49.2.0)
Requirement already satisfied: setuptools>=20.0 in /home/noam/.virtualenvs/tmp-a6fdd377fc9345b0/lib/python3.6/site-packages (from awsebcli>=3.18->-r requirements.txt (line 6)) (49.2.0)
Requirement already satisfied: setuptools>=20.0 in /home/noam/.virtualenvs/tmp-a6fdd377fc9345b0/lib/python3.6/site-packages (from awsebcli>=3.18->-r requirements.txt (line 6)) (49.2.0)
Requirement already satisfied: setuptools>=20.0 in /home/noam/.virtualenvs/tmp-a6fdd377fc9345b0/lib/python3.6/site-packages (from awsebcli>=3.18->-r requirements.txt (line 6)) (49.2.0)
Requirement already satisfied: setuptools>=20.0 in /home/noam/.virtualenvs/tmp-a6fdd377fc9345b0/lib/python3.6/site-packages (from awsebcli>=3.18->-r requirements.txt (line 6)) (49.2.0)
...

[deveshks]

An additional observation. Just using two requirements out of the ones mentions causes a ResolutionImpossible error.

$ cat requirements.txt 
boto3>=1.14.33
awsebcli>=3.18

$ pip install --use-feature=2020-resolver -r requirements.txt 
Collecting boto3>=1.14.33
  Using cached boto3-1.14.33-py2.py3-none-any.whl (129 kB)
Collecting botocore<1.18.0,>=1.17.33
  Using cached botocore-1.17.33-py2.py3-none-any.whl (6.5 MB)
Collecting awsebcli>=3.18
  Using cached awsebcli-3.18.2.tar.gz (246 kB)
Collecting awsebcli>=3.18
  Using cached awsebcli-3.18.1.tar.gz (246 kB)
Collecting awsebcli>=3.18
  Using cached awsebcli-3.18.0.tar.gz (249 kB)
ERROR: Cannot install -r requirements.txt (line 1), -r requirements.txt (line 2), -r requirements.txt (line 1), -r requirements.txt (line 2), -r requirements.txt (line 1) and -r requirements.txt (line 2) because these package versions have conflicting dependencies.

The conflict is caused by:
    boto3 1.14.33 depends on botocore<1.18.0 and >=1.17.33
    awsebcli 3.18.2 depends on botocore<1.16 and >=1.15
    boto3 1.14.33 depends on botocore<1.18.0 and >=1.17.33
    awsebcli 3.18.1 depends on botocore<1.16 and >=1.15
    boto3 1.14.33 depends on botocore<1.18.0 and >=1.17.33
    awsebcli 3.18.0 depends on botocore<1.16 and >=1.15

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible For help visit: https://pip.pypa.io/en/stable/user_guide/#fixing-conflicting-dependencies

[uranusjr]

Yeah, I ran the full requirements set with a few print statements locally, and it seems like the resolver is just backtracking like hell. The problem is that the output pip generates is not doing a good job reporting the situation to the user, and it seems stuck doing repeated things. (They are not the same thing, but different things generating the same output.)

Edit: I’m marking this as an enhancement since pip is arguably doing the correct thing here, but needs to do it better.

Edit 2: For maintainers wondering, the repeated setuptools line is printed because the resolver is trying to build different versions of awsebcli from source.

[pradyunsg]

Thanks for testing out the new resolver and filing an issue @noamkush! ^>^

The problem is that the output pip generates is not doing a good job reporting the situation to the user, and it seems stuck doing repeated things. (They are not the same thing, but different things generating the same output.)

Nice debugging and thanks for looking into this. Looks like this is basically as #8346 -- i.e. improving pip's output during backtracking in dependency resolution? I'm kinda in favor of closing this issue with a note on that one about there being a proper user report + test case in this one.

[uranusjr]

We’ll probably need to revisit the ResolutionTooDeep conversaion. It seems like the user is willing to wait for the resolver (in absolute rounds), the patience threshold is relatively low before the user decides the resolver is “stalling”.

Both this issue and #8705 indicates that the user tends to easily feel the resolver is stuck if it’s not looking for new things (no Collecting xxx and Using cached xxx messages). Maybe that would be a good metric to detect whether the resolver is working too hard, and user intervention (providing a stricter version range) would be useful?

[pradyunsg]

I wonder if what we need here is some output to indicate that the resolver is "doing stuff" in addition to a better metric for "too much work". :)

[uranusjr]

Yeah, it would be helpful to hook into the reporter’s backtracking and pinning_candidate (something like that, I forgot the actual hook name) and output something. That’d help maintainers triage those reports, at least. I doubt that would help user perception much, however, since the user would still see the resolver trying to pin the same package repeatedly, which also feels like stalling.