The dependency resolver fails with ResolutionTooDeep error

[arewm]

Description

When trying to install the azure-cli with pip 21.1.2, install takes a long time and eventually dies. Backing out to pip 21.1.1 seems to resolve this issue.

This might be a reoccurrence of #8314

Expected behavior

Packages should install as requested. In this situation, the install hangs on chardet which is already installed on the system. After backing out the pip version, the output looks something like

[...]
Collecting pynacl>=1.0.1
  Downloading PyNaCl-1.4.0.tar.gz (3.4 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting bcrypt>=3.1.3
  Downloading bcrypt-3.2.0.tar.gz (42 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting PyJWT[crypto]<3,>=1.0.0
  Downloading PyJWT-2.0.1-py3-none-any.whl (15 kB)
  Downloading PyJWT-2.0.0-py3-none-any.whl (15 kB)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/lib/python3.8/site-packages (from requests~=2.22->azure-cli-core==2.25.0->azure-cli) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3.8/site-packages (from requests~=2.22->azure-cli-core==2.25.0->azure-cli) (2.9)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/lib/python3.8/site-packages (from requests~=2.22->azure-cli-core==2.25.0->azure-cli) (1.25.9)
Collecting oauthlib>=3.0.0
  Downloading oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
INFO: pip is looking at multiple versions of urllib3 to determine which version is compatible with other requirements. This could take a while.
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Downloading urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
  Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
  Downloading urllib3-1.25.8-py2.py3-none-any.whl (125 kB)
  Downloading urllib3-1.25.7-py2.py3-none-any.whl (125 kB)
  Downloading urllib3-1.25.6-py2.py3-none-any.whl (125 kB)
  Downloading urllib3-1.25.5-py2.py3-none-any.whl (125 kB)
  Downloading urllib3-1.25.4-py2.py3-none-any.whl (125 kB)
INFO: pip is looking at multiple versions of urllib3 to determine which version is compatible with other requirements. This could take a while.
  Downloading urllib3-1.25.3-py2.py3-none-any.whl (150 kB)
  Downloading urllib3-1.25.2-py2.py3-none-any.whl (150 kB)
  Downloading urllib3-1.24.3-py2.py3-none-any.whl (118 kB)
  Downloading urllib3-1.24.2-py2.py3-none-any.whl (131 kB)
  Downloading urllib3-1.24.1-py2.py3-none-any.whl (118 kB)
[...]

pip version

21.1.2

Python version

3.8.10

OS

alpine

How to Reproduce

I am hitting this bug when trying to install azure-cli in a Jenkins environment. The commands we are running are

apk add jq
apk del python2
apk update
apk add bash py-pip make gcc
apk add --virtual=build libffi-dev musl-dev openssl-dev python3-dev
apk add linux-headers musl-dev musl libc-dev libffi-dev
apk add openssl-dev openssl
apk add --no-cache ca-certificates
apk add --no-cache --virtual .build-deps curl
apk del .build-deps
apk add --no-cache python3
python3 -m ensurepip
rm -r /usr/lib/python*/ensurepip
pip3 install --upgrade pip setuptools
rm -r /root/.cache
pip3 install wheel
pip3 install --upgrade  azure-cli --no-cache-dir

https://github.com/open-cluster-management/bootstrap-ks/blob/main/aks/install.sh#L64-L82

Output

[...]
OK: 12766 distinct packages available
(1/32) Installing libbz2 (1.0.8-r1)
(2/32) Installing libffi (3.3-r2)
(3/32) Installing gdbm (1.13-r1)
(4/32) Installing xz-libs (5.2.5-r0)
(5/32) Installing sqlite-libs (3.32.1-r1)
(6/32) Installing python3 (3.8.10-r0)
(7/32) Installing py3-appdirs (1.4.4-r1)
(8/32) Installing py3-ordered-set (4.0.1-r0)
(9/32) Installing py3-parsing (2.4.7-r0)
(10/32) Installing py3-six (1.15.0-r0)
(11/32) Installing py3-packaging (20.4-r0)
(12/32) Installing py3-setuptools (47.0.0-r0)
(13/32) Installing py3-chardet (3.0.4-r4)
(14/32) Installing py3-idna (2.9-r0)
(15/32) Installing py3-certifi (2020.4.5.1-r0)
(16/32) Installing py3-urllib3 (1.25.9-r0)
[...]
Collecting oauthlib<3.0.0,>=2.1.0
  Downloading oauthlib-2.1.0-py2.py3-none-any.whl (121 kB)
Collecting requests-oauthlib>=0.5.0
  Downloading requests_oauthlib-1.0.0-py2.py3-none-any.whl (21 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
Collecting oauthlib>=0.6.2
  Downloading oauthlib-2.0.7-py2.py3-none-any.whl (124 kB)
  Downloading oauthlib-2.0.6.tar.gz (127 kB)
  Downloading oauthlib-2.0.5.tar.gz (129 kB)
  Downloading oauthlib-2.0.4.tar.gz (127 kB)
  Downloading oauthlib-2.0.3.tar.gz (127 kB)
  Downloading oauthlib-2.0.2.tar.gz (125 kB)
  Downloading oauthlib-2.0.1.tar.gz (122 kB)
  Downloading oauthlib-2.0.0.tar.gz (122 kB)
  Downloading oauthlib-1.1.2.tar.gz (111 kB)
  Downloading oauthlib-1.1.1.tar.gz (108 kB)
  Downloading oauthlib-1.1.0-1.tar.gz (106 kB)
WARNING: Discarding https://files.pythonhosted.org/packages/e5/54/4f96c51b171cf3a64a04b8c5167268803205bc5943b5cdf70bd770727b88/oauthlib-1.1.0-1.tar.gz#sha256=0f786c5573248a38efa86c48c59c0c93140ac836ab2a246aeefd8f9039e999ba (from https://pypi.org/simple/oauthlib/). Requested oauthlib>=0.6.2 from https://files.pythonhosted.org/packages/e5/54/4f96c51b171cf3a64a04b8c5167268803205bc5943b5cdf70bd770727b88/oauthlib-1.1.0-1.tar.gz#sha256=0f786c5573248a38efa86c48c59c0c93140ac836ab2a246aeefd8f9039e999ba (from requests-oauthlib>=0.5.0->msrest>=0.6.21->azure-keyvault-administration==4.0.0b3->azure-cli) has inconsistent version: filename has '1.1.0.post1', but metadata has '1.1.0'
  Downloading oauthlib-1.0.3.tar.gz (109 kB)
  Downloading oauthlib-1.0.2.tar.gz (108 kB)
  Downloading oauthlib-1.0.1.tar.gz (108 kB)
  Downloading oauthlib-1.0.0.tar.gz (108 kB)
  Downloading oauthlib-0.7.2.tar.gz (106 kB)
  Downloading oauthlib-0.7.1.tar.gz (106 kB)
  Downloading oauthlib-0.7.0.tar.gz (105 kB)
  Downloading oauthlib-0.6.3.tar.gz (100 kB)
  Downloading oauthlib-0.6.2.tar.gz (100 kB)
Collecting requests-oauthlib>=0.5.0
  Downloading requests_oauthlib-0.8.0-py2.py3-none-any.whl (23 kB)
  Downloading requests_oauthlib-0.7.0-py2.py3-none-any.whl (22 kB)
  Downloading requests_oauthlib-0.6.2-py2.py3-none-any.whl (22 kB)
  Downloading requests_oauthlib-0.6.1-py2.py3-none-any.whl (21 kB)
INFO: pip is looking at multiple versions of requests-oauthlib to determine which version is compatible with other requirements. This could take a while.
  Downloading requests_oauthlib-0.6.0-py2.py3-none-any.whl (21 kB)
  Downloading requests_oauthlib-0.5.0-py2.py3-none-any.whl (19 kB)
INFO: pip is looking at multiple versions of idna to determine which version is compatible with other requirements. This could take a while.
Collecting idna<3,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Downloading idna-2.9-py2.py3-none-any.whl (58 kB)
  Downloading idna-2.8-py2.py3-none-any.whl (58 kB)
  Downloading idna-2.7-py2.py3-none-any.whl (58 kB)
  Downloading idna-2.6-py2.py3-none-any.whl (56 kB)
  Downloading idna-2.5-py2.py3-none-any.whl (55 kB)
INFO: pip is looking at multiple versions of chardet to determine which version is compatible with other requirements. This could take a while.
Collecting chardet<4,>=3.0.2
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
ERROR: Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 180, in _main
    status = self.run(options, args)
  File "/usr/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 204, in wrapper
    return func(self, options, args)
  File "/usr/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 318, in run
    requirement_set = resolver.resolve(
  File "/usr/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 127, in resolve
    result = self._result = resolver.resolve(
  File "/usr/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 473, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/usr/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 384, in resolve
    raise ResolutionTooDeep(max_rounds)
pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 2000000

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

#10032 should be able to resolve this once and for all.

[uranusjr]

It would be awesome if you could verify whether it works for you (it seems to for me).

[uranusjr]

Side note: this is not likely related to #8314. ResolutioNTooDeep is a symptom for many issues, and I am quite sure the two issues have very different root causes.

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[arewm]

Circling back to this, the issue might have actually been a problem with the alpine image we were using in Jenkins. We fixed it by adding

        apk add linux-headers alpine-sdk jq libc6-compat findutils
        apk add --no-cache python3 py3-pip

so this is working with the latest version of pip. Thanks for the eyes!