pip install cannot resolve dependencies for wheels with extras

[terencehonles]

What did you want to do?

Compiling wheels and then installing those wheels breaks the dependency resolver. I've limited the following example to two packages so I don't need to use a requirements file, but that is where I discovered the issue. I believe it could probably be further reduced with more simple test packages. There may be such packages on PyPI, but I am unaware of them (it probably would be a good idea to have them for other tools to share some smoke tests).

Input:

rm -rf venv wheels \
  && python3 -m venv venv \
  && . venv/bin/activate \
  && python -V \
  && pip install -U pip wheel \
  && pip list \
  && pip wheel --wheel-dir wheels 'bravado-core==5.17.0' 'jsonschema[format]==3.2.0' \
  && pip install wheels/*

Output:

# Python -V
Python 3.8.6

# pip upgrade
Collecting pip
  Using cached pip-20.3.1-py2.py3-none-any.whl (1.5 MB)
Collecting wheel
  Using cached wheel-0.36.0-py2.py3-none-any.whl (34 kB)
Installing collected packages: pip, wheel
  Attempting uninstall: pip
    Found existing installation: pip 20.2.1
    Uninstalling pip-20.2.1:
      Successfully uninstalled pip-20.2.1
Successfully installed pip-20.3.1 wheel-0.36.0

# packages
Package    Version
---------- -------
pip        20.3.1
setuptools 49.2.1
wheel      0.36.0

# creating wheels
Collecting bravado-core==5.17.0
  Using cached bravado_core-5.17.0-py2.py3-none-any.whl (67 kB)
Collecting jsonschema[format]==3.2.0
  Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
Collecting attrs>=17.4.0
  Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)
Collecting idna
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting jsonpointer>1.13
  Using cached jsonpointer-2.0-py2.py3-none-any.whl (7.6 kB)
Collecting jsonref
  Using cached jsonref-0.2-py3-none-any.whl (9.3 kB)
Collecting msgpack>=0.5.2
  Using cached msgpack-1.0.0-cp38-cp38-manylinux1_x86_64.whl (303 kB)
Collecting pyrsistent>=0.14.0
  Using cached pyrsistent-0.17.3-cp38-cp38-linux_x86_64.whl
Collecting python-dateutil
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting pytz
  Using cached pytz-2020.4-py2.py3-none-any.whl (509 kB)
Collecting pyyaml
  Using cached PyYAML-5.3.1-cp38-cp38-linux_x86_64.whl
Collecting rfc3987
  Using cached rfc3987-1.3.8-py2.py3-none-any.whl (13 kB)
Collecting setuptools
  Using cached setuptools-50.3.2-py3-none-any.whl (785 kB)
Collecting simplejson
  Using cached simplejson-3.17.2-cp38-cp38-manylinux2010_x86_64.whl (137 kB)
Collecting six
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting strict-rfc3339
  Using cached strict_rfc3339-0.7-cp38-none-any.whl
Collecting swagger-spec-validator>=2.0.1
  Using cached swagger_spec_validator-2.7.3-py2.py3-none-any.whl (27 kB)
Collecting webcolors
  Using cached webcolors-1.11.1-py3-none-any.whl (9.9 kB)
Saved ./wheels/bravado_core-5.17.0-py2.py3-none-any.whl
Saved ./wheels/attrs-20.3.0-py2.py3-none-any.whl
Saved ./wheels/idna-2.10-py2.py3-none-any.whl
Saved ./wheels/jsonpointer-2.0-py2.py3-none-any.whl
Saved ./wheels/jsonref-0.2-py3-none-any.whl
Saved ./wheels/jsonschema-3.2.0-py2.py3-none-any.whl
Saved ./wheels/msgpack-1.0.0-cp38-cp38-manylinux1_x86_64.whl
Saved ./wheels/pyrsistent-0.17.3-cp38-cp38-linux_x86_64.whl
Saved ./wheels/python_dateutil-2.8.1-py2.py3-none-any.whl
Saved ./wheels/pytz-2020.4-py2.py3-none-any.whl
Saved ./wheels/PyYAML-5.3.1-cp38-cp38-linux_x86_64.whl
Saved ./wheels/rfc3987-1.3.8-py2.py3-none-any.whl
Saved ./wheels/setuptools-50.3.2-py3-none-any.whl
Saved ./wheels/simplejson-3.17.2-cp38-cp38-manylinux2010_x86_64.whl
Saved ./wheels/six-1.15.0-py2.py3-none-any.whl
Saved ./wheels/strict_rfc3339-0.7-cp38-none-any.whl
Saved ./wheels/swagger_spec_validator-2.7.3-py2.py3-none-any.whl

# using wheels
Saved ./wheels/webcolors-1.11.1-py3-none-any.whl
Processing ./wheels/attrs-20.3.0-py2.py3-none-any.whl
Processing ./wheels/bravado_core-5.17.0-py2.py3-none-any.whl
Processing ./wheels/idna-2.10-py2.py3-none-any.whl
Processing ./wheels/jsonpointer-2.0-py2.py3-none-any.whl
Processing ./wheels/jsonref-0.2-py3-none-any.whl
Processing ./wheels/jsonschema-3.2.0-py2.py3-none-any.whl
Processing ./wheels/msgpack-1.0.0-cp38-cp38-manylinux1_x86_64.whl
Processing ./wheels/pyrsistent-0.17.3-cp38-cp38-linux_x86_64.whl
Processing ./wheels/python_dateutil-2.8.1-py2.py3-none-any.whl
Processing ./wheels/pytz-2020.4-py2.py3-none-any.whl
Processing ./wheels/PyYAML-5.3.1-cp38-cp38-linux_x86_64.whl
Processing ./wheels/rfc3987-1.3.8-py2.py3-none-any.whl
Processing ./wheels/setuptools-50.3.2-py3-none-any.whl
Processing ./wheels/simplejson-3.17.2-cp38-cp38-manylinux2010_x86_64.whl
Processing ./wheels/six-1.15.0-py2.py3-none-any.whl
Processing ./wheels/strict_rfc3339-0.7-cp38-none-any.whl
Processing ./wheels/swagger_spec_validator-2.7.3-py2.py3-none-any.whl
Processing ./wheels/webcolors-1.11.1-py3-none-any.whl
Collecting jsonschema[format]>=2.5.1
  Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
  Using cached jsonschema-3.1.1-py2.py3-none-any.whl (56 kB)
  Using cached jsonschema-3.1.0-py2.py3-none-any.whl (56 kB)
  Using cached jsonschema-3.0.2-py2.py3-none-any.whl (54 kB)
  Using cached jsonschema-3.0.1-py2.py3-none-any.whl (54 kB)
  Using cached jsonschema-3.0.0-py2.py3-none-any.whl (54 kB)
  Using cached jsonschema-2.6.0-py2.py3-none-any.whl (39 kB)
  Using cached jsonschema-2.5.1-py2.py3-none-any.whl (38 kB)
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of webcolors to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of swagger-spec-validator to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of strict-rfc3339 to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of six to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of simplejson to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of setuptools to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of rfc3987 to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pyyaml to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pytz to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of python-dateutil to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pyrsistent to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of msgpack to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of jsonschema to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of jsonref to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of jsonpointer to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of idna to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of bravado-core to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of attrs to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install jsonschema 3.2.0 (from /home/terence/projects/NexleafDP/wheels/jsonschema-3.2.0-py2.py3-none-any.whl), jsonschema[format]==2.5.1, jsonschema[format]==2.6.0, jsonschema[format]==3.0.0, jsonschema[format]==3.0.1, jsonschema[format]==3.0.2, jsonschema[format]==3.1.0, jsonschema[format]==3.1.1, jsonschema[format]==3.2.0 and swagger-spec-validator==2.7.3 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested jsonschema 3.2.0 (from /home/terence/projects/NexleafDP/wheels/jsonschema-3.2.0-py2.py3-none-any.whl)
    swagger-spec-validator 2.7.3 depends on jsonschema
    jsonschema[format] 3.2.0 depends on jsonschema 3.2.0 (from https://files.pythonhosted.org/packages/c5/8f/51e89ce52a085483359217bc72cdbf6e75ee595d5b1d4b5ade40c7e018b8/jsonschema-3.2.0-py2.py3-none-any.whl#sha256=4e5b3cf8216f577bee9ce139cbe72eca3ea4f292ec60928ff24758ce626cd163 (from https://pypi.org/simple/jsonschema/))
    The user requested jsonschema 3.2.0 (from /home/terence/projects/NexleafDP/wheels/jsonschema-3.2.0-py2.py3-none-any.whl)
    swagger-spec-validator 2.7.3 depends on jsonschema
    jsonschema[format] 3.1.1 depends on jsonschema 3.1.1 (from https://files.pythonhosted.org/packages/ce/6c/888d7c3c1fce3974c88a01a6bc553528c99d3586e098eee23e8383dd11c3/jsonschema-3.1.1-py2.py3-none-any.whl#sha256=94c0a13b4a0616458b42529091624e66700a17f847453e52279e35509a5b7631 (from https://pypi.org/simple/jsonschema/))
    The user requested jsonschema 3.2.0 (from /home/terence/projects/NexleafDP/wheels/jsonschema-3.2.0-py2.py3-none-any.whl)
    swagger-spec-validator 2.7.3 depends on jsonschema
    jsonschema[format] 3.1.0 depends on jsonschema 3.1.0 (from https://files.pythonhosted.org/packages/11/9c/a0a2c70be340603c8ff5a692a8e6a4997fb858c7fd8701ff2afe087a3b58/jsonschema-3.1.0-py2.py3-none-any.whl#sha256=4f4ddc3d51f33a5363c042dc62c85010e9e3b8353bcf108afff394dde70854b3 (from https://pypi.org/simple/jsonschema/))
    The user requested jsonschema 3.2.0 (from /home/terence/projects/NexleafDP/wheels/jsonschema-3.2.0-py2.py3-none-any.whl)
    swagger-spec-validator 2.7.3 depends on jsonschema
    jsonschema[format] 3.0.2 depends on jsonschema 3.0.2 (from https://files.pythonhosted.org/packages/54/48/f5f11003ceddcd4ad292d4d9b5677588e9169eef41f88e38b2888e7ec6c4/jsonschema-3.0.2-py2.py3-none-any.whl#sha256=5f9c0a719ca2ce14c5de2fd350a64fd2d13e8539db29836a86adc990bb1a068f (from https://pypi.org/simple/jsonschema/))
    The user requested jsonschema 3.2.0 (from /home/terence/projects/NexleafDP/wheels/jsonschema-3.2.0-py2.py3-none-any.whl)
    swagger-spec-validator 2.7.3 depends on jsonschema
    jsonschema[format] 3.0.1 depends on jsonschema 3.0.1 (from https://files.pythonhosted.org/packages/aa/69/df679dfbdd051568b53c38ec8152a3ab6bc533434fc7ed11ab034bf5e82f/jsonschema-3.0.1-py2.py3-none-any.whl#sha256=a5f6559964a3851f59040d3b961de5e68e70971afb88ba519d27e6a039efff1a (from https://pypi.org/simple/jsonschema/))
    The user requested jsonschema 3.2.0 (from /home/terence/projects/NexleafDP/wheels/jsonschema-3.2.0-py2.py3-none-any.whl)
    swagger-spec-validator 2.7.3 depends on jsonschema
    jsonschema[format] 3.0.0 depends on jsonschema 3.0.0 (from https://files.pythonhosted.org/packages/cd/e6/be1b2a6ebebdaf1f790f1e750bb720fbda0335c2a19601ea9d8bb5059f38/jsonschema-3.0.0-py2.py3-none-any.whl#sha256=dd3f8ecb1b52d94d45eedb67cb86cac57b94ded562c5d98f63719e55ce58557b (from https://pypi.org/simple/jsonschema/))
    The user requested jsonschema 3.2.0 (from /home/terence/projects/NexleafDP/wheels/jsonschema-3.2.0-py2.py3-none-any.whl)
    swagger-spec-validator 2.7.3 depends on jsonschema
    jsonschema[format] 2.6.0 depends on jsonschema 2.6.0 (from https://files.pythonhosted.org/packages/77/de/47e35a97b2b05c2fadbec67d44cfcdcd09b8086951b331d82de90d2912da/jsonschema-2.6.0-py2.py3-none-any.whl#sha256=000e68abd33c972a5248544925a0cae7d1125f9bf6c58280d37546b946769a08 (from https://pypi.org/simple/jsonschema/))
    The user requested jsonschema 3.2.0 (from /home/terence/projects/NexleafDP/wheels/jsonschema-3.2.0-py2.py3-none-any.whl)
    swagger-spec-validator 2.7.3 depends on jsonschema
    jsonschema[format] 2.5.1 depends on jsonschema 2.5.1 (from https://files.pythonhosted.org/packages/bd/cc/5388547ea3504bd8cbf99ba2ae7a3231598f54038e9b228cbd174f8ec6a1/jsonschema-2.5.1-py2.py3-none-any.whl#sha256=71e7b3bcf9fca408bcb65bb60892f375d3abdd2e4f296eeeb8fe0bbbfcde598e (from https://pypi.org/simple/jsonschema/))

I've worked around this by specifying --no-deps as mentioned in User Guide: Installation Bundles, but I would expect the resolver to not break on itself by auto adding the following requirements:

jsonschema[format]==2.5.1
jsonschema[format]==2.6.0
jsonschema[format]==3.0.0
jsonschema[format]==3.0.1
jsonschema[format]==3.0.2
jsonschema[format]==3.1.0
jsonschema[format]==3.1.1
jsonschema[format]==3.2.0 

I didn't specify the versions for jsonschema / jsonschema[format] and I would expect it to prefer the wheel it has since that does meet the requirements. Looking at the output from the resolver it seems like it might be treating jsonschema as different from jsonschema[format]:

    The user requested jsonschema 3.2.0 (from /home/terence/projects/NexleafDP/wheels/jsonschema-3.2.0-py2.py3-none-any.whl)
     jsonschema[format] 2.5.1 depends on jsonschema 2.5.1 (from https://files.pythonhosted.org/packages/bd/cc/5388547ea3504bd8cbf99ba2ae7a3231598f54038e9b228cbd174f8ec6a1/jsonschema-2.5.1-py2.py3-none-any.whl#sha256=71e7b3bcf9fca408bcb65bb60892f375d3abdd2e4f296eeeb8fe0bbbfcde598e (from https://pypi.org/simple/jsonschema/))

[brainwane]

Hello and thank you for your bug report! I'm sorry you're having trouble right now. Thank you for sharing your report with us. And I am glad that the pip documentation was able to help you with a temporary workaround.

I'll defer to the new resolver's developers on responding to the issue you're running into.

(If you don't mind, please also tell us what could have happened differently so you could have tested and caught and reported this during the pip resolver beta period.)

[terencehonles]

It's probably unlikely that I would have caught this earlier. This is part of our CI scripts and I figured moving the new resolver to default and adding the ability to turn it off was a later part of the resolver beta (now). The only other way I could have seen this be more of a soft fail is to run both resolvers in parallel and only use the new resolver if there were no issues resolving. Since this is run by our CI that might have been too soft and we wouldn't have noticed the error anyways (unless it did install, but it still exited with an error code)

[elfjes]

We're also running into this issue. We hadn't gotten to convert all our CI pipelines / Dockerfiles to run with the new resolver, and this popped up with pip 20.3. In our case we're depending on apispec[yaml]==4.0.0 but it happens with any wheel with extra requirements

@terencehonles: thanks for the workaround :)

[uranusjr]

The cause to this is #8785. I have modified the title of that issue to reflect the underlying problem better, and will merge conversaion here into it.