pip 20.3.2 eagerly downloads all versions of a package during dependency resolution

[gGonz]

Environment

• pip version: 20.3.2
• Python version: 3.7.2
• OS: Linux

Description
If you use pip to update requirements after installing 20.3.2 it goes into an infinite loop resolving and downloading previous versions of dependencies.

This happened on our CI environment that took 26 minutes to complete a job that takes no longer than 3, but then we were able to replicate it locally by just telling pip to update himself...

Expected behavior

Dependencies should be resolved without seeking all versions of a package.

How to Reproduce

1. pip install -U pip to get to 20.3.2
2. pip install -U pip again

Output

Paste the output of the steps above, including the commands themselves and
pip's output/traceback etc.

First, update pip

$ pip install -U pip
Collecting pip                                                                                            
  Downloading pip-20.3.2-py2.py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 1.2 MB/s 
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.1
    Uninstalling pip-20.1:
      Successfully uninstalled pip-20.1
Successfully installed pip-20.3.2

Then update it again (I had to ctrl+C to stop the loop)

$ pip install -U pip
Requirement already satisfied: pip in /home/ggonz/.pyenv/versions/3.7.2/envs/pip-test/lib/python3.7/site-packages (20.3.2)
Collecting pip
  Using cached pip-20.3.2-py2.py3-none-any.whl (1.5 MB)
  Downloading pip-20.3.2.tar.gz (1.5 MB)
     |████████████████████████████████| 1.5 MB 829 kB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Downloading pip-20.3.1-py2.py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 835 kB/s 
  Downloading pip-20.3.1.tar.gz (1.5 MB)
     |████████████████████████████████| 1.5 MB 1.2 MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Downloading pip-20.3-py2.py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 1.2 MB/s 
  Downloading pip-20.3.tar.gz (1.5 MB)
     |████████████████████████████████| 1.5 MB 718 kB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached pip-20.2.4-py2.py3-none-any.whl (1.5 MB)
  Downloading pip-20.2.4.tar.gz (1.5 MB)
     |████████████████████████████████| 1.5 MB 1.4 MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Downloading pip-20.2.3-py2.py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 1.3 MB/s 
  Downloading pip-20.2.3.tar.gz (1.5 MB)
     |████████████████████████████████| 1.5 MB 1.5 MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Downloading pip-20.2.2-py2.py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 1.9 MB/s 
  Downloading pip-20.2.2.tar.gz (1.5 MB)
     |████████████████████████████████| 1.5 MB 2.0 MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Downloading pip-20.2.1-py2.py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 160 kB/s 
  Downloading pip-20.2.1.tar.gz (1.5 MB)
     |████████████████████████████████| 1.5 MB 44 kB/s 
^C  Installing build dependencies ... canceled
ERROR: Operation cancelled by user

[omry]

Probably related:
With 20.3.2, pip install setuptools --upgrade is iterating all versions and is taking long minutes to not do anything.
With 20.3.1 it finishes in a few seconds.

[ryboe]

@pradyunsg, @uranusjr, and @NoahGorny this is a really bad bug 😬

[shawnbrown]

I just ran into this myself when trying to upgrade my version of pandas (1.1.3) to the latest version (currently 1.1.5). And then saw that it did the same thing when trying to upgrade pip itself.

Out of morbid curiosity, I let the pandas upgrade run its course inside a fresh virtual environment. It took over 35 minutes and downloaded over 650 MB of old packages into the cache directory: https://pastebin.com/V6kg6tTX

Edit: Moved code to as per @omry.

[shawnbrown]

It should be noted, it's not an infinite loop...

Here's the full log of pip 20.3.2 trying to upgrade itself (adds 157 MB of old versions into the cache): https://pastebin.com/AFny1enP

Edit: Moved code to as per @omry.

[omry]

can you use pastebin when pasting very long pastes? it makes looking for comments here very painful.

[hungrybirder]

Oh my god,
Please fix it ASAP.
Thank you.

[wyqsmith]

Gosh...Plz...CI System bombed by pip...

[ryboe]

This is the mitigation I'm using until the bug is fixed.

python -m pip install 'pip==20.3.1'

[Corfucinas]

@wyqsmith

Exactly

[uranusjr]

I did some quick research, and this bug seems to have been introduced by the packaging 20.8 vendor upgrade. This was unfortunately not caught due to the PyPI search API was causing failures on CI, and we missed the test failure in a sea of red. (See update below)

Unfortunately I cannot do anything right now since I don’t have access to PyPI versions, but will try to contact other members who do when they are available. (20.3.2 has now been yanked from PyPI. Thanks @xavfernandez!)