pip 20.3b1 bug with --pre for already installed package

[rgommers]

Environment

• pip version: 20.3b1
• Python version: 3.8
• OS: Linux

Description

After the release of pip 20.3b1 yesterday we started seeing issues on SciPy CI jobs that use pip install --pre --upgrade numpy. Here is a full build log: scipy/scipy#12729 (comment)

There's two exceptions actually, the version normalization seems also to be missing somewhere. But it's the first exception that's the more problematic one.

Expected behavior

Package should be upgraded without any errors.

How to Reproduce

Here is a standalone reproducer:

conda create -n piptest python=3.8 pip=19.3  # start from 19.3 (TravisCI default)
conda activate piptest
pip install -U pip --pre  # installs 20.3b1
pip install numpy==1.17.3  # install an older numpy version, so we can upgrade
pip install --pre --upgrade numpy  -i https://pypi.anaconda.org/scipy-wheels-nightly/simple

Output

Looking in indexes: https://pypi.anaconda.org/scipy-wheels-nightly/simple
Requirement already satisfied: numpy in /home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages (1.17.3)
ERROR: Exception:
Traceback (most recent call last):                                                                                            
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 186, in _merge_into_criterion                                                                                                          
    crit = self.state.criteria[name]                                                                                          
KeyError: 'numpy'                                                                                                             

During handling of the above exception, another exception occurred:                                                           

Traceback (most recent call last):                                                                                            
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 217, in _main                                                                                                                            
    status = self.run(options, args)                                                                                          
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper                                                                                                                           
    return func(self, options, args)                                                                                          
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 318, in run
    requirement_set = resolver.resolve(                                                                                       
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 119, in resolve                                                                                                            
    self._result = resolver.resolve(                                                                                          
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 427, in resolve                                                                                                                        
    state = resolution.resolve(requirements, max_rounds=max_rounds)                                                           
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 294, in resolve                                                                                                                        
    name, crit = self._merge_into_criterion(r, parent=None)                                                                   
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 188, in _merge_into_criterion
    crit = Criterion.from_requirement(self._p, requirement, parent)
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 89, in from_requirement
    if not candidates:
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 96, in __bool__
    return any(self)
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 81, in __iter__
    candidates = _insert_installed(self._installed, self._get_others())
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 41, in _insert_installed
    candidates = sorted(
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 198, in iter_index_candidates
    yield self._make_candidate_from_link(
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 144, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 286, in __init__
    assert str(version) == wheel.version, (
AssertionError: <Version('1.20.0.dev0+20201101041405.5c37621')> != '1.20.0.dev0+20201101041405-5c37621' for wheel numpy

[uranusjr]

Thanks for catching this! According to PEP 440’s Local Version Segments section:

The normal form is using the . character. This allows versions such as 1.0+ubuntu-1 to be normalized to 1.0+ubuntu.1.

So we’re not doing the comparison correctly.

[uranusjr]

For people following packaging discussions, this problem was actually discussed (theoretically at the time) in a recent thread: Escaping versions for wheel, sdist, and .dist-info names. The numpy wheel in question is escaping dot with an underscore, and the spec is vague on whether this should be done or not.

[rgommers]

The numpy wheel in question is escaping dot with an underscore, and the spec is vague on whether this should be done or not.

It seems allowed, but either way it's not a numpy issue. The _ gets normalized in two different ways, once to . and once to -. Which then makes the != check fail.

@uranusjr it's not clear to me that that also explains the first exception - does it, or is that an unrelated resolver issue?

[uranusjr]

By the first exception I assume you mean KeyError: 'numpy'? That’s not a bug, but an expected exception. But the exception is handled like this (very simplified):

try:
    crit = self.state.criteria[name]
except KeyError:
    crit = _calculate_criteria_differently(...) # This eventually calls the above assertion.

And that handler blew up when it shouldn’t.