502 Errors from PyPI are not surfaced

[ewdurbin]

Description

There was a transient outage that caused 502s to be returned from PyPI's CDN edge.

Rather than surfacing this error, it appears that pip instead reported that no distributions were found:

ERROR: Could not find a version that satisfies the requirement billiard==3.6.4.0 (from versions: none)
ERROR: No matching distribution found for billiard==3.6.4.0
Error: Process completed with exit code 1.

Expected behavior

pip should surface the error and perhaps retry rather than reporting that a version is not found.

pip version

23.1.2

Python version

3.11

OS

Ubuntu GHA flavored

How to Reproduce

Attempt to pip install from an index which returns 502 on simple requests.

Observe that rather than surfacing the error, pip reports that no distributions are available.

Output

  pip install -U pip setuptools wheel
  pip install -r requirements.txt --no-deps
  pip install -r requirements/dev.txt
  pip check
  shell: /usr/bin/bash -e {0}
  env:
    BILLING_BACKEND: warehouse.subscriptions.services.MockStripeBillingService api_base=http://localhost:12111 api_version=2020-08-27
    pythonLocation: /opt/hostedtoolcache/Python/3.11.3/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.11.3/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.3/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.3/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.3/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.3/x64/lib
Requirement already satisfied: pip in /opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages (22.3.1)
Collecting pip
  Using cached pip-23.1.2-py3-none-any.whl (2.1 MB)
Requirement already satisfied: setuptools in /opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages (65.5.0)
Collecting setuptools
  Using cached setuptools-67.7.2-py3-none-any.whl (1.1 MB)
Collecting wheel
  Using cached wheel-0.40.0-py3-none-any.whl (64 kB)
Installing collected packages: wheel, setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 65.5.0
    Uninstalling setuptools-65.5.0:
      Successfully uninstalled setuptools-65.5.0
  Attempting uninstall: pip
    Found existing installation: pip 22.3.1
    Uninstalling pip-22.3.1:
      Successfully uninstalled pip-22.3.1
Successfully installed pip-23.1.2 setuptools-67.7.2 wheel-0.40.0
Collecting alembic==1.10.4 (from -r requirements/main.txt (line 7))
  Using cached alembic-1.10.4-py3-none-any.whl (212 kB)
Collecting amqp==5.1.1 (from -r requirements/main.txt (line 11))
  Using cached amqp-5.1.1-py3-none-any.whl (50 kB)
Collecting argon2-cffi==21.3.0 (from -r requirements/main.txt (line 15))
  Using cached argon2_cffi-21.3.0-py3-none-any.whl (14 kB)
Collecting argon2-cffi-bindings==21.2.0 (from -r requirements/main.txt (line 19))
  Using cached argon2_cffi_bindings-21.2.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (86 kB)
Collecting asn1crypto==1.5.1 (from -r requirements/main.txt (line 42))
  Using cached asn1crypto-1.5.1-py2.py3-none-any.whl (105 kB)
Collecting async-timeout==4.0.2 (from -r requirements/main.txt (line 46))
  Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Collecting attrs==23.1.0 (from -r requirements/main.txt (line 50))
  Using cached attrs-23.1.0-py3-none-any.whl (61 kB)
Collecting automat==22.10.0 (from -r requirements/main.txt (line 54))
  Using cached Automat-22.10.0-py2.py3-none-any.whl (26 kB)
Collecting b2sdk==1.21.0 (from -r requirements/main.txt (line 58))
  Using cached b2sdk-1.21.0-py3-none-any.whl
Collecting babel==2.12.1 (from -r requirements/main.txt (line 61))
  Using cached Babel-2.12.1-py3-none-any.whl (10.1 MB)
Collecting bcrypt==4.0.1 (from -r requirements/main.txt (line 65))
  Using cached bcrypt-4.0.1-cp36-abi3-manylinux_2_28_x86_64.whl (593 kB)
ERROR: Could not find a version that satisfies the requirement billiard==3.6.4.0 (from versions: none)
ERROR: No matching distribution found for billiard==3.6.4.0
Error: Process completed with exit code 1.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[woodruffw]

We also observed a variant of this, although our version did actually report the 502 in the pip error:

Collecting sigstore~=1.1.2
  Downloading sigstore-1.1.2-py3-none-any.whl (74 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 74.8/74.8 KB 2.0 MB/s eta 0:00:00
Collecting requests~=2.28
  Downloading requests-2.29.0-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.5/62.5 KB 7.9 MB/s eta 0:00:00
Collecting tuf~=2.1
  Downloading tuf-2.1.0-py3-none-any.whl (45 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.7/45.7 KB 9.7 MB/s eta 0:00:00
Collecting pyOpenSSL>=[23](https://github.com/sigstore/gh-action-sigstore-python/actions/runs/4850935530/jobs/8644308467#step:3:25).0.0
  Downloading pyOpenSSL-23.1.1-py3-none-any.whl (57 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.9/57.9 KB 11.9 MB/s eta 0:00:00
Collecting id>=1.0.0
  Downloading id-1.0.0-py3-none-any.whl (11 kB)
Collecting importlib_resources~=5.7
  Downloading importlib_resources-5.12.0-py3-none-any.whl (36 kB)
Collecting securesystemslib
  Downloading securesystemslib-0.28.0-py3-none-any.whl (917 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 917.8/917.8 KB 50.0 MB/s eta 0:00:00
Collecting appdirs~=1.4
  Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Requirement already satisfied: pyjwt>=2.1 in /usr/lib/python3/dist-packages (from sigstore~=1.1.2->-r /home/runner/work/gh-action-sigstore-python/gh-action-sigstore-python/.//requirements.txt (line 1)) (2.3.0)
Collecting cryptography>=39
  Downloading cryptography-40.0.2-cp36-abi3-manylinux_2_28_x86_64.whl (3.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.7/3.7 MB 103.0 MB/s eta 0:00:00
Collecting sigstore-protobuf-specs~=0.1.0
  ERROR: HTTP error 502 while getting https://files.pythonhosted.org/packages/00/c3/046a392ee177f4558b8301e2ea3599ae64d083be8899a458180eead50ac5/sigstore_protobuf_specs-0.1.0-py3-none-any.whl#sha[25](https://github.com/sigstore/gh-action-sigstore-python/actions/runs/4850935530/jobs/8644308467#step:3:27)6=0e7766add04b5bd145181936e6fedbb[26](https://github.com/sigstore/gh-action-sigstore-python/actions/runs/4850935530/jobs/8644308467#step:3:28)09d7e959f[27](https://github.com/sigstore/gh-action-sigstore-python/actions/runs/4850935530/jobs/8644308467#step:3:29)40051cbca12572b277a2 (from https://pypi.org/simple/sigstore-protobuf-specs/) (requires-python:>=3.7)
ERROR: Could not install requirement sigstore-protobuf-specs~=0.1.0 from https://files.pythonhosted.org/packages/00/c3/046a392ee177f4558b8[30](https://github.com/sigstore/gh-action-sigstore-python/actions/runs/4850935530/jobs/8644308467#step:3:32)1e2ea[35](https://github.com/sigstore/gh-action-sigstore-python/actions/runs/4850935530/jobs/8644308467#step:3:37)99ae64d083be8899a458180eead50ac5/sigstore_protobuf_specs-0.1.0-py3-none-any.whl#sha256=0e7766add04b5bd1451819[36](https://github.com/sigstore/gh-action-sigstore-python/actions/runs/4850935530/jobs/8644308467#step:3:38)e6fedbb2609d7e959f2740051cbca12572b277a2 (from sigstore~=1.1.2->-r /home/runner/work/gh-action-sigstore-python/gh-action-sigstore-python/.//requirements.txt (line 1)) because of HTTP error 502 Server Error: Gateway Error for url: https://files.pythonhosted.org/packages/00/c3/046a[39](https://github.com/sigstore/gh-action-sigstore-python/actions/runs/4850935530/jobs/8644308467#step:3:41)2ee177f4558b8301e2ea3599ae64d083be8899a458180eead50ac5/sigstore_protobuf_specs-0.1.0-py3-none-any.whl for URL https://files.pythonhosted.org/packages/00/c3/046a392ee177f4558b8301e2ea3599ae64d083be8899a458180eead50ac5/sigstore_protobuf_specs-0.1.0-py3-none-any.whl#sha256=0e7766add04b5bd145181936e6fedbb2609d7e959f27[40](https://github.com/sigstore/gh-action-sigstore-python/actions/runs/4850935530/jobs/8644308467#step:3:42)051cbca12572b277a2 (from https://pypi.org/simple/sigstore-protobuf-specs/) (requires-python:>=3.7)
Error: Process completed with exit code 1.

(So maybe this is a place where the errors need to be unified, or possibly a discrepancy between pip versions? I can triage more later.)

Edit: xref https://github.com/sigstore/gh-action-sigstore-python/actions/runs/4850935530/jobs/8644308467#step:3:49

[ewdurbin]

@woodruffw do you know what version of pip was being used? it appears to be a regression in 23.1.x to me.

Edit: I see that that was file downloads (which were also getting transient 502s). I think that's working as expected (though retries would be nice).

[woodruffw]

Yep, I think this was just because the 502 hit during the file download, rather than the index download 🙂

I suspect the pip version in mine is pretty old; probably whatever GHA bundles with their default Python 3 on Ubuntu 22.04. I'll find out now.

Indeed:

DEBUG: pip: pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)

[dstufft]

This is the way it is for historical reasons... maybe it can be changed now?

Historically pip didn't treat failed index / file urls fetches as an error state, but rather it just skipped those urls. This came from the days when we would spider a bunch of URLs and we had no way of knowing in advance whether or not those URLs were expected to work or not.

Now adays we generally expect those URLs to work, so it's less defensible to just skip failing URLs. That being said, it is sometimes used to have pip have multiple repositories configured to use as fallbacks in case there is an outage at one of them, and turning this case into a hard error case would prevent that.

At a minimum warning when we get a failure on a request get seems warranted, and maybe we should move to hard failing.

[alex]

As an initial matter, some heuristics like "if there's only a single index, make errors noisy" seems like a good idea to me, if we want to go for a more middle ground.

As @woodruffw noted, if the errors happen during package download, they are noisy, so it's not clear to me you can do the fallback-in-case-of-outage thing reliably even today.

[pradyunsg]

The assumption there is that either the host server is completely borked or not borked at all. A borked server would trigger a fallback whereas a half-broken server (that serves index pages but not the files successfully) triggers a loud error.

[pradyunsg]

I'm OK with hard failures when we get an error from any index server, or with the proposed error out if the only index returns an error, or with a "all indexes errored out" as well.

[dstufft]

To be clear, I don't have a strong preference one way or the other. I'm just providing historical context for why it is the way it is currently.