Pip 20.3+ break proxy connection

[lex-pec]

Environment

• pip version: 20.2.3
• Python version: 3.9
• OS: Win10 64-bit

Description
When I'm trying to update pip using py -m pip install --upgrade pip a 'ProxySchemeUnknown' error occurs with a plenty of other 'File' errors.

When I was looking for the solution to this problem, I found very similar issues, but they're not the same:
#6555; psf/requests#5297

I think the problem is with my internet provider. I am sharing the internet with my laptop through a portable hotspot in my smartphone. Also, I changed TTL on the laptop.

Expected behavior
pip is updated.

How to Reproduce

1. Open Windows' CommandPrompt.
2. Run py -m pip install --upgrade pip.
3. An error occurs.

Output

C:\Users\User>py -m pip install --upgrade pip
Defaulting to user installation because normal site-packages is not writeable
ERROR: Exception:
Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\cli\base_command.py", line 228, in _main
    status = self.run(options, args)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\cli\req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\commands\install.py", line 323, in run
    requirement_set = resolver.resolve(
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 183, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 388, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 331, in _get_abstract_dist_for
    skip_reason = self._check_skip_installed(req)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 254, in _check_skip_installed
    self.finder.find_requirement(req_to_install, upgrade=True)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\index\package_finder.py", line 898, in find_requirement
    best_candidate_result = self.find_best_candidate(
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\index\package_finder.py", line 881, in find_best_candidate
    candidates = self.find_all_candidates(project_name)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\index\package_finder.py", line 825, in find_all_candidates
    package_links = self.process_project_url(
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\index\package_finder.py", line 790, in process_project_url
    html_page = self._link_collector.fetch_page(project_url)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\index\collector.py", line 643, in fetch_page
    return _get_html_page(location, session=self.session)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\index\collector.py", line 455, in _get_html_page
    resp = _get_html_response(url, session=session)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\index\collector.py", line 152, in _get_html_response
    resp = session.get(
  File "C:\Program Files\Python39\lib\site-packages\pip\_vendor\requests\sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\network\session.py", line 421, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "C:\Program Files\Python39\lib\site-packages\pip\_vendor\requests\sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Program Files\Python39\lib\site-packages\pip\_vendor\requests\sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "C:\Program Files\Python39\lib\site-packages\pip\_vendor\cachecontrol\adapter.py", line 53, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "C:\Program Files\Python39\lib\site-packages\pip\_vendor\requests\adapters.py", line 412, in send
    conn = self.get_connection(request.url, proxies)
  File "C:\Program Files\Python39\lib\site-packages\pip\_vendor\requests\adapters.py", line 309, in get_connection
    proxy_manager = self.proxy_manager_for(proxy)
  File "C:\Program Files\Python39\lib\site-packages\pip\_vendor\requests\adapters.py", line 193, in proxy_manager_for
    manager = self.proxy_manager[proxy] = proxy_from_url(
  File "C:\Program Files\Python39\lib\site-packages\pip\_vendor\urllib3\poolmanager.py", line 492, in proxy_from_url
    return ProxyManager(proxy_url=url, **kw)
  File "C:\Program Files\Python39\lib\site-packages\pip\_vendor\urllib3\poolmanager.py", line 429, in __init__
    raise ProxySchemeUnknown(proxy.scheme)
pip._vendor.urllib3.exceptions.ProxySchemeUnknown: Not supported proxy scheme None
WARNING: You are using pip version 20.2.3; however, version 20.3.1 is available.
You should consider upgrading via the 'C:\Program Files\Python39\python.exe -m pip install --upgrade pip' command.

[brainwane]

I'll defer to other people in responding to the substance of your issue, but: congratulations on filing your first GitHub issue! And thanks for the detailed report!

[jw120]

I resolved the same problem by changing my https_proxy environement variable from : to https://:

Perhaps urllib got stricter

[uranusjr]

Combined with #9190, it indeed seems like urllib3 made changes to the proxy code that broke some stuff in pip.

[lex-pec]

UPD: I don't know why, but now I have a different problem, but it's also related to the connection. When I am trying to update pip or to install a package, I have another error:

(base) C:\Users\User>pip install --upgrade pip
Collecting pip
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /packages/ab/11/2dc62c5263d9eb322f2f028f7b56cd9d096bb8988fcf82d65fa2e4057afe/pip-20.3.1-py2.py3-none-any.whl
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /packages/ab/11/2dc62c5263d9eb322f2f028f7b56cd9d096bb8988fcf82d65fa2e4057afe/pip-20.3.1-py2.py3-none-any.whl
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /packages/ab/11/2dc62c5263d9eb322f2f028f7b56cd9d096bb8988fcf82d65fa2e4057afe/pip-20.3.1-py2.py3-none-any.whl
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /packages/ab/11/2dc62c5263d9eb322f2f028f7b56cd9d096bb8988fcf82d65fa2e4057afe/pip-20.3.1-py2.py3-none-any.whl
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /packages/ab/11/2dc62c5263d9eb322f2f028f7b56cd9d096bb8988fcf82d65fa2e4057afe/pip-20.3.1-py2.py3-none-any.whl
ERROR: Could not install packages due to an EnvironmentError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/ab/11/2dc62c5263d9eb322f2f028f7b56cd9d096bb8988fcf82d65fa2e4057afe/pip-20.3.1-py2.py3-none-any.whl (Caused by ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)))

I've started to look for the solution and found a treatment for this problem on a chinese blog:
https://www.cnblogs.com/temari/p/13452629.html
The point was to add a trusted host and use -i parameter.

So, instead of pip install --upgrade pip I typed pip install --upgrade pip -i http://pypi.douban.com/simple --trusted-host pypi.douban.com like in the post and it worked! But I don't want to use these parameters everytime I need to install a package or update pip..

I can't make any conclusions, because I don't know how pip works. I've tryed to reach pypi.python.org and pypi.org in a browser, everything was fine.

May be @jw120 is right and the problem is with http:// (or https://) before proxy.

[pradyunsg]

@lex-pec: Could you share the output of pip config list, with the non-working vs working configurations?

[FDUZS]

same here.

Zheng ~ ❯❯❯ pip --version
pip 20.3.1 from d:\scoop\apps\python\current\lib\site-packages\pip (python 3.7)
Zheng ~ ❯❯❯ pip config list
global.index-url='https://pypi.tuna.tsinghua.edu.cn/simple'

without http_proxy and https_proxy:

Zheng ~\..\pywonderland git: master ≣ ❯❯❯ pip install -r .\requirements.txt
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError(0, 'Error'))': /simple/colour/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError(0, 'Error'))': /simple/colour/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError(0, 'Error'))': /simple/colour/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError(0, 'Error'))': /simple/colour/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError(0, 'Error'))': /simple/colour/
ERROR: Could not find a version that satisfies the requirement colour
ERROR: No matching distribution found for colour

after setting up those two environment variables:

Zheng ~\..\pywonderland git: master ≣ ❯❯❯ proxy

Name                           Value
----                           -----
http_proxy                     127.0.0.1:1080
https_proxy                    127.0.0.1:1080

Zheng ~\..\pywonderland git: master ≣ ❯❯❯ pip install -r .\requirements.txt
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
ERROR: Could not install packages due to an EnvironmentError: Not supported proxy scheme None

[sethmlarson]

Hello, urllib3 dev here.

• The ProxySchemeUnknown error has been in urllib3 in its current state for 6 years and wasn't changed between 1.25.9 and 1.26.2. Diff for these versions is here: urllib3/urllib3@1.25.9...1.26.2, ProxySchemeUnknown is raised here: e76b1dd#diff-42d9ff3a3dc18438ae9ad3552070169bc4140b54b61529e8a2ac7708352e1c5dL428 and is unchanged.

  My only thoughts on the ProxySchemeUnknown issue is that proxies that previously weren't being detected and passed to urllib3 now are?

• As for connection issues with a proxy: Users that have their proxy configured as proxy_url=https://... and connecting to https://pypi.org before were actually using their proxy in an HTTP configuration (urllib3 didn't support HTTPS in HTTPS before v1.26 and would silently connect to the proxy via HTTP) so should update their proxy to be "http://..." or change their proxy to use HTTPS if that's desired.

  We added a InvalidProxyConfigurationWarning for this situation in 1.25.9 to warn users of the upcoming change to make HTTPS+HTTPS proxy requests work as expected in 1.26, we didn't receive many hits on the linked issue so assumed this issue wasn't widespread.

  Comments like these: Support for web proxies is broken in pip 20.3 #9190 (comment) Support for web proxies is broken in pip 20.3 #9190 (comment) seem to indicate this is the case for some reporting users.

  My recommendation to all users that know they have a proxy configured in 20.2 of HTTPS_PROXY=https://<host> should actually be HTTPS_PROXY=http://<host>. I'm interested in seeing people's issues that don't fit this criteria though, if any.

cc @jalopezsilva

[uranusjr]

Hmm, the 20.2 series was on 1.25.9, but I don’t recall anyone reporting issues on this either. Maybe our warning settings did not bubble them correct? I have to admit I’ve never understood how warnings in Python work. Either way, thanks for the recommendation! I’m changing the title and pinning this issue for better discovery. (Edit: oops, looks like we have too many important issues, and GitHub only allows 3. So we’ll have to do without pinning.)