Support system certificate store

[datamoc]

Description

Hello,

We don’t have direct Internet access in my company main network. But we have local pip repository.
Sadly pip on windows does not recognized the company certificates.
I was able to use pip with a ca-bundle.crt created with a very simple script:
import ssl

context = ssl.create_default_context()
der_certs = context.get_ca_certs(binary_form=True)
pem_certs = [ssl.DER_cert_to_PEM_cert(der) for der in der_certs]

with open('C:\Users\user\certs\ca-bundle.crt', 'w') as outfile:
for pem in pem_certs:
outfile.write(pem + '\n')

In _ssl.py file it’s written that is equivalent to ssl.create_default_context() but it’s not the case.
I was not able to change the _ssl.py file in order to have the expected behavior.

Regards

Michel

Expected behavior

pip should be able to use the windows certificates without any user action

pip version

21.3.1

Python version

3.6.6

OS

Windows 10 1909

How to Reproduce

add ini C:\Users\User\pip\pip.ini

[global]
index-url = htts://locarepo.company/pypi/simple

(you have to create a local server)
in Powershell

PS C:\Users\user\Documents\My Apps> & 'C:\Program Files\Python36\python.exe' -m pip install toto --user

result:
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)': /api/pypi/python_pypi/simple/toto/

Output

PS C:\Users\user\Documents\My Apps> & 'C:\Program Files\Python36\python.exe' -m pip install toto --user

Looking in indexes: https://repos.company/api/pypi/python_pypi/simple
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)': /api/pypi/python_pypi/simple/toto/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)': /api/pypi/python_pypi/simple/toto/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)': /api/pypi/python_pypi/simple/toto/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)': /api/pypi/python_pypi/simple/toto/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)': /api/pypi/python_pypi/simple/toto/
Could not fetch URL https:// repos.company/api/pypi/python_pypi/simple/toto/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host= 'repos.company', port=443): Max retries exceeded with url: /api/pypi/python_pypi/simple/toto/ (Caused by SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)) - skipping
ERROR: Could not find a version that satisfies the requirement toto (from versions: none)
ERROR: No matching distribution found for toto

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

I seem to recall another entry on this but couldn’t find it, so I changed the title to more match the thing we want. There’s no active effort toward this though, so feel free to try your hands on it.

[pradyunsg]

Closing as a duplicate of #10961.

That issue has notes on what the appropriate changes would be, to make this happen.

[notatallshaw]

@uranusjr @pradyunsg I have a question on this, is it possible to add something as a non-supported option?

As discussed in #10961 adding an option for the requests ssl context with load_default_certs would definitely help most cases where the certs are in the system store but not in certifi. But there is an edge cases where Windows has not yet lazily loaded the certs in to system store and calling this method does not trigger that. However in most cases the user will have already triggered this by opening a web browser or using any other tool which uses Schannel to make an HTTPS request.

So would it be possible to add this feature optionally but add a warning that says it is not supported? Or is this too nuanced for user support for pip maintainers?

[pradyunsg]

That's... odd. I don't think it makes sense to add a feature only to call it unsupported. I'm definitely not comfortable adding such an option, since it does seem nuanced in a way that increases the support burden.

That said, let's move the discussion to that other issue, and see if other maintainers feel differently. :)

[uranusjr]

Is “that other issue” #10961? Should we unlock it then?

Another thing I feel is it’s difficult to gauge the impact (both directions) without the code. If this could be done similarly to keyring support, I’d probably be fine adding it; but if it needs to take the form of a hidden option or environment variable, probably less so.

[pfmoore]

Agreed I don't like the idea of adding a deliberately unsupported feature. We have enough trouble with existing unsupported behaviour that we didn't explicitly add...

[notatallshaw]

Maybe I should make a new issue? I'm thinking of rather than having a flag that says "Pip uses System certificate store" it could be a flag that says "Pip uses 'load default certs' for the SSL Context as provided by Python's standard library"

Then users can only expect whatever behavior the standard library provides and not any specific claims about what that it will actually do for them beyond what's documented in the standard library.

Edit: I just made a new issue to match the proposal: #11038