Keyring prevent some package uninstall

[pierreluctg]

Environment

• pip version: 20.1.1
• Python version: 3.7
• OS: Windows

Description

Optional pip keyring support is cause some resource lock while uninstalling cffi

Expected behavior

pip should not lock any resources and allow clean uninstall of cffi

How to Reproduce

# Install keyring to enable keyring feature in pip
pip install keyring -q
# install cffi
pip install cffi -q
# uninstall cffi
pip uninstall cffi -y

Output

Found existing installation: cffi 1.14.0
Uninstalling cffi-1.14.0:
  Successfully uninstalled cffi-1.14.0
ERROR: Exception:
Traceback (most recent call last):
  File "c:\projects\.venv_pip_keyring\lib\site-packages\pip\_internal\cli\base_command.py", line 188, in _main
    status = self.run(options, args)
  File "c:\projects\.venv_pip_keyring\lib\site-packages\pip\_internal\commands\uninstall.py", line 89, in run
    uninstall_pathset.commit()
  File "c:\projects\.venv_pip_keyring\lib\site-packages\pip\_internal\req\req_uninstall.py", line 450, in commit
    self._moved_paths.commit()
  File "c:\projects\.venv_pip_keyring\lib\site-packages\pip\_internal\req\req_uninstall.py", line 290, in commit
    save_dir.cleanup()
  File "c:\projects\.venv_pip_keyring\lib\site-packages\pip\_internal\utils\temp_dir.py", line 196, in cleanup
    rmtree(self._path)
  File "c:\projects\.venv_pip_keyring\lib\site-packages\pip\_vendor\retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "c:\projects\.venv_pip_keyring\lib\site-packages\pip\_vendor\retrying.py", line 212, in call
    raise attempt.get()
  File "c:\projects\.venv_pip_keyring\lib\site-packages\pip\_vendor\retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "c:\projects\.venv_pip_keyring\lib\site-packages\pip\_vendor\six.py", line 703, in reraise
    raise value
  File "c:\projects\.venv_pip_keyring\lib\site-packages\pip\_vendor\retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "c:\projects\.venv_pip_keyring\lib\site-packages\pip\_internal\utils\misc.py", line 136, in rmtree
    onerror=rmtree_errorhandler)
  File "c:\python37-32\lib\shutil.py", line 516, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "c:\python37-32\lib\shutil.py", line 400, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "c:\python37-32\lib\shutil.py", line 398, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\*****\\AppData\\Local\\Temp\\pip-uninstall-l1br_4h0\\_cffi_backend.cp37-win32.pyd'

To show that pip keyring feature is in cause here, setting the keyring backend to null (set PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring) is preventing this issue.

[pierreluctg]

@pradyunsg what do I need to do to get this triaged?

[deveshks]

May I also suggest filing an issue with keyring on this?

I also found some closed issues related to cffi and Access is denied which weren't related to keyring.

Also I couldn't reproduce this issue in OSX Catalina 10.15.4

$ pip --version
pip 20.1.1 from ...... (python 3.8)

$ python --version
Python 3.8.2

$ pip install keyring
Collecting keyring
  Using cached keyring-21.2.1-py3-none-any.whl (31 kB)
Installing collected packages: keyring
Successfully installed keyring-21.2.1

$ pip install cffi
Collecting cffi
  Using cached cffi-1.14.0-cp38-cp38-macosx_10_9_x86_64.whl (175 kB)
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Installing collected packages: pycparser, cffi
Successfully installed cffi-1.14.0 pycparser-2.20

$ pip uninstall cffi -y
Found existing installation: cffi 1.14.0
Uninstalling cffi-1.14.0:
  Successfully uninstalled cffi-1.14.0

[pfmoore]

Installers like pip have very strict constraints on what they can do. One of the reasons we vendor all of our dependencies is so that we don't typically hit issues like this. Keyring is an exception, being an "optional" dependency, but in this case the issue is that it doesn't conform to the restrictions that "being part of pip" impose (specifically, it depends on an external C extension.

I don't think we can expect keyring to work under the limitations needed to make it safe for pip to use, unfortunately.

One thing that might improve this issue is for pip to lazily import keyring only when needed. This would at least avoid the problem when keyring isn't actually used by pip (although there will still be potential issues - users who need keyring to access an index won't be able to upgrade cfffi without disabling keyring, for example).

The issue is Windows-only because Windows won't allow you to delete a shared library that's loaded in a running process. Unix will allow you to do this, and so uninstalling cffi while using it is OK on Unix but not on Windows.

[pierreluctg]

@deveshks, the issue is not in keyring it's in pip and was introduced by #5952.
The issue may be on windows only, you will only hit the issue when the keyring backend uses cffi.

Uninstalling a package using pip should not make use of any external (not in pip vendor). This is causing the issue that we have here.

[pierreluctg]

Thank you @pfmoore, yes, lazily importing keyring only when needed will fix the uninstall issue. But potentially still cause an issue on updates.

However, for the update case, it will be probably be easier to diagnose (for a user that uses keyring to store pip related credentials) versus a user that have no idea keyring is installed (potentially via a dependency) and hits this issues...

[pfmoore]

it's in pip and was introduced by #5952

It's possible that some additional thought is needed around how keyring support is activated. I didn't get involved in the discussions at the time so I can't say whether a manual opt-in (beyond simply "having keyring installed") was considered and/or found unacceptable.

[pierreluctg]

Can any of the Member look at this issue and provide a fix?

[pfmoore]

Please understand that keyring support is optional and the reason it's not built in is precisely because it can cause issues like this. There's no "fix" needed here. If you want to uninstall a package that keyring relies on, you may have to disable or remove keyring while you do so.

Repeatedly insisting that someone else fix this issue isn't going to alter that fact.

I've offered some suggestions as to how someone could try to reduce the number of times this is an issue, but I'm certainly not offering to do that work. And on reflection, I no longer consider the current behaviour a pip bug.

As has been already suggested, another approach you could take is to raise an issue with keyring, saying that its use of C extensions causes this problem when it gets used in pip. But I doubt the keyring maintainers are likely to remove all C dependencies just for this very niche problem.

I'm going to mark this issue as a feature request, and "Deferred till PR", to reflect the fact that there doesn't appear to be much more to say here until someone (quite likely not one of the pip committers, but a user who is sufficiently interested in improving the behaviour in this situation to work on it) submits a PR that can act as a starting point for further discussion.

I'm happy if any of the other @pypa/pip-committers want to pick this up and handle it differently, but I will leave the discussion at this point.

[pierreluctg]

@pfmoore I fully agree that keyring support should be optional. Even more when you try to use pip uninstall, since you do not need any secret to be retrieved.

However, currently because keyring and it's dependencies are not vendored like most pip dependencies. This is preventing users to uninstalling cffi, upgrading cffi or installing any package that need a different version of cffi.

I am sure there are good reasons why keyring is not vendored (some are mentioned in #5952).

The issue is not in keyring, the issue was clearly introduce by #5952.

This is not how a optional feature should work.