virtualenv can't be installed as `--user` when an older global installation exists.

[mboisson]

Issue

I have the following:

$ virtualenv --version
virtualenv 20.0.18 from /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.10/lib/python3.8/site-packages/virtualenv/__init__.py
$ virtualenv
usage: virtualenv [--version] [--with-traceback] [-v | -q] [--app-data APP_DATA] [--clear-app-data] [--discovery {builtin}] [-p py] [--creator {builtin,cpython3-posix,venv}] [--seeder {app-data,pip}] [--no-seed]
                  [--activators comma_sep_list] [--clear] [--system-site-packages] [--symlinks | --copies] [--download | --no-download] [--extra-search-dir d [d ...]] [--pip version] [--setuptools version] [--wheel version] [--no-pip]
                  [--no-setuptools] [--no-wheel] [--symlink-app-data] [--prompt prompt] [-h]
                  dest
virtualenv: error: the following arguments are required: dest

if I now try to upgrade virtualenv in my user account:

$ pip install --user --upgrade virtualenv
Ignoring pip: markers 'python_version < "3"' don't match your environment
Looking in links: /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/avx2, /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic, /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/virtualenv-20.13.1+computecanada-py2.py3-none-any.whl
Collecting filelock<4,>=3.2
  Using cached filelock-3.5.0-py3-none-any.whl (10.0 kB)
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/platformdirs-2.5.0+computecanada-py3-none-any.whl
Requirement already satisfied, skipping upgrade: six<2,>=1.9.0 in /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.10/lib/python3.8/site-packages (from virtualenv) (1.14.0)
Collecting distlib<1,>=0.3.1
  Using cached distlib-0.3.4-py2.py3-none-any.whl (461 kB)
Installing collected packages: filelock, platformdirs, distlib, virtualenv
Successfully installed distlib-0.3.4 filelock-3.5.0 platformdirs-2.5.0+computecanada virtualenv-20.13.1+computecanada

and then rerun the same:

$ virtualenv --version
virtualenv 20.13.1 from /home/mboisson/.local/lib/python3.8/site-packages/virtualenv/__init__.py
$ virtualenv
ModuleNotFoundError: No module named 'virtualenv.seed.via_app_data'

Environment

Provide at least:

• OS: irrelevant

$ pip list
Package        Version
-------------- ---------------------
appdirs        1.4.3
distlib        0.3.4
filelock       3.5.0
more-itertools 8.2.0
pip            20.0.2
platformdirs   2.5.0+computecanada
setuptools     45.2.0
setuptools-scm 3.5.0
six            1.14.0
virtualenv     20.13.1+computecanada
wheel          0.34.2

I believe this issue is caused by the switch from importlib_metadata to importlib.metadata in PR #2246

In importlib_metadata's version of entry_points, there is a step to make the imported entry points unique:
https://github.com/python/importlib_metadata/blob/main/importlib_metadata/__init__.py#L995

This ensures that only the highest priority entry_points.txt from virtualenv*.dist_info is considered.

This is also the case in importlib.metadata's latest version:
https://github.com/python/cpython/blob/main/Lib/importlib/metadata/__init__.py

but it was not the case in:
Python 3.8: https://github.com/python/cpython/blob/3.8/Lib/importlib/metadata.py#L541
Python 3.9: https://github.com/python/cpython/blob/3.9/Lib/importlib/metadata.py#L569
It was not until Python 3.10 that this was taken up:
https://github.com/python/cpython/blob/3.10/Lib/importlib/metadata/__init__.py#L978

I believe importlib_metadata should be used for Python < 3.10.

[mboisson]

Some additional info. Here are the dist-info/entry_points.txt that are found, comparing with importlib_metadata vs importlib.metadata:

With importlib_metadata:

$ strace -f -e trace=file python -c "from importlib_metadata import entry_points; print(str(entry_points()))" |& grep entry_points | grep "virtualenv-.*dist-info"
openat(AT_FDCWD, "/home/mboisson/.local/easybuild/software/2020/avx2/Core/python-build-bundle/2022a/lib/python3.8/site-packages/virtualenv-20.13.1.dist-info/entry_points.txt", O_RDONLY|O_CLOEXEC) = 3

With importlib.metadata with python 3.8:

$ strace -f -e trace=file python -c "from importlib.metadata import entry_points; print(str(entry_points()))" |& grep entry_points | grep "virtualenv-.*dist-info"
openat(AT_FDCWD, "/home/mboisson/.local/easybuild/software/2020/avx2/Core/python-build-bundle/2022a/lib/python3.8/site-packages/virtualenv-20.13.1.dist-info/entry_points.txt", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/home/mboisson/.local/easybuild/software/2020/avx2/Core/python/3.8.2/lib/python3.8/site-packages/virtualenv-20.0.18.dist-info/entry_points.txt", O_RDONLY|O_CLOEXEC) = 3

With importlib.metdata with python 3.10:

$ strace -f -e trace=file python -c "from importlib.metadata import entry_points; print(str(entry_points()))" |& grep entry_points | grep "virtualenv-.*dist-info"
openat(AT_FDCWD, "/home/mboisson/.local/easybuild/software/2020/avx2/Core/python-build-bundle/2022a/lib/python3.10/site-packages/virtualenv-20.13.1.dist-info/entry_points.txt", O_RDONLY|O_CLOEXEC) = 3

[gaborbernat]

A docker image replicating this would be helpful. A PR addressing this would also be appreciated.

[mboisson]

I don't have docker, but from a fresh bare ubuntu 20.04 VM:

ubuntu@test:~$ sudo apt install python3-pip
...... [truncated]

ubuntu@test:~$ sudo pip install --upgrade "virtualenv<20.1"
Collecting virtualenv<20.1
  Downloading virtualenv-20.0.35-py2.py3-none-any.whl (4.9 MB)
     |████████████████████████████████| 4.9 MB 10.4 MB/s
Requirement already satisfied, skipping upgrade: distlib<1,>=0.3.1 in /usr/local/lib/python3.8/dist-packages (from virtualenv<20.1) (0.3.4)
Requirement already satisfied, skipping upgrade: six<2,>=1.9.0 in /usr/lib/python3/dist-packages (from virtualenv<20.1) (1.14.0)
Collecting appdirs<2,>=1.4.3
  Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Requirement already satisfied, skipping upgrade: filelock<4,>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from virtualenv<20.1) (3.5.1)
Installing collected packages: appdirs, virtualenv
  Attempting uninstall: virtualenv
    Found existing installation: virtualenv 20.13.1
    Uninstalling virtualenv-20.13.1:
      Successfully uninstalled virtualenv-20.13.1
Successfully installed appdirs-1.4.4 virtualenv-20.0.35
ubuntu@test:~$ virtualenv --version
virtualenv 20.0.35 from /usr/local/lib/python3.8/dist-packages/virtualenv/__init__.py

ubuntu@test:~$ pip install --upgrade --user virtualenv
Collecting virtualenv
  Downloading virtualenv-20.13.1-py2.py3-none-any.whl (8.6 MB)
     |████████████████████████████████| 8.6 MB 12.0 MB/s
Requirement already satisfied, skipping upgrade: distlib<1,>=0.3.1 in /usr/local/lib/python3.8/dist-packages (from virtualenv) (0.3.4)
Requirement already satisfied, skipping upgrade: platformdirs<3,>=2 in /usr/local/lib/python3.8/dist-packages (from virtualenv) (2.5.0)
Requirement already satisfied, skipping upgrade: six<2,>=1.9.0 in /usr/lib/python3/dist-packages (from virtualenv) (1.14.0)
Requirement already satisfied, skipping upgrade: filelock<4,>=3.2 in /usr/local/lib/python3.8/dist-packages (from virtualenv) (3.5.1)
Installing collected packages: virtualenv
  WARNING: The script virtualenv is installed in '/home/ubuntu/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed virtualenv-20.13.1
ubuntu@test:~$ virtualenv --version
virtualenv 20.13.1 from /home/ubuntu/.local/lib/python3.8/site-packages/virtualenv/__init__.py
ubuntu@test:~$ virtualenv
ModuleNotFoundError: No module named 'virtualenv.activation.xonsh'

Different error message, because different versions are involved, but the source is the same: entry_points from python3.X for X<10 return multiple entries for virtualenv instead of returning just the highest priority one (the one in --user).

[gaborbernat]

I now understand the issue better. Sorry, it took me so long. IMHO this is something we just don't want (should) support. Installing the virtualenv package twice (once globally and once locally via user) is likely to cause more issues than worth. I think people should install virtualenv via pipx into an isolated environment and not overlay it on the system via --user flag.

[mboisson]

Not supporting it is simply something not possible in cluster and in shared environments. Having some central packages and some user-packages is common place. This behavior is simply broken for virtualenv. It works for everything else, and it works with python 3.10, so I'm not sure what is the issue.

[gaborbernat]

Not supporting it is simply something not possible in cluster and in shared environments.

pipx installs content into user space virtual environments, so solves the problem better than the --user flag. I know it's commonplace but it's not a good solution to the problem. I rather not add support for something that happens to work now, but is brittle in general.