Notes on pipsi compatibility & usability challenges

[ncoghlan]

Kenneth recently dropped the pipsi based installation instructions, presumably because they were causing confusion (65a2106#diff-62c330cd8da31d476bb63b51bb8b1307).

As far as I'm aware, there are three main ways for things to go wrong with pipsi:

1. The same PATH bootstrapping issues that affect all Python CLI applications

See pypa/packaging.python.org#396 for more discussion of that problem.

2. Getting it to use the right Python version

If you use python3 -m pip --user pipsi (or an equivalent) to bootstrap it, then it will default to creating Python 3.x virtual environments. However, if pipsi ends up being installed with Python 2.7, then it will also default to that for all of the applications it installs (which is a hassle for pipenv, since it really prefers to have Python 3.x as the host Python).

3. Getting the pew CLI installed correctly

Even if folks don't run into the above two issues, pipsi install pipenv doesn't quite work as desired, because pipenv's shell integration also has a CLI dependency on pew, not just a Python API dependency.

That distinction doesn't matter for a regular pipenv installation with pip, since that installs everything to a common destination, so the pew CLI automatically comes along with the Python level dependency declaration.

However, it doesn't work for pipsi, since breaking the per-app venv isolation is an extra step in the installation process, so the installation currently needs to be spelled as pipsi install pew && pipsi install pipenv.

This isn't an urgent issue, but I think it could be a good design driver for adding "CLI dependency declaration" support to the packaging metadata formats, as we some clear constraints that we'd like to adhere to in solving this problem:

• updating pipenv to publish new CLI dependency declaration metadata would be fine
• updating pipsi to consume new CLI dependency declaration metadata would be fine
• requiring updates to pew to make things work would not be fine (since it creates a major barrier to adoption if a dependency management feature doesn't already work with previously published packages - a concrete example of that would be PEP 345's failed attempt to move away from the de facto standard established by setuptools for spelling dependency declarations vs PEP 508's pragmatic adoption of the established setuptools conventions)

https://github.com/pypa/interoperability-peps/pull/30/files would be an interesting proposal to consider on this front, as it aimed to tackle the problem of declaring external dependencies in general. As far as I can tell, given suitable adjustments to pipsi, it could potentially work for pipenv's use case as long as external requirements were listed in a separate field:

• pipenv would add bin!pew as a new external requirement
• pip and other existing installers would ignore it, since it would be in a different field added by metadata 2.2
• pipsi could be updated to check for the external binary dependencies listed in the installed project, and if they weren't already available on the system, check if they could be exported from one of the app's declared Python level dependencies.

[uranusjr]

Didn’t you purge the CLI dependency to pew @techalchemy?

[ncoghlan]

From what I could see searching the repo, the Python code now consistently accesses pew via its Python API.

The remaining issue is with some of the shell integrations, which still access pew as a command line application.

I didn't look into whether or not there might be a way to replace those remaining pew CLI references with pipenv subcommands, though.

[uranusjr]

I believe there are also references to pewtwo, which is a vendored version of pew, but Pipsi should be able to pick it up since it is published in Pipenv’s setup.py.

[techalchemy]

@ncoghlan since I removed psutil from pew in a patch that is not yet accepted upstream, we have had pew vendored and accessible as pewtwo as an entry point. See https://github.com/pypa/pipenv/blob/master/tasks/vendoring/patches/patched/pew_psutil.patch#L17 for that. Our api calls pick up the version in the patched directory and in most cases at least where I personally switched to api calls it was for performance gains or behavior issues related to subprocess forking.

Tl;dr pewtwo exists and pipenv has no setuptools level dependency on pew (you can uninstall it even). I’ll try this all out and then we can update the docs

[ncoghlan]

Oh, cool :)

Looking more closely at the hits I found when searching the repo, it turns out they're only missing patches for pew's native shell-integrations, and I don't believe pipenv relies on those for anything:

• https://github.com/pypa/pipenv/search?l=fish&q=pew&type=
• https://github.com/pypa/pipenv/search?l=Shell&q=pew&type=

(I didn't look closely enough at the paths when initially scanning the search results - I only looked at the file names)

[techalchemy]

@ncoghlan do these include activate.fish and such? If so they run when we run pipenv.pew.pew.workon(whatever)

[ncoghlan]

The ones I'm seeing are the code completion scripts for fish, zsh, and bash, plus the prompt modification script for fish.

[techalchemy]

@ncoghlan to clarify, you're saying that we don't include patches for pew's native shell integrations? they are included in our patched/pew tree but I'm not sure if there would be something else you're saying we need to do to accomplish this--

https://github.com/pypa/pipenv/tree/master/pipenv/patched/pew/shell_config

[ncoghlan]

Right, I'm saying those definitely aren't patched yet, but I don't know if there's a code path anywhere in pipenv where that actually matters (I haven't looked into exactly which aspects of those scripts pipenv's own shell integration relies on).

[uranusjr]

I just did a quick grep on the source and it seems we are only using Pew for four things at this point:

1. Create virtualenvs (new_cmd)
2. Global site packages toggle (toggleglobalsidepackages)
3. List virtual environments, to find the one matching the current project (lsenvs)
4. Shell (workon_cmd)

The first three are all quite trivial, and the fourth is already very heavily patched by Pipenv, and has some more patching planned. I feel maybe it is time to just drop dependency to Pew altogether. Without its presence Pipsi compatibility would largely fix itself. We can suggest users to install Pipenv with pipsi install pipenv --python=python3 to make sure it always runs on Python 3.