Having a runtime dependency on pip: make it explicit whether this is supported or not

[pfmoore]

What's the problem this feature will solve?

See https://discuss.python.org/t/pip-plans-to-introduce-an-alternative-zipapp-deployment-method/17431/22 for context. Basically, people are suggesting that if we distribute a zipapp version of pip and as a result, environments without pip become more common, they will add pip as a runtime dependency to force the environment to contain pip.

While locating a copy of pip to run programmatically is a non-trivial problem (this is one of the reasons we're hesitating over making an official zipapp), if projects start to add a dependency on pip, this will block the option of finding other, less disruptive, ways of achieving pip-less environments¹.

In addition, pip has no programmatic API, and the normal use of declaring a dependency is for your code to call a library from Python. People don't declare a dependency on bzr so they can run bzr commands using subprocess, any more than they do for git. Why should pip be different?

Describe the solution you'd like

Explicitly state in the documentation, alongside the statement that pip has no supported Python API, that we don't support including pip in a project's runtime dependencies.

Alternative Solutions

Leave our position unstated and implicit.

Additional context

Some of the alternative approaches to relying on pip being in the environment include:

1. Projects assume the existence of a pip command, and run that (the same as they would for something like git). This is a reversion of our existing advice that people should run pip via subprocess.run([sys.executable, "-m", "pip"]) which would be frustrating churn for our users. Otherwise, though, it's exactly what we'd expect people to do for any other command line tool like git.
2. Projects add a configuration option to allow the user to state how pip should be invoked (with the default remaining python -m pip, at least for now). Not all projects have a configuration system, so this might be a bigger change than is ideal.
3. Projects try to auto-discover a suitable pip command. This is likely both costly and error prone.

So I think that relying on pip being in the environment may need to remain the norm for now. And that may well impact the advisability of shipping an official zipapp. However, I still don't think that projects adding a dependency on pip is a good idea.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

Footnotes

1. Which I believe are important - py -m venv takes 6 seconds on my machine, whereas py -m venv --without-pip takes under 0.1 second. ↩

[pfmoore]

To support people who would otherwise want to add a dependency on pip, we could create a separate library that does the "heavy lifting" of locating pip and runs pip. People can depend on that instead.

[notatallshaw]

FYI one user of needing to know the location of Pip and be able to guarantee it exists in an environment is Conda.

In a Conda requirements file you can explicitly ask for pip to install packages, this is handled through Conda calling a subprocess to run Pip: https://github.com/conda/conda/blob/main/conda_env/pip_util.py#L26

Not sure it's helpful but maybe if you need a downstream user of Pip to test some change out to how pip should be called it might be worth contacting them.

[uranusjr]

needing to know the location of Pip and be able to guarantee it exists in an environment is Conda

Wouldn’t this be covered by Conda? Conda’s pip is a re-distribution, and re-distributors by definition can have different rules they can maintain themselves. This is similar to how pip can be installed with apt install python3-pip, and we certainly won’t go so far to say it’s a wrong thing for apt to support, just not for us to support. That’s what I understand from this.

[notatallshaw]

Wouldn’t this be covered by Conda? Conda’s pip is a re-distribution, and re-distributors by definition can have different rules they can maintain themselves. This is similar to how pip can be installed with apt install python3-pip, and we certainly won’t go so far to say it’s a wrong thing for apt to support, just not for us to support. That’s what I understand from this.

Agreed but I was thinking, maybe incorrectly, once a standalone install version of Pip is available it might itself be useful as a conda package. Much like the way conda itself is only installed in to the base environment then some users might want to install pip only once in to the base environment and let other environments use that one pip (much the same way is being discussed about virtual environments).

Conda developers would need to explicitly support that style of calling Pip in the pip_util.py file I linked and to do that they would need a reliable way to find where Pip is, Maybe though conda developers have enough control over their environment that they would just write their own. It was just a thought.

[pradyunsg]

I have an elaborate note somewhere about this -- but I don't think I understand the rationale here. IIUC, the proposal is to stop people from treating pip like any other Python package, because we're exploring alternative ways to distribute pip?

[pradyunsg]

To make my position explicit: I think users depending on pip as {sys.executable} -m pip and having a runtime dependency on pip is perfectly reasonable. Making pip a weird special case in more ways is not a good idea, at least, not for the motivation provided here. I don't think a new distribution mechanism should result in new restrictions being placed on existing mechanisms. And if we're not doing anything in implementation to prevent this, this is a docs-only "don't do this" which... uh... won't work and will only serve to create friction?

the normal use of declaring a dependency is for your code to call a library from Python.

No? We explicitly have packages that only ship a CLI that are on PyPI that users use. A package doesn't need to have a usable Python API for it to be packaged for use by Python code.

[pfmoore]

Fair points. I've revised my opinion here, and I'm a little less concerned about people depending on pip, as long as it's for the right reasons.

I do think pip is a "weird special case" in a lot of ways, though, and one of my concerns is that we don't make it harder to move away from being a special case. Specifically, pip is one of the few tools that (currently) absolutely cannot be run except when installed in the environment. Our policy that people should run pip as python -m pip comes from that, historically, and the Discourse thread established that we've already got to a point where changing that to a more normal "just run the pip command" approach is going to be really hard to change. Making it the norm that "depend on pip" is the way to make sure that pip is available, adds just one more roadblock on the route to letting people use a standalone pip command. We may choose not to follow that route, but not being able to is definitely unusual compared to other Python tools like tox, nox, flake8, black, hatch, poetry, etc.

I also think it's important to remember that "having a dependency on pip" is not the norm at the moment. People just expect pip to be present - because it always is. The idea of depending on pip came up as a reaction to the thought that pip-less environments may become less common, as a way of (essentially) preventing that happening. IMO that was mostly an over-reaction by people who mistakenly thought we were going to suddenly stop adding pip to environments, but nevertheless now that the idea has been suggested, I don't think it's unreasonable for us say explicitly that we prefer people to maintain the current status quo, at least while we explore what possibilities there are for changing things without breaking people's usage of python -m pip. But I concede that explicitly not supporting it is too harsh an approach. Maybe just a note in the section about calling pip from your program to say something like "It is OK to assume that pip is installed when you run python -m pip. Whenever a Python environment is created, pip is automatically installed by default, so there is no need to take special measures to ensure pip is present."

Overall, though, I guess I'm OK with leaving things as they are, as long as we don't see a sudden surge in people adding dependencies on pip (at which point we can maybe just have a conversation as to why they feel this is needed, given that we're not suddenly introducing pip-less environments).

[sbidoul]

Actually, with #11320, I can totally see myself making pip-deepfreeze depend on pip>=22.3 and then using {sys.executable} -m pip --python {target_python} inspect and other CLI goodies pip has to offer.

I guess it will equally apply to hatch, pip-deptree, pip-audit and more.

[pfmoore]

Yeah, it makes a lot of sense for tools like that which would get installed using pipx to depend on pip. I'm happy to withdraw this suggestion on that basis.

One point I made over on Discourse - pipx exposes a shared pip in every venv it manages (to reduce the overhead of installing pip everywhere). It's not immediately obvious to me how that would interact with a package that depends on pip (probably perfectly fine, but if the shared pip doesn't match the application's dependency spec things might get tricky). It's probably something you should test before relying on it. Conversely, of course, it would now make a lot of sense for pipx to do the same, and install a private copy of pip and use --python. Maybe I'll look at that next 🙂

[pradyunsg]

It’s resolved then, that users are ok to declare a runtime dependency on pip. :)

[pfmoore]

Yes, although I still feel that it should be the exception rather than the rule 🙂