Toward PEP 518

[pradyunsg]

I'm AWOL this weekend but as I understand it, there needs to be a discussion toward PEP 518 and implementation of it in pip.

I've opened this issue, because I couldn't locate where the discussion was happening, if it is. Plus, having it in one place that's not pypa-dev/distutils-sig would be nice-ish?

[pfmoore]

#4799 is where a lot of the debate is happening. It was mostly prompted by:

1. I understood that the only outstanding blocker to PEP 518 support being in was Do wheel installs from VCS/directories using ephemeral caching #4764 (via Install build dependencies as specified in PEP 518 #4144 (comment))
2. Then Install build requirements before calling setup.py #4799 came up, and I looked at it to see if I could make some sense of all the in-progress work @xoviat was doing.
3. In the course of that, PEP 518 support: setup.py is called before build requirements have been installed #4647 popped up as a release blocker saying that PEP 518 support was broken.

As I dug into trying to work out what @xoviat was saying in #4799, it became obvious that we had some issues around recursive building of build environments (X needs Y to build, and Y needs Z, ...) although I'm still unclear as to whether these are implementation bugs, deeper design problems, or nasty corner cases that we can defer without too much issue.

Personally, I'm at the point where I'm out of my depth. I don't understand @xoviat's implementation to judge if it's needed, or if we still just need to merge #4764 and fix #4647 to be ready to go. Nor do I know how easy it'll be to fix #4647 (@xoviat seems to be saying that we need to merge #4799 to fix #4647, but that brings its own problems).

I've run out of time and energy to take the discussion any further this weekend, so I'm going to drop out at this point (at least for a while). For me, the key point is that we want an acceptable level of PEP 518 support for pip 10. I'd like someone to give me a feel for whether we're nearly there, or whether we're weeks away, so that I can avoid getting people fired up that pip 10 is coming only to then say it won't be till the new year...

[pradyunsg]

Thanks for a very helpful summary @pfmoore.

@ncoghlan @dstufft @xoviat Can we please bring the discussion here? Doing it on a closed PR feels weird to me. ._.

Sure thing.

@pradyunsg I know you don't have time for this. But you have been more successful at getting PRs approved than I have, so if you want, I'll be more than happy to walk you through the current implementation, how it works, and the potential problems that it has. I'll explain how I solved some (but not all) of these problems and my ideas for a complete fix (which again, I may do after PEP 517 if it's not done). I honestly don't care who does the work as long as it's done.

Actually, you're AWOL, so let me write a summary:

pip has an object hierarchy, as is common in most Python projects. It all starts the the command, which creates new references to objects, which create new references to lower objects. It's like a tree.

I'll define the "scope" of an object as a sort of lifetime. It's the duration that an object exists. Right now, the scope of PEP 518 in pip is the WheelBuilder. The PEP 518 environment is set up for bdist_wheel, then bdist_wheel is run within that environment, and then the environment is torn down.

So what's the problem with that? The problem is that the scope of the PEP 518 environment needs to be equal or greater than the scope of all calls to setup.py. More specifically, it needs to encapsulate an object that exists throughout the duration of setup.py calls. That object is Requirement.

The first obvious decision that you will encounter is: what should have a reference to BuildEnvironment, and the Requirement is a good a place as any. In fact, it's the best place IMHO to put the reference because setup.py is called if and only if Requirement exists.

The next problem you may encounter is this: how do we install the BuildEnvironment requirements? We could just shell out to pip. And that's the decision that was made by the original implementer. But there's a problem with that: pip has no way of knowing how many shell calls it's making because each pip could call itself again. In fact, a maliciously constructed package with circular dependencies could crash someone's computer if pip spawned too many processes.

Another problem is what shell call should we make? It's actually trickier than you might think, because getting the command-line parameters is frankly a PITA where you need to make that call. So you might have trouble passing the original parameters that the user passed to the child. The solution used by the original implementer involved using the finder, but I think you know the problem with that.

Shelling out a child of yourself without some kind of manager class that can kill children when the user presses ctrl+C isn't just wrong, it's malicious, especially when you don't know how many processes you've spawned. I personally don't know whether the children die in the current implementation (this might be FUD), but if they don't, the current implementation IMHO is wrong (aside from the other concerns).

Some possible solutions to that issue are the following:

1. If you want to get PEP 518 out the door, your best bet is probably some kind of lockfile that only allows up to 10 locks or so to make sure that pip isn't multiplying infinitely. Then you can just pass the exact requirements down to the child along with the command-line arguments.

2. A proper solution, which I'd like to implement after PEP 517, is to have a BuildEnvironmentManager class that's initialized directly in the install command. the BuildEnvironmentManager would have a reference to all of the objects there (RequirementPreparer, WheelBuilder, etc.), and would have a single method: get_build_environment(requirement_set). You could then implement a method on RequirementPreparer that's something like set_build_environment_manager, which it can then use to obtain build environments. The BuildEnvironmentManager could even detect multiple uses of the same environment (most commonly ['setuptools', 'wheel']) and provide the same environment if it's needed multiple times so that you don't need to create it (very common initially with projects having no pyproject.toml). Ideally there would also be some OOP design to try to remove the circular references (not trivial).

[ncoghlan]

@xoviat While it may not cover the case of deliberate malice, am I right in thinking that a build cache (which was used even when --no-binary :all: was specified) with the ability to track not only completed builds, but also in-progress ones, would be sufficient to ensure that build dependency cycles terminated? This would be a variant of your first suggestion (a cross-process limit on the number of concurrent pip invocations), but reformulated as:

1. Only one process on a machine is permitted to be building the same package at the same time
2. There's a top level "build ID" that pip passes down to any sub-builds that it spawns (e.g. the PID of the top level process combined with the name of the package being built)
3. If the build cache indicates that something is being built by a different build ID, then wait for that build to finish
4. If the build cache indicates that something is already being built for the same build ID, then bail out with an error reporting the circular dependency, and indicating that --binary <name> is going to be required for the build to work

pip would also need to implement @pfmoore's suggestion of exempting setuptools & wheel from the default logic of needing both setuptools and wheel as build dependencies, otherwise the implicit build dependency injection would inherently trigger the circular dependency detection logic.

Using the disk to avoid having to figure out OOP design problems is not a bad idea. That's like an intermediate option between implementing PEP 518 completely correctly and just hacking something together.

[ncoghlan]

It would also interact nicely with containerised environments and chroots in general, since we'd be able to use operating system level tools to keep different Python level builds isolated from each other, so pip would just need to figure out how to ensure its own subprocesses cooperate with each other.

[pfmoore]

@xoviat thanks for the summary above. As I'd said, I'd reached the limit of my understanding of the code in this area, and your explanation has helped me enormously.

[pradyunsg]

I had not actually looked at #4144's code before. I just did and I really don't want that to be shipping.

implementing PEP 518 completely correctly and just hacking something together.

Honestly, I think it comes down to this. Implementing PEP 518 completely and properly is a task which would/might delay pip 10 a (fair) bit if we go that way.

I think a safe-ish middle ground here would be to require build-dependencies to be available as wheels. That way, we can avoid the recursion problem since the build-dependencies (and all of their dependencies) would not need to be built via this process.

How does this sound?

[pradyunsg]

It's restricted but I think a restricted first implementation is better than a you-can-shoot-yourself-in-the-foot-if-you-are-not-careful first implementation.

[pfmoore]

Implementing PEP 518 completely and properly is a task which would/might delay pip 10 a (fair) bit if we go that way.

Thanks for confirming that - that was my fear.

However, now I am confused, as I thought that at least partial PEP 518 was already in master. Specifically, as I understand it #4647 demonstrates a bug in the PEP 518 support in master - so clearly we have something already, as whatever it is isn't correct...

So we have to do something, and it seems the options are:

1. Rip out whatever we have for PEP 518 support at the moment.
2. Tidy up what we have and ship partial support.
3. Implement PEP 518 fully before we release pip 10.

As you say (3) means a long delay before pip 10 and we have other fixes that I'd really like to see released (Unicode fixes being one we're regularly getting issues over). So I'm not keen on that. You didn't mention (1), and I'm not sure whether that's because you thought we had no PEP 518 support in place, or whether you assumed backing it out wasn't an option. Personally, I don't like the idea - it's a backward step, and it sends a pretty negative message about the PEP itself, if it's that hard to implement correctly. But I think we should be explicit about rejecting it.

Your proposal for (2), that we ship a version of PEP 518 that only supports wheels as build-dependencies (we'd still need to fix #4647, as the demonstration of that uses a build dependency that is a wheel) seems reasonable, in the sense that it's practical for us to implement. My main reservation is that I've no idea how problematic that restriction would be for people who want to use PEP 518.

So I guess I feel like we're stuck whatever we do, but partial support covering only wheels as build dependencies is the best option out of a bad lot :-(

The PEP authors (in #4799 (comment) and #4799 (comment) in response to my question in #4799 (comment)) were pretty definite that full PEP support required building any build dependencies, though, so it has to be only a stopgap.

But it's not me that will be implementing this, so I'm happy to go with the judgement of whoever does. One thing I have done is create #4803 and mark it as a release blocker, as a reminder that we should document how we deviate from the spec, if we need to.

(And off-topic for this issue, but can I suggest that we are careful not to make the same mistakes when we start implementing PEP 517? Let's make sure we understand all of the implementation implications before we get too deep into coding - my instinct is that PEP 517 is going to be an even more complex design problem than PEP 518...)

[ncoghlan]

I'm most familiar with distros when it comes to the "build everything from source" perspective, and we definitely separate the process of "bootstrapping the buildroot" from that of regular package builds. Full auto-bootstrapping from source is hard, since you end up having to do things like bootstrap your C compiler.

So for pip, I think it's reasonable to say that build dependencies will always be installable from wheel files. The refinement you can introduce post 10.x is to have a build cache that's distinct from the regular wheel cache, such that users of the cache can be sure that all the wheels in that were built in a controlled environment, rather than downloaded from PyPI or another index server.

Personally, I don't like the idea - it's a backward step, and it sends a pretty negative message about the PEP itself, if it's that hard to implement correctly.

I don't necessarily agree with that. It's hard to implement correctly for pip. But pip as stated in the PEP is one of the only front ends that people are going to use. I think it's our job as language implementers, and pip really is a language that people use to package their python projects, to make it as simple as possible to create a build system without having to think about all of these difficult issues. For them it should just work seamlessly because we've done the hard work.