Bootstrapping problem (how to bootstrap all-source environments)

[jaraco]

In this comment, @pfmoore suggests that we may need to re-think --no-binary, specifically around wheel but more generally around any or all build dependencies. In pypa/pip#7831, we've essentially concluded that --no-binary :all: is unsupported for bootstrapping issues. As reported in pypa/wheel#344 and and pypa/wheel#332, wheel is essentially blocked from adopting any form of PEP 518, or any other tool that requires pyproject.toml (because presence of a pyproject.toml implies PEP 518) unless wheel can elect to break --no-binary wheel installs.

Furthermore, the backend-path solution is likely not to be viable. Even at it's face backend-path adds a path to the PYTHONPATH when building, but unless a source checkout of wheel has had setup.py egg-info run on it, it won't have the requisite metadata to advertise its distutils functionality.

@pfmoore has suggested that it's not the responsibility of the PyPA or its maintainers to solve the issue, but that the downstream packagers should propose a solution. To that end, should wheel restore support for pyproject.toml and break --no-binary workflows, thus breaking the world for downstream packagers and incentivizing them to come up with a solution (that seems inadvisable)? Should wheel (and other use-cases) remain encumbered with this (undocumented) constraint in order to force legacy behavior when building wheel, thus alleviating the incentive to fix the issue (also sub-optimal)?

Perhaps solving the "bootstrapping from source" problem would be a good one for the PSF to sponsor.

[pradyunsg]

Perhaps solving the "bootstrapping from source" problem would be a good one for the PSF to sponsor.

@brainwane @ewdurbin @di @dstufft ^ for the Packaging-WG "Fundable Packaging Improvements" page.

[brainwane]

I'm interested in adding this to the list of fundable packaging improvements, but I need help with wording. Try answering each of these questions in about 20-200 words each:

• What is the current situation/context? Example: "Scientists need to install some Python packages via pip and others with conda."

• What ought to be fixed, made, or implemented?

• What problems would this solve, and what new capabilities would it cause?

[brainwane]

Anyone who wants to add this to the list of fundable packaging improvements can now submit a pull request on https://github.com/psf/fundable-packaging-improvements .

[jaraco]

This issue is more general than just wheel and affects a number of packages beyond setuptools and wheel.

In pypa/setuptools#980, I learned of the issue in build tools. If build tools have dependencies, and those dependencies use the build tool, it's not possible to build the dependencies from source. For example,

• Setuptools depends on appdirs.
• Appdirs is built by setuptools.

An attempt to build/install Setuptools or Appdirs from source rather than wheels (as many platforms prefer to do) will fail.

Setuptools has worked around this issue by vendoring all of its dependencies, although it has a stated goal to stop vendoring (pypa/setuptools#2825).

However, in python/importlib_metadata#392 today, I've learned of even more places where these circular dependencies can manifest. In this case:

• setuptools_scm depends on importlib_metadata.
• importlib_metadata relies on setuptools_scm to build.

So unless setuptools_scm is pulled pre-built, when it attempts to build from source, it also pulls importlib_metadata which requires setuptools_scm to build.

I've probably stated before, I wish for libraries like setuptools_scm to be able to adopt dependencies at a whim (and not have users encountering errors).

These new cases lead me to believe the problem is not solveable by downstream packagers except by forcing all build tools (and their plugins) to vendor their dependencies.

I'd like to avoid each of these projects needing to come up with a bespoke hack to work around this issue and eliminate this footgun that will reduce adoption due to these bootstrapping issues.

[jaraco]

What is the current situation/context? Example: "Scientists need to install some Python packages via pip and others with conda."

The situation is in any context where in a project, any of the PEP 518 build-system.requires has a dependency whose build-system.requires also leads back to the original project. For the general case, this situation is not a problem as the pip resolver can resolve dependencies and pull in their pre-built versions, but when users wish to build from source, the circular dependency prevents building.

What ought to be fixed, made, or implemented?

First, it's not obvious what needs to be implemented. This project still needs work to explore the problem space and propose options to be considered by the PyPA. It's going to require coordination across many projects and may require a PEP.

A rough sketch of some options to consider:

• Implement a bootstrap handler that will identify and intercept these circular dependencies and provide an alternate way to make the functionality available without building (such as by adding the project's sources root and src/ directory to sys.path).
• Require the builders to keep blessed, vendored copies of dependencies of build tools.
• Explicitly disallow building these projects from source and require downstream packagers that wish to bootstrap a system from source to own the problem and devise their own workaround.
• Require build tools not to have circular dependencies. Publish this formal requirement.

What problems would this solve, and what new capabilities would it cause?

It would solve the class of problems where errors occur when users attempt to build from sources but dependencies exist in the build dependencies. It would mean that build tools like setuptools and setuptools_scm could naturally depend on other libraries. It would mean that other projects could rely on these build tools without getting reports of failed builds.

[jaraco]

Anyone who wants to add this to the list of fundable packaging improvements can now submit a pull request on https://github.com/psf/fundable-packaging-improvements .

Based on the criteria in the readme there, I don't believe this project is yet ready for a fundable project. There is not yet consensus or an accepted PEP. To the contrary, the initial feedback has been negative, meaning there's probably a need to develop that consensus first. Maybe the additional use case described above will help nudge us in that direction.

In the meantime, I'm slightly inclined not to workaround the issues for users, thus masking the root cause.

[pfmoore]

I think there's a good fundable project in here, but it's a research project. I think the problem is that we are getting feedback that people want to be able to "build everything from source", but that causes an issue because bootstrapping involves circular dependencies. This sounds pretty much identical to the issues of building a C compiler, so we could probably learn a lot from how people solved that problem. Also, presumably the people wanting to build Python packages from scratch stop at some point - do they need to build the Python interpreter itself from sources, for example?

I'd suggest proposing a research project to find out why people want to build with --no-binary :all:, and in particular what are the precise constraints they are working to. This could involve user surveys, requirements collection, etc. That's one of the reasons I think this would make a good fundable project - the skills needed to do a good job here are not common in the Python packaging community, so hiring specialists would be very cost effective.

The deliverables from such a project could be a requirement spec for "build everything from source" activities and in particular a listing of the constraints that need to be satisfied for any solution.

Once we have that documentation, follow-up projects to actually design and/or implement a solution will be much easier to specify. Or it may be that the negative reception that's happened so far can be crystallised into specific issues that we (the Python packaging community) aren't willing to support, which will give users much more actionable feedback at least.

[jaraco]

In pypa/setuptools#4457 (comment), I describe what appears to be a distilled version of the implicit constraint:

• Build backends can only have (build or runtime) dependencies on projects that don't rely on that build backend.

I wonder if instead the constraint should be:

• Environments that rely exclusively on building from source (e.g. --no-binary :all:) must use a frontend that implements caching of built artifacts in order to break cycles in build dependencies.