Install install_requires before running python setup.py bdist_wheel

[paugier]

Environment

• pip version: 19.0.1
• Python version: CPython 3.7

Description

pip executes python setup.py bdist_wheel before installing the packages listed in install_requires.

For a simple package depending on numpy, using numpy in setup.py but such that python setup.py egg_info works fine without numpy, it shouldn't be necessary to use setup_requires (and to have a setup.py compatible with setup_requires, which is not easy).

From https://pip.pypa.io/en/latest/reference/pip_install/#installation-order

Although the new install order is not intended to replace (and does not replace) the use of setup_requires to declare build dependencies, it may help certain projects install from sdist (that might previously fail) that fit the following profile:

• They have build dependencies that are also declared as install dependencies using install_requires.
• python setup.py egg_info works without their build dependencies being installed.
• For whatever reason, they don’t or won’t declare their build dependencies using setup_requires.

For example, for this setup.py

import sys
from setuptools import setup

if "egg_info" in sys.argv:
    setup()
    sys.exit()

import numpy as np

setup()

and this setup.cfg:

[metadata]
name = irequires

[options]
install_requires =
    numpy

the command pip install . fails with

Processing /home/users/augier3pi/Dev/try_py_dependencies/irequires
Collecting numpy (from irequires==0.0.0)
  Using cached https://files.pythonhosted.org/packages/3d/10/62224c551acfd3a3583ad16d1e0f1c9e9c333e74479dc51977c31836119c/numpy-1.16.0-cp37-cp37m-manylinux1_x86_64.whl
Building wheels for collected packages: irequires
  Building wheel for irequires (setup.py) ... error
  Complete output from command /home/users/augier3pi/tmp/tmp_venv/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-_0f8h24y/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-6n4locuo --python-tag cp37:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-_0f8h24y/setup.py", line 10, in <module>
      import numpy as np
  ModuleNotFoundError: No module named 'numpy'
  
  ----------------------------------------
  Failed building wheel for irequires
  Running setup.py clean for irequires
  Complete output from command /home/users/augier3pi/tmp/tmp_venv/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-_0f8h24y/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" clean --all:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-_0f8h24y/setup.py", line 10, in <module>
      import numpy as np
  ModuleNotFoundError: No module named 'numpy'
  
  ----------------------------------------
  Failed cleaning build dir for irequires
Failed to build irequires
Installing collected packages: numpy, irequires
  Running setup.py install for irequires ... done
Successfully installed irequires-0.0.0 numpy-1.16.0

Expected behavior

pip should install numpy before executing python setup.py bdist_wheel so that this simple setup works fine as explained in https://pip.pypa.io/en/latest/reference/pip_install/#installation-order.

Note

Listing packages both in install_requires and setup_requires don't work well. See pypa/setuptools#209 and pypa/setuptools#391

[paugier]

Note that we still get exactly the same error even by adding setup_requires in setup.cfg,

[metadata]
name = irequires

[options]

install_requires =
    numpy

setup_requires =
    numpy

[RonnyPfannschmidt]

setup_requires does not match with PEP-517, so thats not supportable for pip
install_requires lists the things a package needs when it was installed, so thats explicitly distinct from what it needs for building

if you need numpy for the build of a package in isolation, use the PEP-517/518 metadata or set up a custom build without isolation

[paugier]

Few days ago, I opened issue #6144 on PEP-517. I think this PEP (isolated build) is a problem for some packages (and also for compatibility with conda) so many packages won't be able to use it.

It's really a problem that the very simple package described in the issue cannot be installed with just one simple pip command.

It would be so simple for pip! The numpy wheel is ready.

I really don't understand what would be the problem with installing the wheels as soon as they are ready and not to build all wheels and then installing them.

Otherwise, why is there the sentence I quoted in https://pip.pypa.io/en/latest/reference/pip_install/#installation-order ?

[RonnyPfannschmidt]

build isolation builds things in a completely pristine environment, - its a positive key property for repeatability and comprehension

a key need is not having random things affect the state, which seems to be your desire to get as a feature - my personal opinion on that is -- sorry no go, please try again with something reliable

[paugier]

I really understand that isolation build is very good and useful for most packages.

I also think that many good things in Python have been done before without isolation build and that it's not a good idea to force people to use isolation build if you want to impose a new standard (PEP 518, pyproject.toml), but anyway.

You also have to understand that there are reasonable reasons to avoid isolation builds (for example compatibility with conda [which is not a small subject in numerical / scientific / data Python], or using packages for which there are no wheels on PyPI, as mpi4py).

So simple things have to work without build isolation and pyproject.toml. In particular, things like what I describe in this issue should work. At least, there should have a pip option to get this nice behavior (which is not crazy).

It really seems that it is not even technically difficult from the pip point of view.

What is the drawback of installing wheels as soon as they are ready? It seems to me that pipenv does it (at least tox-pipenv). Anyway, isolation build won't become the default for package without pyproject.toml.

What is the advantage of building the wheels from all collected sdist with the old environment?

Also, it would be nice to answer about the sentence at the end of the paragraph https://pip.pypa.io/en/latest/reference/pip_install/#installation-order

Is it just wrong? If it is wrong, should it be removed from the documentation?

Moreover, why was it included in the official pip documentation if it is a such a terrible practice to rely on non-isolated builds and on the evolution of the environment during the pip command execution?

[pradyunsg]

The guarantee is about installation order; not build order. It does not suggest that it would install a install_requires entry before a setup.py is executed.

I suggest you use PEP 518, or disable isolation with --no-build-isolation, if you have a build-time dependency.

[paugier]

It does not suggest that it would install a install_requires entry before a setup.py is executed.

Yes it does! 🙂

I'm sure that the last paragraph of https://pip.pypa.io/en/latest/reference/pip_install/#installation-order is really about this issue. From what is written (see my quote in the first message of the issue), the example in this issue should work. It's clear.

I have to repeat that PEP 518 (pyproject.toml) implies PEP 517 (build isolation) and that build isolation is not compatible with conda environment (so pyproject.toml won't be considered in conda env, so the problem won't be solved in conda env) and cannot be used for many projects (for example all projects using mpi4py in their setup.py).

Using --no-build-isolation leads to the same result as without pyproject.toml, i.e. pip is not able to install the example package in a fresh environment.

So unfortunately, pyproject.toml is not a nice universal solution.

A simple solution would be that pip installs each wheel that has to be installed just when it is ready (before the next build) and not at the end, when all wheels have been built. What is the problem of doing that?

[dstufft]

install_requires are runtime dependencies, not build time, and pip doesn't install them prior to trying to build the package in question.

[paugier]

@dstufft

I know. I try to tell you that there is a problem with the new standard (pyproject.toml / build isolation) of how to declare build dependencies and that it would be good if what is written in the official documentation works.

From https://pip.pypa.io/en/latest/reference/pip_install/#installation-order

Although the new install order is not intended to replace (and does not replace) the use of setup_requires to declare build dependencies, it may help certain projects install from sdist (that might previously fail) that fit the following profile:

• They have build dependencies that are also declared as install dependencies using install_requires.
• python setup.py egg_info works without their build dependencies being installed.
• For whatever reason, they don’t or won’t declare their build dependencies using setup_requires.

If there a problem with my English? Why don't you understand me? Isn't it clear that if we take seriously what is written in this paragraph, the example in this issue should work.

• the build dependency (here, numpy) is declared in install_requires.
• python setup.py egg_info works without numpy
• For whatever reason (in particular because it leads to problems to declare the same package in install_requires and in setup_requires, as already explained above), we don't use setup_requires

I have to admit that I am sad about how my remarks (this issue and issue #6144) are treated. In scientific and data applications, conda is used a lot. My reasoning is that it is not a great thing that the Python community is split in terms of installation tools, so I tried a lot to be also able to work without conda. I thought that pip could also be adapted for our applications. But now, I start to see how concerns from my community are treated by pip core developers.

[pfmoore]

Maybe the wording of that document is a little unclear. But the principle is simple:

• Build dependencies are declared via setup_requires (or since PEP 518, via requires in pyproject.toml).
• Runtime requirements (or "install dependencies") are declared via install_requires.

Those terms are possibly a little confusing (they confuse me!) but they have been standard for many, many years.

"They have build dependencies that are also declared as install dependencies using install_requires" means that the project declares that foo is both a build dependency and a runtime dependency (by including it in both pyproject.toml and install_requires). There's no implication here (or anywhere else) that a runtime dependency will automatically be available as a build dependency. Since PEP 518 support in pip introduced build isolation, clearly understanding the distinction is crucial if you expect your build step to work properly.

You say that "For whatever reason (in particular because it leads to problems to declare the same package in install_requires and in setup_requires, as already explained above), we don't use setup_requires". That's fine - setup_requires is now obsolete, precisely because it had issues (the build requirements were downloaded by setuptools and not by pip, for example). But that means you have to use pyproject.toml and the requires key, not that you can expect pip (or any other tool) to automatically infer your build dependencies¹. But before pyproject.toml, you'd have either had to use setup_requires, or rely on some sort of undocumented and accidental behaviour.

I have to admit that I am sad about how my remarks are treated

All I can say is that we're trying. Personally, I'm struggling to understand what you want and why the approaches that have been suggested to you are so unacceptable. I will say that "because conda needs pip to work like this" is not a compelling argument, at least to me. While I'm 100% in favour of better interoperability between conda and pip, I don't think that will be achieved by simply making pip change "to work with conda". What's needed is a better understanding of the shared requirements of the two ecosystems and how both tools can compromise to make things work better - while still remaining true to their core goals and user communities.

¹ It's possible that the setuptools backend is introspecting your setup.py and converting setup_requires requirements into build requirements as part of the get_requires_for_build_wheel hook. But I don't understand that code so I might be wrong, and even if I'm not that's something specific to setuptools, not anything that pip is involved with.