Unable to build no-binary :all: with use-pep517

[jaraco]

Environment

• pip version: 20.0.2
• Python version: 3.8.1
• OS: macOS 10.15.3

Description

As distlib doesn't publish a wheel, and because it has an implicit dependency on setuptools, I'm attempting to use --use-pep517 to force pip to ensure that setuptools/wheel are present for building, but when I do, pip fails to install wheel:

~ $ pip install --use-pep517 --no-binary :all: distlib                                                                                                                        
Collecting distlib
  Using cached distlib-0.3.0.zip (571 kB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-6j9zlufg/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel
       cwd: None
  Complete output (13 lines):
  Collecting setuptools>=40.8.0
    Using cached setuptools-45.3.0.zip (862 kB)
  Collecting wheel
    Using cached wheel-0.34.2.tar.gz (58 kB)
      ERROR: Command errored out with exit status 1:
       command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-hd2m84od/wheel/setup.py'"'"'; __file__='"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-hd2m84od/wheel/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-hd2m84od/wheel/pip-egg-info
           cwd: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-hd2m84od/wheel/
      Complete output (3 lines):
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
      ModuleNotFoundError: No module named 'setuptools'
      ----------------------------------------
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-6j9zlufg/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel Check the logs for full command output.
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/bin/pip-run", line 8, in <module>
    sys.exit(run())
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/__init__.py", line 18, in run
    with deps.load(*deps.not_installed(pip_args)) as home:
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/deps.py", line 75, in load
    _installable(args) and subprocess.check_call(cmd)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/local/bin/python', '-m', 'pip', 'install', '-t', '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-mzykfthi', '--use-pep517', '--no-binary', ':all:', 'distlib')' returned non-zero exit status 1.

Expected behavior

pip should be able to assemble a build environment, either by disabling the no-binary option for those requirements or (preferably) by providing a bootstrap environment for these build dependencies.

Note: This may be an issue that can be addressed with wheel.

[jaraco]

I attempted to address this within wheel at pypa/wheel#344, but it seems that even declaring the build dependencies, pip attempts to install wheel from sdist and fails when setuptools isn't already present.

$ pip-run --no-binary :all: git+https://github.com/pypa/wheel@bugfix/344-explicit-setuptools                                                                            
  Running command git clone -q https://github.com/pypa/wheel /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-req-build-d31ewf1h
Collecting git+https://github.com/pypa/wheel@bugfix/344-explicit-setuptools
  Cloning https://github.com/pypa/wheel (to revision bugfix/344-explicit-setuptools) to /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-req-build-d31ewf1h
  Running command git checkout -b bugfix/344-explicit-setuptools --track origin/bugfix/344-explicit-setuptools
  Switched to a new branch 'bugfix/344-explicit-setuptools'
  Branch 'bugfix/344-explicit-setuptools' set up to track remote branch 'bugfix/344-explicit-setuptools' from 'origin'.
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-p3mtk_lh/normal --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- wheel
       cwd: None
  Complete output (11 lines):
  Collecting wheel
    Using cached wheel-0.34.2.tar.gz (58 kB)
      ERROR: Command errored out with exit status 1:
       command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-eouk8ka8/wheel/setup.py'"'"'; __file__='"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-eouk8ka8/wheel/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-eouk8ka8/wheel/pip-egg-info
           cwd: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-eouk8ka8/wheel/
      Complete output (3 lines):
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
      ModuleNotFoundError: No module named 'setuptools'
      ----------------------------------------
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-p3mtk_lh/normal --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- wheel Check the logs for full command output.
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/bin/pip-run", line 8, in <module>
    sys.exit(run())
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/__init__.py", line 18, in run
    with deps.load(*deps.not_installed(pip_args)) as home:
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/deps.py", line 75, in load
    _installable(args) and subprocess.check_call(cmd)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/local/bin/python', '-m', 'pip', 'install', '-t', '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-trtwrd5a', '--no-binary', ':all:', 'git+https://github.com/pypa/wheel@bugfix/344-explicit-setuptools')' returned non-zero exit status 1.

I think pip probably should do something here to allow wheel to build when installed from source.

[pfmoore]

Is this not related to in-tree backend support (which pip doesn't yet include, IIRC)? I thought the idea of that PEP update was precisely to allow projects to be defined in a way that meant they worked even if the user requested that everything was built from source.

If it's not then I think we need to get a much better understanding of what pip is expected to do in the face of --no-binary :all: before deciding on a solution, rather than firefighting individual use cases when they arise.

[jaraco]

I believe this issue is unrelated to in-tree backend support, though I do think that support is present in pip 20). The issue may be described thus:

Attempting to build any package with --use-pep517 and --no-binary :all:, pip downloads setuptools and wheel and attempts to build those (not from wheels). When attempting to build wheel, it fails because setuptools is not present.

It might just be the case that when using pep517 to build a package, it should also use pep517 to build the package's dependencies also... except that wheel would need wheel, leading to an infinite loop.

we need to get a much better understanding of what pip is expected to do

Agreed. Maybe the answer is that no-binary :all: is unsupported.

[pfmoore]

It might just be the case that when using pep517 to build a package, it should also use pep517 to build the package's dependencies also... except that wheel would need wheel, leading to an infinite loop.

Eww. Yes, that is a variant of the bootstrapping issue that in-tree backend support was addressing, but sufficiently different to be a new problem in its own right.

Maybe the answer is that no-binary :all: is unsupported.

That would be my strong preference. I honestly don't see why pip has to be the tool to handle the "build everything from raw source" problem. But there were fairly strong voices arguing that we had to support that case, so I think they should be the ones defining the solution here.

[jaraco]

In my use-case, I wasn't concerned with the "build everything from raw source" use case, but was rather attempting to build a single package from source and I used --no-binary :all: as a shorthand for "please build from source". In my case, I could have used --no-binary distlib, which would have been more selective.

[jaraco]

In the referenced vendoring issue, I learned that it's not just system packagers that want to build from source but also packages that wish to verify licenses.

[McSinyx]

Coming from ActiveState/appdirs#141,

Attempting to build any package with --use-pep517 and --no-binary :all:, pip downloads setuptools and wheel and attempts to build those (not from wheels). When attempting to build wheel, it fails because setuptools is not present.

How about letting setuptools and wheel be hard runtime dependencies? Just like we don't build pip when we want to use pip to build other wheels, then by the same logic, other build tools should not be built during the build process of other packages.

[uranusjr]

I think the whole idea of build isolation is to ultimately remove setuptools and wheel from needing to present in the runtime environment, because we can pull it if needed in the isolated build environment instead.

[McSinyx]

Thank you @uranusjr, I totally missed that point.

[jaypadariya]

chatboat python

if you want to install chatterbot

pip install chatterbot --no-binary :all:

you should use "pip install chatterbot --no-binary :all:"

before that run under three codes and install it.

conda remove --force PyYAML
pip install pint
pip install mathparse