Load build backend in isolated build environment with --no-index given

[cyrixsimon]

• Pip version: 10.0.1
• Python version: 3.6.5
• Operating system: openSUSE

Description:

Since version 10 pip raises an exception during installation for some python packages.

pip._internal.exceptions.DistributionNotFound: No matching distribution found for setuptools

Although the setuptools are installed.

setuptools          39.1.0     
setuptools-scm      2.0.0  

With pip version 9 everything works fine.

What I've run:

One python package which fails with pip installation is dateutil.

# /usr/local/python/bin/pip3.6 install --verbose --no-index .
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /tmp/pip-ephem-wheel-cache-titakidg
Created temporary directory: /tmp/pip-install-idx4_e_v
Processing /tmp/workspace/python-dateutil-2.7.2
  Created temporary directory: /tmp/pip-req-build-h_jueb1s
  Created temporary directory: /tmp/pip-build-env-lz7qvsgc
  0 location(s) to search for versions of setuptools:
  Could not find a version that satisfies the requirement setuptools (from versions: )
Cleaning up...
  Removing source in /tmp/pip-req-build-h_jueb1s
No matching distribution found for setuptools
Exception information:
Traceback (most recent call last):
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/commands/install.py", line 291, in run
    resolver.resolve(requirement_set)
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/resolve.py", line 103, in resolve
    self._resolve_one(requirement_set, req)
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/resolve.py", line 257, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/resolve.py", line 210, in _get_abstract_dist_for
    self.require_hashes
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/operations/prepare.py", line 324, in prepare_linked_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/operations/prepare.py", line 150, in prep_for_dist
    build_requirements)
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/operations/prepare.py", line 59, in _install_build_reqs
    for r in build_requirements
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/operations/prepare.py", line 59, in <listcomp>
    for r in build_requirements
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/index.py", line 533, in find_requirement
    'No matching distribution found for %s' % req
pip._internal.exceptions.DistributionNotFound: No matching distribution found for setuptools

[jaimguer]

I'm seeing the same with dateutil, --no-index, and --find-links on Amazon Linux 2017.03 and Python 3.4.3. I have setuptools-39.1.0.zip downloaded in the --find-links directory as well.

sudo /usr/local/bin/pip3 install -vvv -r requirements.txt --no-index --find-links /tmp/deps/                                                                                                                  
Collecting python-dateutil==2.7.3 (from -r /tmp/requirements.txt (line 58))                                                                                                                                   
  0 location(s) to search for versions of python-dateutil:                                                                                                                                                    
  Found link file:///mnt/tmp/deps/python-dateutil-2.7.3.tar.gz, version: 2.7.3                                                                                                                                
  Local files found: /mnt/tmp/deps/python-dateutil-2.7.3.tar.gz                                                                                                                                               
  Using version 2.7.3 (newest of versions: 2.7.3)                                                                                                                                                             
  Created temporary directory: /mnt/tmp/pip-build-env-pm8ysmpg                                                                                                                                                
  0 location(s) to search for versions of setuptools:                                                                                                                                                         
  Could not find a version that satisfies the requirement setuptools (from versions: )                                                                                                                        
Cleaning up...                                                                                                                                                                                                
  Removing source in <dirs>...                                                                                                                                                                                
No matching distribution found for setuptools                                                                                                                                                                 
Exception information:                                                                                                                                                                                        
Traceback (most recent call last):                                                                                                                                                                            
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/basecommand.py", line 228, in main                                                                                                               
    status = self.run(options, args)                                                                                                                                                                          
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/commands/install.py", line 291, in run                                                                                                           
    resolver.resolve(requirement_set)                                                                                                                                                                         
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/resolve.py", line 103, in resolve                                                                                                                
    self._resolve_one(requirement_set, req)                                                                                                                                                                   
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/resolve.py", line 257, in _resolve_one                                                                                                           
    abstract_dist = self._get_abstract_dist_for(req_to_install)                                                                                                                                               
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/resolve.py", line 210, in _get_abstract_dist_for                                                                                                 
    self.require_hashes                                                                                                                                                                                       
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/operations/prepare.py", line 324, in prepare_linked_requirement                                                                                  
    abstract_dist.prep_for_dist(finder, self.build_isolation)                                                                                                                                                 
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/operations/prepare.py", line 150, in prep_for_dist                                                                                               
    build_requirements)                                                                                                                                                                                       
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/operations/prepare.py", line 59, in _install_build_reqs                                                                                          
    for r in build_requirements                                                                                                                                                                               
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/operations/prepare.py", line 59, in <listcomp>                                                                                                   
    for r in build_requirements                                                                                                                                                                               
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/index.py", line 533, in find_requirement                                                                                                         
    'No matching distribution found for %s' % req                                                                                                                                                             
pip._internal.exceptions.DistributionNotFound: No matching distribution found for setuptools

[hynek]

Just as a data point, I ran into this error by adding a pyproject.toml file that only contained configuration for black. Pinning pip to <10 fixed it and sadly I can't investigate any further.

[dstufft]

I think if you add a pyproject.toml, you need to add the build dependencies too, so adding

[build-system]
requires = ["setuptools", "wheel"]

should fix it.

[benoit-pierre]

The presence of pyproject.toml triggers PEP 518 support, and build isolation. You won't be able to install such a project with pip if wheels for setuptools/wheel are not available: no index and not in cache, or forcing source installs. You can disable build isolation with --no-build-isolation.

[hynek]

Oh I think @benoit-pierre has it. All of us seem to be running --no-index and that can’t work anymore. So if the others have a pyproject.toml too, this is rather about docs and/or refusing no-index in in build isolation mode?

[benoit-pierre]

Using --no-index does not necessarily mean it will fails: if you use --find-links with a directory providing wheels for setuptools/wheel. Next version of pip should also support installing build dependencies from source (see #5336).

[pradyunsg]

Indeed. This is exactly the case that makes me think #5416 would be a good idea.

[pfmoore]

I don't think that #5416 is relevant here. If you have a pyproject.toml you need to be able to install your build tool, and --no-index precludes that, unless you have a local copy of the build tool wheels. It's an inconvenient (and may be unexpected) side effect of PEP 518 plus build isolation, but it's essentially "by design".

It may be that we can find some way of mitigating the impact (try again with build isolation disabled if the build tools aren't available for install, for example, or allow locally installed copies of the requested build tools "leak through" into the isolated build environment) but ultimately we have to bite the bullet and accept that PEP 517 and 518 are moving us into a world where we don't assume build tools are available, and that making sure they are available becomes the responsibility of the user running the build (whether by having local wheels or by manually specifying --no-build-isolation).

[asottile]

At least with python-dateutil 2.7.1 (which does not specify PEP518 information in pyproject.toml) this is an issue (python-dateutil has since added PEP518 information to pyproject.toml, perhaps making this issue more confusing)

[pfmoore]

Yeah, if projects include invalid pyproject.toml files (the build-system.requires key is mandatory according to PEP 518) then that's a different issue (and one that I argue in #5416 that we should just reject).

[benoit-pierre]

Should the download command include build dependencies, so pip download -d dir project && pip wheel/install --no-index -f dir project work?

[asottile]

hmmm, I was under the impression that pyproject.toml was "the replacement for setup.cfg" -- a generic place to put configuration information. Seems odd to require pip build information to be there?

[benoit-pierre]

It's not pip build information, it's the project build requirements, a (working) replacement for setup_requires support when using setuptools. (As well as a way to support other build systems with PEP 517).