Avoid generating metadata in `pip download --no-deps ...`

[tlynn]

"pip install xxx --download yyy --no-deps" runs "python setup.py egg_info" after downloading (to generate the requires.txt dependency list?). This seems unnecessary.

This is a problem in the case of "cryptography", since its setup.py uses setup_requires, which automatically downloads its dependencies at the slightest nudge, even just --help (cryptography issue #716).

This impedes using pip install --download to build a collection of trusted (checksummed) modules for later installation.

[xolox]

The python setup.py egg_info command is run for metadata discovery. It's an unfortunate fact of the Python packaging ecosystem that anything related to packaging always involves arbitrary code execution (referring to setup.py). The setuptools setup_requires feature is a particularly nasty example of this (don't get me wrong, I understand it has its uses, but it's still nasty). So I believe this should be fixed on the side of the cryptography package, because we can't expect pip to reasonably deal with setup_requires:

• Setuptools downloads and unpacks setup requirements on demand every time a setup.py script is evaluated in any way, even if you run something very simple which seems like it should be a read only operation, e.g. python setup.py --help or python setup.py --version.

  _{As a bonus setuptools uses easy_install for setup requirements which can cause several problems: Using pip install --no-index will not stop easy_install from downloading remote package archives and using pip install --index-url=... will not instruct easy_install to use the custom index URL, instead the URL has to be set in ~/.pydistutils.cfg (you can't even configure it using an environment variable).}

• I assume setuptools always installs setup requirements as the very first thing because setup requirements can for example "install" custom setup commands that should appear in the output of python setup.py --help-commands. Setuptools can't decide what setup requirements will do (the possibilities are unbounded :-) so setuptools errs on the side of caution and always installs setup requirements before doing anything else with setup.py.

• The only people in a position to judge which setup commands actually need setup requirements are the developers of the package that defines setup requirements. For this reason I created pull request Work around for side effects in setup.py script pyca/cryptography#1257 to get this fixed on the side of the cryptography package (it seems like it's possible to fix this with a not-too-big change to the setup.py script).

It is a shame that the setup_requires feature of setuptools is so rudimentary, it would have been nice if there was a canonical and supported way to tell setuptools which setup commands really need setup requirements. That might have avoided this mess.

[sholsapp]

@xolox thanks for the detailed description, that helped me understand what was going on here.

Another unfortunate side effect of the behavior reported by @tlynn is that we can't run parallel downloads for packages that share transitive dependencies. What one sees is that when using something like
pip install --exists-action=i --download-dir ./download-cache <lots of dependencies that share transitive dependencies> are lots of OSErrors about paths not existing, presumably because invocations of setup.py egg_info on packages that are in the process of being downloaded result in files not being found.

This is probably a convoluted example, but I figured I'd share my use case here.

[dstufft]

When we implement this, we should take care to prevent path traversal bugs like mentioned in #731.

[pradyunsg]

@dstufft Can this be closed since the --download flag has been removed?

[dstufft]

I think the issue is still valid for pip download --no-deps?

[pradyunsg]

I guess so.

[pdecat]

What about not running setup.py egg_info on pip install --no-deps ? (Even without --no-download)

That would allow working around a current pylxd issue canonical/pylxd#308 (comment)

[RonnyPfannschmidt]

how about additionally not running egg-info if the tarball ships an egg_info file to begin with
i believe these days the setuptools-generated packages can be trusted

however this should be introduced over a grace period where pip warns about differences between shipped and freshly generated

[RonnyPfannschmidt]

i just ran into this when trying to use pip as sane caching downloader of other stuff
(its just really neat for travis use since the download uses the pip cache which is cached + all the bells and whistles)

[RonnyPfannschmidt]

i just tried to disable this for downloading and the issue spans resolver, preparer and a few other utilities - i will need to retry with more time