dependencies listed in both setup_requires and install_requires install to temporary .egg dir

Originally reported by: james_salter (Bitbucket: james_salter, GitHub: Unknown)

---

Found on setuptools 16.0, python 2.7.9, windows 8.

Test script:

from setuptools import setup

package = "football-data"

setup(
    setup_requires=[package],
    install_requires=[package],
    name="test_setup_requires"
)

After running this script from c:\dev, my c:\python27\lib\site-packages\easy-install.pth contains:

c:/dev/.eggs/football_data-0.1.2-py2.7.egg

i.e. the package has been installed with its location in the temporary .eggs directory used to fulfil the install_requires directive.

This doesn't happen if the setup_requires key is omitted - the package is then installed to site-packages as expected. And it isn't installed at all if install_requires is omitted, as expected.

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/391

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

This was supposed to have been fixed in #209.

Original comment by anthrotype (Bitbucket: anthrotype, GitHub: anthrotype):

---

I'm afraid this is not fixed yet.

In particular, it affects all packages that use CFFI to build extension modules, such as cryptography for example.
CFFI in fact is required both in setup_requires and in install_requires.

If I clone, say, cryptography repo, and do python setup.py install, then cffi and pycparser are downloaded to a temp .eggs/ folder, in order to build cryptography's CFFI extension modules; but then, when it comes to installing them, setuptools simply adds a link in easy-install.pth to the packages which were already downloaded by setup_requires

I'm using the latest setuptools version 18.3.2 on Python 3.4.3 (OS X Yosemite). It's worth mentioning that if I do pip install -v . instead, everything is installed correctly in site-packages, including cffi and pycparser.

Here's how the easy-install.pth looks like after I do python setup.py install for cryptography package.

import sys; sys.__plen = len(sys.path)
./cryptography-1.1.dev1-py3.4-macosx-10.10-x86_64.egg
./six-1.9.0-py3.4.egg
./pyasn1-0.1.9-py3.4.egg
./idna-2.0-py3.4.egg
/Users/cosimolupo/Documents/Github/cryptography/.eggs/cffi-1.2.1-py3.4-macosx-10.10-x86_64.egg
/Users/cosimolupo/Documents/Github/cryptography/.eggs/pycparser-2.14-py3.4.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

There was a ticket open on cryptography repository about this, but they closed it as they "don't think it's actionable as a cryptography bug", but is a setuptools thing.
I hope it'll get fixed soon, as it makes distributing cffi extension modules a bit harder.

Thank you.

Cosimo

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I agree. This appears not to have been fixed. Rather than re-open #209, I'm going to continue to track the issue here.

Original comment by anthrotype (Bitbucket: anthrotype, GitHub: anthrotype):

---

Any news on this? I confirm it's still happening as of the current version.

git clone https://github.com/pyca/cryptography.git
cd cryptography
virtualenv test
source test/bin/activate
pip install --upgrade setuptools
python setup.py install
python -c "import cffi; print(cffi.__file__)"

cffi (and pycparser) gets installed only via easy-install.pth links to the temporary invisible .eggs/ subfolder, instead of being copied to site-packages.

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

No work has been done on this, other than exploratory efforts to obviate setup_requires, and even the techniques there wouldn't help here.

The fundamental issue is that at the time easy_install runs to install the install_requires packages, those packages are already present in the current runtime environment. It's interesting to me that it creates a link to the existing .egg, which suggests to me there may be a code point where creating that link could be replaced with copying the file instead of linking it.

I welcome proposals or pull requests.

Original comment by andi0661 (Bitbucket: andi0661, GitHub: Unknown):

---

The whole point of setup_requires is to be able to use the package, so regardless of where it is being installed, it is necessarily imported while the setup script runs. So as long as the install_requires logic skips the install based on finding the package imported, and as long as the setup_requires logic installs the package to a place other than the desired one the install_requires logic would install to if it did, we are caught in a circle.

Based on this description of the circle, here are some options on how it might be broken. Just as some food for thought - I did not dig deep enough into distutils, setuptools, etc to claim any of them would work:

• Change the install_requires logic to skip installation of the package if it is currently loaded, to skip if already installed at the place it would install to.
• Change the setup_requires logic to install to the same place install_requires would install to. This is probably not a desirable goal, but a variation of that idea would be to introduce a new keyword e.g. setup_and_install_requires that does that. If a package is needed both during setup and after install, it would be specified with setup_and_install_requires (and not any of the other two), causing it to be installed to the normal destination install_requires would install to, and it would be imported for use during the setup script run.