EnvironmentError when installing a package with root-owned directories under excluded packages

[richin13]

Environment

• pip version: pip 19.0.1 (tried with 19.0.2 as well)
• Python version: ^3.7 (tried with 3.5.6 as well)
• OS: Linux 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux (Linux Mint 19.1 Tessa)
• Example package tree:

.
├── Dockerfile
├── examples
│   └── library_docker
│       └── volumes
│           └── postgresql
│               └── local_data [error opening dir]
├── <library-code>
│   └── ...
├── mypy.ini
├── README.md
├── requirements-test.txt
├── setup.py
└── tox.ini

• setup.py relevant section:

setup(
    ...,
    packages=find_packages(exclude=[
        '*.egg-info.*',
        '.tox*',
        'test.*',
        'examples.*',  # <---- Notice here!
        'venv.*',
    ]),
)

Description

Pip fails to install a package when one of its subdirectories has non-read permissions even when that subdirectory is listed as excluded in the packages section of the setup() function
In the directory tree shown above, examples/library_docker/volumes/postgresql/local_data is a mounted docker volume with a set of permissions drwx------ and owned by root user

Expected behavior
The package should install no matter what the permissions of subdirectories in excluded packages are

How to Reproduce
So replicating this shouldn't be hard; I tried the following:

mkdir my-library
cd my-library
mkdir library
mkdir -p examples/docker_examples/volumes/data
chown root:root examples/docker_examples/volumes/data # might need sudo for this
chmod 600 examples/docker_examples/volumes/data # might need sudo for this
touch library/__init__.py setup.py
cat > setup.py << EOF
from setuptools import setup, find_packages

setup(
    name='dummy',
    version='0.1.1',
    description='Dummy library',
    long_description='',
    author='Ricardo Madriz',
    license='MIT',
    install_requires=[],
    packages=find_packages(exclude=[
        '*.egg-info.*',
        '.tox*',
        'test.*',
        'examples.*',
        'venv.*',
    ]),
    keywords='dummy',
)
EOF

pip --verbose install .

Output

Created temporary directory: /tmp/pip-ephem-wheel-cache-1ozi4fjy
Created temporary directory: /tmp/pip-req-tracker-2elnf8wu
Created requirements tracker '/tmp/pip-req-tracker-2elnf8wu'
Created temporary directory: /tmp/pip-install-pygw0ny9
Processing /path/to/codebase
  Created temporary directory: /tmp/pip-req-build-as_xjkei
Could not install packages due to an EnvironmentError.
Traceback (most recent call last):
  File "/home/richin13/.pyenv/versions/3.7.0/envs/library-gu2to/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 315, in run
    resolver.resolve(requirement_set)
  File "/home/richin13/.pyenv/versions/3.7.0/envs/library-gu2to/lib/python3.7/site-packages/pip/_internal/resolve.py", line 131, in resolve
    self._resolve_one(requirement_set, req)
  File "/home/richin13/.pyenv/versions/3.7.0/envs/library-gu2to/lib/python3.7/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/richin13/.pyenv/versions/3.7.0/envs/library-gu2to/lib/python3.7/site-packages/pip/_internal/resolve.py", line 242, in _get_abstract_dist_for
    self.require_hashes
  File "/home/richin13/.pyenv/versions/3.7.0/envs/library-gu2to/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 334, in prepare_linked_requirement
    progress_bar=self.progress_bar
  File "/home/richin13/.pyenv/versions/3.7.0/envs/library-gu2to/lib/python3.7/site-packages/pip/_internal/download.py", line 865, in unpack_url
    unpack_file_url(link, location, download_dir, hashes=hashes)
  File "/home/richin13/.pyenv/versions/3.7.0/envs/library-gu2to/lib/python3.7/site-packages/pip/_internal/download.py", line 734, in unpack_file_url
    shutil.copytree(link_path, location, symlinks=True)
  File "/home/richin13/.pyenv/versions/3.7.0/envs/library-gu2to/lib/python3.7/shutil.py", line 359, in copytree
    raise Error(errors)
shutil.Error: [('/path/to/codebase/examples/library_docker/volumes/postgresql/local_data', '/tmp/pip-req-build-as_xjkei/examples/library_docker/volumes/postgresql/local_data', "[Errno 13] Permission denied: '/path/to/codebase/examples/library_docker/volumes/postgresql/local_data'")]
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-2elnf8wu'

[cjerdonek]

Did this ever work with pip (e.g. 18.x release)?

[richin13]

Nope, just tested it with 18.1 with the same results

[richin13]

Further reference:

pypa/setuptools#733

pypa/setuptools#809

[pradyunsg]

I can't reproduce this with the given example.

[richin13]

Humm weird. I just run the example and the error is still there. Are you sure you used sudo for changing the owner and permissions of the examples/docker_examples/volumes/data folder?

[chrahunt]

Hi @richin13. I was able to reproduce this and it looks like it is caused by a known pip behavior.

The underlying issue is that pip makes a full copy of the source directory into a temporary directory before building the project into a wheel (which we do before installation). This will be fixed by generating an sdist first (which shouldn't include your example directories), then extracting that into the temporary directory that we use for the build. That activity is tracked by #2195.

Since we're already tracking the fix there, I will close this issue. Please don't hesitate to correct anything above or let us know if you have any other issues. Thanks!