[BUG] setuptools 64.0.0 breaks editable install

[ajasmin]

setuptools version

setuptools==64.0.0

Python version

Python 3.10.6

OS

Ubuntu jammy

Additional environment information

No response

Description

After creating a simple project and installing it as editable with pip install -e, I'm unable to import modules from the package.

Expected behavior

We should be able to import a package installed as editable.

How to Reproduce

1. Create this simple project structure

pyproject.toml

[build-system]
requires = [
  "setuptools==64.0.0",
  "wheel",
]
build-backend = "setuptools.build_meta"

[project]
name = "foo"
version = "1.0.0"

[tool.setuptools]
packages.find.include = ["foo.*"]

setup.py

from setuptools import setup

setup()

foo/__init__.py

__version__ = "1.0.0"

2. Create and activate a venv:

python -mvenv venv
. venv/bin/activate

3. Install the project as editable:

python install -e .

Change to another directory and try to import the package:

python -c "import foo"

Output

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'foo'

[abravalheri]

Hi @ajasmin thank you very much for reporting this.

I went through your reproduction steps, but I have a doubt: have you activated your virtual environment? (It does not seem to be part of the steps).

Instead of relying on the activation, I tried to reproduce the same experiment, but using the path to the Python executable directly. This is what I run in my terminal:

cd /tmp
rm -rf foo
mkdir -p foo/foo
cd foo
cat <<EOF > pyproject.toml
[build-system]
requires = [
  "setuptools==64.0.0",
  "wheel",
]
build-backend = "setuptools.build_meta"

[project]
name = "foo"
version = "1.0.0"

[tool.setuptools]
packages.find.include = ["foo.*"]
EOF

cat <<EOF > setup.py
from setuptools import setup

setup()
EOF

echo '__version__ = "1.0.0"' > foo/__init__.py

python3.8 -mvenv venv

./venv/bin/python -m pip install -e .

cd /tmp

/tmp/foo/venv/bin/python -c "import foo"

... I cannot see any error with these steps.

Could you please verify if the same problem persists if you use:

a) the activation script before running Python
b) the explicit path for the Python executable

?

If you still see the error, maybe we will need to create a reproducer using something like a container...

[ajasmin]

I went through your reproduction steps, but I have a doubt: have you activated your virtual environment? (It does not seem to be part of the steps).

I activated the venv in my tests. It was missing from the description.

If you still see the error, maybe we will need to create a reproducer using something like a container...

Thanks. I will try to do so

[abravalheri]

I activated the venv in my tests it was missing from the description.

Umm... Could you also try using the explicit path instead of the activation script? (Sometimes the activation script doesn't work very well in my setup, so I got used to just call everything with the explicit path to ensure I am using the correct environment).

[abravalheri]

Sorry @ajasmin, my bad! I don't trust the activation script in my own machine so I biased against it.

There was an error when I tried to replicate the error: when I cd into the tmp folder, the root directory of the project was also named foo and that created another accidental namespace package...

I was able to replicate your error by renaming the root directory to a name different than the package.

---

What is happening here is not related to the editable installation, you should be able to have the same error even if you omit -e and use v63 of setuptools, for example by running:

cd /tmp
rm -rf proj
mkdir -p proj/foo
cd proj
cat <<EOF > pyproject.toml
[build-system]
requires = [
  "setuptools==63.0.0",
  "wheel",
]
build-backend = "setuptools.build_meta"

[project]
name = "foo"
version = "1.0.0"

[tool.setuptools]
packages.find.include = ["foo.*"]
EOF

cat <<EOF > setup.py
from setuptools import setup

setup()
EOF

echo '__version__ = "1.0.0"' > foo/__init__.py

python3.10 -mvenv venv

/tmp/proj/venv/bin/python -m pip install .

cd /tmp

/tmp/proj/venv/bin/python -c "import foo"

The reason why the experiment is failing is related to the line:

packages.find.include = ["foo.*"]

include simply runs a glob pattern matcher with the names of the packages. In your case the package name is foo which does not match foo.* because it does not contain a ..

If you replace:

- packages.find.include = ["foo.*"]
+ packages.find.include = ["foo*"]

things should work.

---

This behaviour is explained in our docs:

include and exclude accept strings representing glob patterns. These patterns should match the full name of the Python module (as if it was written in an import statement).

For example if you have util pattern, it will match util/__init__.py but not util/files/__init__.py.

The fact that the parent package is matched by the pattern will not dictate if the submodule will be included or excluded from the distribution. You will need to explicitly add a wildcard (e.g. util*) if you want the pattern to also match submodules.

[abravalheri]

Please let me know if this works for you!

[NickCrews]

EDIT: I think my problem is actually #3537

@abravalheri I think it is a difference between 63 and 64, I think I'm getting hit with this. Sorry, the following is not minimally reproducible, but I think it is pretty good evidence of the difference between 63 and 64.

This PR, which uses setuptools>=64, fails CI with AttributeError: module 'dedupe' has no attribute 'Dedupe'.

So I made a followup PR, identical to the other except I swapped to setuptools==63, and all the corresponding CI checks pass (there is one failing CI check, but that's unrelated)

[ajasmin]

include simply runs a glob pattern matcher with the names of the packages. In your case the package name is foo which does not match foo.* because it does not contain a ..

Thank you! I will make the change.

Though, for what's it worth. there is a behavior change in 64. I've written a Dockerfile.

Changing the setuptools==64.0.0to setuptools==63.* will make the import work.

# syntax=docker/dockerfile:1.4

FROM ubuntu:jammy

RUN export DEBIAN_FRONTEND=noninteractive && \
	apt-get update -yq && \
	apt-get install -yq python3-full

RUN mkdir -p my_project/foo/bar
RUN printf '%s\n' \
'[build-system]' \
'requires = [' \
'  "setuptools==64.0.0",' \
'  "wheel",' \
']' \
'build-backend = "setuptools.build_meta"' \
'' \
'[project]' \
'name = "foo"' \
'version = "1.0.0"' \
'' \
'[tool.setuptools]' \
'packages.find.include = ["foo.*"]' \
> my_project/pyproject.toml

RUN printf '%s\n' \
'from setuptools import setup' \
'' \
'setup()' \
> my_project/setup.py

RUN echo '__version__ = "1.0.0"' > my_project/foo/bar/__init__.py

RUN tail -n +1 $(find my_project -type f)

RUN python3 -mvenv venv

RUN venv/bin/python -m pip install -U pip==22.2.1
RUN venv/bin/python -m pip install -e my_project

RUN mkdir t

RUN cd t && ../venv/bin/python -c "import foo.bar"