ModuleNotFoundError during plugin execution on submodule of namespace package.

[jaraco]

In jaraco/jaraco.site#1, I stumbled onto another obscure error implicating namespace packages.

In this branch, I pared the issue down to a minimal repro.

jaraco.site minimize-issue-1 $ tree
.
├── jaraco
│   └── site
│       ├── __init__.py
│       └── projecthoneypot
│           ├── __init__.py
│           └── croakysteel.py
└── tox.ini

3 directories, 4 files
jaraco.site minimize-issue-1 $ cat tox.ini
[tox]
envlist = python

[testenv]
skip_install = True
deps =
        pytest
        pytest-black
commands =
        pytest --black
jaraco.site minimize-issue-1 $ find jaraco -type f | xargs cat

$ tox
python installed: appdirs==1.4.4,attrs==20.3.0,black==20.8b1,click==7.1.2,iniconfig==1.1.1,mypy-extensions==0.4.3,packaging==20.9,pathspec==0.8.1,pluggy==0.13.1,py==1.10.0,pyparsing==2.4.7,pytest==6.2.2,pytest-black==0.3.12,regex==2020.11.13,toml==0.10.2,typed-ast==1.4.2,typing-extensions==3.7.4.3
python run-test-pre: PYTHONHASHSEED='3416208206'
python run-test: commands[0] | pytest --black
============================= test session starts ==============================
platform darwin -- Python 3.9.0, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
cachedir: .tox/python/.pytest_cache
rootdir: /Users/jaraco/code/main/jaraco.site
plugins: black-0.3.12
collected 3 items

jaraco/site/__init__.py s                                                [ 33%]
jaraco/site/projecthoneypot/__init__.py s                                [ 66%]
jaraco/site/projecthoneypot/croakysteel.py E                             [100%]

==================================== ERRORS ====================================
_____________________ ERROR at setup of Black format check _____________________

self = <Package projecthoneypot>

    def _importtestmodule(self):
        # We assume we are only called once per module.
        importmode = self.config.getoption("--import-mode")
        try:
>           mod = import_path(self.fspath, mode=importmode)

.tox/python/lib/python3.9/site-packages/_pytest/python.py:578: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

p = local('/Users/jaraco/code/main/jaraco.site/jaraco/site/projecthoneypot/__init__.py')

    def import_path(
        p: Union[str, py.path.local, Path],
        *,
        mode: Union[str, ImportMode] = ImportMode.prepend,
    ) -> ModuleType:
        """Import and return a module from the given path, which can be a file (a module) or
        a directory (a package).
    
        The import mechanism used is controlled by the `mode` parameter:
    
        * `mode == ImportMode.prepend`: the directory containing the module (or package, taking
          `__init__.py` files into account) will be put at the *start* of `sys.path` before
          being imported with `__import__.
    
        * `mode == ImportMode.append`: same as `prepend`, but the directory will be appended
          to the end of `sys.path`, if not already in `sys.path`.
    
        * `mode == ImportMode.importlib`: uses more fine control mechanisms provided by `importlib`
          to import the module, which avoids having to use `__import__` and muck with `sys.path`
          at all. It effectively allows having same-named test modules in different places.
    
        :raises ImportPathMismatchError:
            If after importing the given `path` and the module `__file__`
            are different. Only raised in `prepend` and `append` modes.
        """
        mode = ImportMode(mode)
    
        path = Path(str(p))
    
        if not path.exists():
            raise ImportError(path)
    
        if mode is ImportMode.importlib:
            module_name = path.stem
    
            for meta_importer in sys.meta_path:
                spec = meta_importer.find_spec(module_name, [str(path.parent)])
                if spec is not None:
                    break
            else:
                spec = importlib.util.spec_from_file_location(module_name, str(path))
    
            if spec is None:
                raise ImportError(
                    "Can't find module {} at location {}".format(module_name, str(path))
                )
            mod = importlib.util.module_from_spec(spec)
            spec.loader.exec_module(mod)  # type: ignore[union-attr]
            return mod
    
        pkg_path = resolve_package_path(path)
        if pkg_path is not None:
            pkg_root = pkg_path.parent
            names = list(path.with_suffix("").relative_to(pkg_root).parts)
            if names[-1] == "__init__":
                names.pop()
            module_name = ".".join(names)
        else:
            pkg_root = path.parent
            module_name = path.stem
    
        # Change sys.path permanently: restoring it at the end of this function would cause surprising
        # problems because of delayed imports: for example, a conftest.py file imported by this function
        # might have local imports, which would fail at runtime if we restored sys.path.
        if mode is ImportMode.append:
            if str(pkg_root) not in sys.path:
                sys.path.append(str(pkg_root))
        elif mode is ImportMode.prepend:
            if str(pkg_root) != sys.path[0]:
                sys.path.insert(0, str(pkg_root))
        else:
            assert_never(mode)
    
>       importlib.import_module(module_name)

.tox/python/lib/python3.9/site-packages/_pytest/pathlib.py:531: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

name = 'site.projecthoneypot', package = None

    def import_module(name, package=None):
        """Import a module.
    
        The 'package' argument is required when performing a relative import. It
        specifies the package to use as the anchor point from which to resolve the
        relative import to an absolute import.
    
        """
        level = 0
        if name.startswith('.'):
            if not package:
                msg = ("the 'package' argument is required to perform a relative "
                       "import for {!r}")
                raise TypeError(msg.format(name))
            for character in name:
                if character != '.':
                    break
                level += 1
>       return _bootstrap._gcd_import(name[level:], package, level)

/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

name = 'site.projecthoneypot', package = None, level = 0

>   ???

<frozen importlib._bootstrap>:1030: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

name = 'site.projecthoneypot'
import_ = <function _gcd_import at 0x7fb57cd8c310>

>   ???

<frozen importlib._bootstrap>:1007: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

name = 'site.projecthoneypot'
import_ = <function _gcd_import at 0x7fb57cd8c310>

>   ???
E   ModuleNotFoundError: No module named 'site.projecthoneypot'; 'site' is not a package

<frozen importlib._bootstrap>:981: ModuleNotFoundError

The above exception was the direct cause of the following exception:

self = <_HookCaller 'pytest_runtest_setup'>, args = ()
kwargs = {'item': <BlackItem croakysteel.py>}, notincall = set()

    def __call__(self, *args, **kwargs):
        if args:
            raise TypeError("hook calling supports only keyword arguments")
        assert not self.is_historic()
        if self.spec and self.spec.argnames:
            notincall = (
                set(self.spec.argnames) - set(["__multicall__"]) - set(kwargs.keys())
            )
            if notincall:
                warnings.warn(
                    "Argument(s) {} which are declared in the hookspec "
                    "can not be found in this hook call".format(tuple(notincall)),
                    stacklevel=2,
                )
>       return self._hookexec(self, self.get_hookimpls(), kwargs)

.tox/python/lib/python3.9/site-packages/pluggy/hooks.py:286: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest.config.PytestPluginManager object at 0x7fb581d7ee20>
hook = <_HookCaller 'pytest_runtest_setup'>
methods = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/Users/jaraco/code/main/jaraco.site/.tox/python/lib...=None>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7fb581f17a30>>, ...]
kwargs = {'item': <BlackItem croakysteel.py>}

    def _hookexec(self, hook, methods, kwargs):
        # called from all hookcaller instances.
        # enable_tracing will set its own wrapping function at self._inner_hookexec
>       return self._inner_hookexec(hook, methods, kwargs)

.tox/python/lib/python3.9/site-packages/pluggy/manager.py:93: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

hook = <_HookCaller 'pytest_runtest_setup'>
methods = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/Users/jaraco/code/main/jaraco.site/.tox/python/lib...=None>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7fb581f17a30>>, ...]
kwargs = {'item': <BlackItem croakysteel.py>}

>   self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
        methods,
        kwargs,
        firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
    )

.tox/python/lib/python3.9/site-packages/pluggy/manager.py:84: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

item = <BlackItem croakysteel.py>

    def pytest_runtest_setup(item: Item) -> None:
        _update_current_test_var(item, "setup")
>       item.session._setupstate.prepare(item)

.tox/python/lib/python3.9/site-packages/_pytest/runner.py:150: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest.runner.SetupState object at 0x7fb581f7f2e0>
colitem = <BlackItem croakysteel.py>

    def prepare(self, colitem) -> None:
        """Setup objects along the collector chain to the test-method."""
    
        # Check if the last collection node has raised an error.
        for col in self.stack:
            if hasattr(col, "_prepare_exc"):
                exc = col._prepare_exc  # type: ignore[attr-defined]
                raise exc
    
        needed_collectors = colitem.listchain()
        for col in needed_collectors[len(self.stack) :]:
            self.stack.append(col)
            try:
                col.setup()
            except TEST_OUTCOME as e:
                col._prepare_exc = e  # type: ignore[attr-defined]
>               raise e

.tox/python/lib/python3.9/site-packages/_pytest/runner.py:452: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest.runner.SetupState object at 0x7fb581f7f2e0>
colitem = <BlackItem croakysteel.py>

    def prepare(self, colitem) -> None:
        """Setup objects along the collector chain to the test-method."""
    
        # Check if the last collection node has raised an error.
        for col in self.stack:
            if hasattr(col, "_prepare_exc"):
                exc = col._prepare_exc  # type: ignore[attr-defined]
                raise exc
    
        needed_collectors = colitem.listchain()
        for col in needed_collectors[len(self.stack) :]:
            self.stack.append(col)
            try:
>               col.setup()

.tox/python/lib/python3.9/site-packages/_pytest/runner.py:449: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Package projecthoneypot>

    def setup(self) -> None:
        # Not using fixtures to call setup_module here because autouse fixtures
        # from packages are not called automatically (#4085).
        setup_module = _get_first_non_fixture_func(
>           self.obj, ("setUpModule", "setup_module")
        )

.tox/python/lib/python3.9/site-packages/_pytest/python.py:644: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Package projecthoneypot>

    @property
    def obj(self):
        """Underlying Python object."""
        obj = getattr(self, "_obj", None)
        if obj is None:
>           self._obj = obj = self._getobj()

.tox/python/lib/python3.9/site-packages/_pytest/python.py:291: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Package projecthoneypot>

    def _getobj(self):
>       return self._importtestmodule()

.tox/python/lib/python3.9/site-packages/_pytest/python.py:500: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Package projecthoneypot>

    def _importtestmodule(self):
        # We assume we are only called once per module.
        importmode = self.config.getoption("--import-mode")
        try:
            mod = import_path(self.fspath, mode=importmode)
        except SyntaxError as e:
            raise self.CollectError(
                ExceptionInfo.from_current().getrepr(style="short")
            ) from e
        except ImportPathMismatchError as e:
            raise self.CollectError(
                "import file mismatch:\n"
                "imported module %r has this __file__ attribute:\n"
                "  %s\n"
                "which is not the same as the test file we want to collect:\n"
                "  %s\n"
                "HINT: remove __pycache__ / .pyc files and/or use a "
                "unique basename for your test file modules" % e.args
            ) from e
        except ImportError as e:
            exc_info = ExceptionInfo.from_current()
            if self.config.getoption("verbose") < 2:
                exc_info.traceback = exc_info.traceback.filter(filter_traceback)
            exc_repr = (
                exc_info.getrepr(style="short")
                if exc_info.traceback
                else exc_info.exconly()
            )
            formatted_tb = str(exc_repr)
>           raise self.CollectError(
                "ImportError while importing test module '{fspath}'.\n"
                "Hint: make sure your test modules/packages have valid Python names.\n"
                "Traceback:\n"
                "{traceback}".format(fspath=self.fspath, traceback=formatted_tb)
            ) from e
E           _pytest.nodes.Collector.CollectError: ImportError while importing test module '/Users/jaraco/code/main/jaraco.site/jaraco/site/projecthoneypot/__init__.py'.
E           Hint: make sure your test modules/packages have valid Python names.
E           Traceback:
E           /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module
E               return _bootstrap._gcd_import(name[level:], package, level)
E           E   ModuleNotFoundError: No module named 'site.projecthoneypot'; 'site' is not a package

.tox/python/lib/python3.9/site-packages/_pytest/python.py:603: CollectError
=========================== short test summary info ============================
ERROR jaraco/site/projecthoneypot/croakysteel.py::BLACK - _pytest.nodes.Colle...
========================= 2 skipped, 1 error in 0.28s ==========================
ERROR: InvocationError for command /Users/jaraco/code/main/jaraco.site/.tox/python/bin/pytest --black (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   python: commands failed

As you can see, even though jaraco is a namespace package (works for import jaraco.site.projecthoneypot.croakysteel), when pytest attempts to import croakysteel, it incorrectly detects that jaraco is part of the package ancestry, so incorrectly imports site as a top-level package and not jaraco.site.

This issue goes away if pytest-black isn't used, but is also triggered by other plugins like pytest-mypy or pytest-flake8.

This issue is similar to #3396, but doesn't involve doctests. It's similar to #5147, but only affects plugins.

[nicoddemus]

Thanks @jaraco,

Indeed might be the time to retake the discussion started in #5147, now that we have --import-mode=importlib and only support Python 3.6+.

cc @asottile @RonnyPfannschmidt @0cjs

[RonnyPfannschmidt]

It's not something I can dig into near term, sorry

[asottile]

hmm, if it works correctly for normal pytest items but not for custom plugin items, then there's probably some missing gap that makes the two different that we should enhance for plugin authors (as much as I personally think pytest-flake8 / pytest-black / pytest-mypy / etc. are bad ideas and this is just an expected side-effect of putting a square peg into a round hole)

[jaraco]

The issue appears to be in

pytest/src/_pytest/pathlib.py

Lines 570 to 584 in 12e7db8

	def resolve_package_path(path: Path) -> Optional[Path]:
	"""Return the Python package path by looking for the last
	directory upwards which still contains an __init__.py.
	Returns None if it can not be determined.
	"""
	result = None
	for parent in itertools.chain((path,), path.parents):
	if parent.is_dir():
	if not parent.joinpath("__init__.py").is_file():
	break
	if not parent.name.isidentifier():
	break
	result = parent
	return result

When called for a file in the subpackage, it infers the incorrect directory:

>>> resolve_package_path(pathlib.Path('/Users/jaraco/code/main/jaraco.site/jaraco/site/projecthoneypot/__init__.py'))
PosixPath('/Users/jaraco/code/main/jaraco.site/jaraco/site')

In the case of the affected projects, I'd expect that function to return .../jaraco.site/jaraco, the namespace package. The problem is, the function is required to guess which directory is the top-level package by discovering directories containing __init__.py, but a PEP 420 namespace package doesn't have that file.

Indeed, that's the same conclusion in the last comment of #5147.