`importlib.resources` support in editable mode

[LecrisUT]

For now I want to gather a few notes about my experimentation with importlib.resources:

• scikit-build-core strips out the suffix of the library, i.e. in the _editable.py file

  install(
    {...},
    {...,
      'spglib.lib.libsymspg': 'spglib/lib/libsymspg.so.2.1.0'
    }, ...
    )

  In principle that seems really nice, but the name would be inconsistent on unix and windows anyway (prefix lib or not). At which point the cmake author would have to make a new library with fixed OUTPUT_NAME and suffixes, etc. But in that case it is unnecessary to sanitize the file anymore right? I think it is better to just export all files as is and handle these on the python side
• After running importlib.resources.files() the files are not picked when using iterdir(). Only the files that are actually in that folder and pycache are detected. Presumably the paths from _editable.py have to be injected there? Sure enough when doing importlib.resources.as_file it redirects to the original source path.

Any clues on where to start to help implement this?

[LecrisUT]

I was trying to investigate how hatchling handles redirection in editable installs, i.e. if I have something like:

[tool.hatch]
build.include = [
    "python/beakerlib",
    "src",
]

[tool.hatch.build.sources]
"python/beakerlib" = "beakerlib"
"src" = "beakerlib/src"

This should be internally a similar issue, although in this case it should be more straightforward. Apparently that is not solved in hatchling either:

ValueError: Dev mode installations are unsupported when any path rewrite in the `sources` option changes a prefix rather than removes it, see: https://github.com/pfmoore/editables/issues/20

[henryiii]

I think the first step will be to rewrite the editable support classes, probably looking more like https://github.com/mesonbuild/meson-python/blob/main/mesonpy/_editable.py. Currently we are reusing the file based finders, but we need a custom finder that supports the resources protocol. That will have access to that list, and will produce the correct items when requested by importlib.resources.

[LecrisUT]

I've found a similar issue in editables hopefully that project is useful as a more light-weight reference.

Also thanks for the reference. Seems quite doable at first glance.

[LecrisUT]

Oh, I just did a quick test with setuptools backend and it manages to pick up files and put them in correct subpackages and they are accessible in editable install. Looking at the implementations, the only difference I see is:

setuptools implementation

class _EditableNamespaceFinder:  # PathEntryFinder
    @classmethod
    def _path_hook(cls, path):
        if path == PATH_PLACEHOLDER:
            return cls
        raise ImportError

    @classmethod
    def _paths(cls, fullname):
        # Ensure __path__ is not empty for the spec to be considered a namespace.
        return NAMESPACES[fullname] or MAPPING.get(fullname) or [PATH_PLACEHOLDER]

    @classmethod
    def find_spec(cls, fullname, target=None):
        if fullname in NAMESPACES:
            spec = ModuleSpec(fullname, None, is_package=True)
            spec.submodule_search_locations = cls._paths(fullname)
            return spec
        return None

    @classmethod
    def find_module(cls, fullname):
        return None


def install():
    if not any(finder == _EditableFinder for finder in sys.meta_path):
        sys.meta_path.append(_EditableFinder)

    if not NAMESPACES:
        return

    if not any(hook == _EditableNamespaceFinder._path_hook for hook in sys.path_hooks):
        # PathEntryFinder is needed to create NamespaceSpec without private APIS
        sys.path_hooks.append(_EditableNamespaceFinder._path_hook)
    if PATH_PLACEHOLDER not in sys.path:
        sys.path.append(PATH_PLACEHOLDER)  # Used just to trigger the path hook

Is this part relevant? Can I edit the _editable.py without installing to test the difference?

Edit: Oooh I get it. The sys.meta_path and sys.path_hooks are not the ones responsible when calling importlib.resources. Instead we need to add a importlib.resources.abc.ResourceReader somewhere (probably in the module loader?). The documentation does not actually point to where this should be implemented in

[henryiii]

For playing with this, I generally do pip install -e . then edit the editable file installed into site-package directly.

Documentation is very limited on how to do this sort of thing! :)

[LecrisUT]

Well, I managed to get at least the backbone for this:

class TestLoader(importlib.machinery.SourceFileLoader):
    def __init__(self, name: str, path: str):
        super().__init__(name,path)

    def set_data(self, path: importlib.abc._Path, data: bytes, *, _mode: int = ...) -> None:
        pass

    def get_resource_reader(self, module):
        return TestReader(self)

# Traversable
class TestPath(type(pathlib.Path())):
    def iterdir(self: Self) -> Traversable:
        for k,v in NAMESPACES.items():   # NAMESPACES = self.known_wheel_files
            yield pathlib.Path(os.path.join(DIR, v))
        yield from super().iterdir()

# TraversableResources
class TestReader(importlib.readers.FileReader):
    def __init__(self, loader: TestLoader):
        super().__init__(loader)

    def files(self) -> Traversable:
        return TestPath(self.path)

With a hook at:

        if fullname in self.known_source_files:
            redir = self.known_source_files[fullname]
            spec = importlib.util.spec_from_file_location(fullname, redir)
            spec.loader = TestLoader(fullname, redir)
            return spec

Well, that hook doesn't work because it should be in self.known_wheel_files. But over there find_spec returns null.

It needs proper logic to handle python packages and modules, and much more that the meson one does

[LecrisUT]

Well, this got complicated fast but I guess the main logic that is missing:

• Generate a tree of all of the importable modules, i.e. those that can accept importlib.resources.files(). The leafs should not count, e.g. currently there is an implication that spglib/include/spglib.h -> spglib.include.spglib, but that should not be the case, only the parent should be allowed (spglib/include -> spglib.include)
• In the find_spec method, use an appropriate typed loader depending on if the module is a python source file or an intermediary node. In both cases the corresponding Reader must inject/replace the leaves of the node

The main complication here is getting the correct type of the loaders in order for the source code to be executed and creating the navigable object. Meson implementation has got all of the points addressed, but I think this can be simplified to piggyback onto pathlib.Path and the importlib.util.spec_from_file_location.