[MLIR][Python] enable type stub auto-generation #155741
Conversation
makslevental
force-pushed
the
users/makslevental/stubgen
branch
30 times, most recently
from
92b0650 to
7e6569c
Compare
This reverts commit 22c2e15.
This reverts commit 22c2e15.
This reverts commit 22c2e15.
This reverts commit 22c2e15.
This reverts commit 22c2e15.
Yes I can add dep files like we do for the generated builder sources.
Many people rely/relied on these type stubs as de facto docs for the bindings - that's why this choice was made.
|
I don't follow what you mean? If you rely on these, surely you'll get these in the install package after they get generated? If this work needs to have the build configuration generate file in the source directory, I would claim there is something deeply broken in the whole design, and I'll request a revert until we can figure out a way forward. |
I don't understand what you don't understand - these type stubs used to be in the source directory. They functioned as a reference for users and developers. Now they're no longer checked in. I had several people contact me confused about their disappearance.
Do you have a technical reason for calling this "deeply broken" or if it just FUD? The files are non functional outside of a type checking context, they are generated into an empty directory, and they are in the gitignore. |
OK, they use to be there, now they are not, so they will be ... in the build/install directory. I mean the same can be said about when we moved things to be ODS generated, many headers ended up not in the source directory anymore.
I want to be able to mount my source directory in a read-only way in a docker container where my cmake build operates in an isolated build directory. This is an invariant of the system that has always existed as far as I know, is there a precedent where this isn't the case? I also have multiple independent build directory for the same source, and these need to be independent and shouldn't interact with each other under the hood. |
|
Fair enough. I will send a PR momentarily to dump them into the build directory. |
|
Fixed here #157697 |
This reverts commit 22c2e15.
This reverts commit 22c2e15.
This reverts commit 22c2e15.
This reverts commit 22c2e15.
Despite several hotfixes, things remain broken, in particular: - installation/distribution (`ninja install / install-distribution`); - downstream projects with bindings exposed. See #157583 (comment) for more details. Reverts #155741, #157583, #157697. Let's make sure things are fixed and re-land as a unit.
…lvm#157173)" This reverts commit 46d8fdd. The whole set of commits got reverted in llvm#157831, reverting this one too.
This a reland of #155741 which was reverted at #157831. This version is narrower in scope - it only turns on automatic stub generation for `MLIRPythonExtension.Core._mlir` and **does not do anything automatically**. Specifically, the only CMake code added to `AddMLIRPython.cmake` is the `mlir_generate_type_stubs` function which is then used only in a manual way. The API for `mlir_generate_type_stubs` is: ``` Arguments: MODULE_NAME: The fully-qualified name of the extension module (used for importing in python). DEPENDS_TARGETS: List of targets these type stubs depend on being built; usually corresponding to the specific extension module (e.g., something like StandalonePythonModules.extension._standaloneDialectsNanobind.dso) and the core bindings extension module (e.g., something like StandalonePythonModules.extension._mlir.dso). OUTPUT_DIR: The root output directory to emit the type stubs into. OUTPUTS: List of expected outputs. DEPENDS_TARGET_SRC_DEPS: List of cpp sources for extension library (for generating a DEPFILE). IMPORT_PATHS: List of paths to add to PYTHONPATH for stubgen. PATTERN_FILE: (Optional) Pattern file (see https://nanobind.readthedocs.io/en/latest/typing.html#pattern-files). Outputs: NB_STUBGEN_CUSTOM_TARGET: The target corresponding to generation which other targets can depend on. ``` Downstream users should use `mlir_generate_type_stubs` in coordination with `declare_mlir_python_sources` to turn on stub generation for their own downstream dialect extensions and upstream dialect extensions if they so choose. Standalone example shows an example. Note, downstream will also need to set `-DMLIR_PYTHON_PACKAGE_PREFIX=...` correctly for their bindings.
…7930) This a reland of llvm/llvm-project#155741 which was reverted at llvm/llvm-project#157831. This version is narrower in scope - it only turns on automatic stub generation for `MLIRPythonExtension.Core._mlir` and **does not do anything automatically**. Specifically, the only CMake code added to `AddMLIRPython.cmake` is the `mlir_generate_type_stubs` function which is then used only in a manual way. The API for `mlir_generate_type_stubs` is: ``` Arguments: MODULE_NAME: The fully-qualified name of the extension module (used for importing in python). DEPENDS_TARGETS: List of targets these type stubs depend on being built; usually corresponding to the specific extension module (e.g., something like StandalonePythonModules.extension._standaloneDialectsNanobind.dso) and the core bindings extension module (e.g., something like StandalonePythonModules.extension._mlir.dso). OUTPUT_DIR: The root output directory to emit the type stubs into. OUTPUTS: List of expected outputs. DEPENDS_TARGET_SRC_DEPS: List of cpp sources for extension library (for generating a DEPFILE). IMPORT_PATHS: List of paths to add to PYTHONPATH for stubgen. PATTERN_FILE: (Optional) Pattern file (see https://nanobind.readthedocs.io/en/latest/typing.html#pattern-files). Outputs: NB_STUBGEN_CUSTOM_TARGET: The target corresponding to generation which other targets can depend on. ``` Downstream users should use `mlir_generate_type_stubs` in coordination with `declare_mlir_python_sources` to turn on stub generation for their own downstream dialect extensions and upstream dialect extensions if they so choose. Standalone example shows an example. Note, downstream will also need to set `-DMLIR_PYTHON_PACKAGE_PREFIX=...` correctly for their bindings.
This a reland of llvm/llvm-project#155741 which was reverted at llvm/llvm-project#157831. This version is narrower in scope - it only turns on automatic stub generation for `MLIRPythonExtension.Core._mlir` and **does not do anything automatically**. Specifically, the only CMake code added to `AddMLIRPython.cmake` is the `mlir_generate_type_stubs` function which is then used only in a manual way. The API for `mlir_generate_type_stubs` is: ``` Arguments: MODULE_NAME: The fully-qualified name of the extension module (used for importing in python). DEPENDS_TARGETS: List of targets these type stubs depend on being built; usually corresponding to the specific extension module (e.g., something like StandalonePythonModules.extension._standaloneDialectsNanobind.dso) and the core bindings extension module (e.g., something like StandalonePythonModules.extension._mlir.dso). OUTPUT_DIR: The root output directory to emit the type stubs into. OUTPUTS: List of expected outputs. DEPENDS_TARGET_SRC_DEPS: List of cpp sources for extension library (for generating a DEPFILE). IMPORT_PATHS: List of paths to add to PYTHONPATH for stubgen. PATTERN_FILE: (Optional) Pattern file (see https://nanobind.readthedocs.io/en/latest/typing.html#pattern-files). Outputs: NB_STUBGEN_CUSTOM_TARGET: The target corresponding to generation which other targets can depend on. ``` Downstream users should use `mlir_generate_type_stubs` in coordination with `declare_mlir_python_sources` to turn on stub generation for their own downstream dialect extensions and upstream dialect extensions if they so choose. Standalone example shows an example. Note, downstream will also need to set `-DMLIR_PYTHON_PACKAGE_PREFIX=...` correctly for their bindings.
This PR turns on automatic type stub generation (rather than relying on hand-written/updated stubs). It uses nanobind's stubgen facility. If you would like to enable this functionality you can add
GENERATE_TYPE_STUBStodeclare_mlir_python_extension.