Implement hermetic toolchain with bzlmod - toolchain not configurable work on different platforms

[chrislovecnm]

🚀 feature request

Relevant Rules

Existing rules with bzlmod

Description

I am working on the bzlmod example and running into a problem. I will update the documentation and also add this to the example.

When using bzlmod the toolchain registration does not download a hermetic version of Python. When using an extension and MODULES.bzl, the extension code cannot register a native toolchain because it runs in a different thread by design.

But how do we do this? I am getting an error that tells me to register a toolchain in the BUILD.bazel file when I try to use the extension to download a hermetic version of Python.

So I am trying:

I am using the example for bzlmod and have added a file that uses native.register_toolchains.

PLATFORMS = ["aarch64-apple-darwin", "aarch64-unknown-linux-gnu", "x86_64-apple-darwin", "x86_64-pc-windows-msvc", "x86_64-unknown-linux-gnu"]

def setup_toolchains(name):
    toolchain_repo_name = "{name}_toolchains".format(name = name)
    for platform in PLATFORMS:
        native.register_toolchains("@{toolchain_repo_name}//:{platform}_toolchain".format(
            toolchain_repo_name = toolchain_repo_name,
            platform = platform,
        ))

In the BUILD.bazel file, I am loading this function and using it.

load(":setup_python_toolchain.bzl", "setup_toolchains")
setup_toolchains(
    name = "python_39",
)

But I am getting "Error: no native function or rule 'register_toolchains'".

ERROR: package contains errors: : Traceback (most recent call last):
        File "/home/clove/Workspace/src/github.com/bazelbuild/rules_python/examples/bzlmod/BUILD.bazel", line 7, column 17, in <toplevel>
                setup_toolchains(
        File "/home/clove/Workspace/src/github.com/bazelbuild/rules_python/examples/bzlmod/setup_python_toolchain.bzl", line 6, column 15, in setup_toolchains
                native.register_toolchains("@{toolchain_repo_name}//:{platform}_toolchain".format(
Error: no native function or rule 'register_toolchains'
Available attributes: aar_import, action_listener, alias, android_binary, android_device, android_device_script_fixture, android_host_service_fixture, android_instrumentation_test, android_library, android_local_test, android_sdk, android_tools_defaults_jar, apple_cc_toolchain, available_xcodes, cc_binary, cc_host_toolchain_alias, cc_import, cc_libc_top_alias, cc_library, cc_proto_library, cc_shared_library, cc_shared_library_permissions, cc_test, cc_toolchain, cc_toolchain_alias, cc_toolchain_suite, config_feature_flag, config_setting, constraint_setting, constraint_value, environment, existing_rule, existing_rules, exports_files, extra_action, fdo_prefetch_hints, fdo_profile, filegroup, genquery, genrule, glob, j2objc_library, java_binary, java_import, java_library, java_lite_proto_library, java_package_configuration, java_plugin, java_plugins_flag_alias, java_proto_library, java_runtime, java_test, java_toolchain, label_flag, label_setting, objc_import, objc_library, package, package_group, package_name, platform, propeller_optimize, proto_lang_toolchain, proto_library, py_binary, py_library, py_runtime, py_test, repository_name, sh_binary, sh_library, sh_test, subpackages, test_suite, toolchain, toolchain_type, xcode_config, xcode_config_alias, xcode_version
ERROR: Skipping '//...': Error evaluating '//...': error loading package '': Package '' contains errors
WARNING: Target pattern parsing failed.
ERROR: Error evaluating '//...': error loading package '': Package '' contains errors
INFO: Elapsed time: 0.088s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)

Describe the solution you'd like

I will document the solution.

Describe alternatives you've considered

Use the BUILD files, but it is not working. I would rather not use a WORKSPACE.bzl file, which seems to defeat the purpose of using a MODULES.bzl file and bzlmod.

History

So when extensions.bzl was added, we have the comment:

https://github.com/bazelbuild/rules_python/blob/b80b8fde601f3bee9a4174c4aef03a3811912b93/python/extensions.bzl#L30

I don't understand the comment.

Previously when using a WORKSPACE file the method python_register_toolchains is where a hermetic version of Python is downloaded. When you have:

python_register_toolchains(
    name = "python39",
    python_version = "3.9",
)

Now we have:

# Register an already-defined toolchain so that Bazel can use it during toolchain resolution.
register_toolchains(
    "@python3_9_toolchains//:all",
)

Which does not download Python.

[chrislovecnm]

Paging @alexeagle, @kormide, and @aignas to see if anyone has ideas on how to do this.

[chrislovecnm]

I have verified this problem on both Linux and Windows.

I am getting the error:

ERROR: /home/clove/.cache/bazel/_bazel_clove/4d81ba2c4676ec44765d3282b1133dda/external/rules_python~override~pip~pip/tabulate/BUILD.bazel:3:6: @rules_python~override~pip~pip//tabulate:tabulate depends on @rules_python~override~pip~pip_tabulate//:pkg in repository @rules_python~override~pip~pip_tabulate which failed to fetch. no such package '@rules_python~override~pip~pip_tabulate//': python interpreter `python3` not found in PATH
ERROR: Analysis of target '//:bzlmod' failed; build aborted:

When I don't have a python binary installed.

[chrislovecnm]

This is not implemented yet. See bazelbuild/rules_rust#1528 for how rust implemented the functionality

[gferon]

Just curious: would this be the reason why pip.parse() is using the system interpreter instead of the registered Python toolchain? I don't see any obvious way to populate the python_interpreter_target of pip.parse() either.

[chrislovecnm]

@gferon it allows you to have 3.10 installed on the host but use 3.9 in the repo for example. It also allows you to register an OS specific toolchain.

[chrislovecnm]

Here is what I have discovered. The pip extension does not have the capability to have the tool chain set dynamically. A use case that you would want the tool chain set dynamically is when you would have developers or the project built on different operating systems. There are multiple factors in considering this support.

1. Our toolchain is coupled to our pip declaration and we also support different python versions in the same repo
2. Bazel does not allow the use of setting use_repos dynamically. Support is available for this in 6.2.
3. currently the pip arguments and our extension does not allow a user to dynamically configure the interpreter.

I have started to assess these features in #1155

From that PR I have added a parameter to the pip arguments that allows a user to configure a toolchain. The user rights an extension and a rule which creates a text file which contains the name of the Label for the interpreter. Not happy with the elegance of the solution, but it works. The challenge is also we have a call to load the toolchain for every OS that the project supports.

Bazel as of 6.2 will have the support of dynamically adding “use_repo” statements, so a user code write a rule to do this dynamically.

But imo is a lot for the user to do. All of this was supported previous to bzlmod, but now bzlmod does not have support for native.register_tooclhain. As designed.

Options

Other rules have added support for bzlmod in a hub spoke manner where they are confirguring the toolchain for the user bazelbuild/rules_rust#1528

But this may not work for us, with our multiple python version support.

What I recommend initially is that we move more of the grunt work for a user. The work in my PR can move inside our extension and the user tells the tule to do it. Once blzmod supports dynamically adding “use_repos” we then do that.

[chrislovecnm]

#1155 starts the work. We may have to have an extension in the user's module because the interpreter label is not set. We might be able to do it internally, but I do not know where.

[gferon]

@gferon it allows you to have 3.10 installed on the host but use 3.9 in the repo for example. It also allows you to register an OS specific toolchain.

Perfect, this is exactly what I was looking for. I'll wait for #1155 then.