Add a CLI subcommand for bootstrapping the Python installation

Author: chrisrink10

CHANGELOG.md

@@ -9,7 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
  * Added support for passing through `:tag` metadata to the generated Python AST (#354)
  * Added support for calling symbols as functions on maps and sets (#775)
  * Added support for passing command line arguments to Basilisp (#779)
- * Added support for autocompleting names in the `python/` pseudo-namespace for Python builtins at the REPL (#787) 
+ * Added support for autocompleting names in the `python/` pseudo-namespace for Python builtins at the REPL (#787)
+ * Added a subcommand for bootstrapping the Python installation with Basilisp (#7??)
 
 ### Changed
  * Optimize calls to Python's `operator` module into their corresponding native operators (#754)

Makefile

@@ -8,6 +8,17 @@ format:
 	@poetry run sh -c 'isort . && black .'
 
 
+.PHONY: check
+check:
+	@rm -f .coverage*
+	@TOX_SKIP_ENV='pypy3|safety|coverage' poetry run tox run-parallel -p auto
+
+
+.PHONY: lint
+lint:
+	@poetry run tox run-parallel -m lint
+
+
 .PHONY: repl
 repl:
 	@BASILISP_USE_DEV_LOGGER=true poetry run basilisp repl
@@ -16,7 +27,12 @@ repl:
 .PHONY: test
 test:
 	@rm -f .coverage*
-	@TOX_SKIP_ENV='pypy3|safety|coverage' poetry run tox run-parallel -p 4
+	@TOX_SKIP_ENV='pypy3' poetry run tox run-parallel -m test
+
+
+.PHONY: type-check
+type-check:
+	@poetry run tox run-parallel -m mypy
 
 
 lispcore.py:

docs/cli.rst

@@ -82,4 +82,22 @@ If you installed the `PyTest <https://docs.pytest.org/en/7.0.x/>`_ extra, you ca
 
    basilisp test
 
-Because Basilisp defers all testing logic to PyTest, you can use any standard PyTest arguments and flags from this entrypoint.
+Because Basilisp defers all testing logic to PyTest, you can use any standard PyTest arguments and flags from this entrypoint.
+
+.. _bootstrap_cli_command:
+
+Bootstrap Python Installation
+-----------------------------
+
+For some installations, it may be desirable to have Basilisp readily importable whenever the Python interpreter is started.
+You can enable that as described in :ref:`bootstrapping`:
+
+.. code-block:: bash
+
+   basilisp bootstrap
+
+If you would like to remove the bootstrapped Basilisp from your installation, you can remove it:
+
+.. code-block:: bash
+
+   basilisp bootstrap --uninstall

docs/contributing.rst

@@ -84,7 +84,15 @@ All three steps can be performed across all supported versions of CPython using
 
 .. code-block:: bash
 
+   make check
+
+Likewise, individual steps can be run across all supported verions using their respective targets:
+
+.. code-block::
+
+   make lint
    make test
+   make type-check
 
 To run a more targeted CI check directly from within the Poetry shell, developers can use ``tox`` commands directly.
 For instance, to run only the tests for ``basilisp.io`` on Python 3.12, you could use the following command:

docs/gettingstarted.rst

@@ -132,4 +132,25 @@ For systems where the shebang line allows arguments, you can use ``#!/usr/bin/en
 .. code-block:: clojure
 
    #!/usr/bin/env basilisp-run
-   (println "Hello world!")
+   (println "Hello world!")
+
+Finally, Basilisp has a command line option to bootstrap your Python installation such that Basilisp will already be importable whenever Python is started.
+This takes advantage of the ``.pth`` file feature supported by the `site <https://docs.python.org/3/library/site.html>`_ package.
+Specifically, any file with a ``.pth`` extension located in any of the known ``site-packages`` directories will be read at startup and, if any line of such a file starts with ``import``, it is executed.
+
+.. code-block:: bash
+
+   $ basilisp bootstrap
+   Your Python installation has been bootstrapped! You can undo this at any time with with `basilisp bootstrap --uninstall`.
+   $ python
+   Python 3.12.1 (main, Jan  3 2024, 10:01:43) [GCC 11.4.0] on linux
+   Type "help", "copyright", "credits" or "license" for more information.
+   >>> import importlib; importlib.import_module("basilisp.core")
+   <module 'basilisp.core' (/home/chris/Projects/basilisp/src/basilisp/core.lpy)>
+
+.. warning::
+
+   Code in ``.pth`` files is executed each time the Python interpreter is started.
+   The Python ``site`` documentation warns that "[i]ts impact should thus be kept to a minimum".
+   Bootstrapping Basilisp can take as long as 30 seconds (or perhaps longer) on the first run due to needing to compile :lpy:ns:`basilisp.core` to Python bytecode.
+   Subsequent startups should be considerable faster unless users have taken any measures to disable :ref:`namespace_caching`.

src/basilisp/cli.py

@@ -2,7 +2,9 @@
 import importlib.metadata
 import io
 import os
+import site
 import sys
+import textwrap
 import traceback
 import types
 from pathlib import Path
@@ -242,6 +244,61 @@ def _wrapped_subcommand(subparsers: "argparse._SubParsersAction"):
     return _wrap_add_subcommand
 
 
+def bootstrap_basilisp_installation(_, args: argparse.Namespace) -> None:
+    if args.quiet:
+        print_ = lambda v: v
+    else:
+        print_ = print
+
+    if args.uninstall:
+        if not (removed := basilisp.unbootstrap_python()):
+            print_("No Basilisp bootstrap files were found.")
+        else:
+            for file in removed:
+                print_(f"Removed '{file}'")
+    else:
+        basilisp.bootstrap_python()
+        print_(
+            "Your Python installation has been bootstrapped! You can undo this at any "
+            "time with with `basilisp bootstrap --uninstall`."
+        )
+
+
+@_subcommand(
+    "bootstrap",
+    help="bootstrap the Python installation to allow importing Basilisp namespaces",
+    description=textwrap.dedent(
+        """Bootstrap the Python installation to allow importing Basilisp namespaces"
+        without requiring an additional bootstrapping step.
+        
+        Python installations are bootstrapped by installing a `basilispbootstrap.pth`
+        file in your `site-packages` directory. Python installations execute `*.pth`
+        files found at startup.
+        
+        Bootstrapping your Python installation in this way can help avoid needing to
+        perform manual bootstrapping from Python code within your application.
+        
+        On the first startup, Basilisp will compile `basilisp.core` to byte code
+        which could take up to 30 seconds in some cases depending on your system and
+        which version of Python you are using. Subsequent startups should be
+        considerably faster so long as you allow Basilisp to cache bytecode for"""
+    ),
+    handler=bootstrap_basilisp_installation,
+)
+def _add_bootstrap_subcommand(parser: argparse.ArgumentParser) -> None:
+    parser.add_argument(
+        "--uninstall",
+        action="store_true",
+        help="if true, remove any `.pth` files installed by Basilisp in all site-packages directories",
+    )
+    parser.add_argument(
+        "-q",
+        "--quiet",
+        action="store_true",
+        help="if true, do not print out any",
+    )
+
+
 def nrepl_server(
     _,
     args: argparse.Namespace,
@@ -418,6 +475,10 @@ def run(
     with runtime.ns_bindings(args.in_ns) as ns:
         ns.refer_all(core_ns)
 
+        main_ns_var = core_ns.find(sym.symbol(runtime.MAIN_NS_VAR_NAME))
+        assert main_ns_var is not None
+        main_ns_var.bind_root(sym.symbol(args.in_ns))
+
         if args.args:
             cli_args_var = core_ns.find(sym.symbol(runtime.COMMAND_LINE_ARGS_VAR_NAME))
             assert cli_args_var is not None
@@ -516,6 +577,7 @@ def invoke_cli(args: Optional[Sequence[str]] = None) -> None:
     )
 
     subparsers = parser.add_subparsers(help="sub-commands")
+    _add_bootstrap_subcommand(subparsers)
     _add_nrepl_server_subcommand(subparsers)
     _add_repl_subcommand(subparsers)
     _add_run_subcommand(subparsers)

src/basilisp/importer.py

@@ -4,11 +4,12 @@
 import os
 import os.path
 import sys
+import traceback
 import types
 from functools import lru_cache
 from importlib.abc import MetaPathFinder, SourceLoader
 from importlib.machinery import ModuleSpec
-from typing import Iterable, List, Mapping, MutableMapping, Optional, cast
+from typing import Iterable, List, Mapping, MutableMapping, Optional, Sequence, cast
 
 from basilisp.lang import compiler as compiler
 from basilisp.lang import reader as reader
@@ -133,7 +134,7 @@ def __init__(self):
     def find_spec(
         self,
         fullname: str,
-        path,  # Optional[List[str]] # MyPy complains this is incompatible with supertype
+        path: Optional[Sequence[str]],
         target: Optional[types.ModuleType] = None,
     ) -> Optional[ModuleSpec]:
         """Find the ModuleSpec for the specified Basilisp module.
@@ -214,9 +215,61 @@ def get_filename(self, fullname: str) -> str:  # pragma: no cover
         try:
             cached = self._cache[fullname]
         except KeyError as e:
-            raise ImportError(f"Could not import module '{fullname}'") from e
-        spec = cached["spec"]
-        return spec.loader_state.filename
+            if (spec := self.find_spec(fullname, None)) is None:
+                raise ImportError(f"Could not import module '{fullname}'") from e
+        else:
+            spec = cached["spec"]
+            assert spec is not None, "spec must be defined here"
+        return spec.loader_state["filename"]
+
+    def get_code(self, fullname: str) -> Optional[types.CodeType]:
+        """Return code to load a Basilisp module.
+
+        This function is part of the ABC for `importlib.abc.ExecutionLoader` which is
+        what Python uses to execute modules at the command line as `python -m module`.
+        """
+        core_ns = runtime.Namespace.get(runtime.CORE_NS_SYM)
+        assert core_ns is not None
+
+        with runtime.ns_bindings("basilisp.namespace-executor") as ns:
+            ns.refer_all(core_ns)
+
+            # Set the *main-ns* variable to the current namespace.
+            main_ns_var = core_ns.find(sym.symbol(runtime.MAIN_NS_VAR_NAME))
+            assert main_ns_var is not None
+            main_ns_var.bind_root(sym.symbol(fullname))
+
+            # Basilisp can only ever product multiple `types.CodeType` objects for any
+            # given module because it compiles each form as a separate unit, but
+            # `ExecutionLoader.get_code` expects a single `types.CodeType` object. To
+            # simulate this requirement, we generate a single `(load "...")` to execute
+            # in a synthetic namespace.
+            #
+            # The target namespace is free to interpret
+            code: List[types.CodeType] = []
+            path = "/" + "/".join(fullname.split("."))
+            forms = cast(
+                List[ReaderForm],
+                list(
+                    reader.read_str(f'(load "{path}")', resolver=runtime.resolve_alias)
+                ),
+            )
+            assert len(forms) == 1
+            try:
+                compiler.compile_and_exec_form(
+                    forms[0],
+                    compiler.CompilerContext(
+                        filename="<Basilisp Namespace Executor>",
+                        opts=runtime.get_compiler_opts(),
+                    ),
+                    ns,
+                    collect_bytecode=code.append,
+                )
+            except Exception as e:
+                raise ImportError(f"Could not import module '{fullname}'") from e
+            else:
+                assert len(code) == 1
+                return code[0]
 
     def create_module(self, spec: ModuleSpec):
         logger.debug(f"Creating Basilisp module '{spec.name}'")
@@ -234,7 +287,7 @@ def _exec_cached_module(
         loader_state: Mapping[str, str],
         path_stats: Mapping[str, int],
         ns: runtime.Namespace,
-    ):
+    ) -> None:
         """Load and execute a cached Basilisp module."""
         filename = loader_state["filename"]
         cache_filename = loader_state["cache_filename"]
@@ -264,7 +317,7 @@ def _exec_module(
         loader_state: Mapping[str, str],
         path_stats: Mapping[str, int],
         ns: runtime.Namespace,
-    ):
+    ) -> None:
         """Load and execute a non-cached Basilisp module."""
         filename = loader_state["filename"]
         cache_filename = loader_state["cache_filename"]
@@ -302,7 +355,7 @@ def _exec_module(
         )
         self._cache_bytecode(filename, cache_filename, cache_file_bytes)
 
-    def exec_module(self, module):
+    def exec_module(self, module: types.ModuleType) -> None:
         """Compile the Basilisp module into Python code.
 
         Basilisp is fundamentally a form-at-a-time compilation, meaning that

src/basilisp/lang/runtime.py

@@ -86,6 +86,7 @@
 DEFAULT_READER_FEATURES_VAR_NAME = "*default-reader-features*"
 GENERATED_PYTHON_VAR_NAME = "*generated-python*"
 PRINT_GENERATED_PY_VAR_NAME = "*print-generated-python*"
+MAIN_NS_VAR_NAME = "*main-ns*"
 PRINT_DUP_VAR_NAME = "*print-dup*"
 PRINT_LENGTH_VAR_NAME = "*print-length*"
 PRINT_LEVEL_VAR_NAME = "*print-level*"
@@ -2106,6 +2107,25 @@ def in_ns(s: sym.Symbol):
         ),
     )
 
+    # Dynamic Var containing command line arguments passed via `basilisp run`
+    Var.intern(
+        CORE_NS_SYM,
+        sym.symbol(MAIN_NS_VAR_NAME),
+        None,
+        dynamic=True,
+        meta=lmap.map(
+            {
+                _DOC_META_KEY: (
+                    "The name of the main namespace as a symbol if this process was "
+                    "executed as ``basilisp run {file}`` or ``python -m {namespace}`` "
+                    "or ``nil`` otherwise.\n\n"
+                    "This can be useful for detecting scripts similarly to how Python "
+                    'scripts use ``if __name__ == "__main__":``.'
+                )
+            }
+        ),
+    )
+
     # Dynamic Var for introspecting the default reader featureset
     Var.intern(
         CORE_NS_SYM,