Format Python and Bash scripts using ruff and shfmt

[eramongodb]

Extends automated formatting to Python and Bash scripts using the existing uv project file.

Important

This PR only selects and provides the tools required for automated formatting of Python and Bash scripts. However, it does not check or enforce that these files are correctly formatted. The lint task is untouched except to use the new etc/format.py script. The etc/format.py itself script remains specific to clang-format only.

The new etc/ruff-format-all.sh and etc/shfmt-format-all.sh scripts are included for convenience and reference ("how to use these tools"). They do not need to be used directly (e.g. directly running uv run ruff format is just fine). I expect/hope these will be integrated into etc/format.py (similar to how it replaces the etc/clang-format-all.sh script) in the near future.

Similar changes are planned for the C Driver shortly following this PR.

---

For C and C++ source files, tools/format.py from mongodb/mongo-c-driver#2050 is imported and reused here as etc/format.py, with minor tweaks to support the C++ Driver directory structure. This replaces the etc/clang-format-all.sh script. Relevant documentation and references have been updated accordingly.

Note

Adding support for uv run format, similar to uv run mc-evg-generate, was considered but postponed for now.

---

For Python files, this PR proposes using Ruff. For consistency with our updated ClangFormat preferences, the column width is set to 120. For consistency with recent Python code, quote-style = "single" and indent-style = "space" are also set. Note multiline strings unconditionally use double-quotes despite quote-style = "single" for conformance with PEP 8 and PEP 257.

The expected commands to format all Python files is documented and supported by etc/ruff-format-all.sh. Note the ruff check --fix command required to sort and group import statements (see: astral-sh/ruff#8232), first by "standard library", then "third party", then "first party" (see: Ruff FAQ). I don't believe this ordering is customizeable (see: astral-sh/ruff#7615).

Unfortunately I don't think there's anything we can do about the "two-space lead for trailing comments" formatting either (e.g. print("example") # comment). There is a massive semi-related unresolved issue astral-sh/ruff#7684 concerning formatting of comments. The default behavior is left as-is for now.

Formatting of pyproject.toml itself (as a TOML file) is also deferred, as there does not seem to be an immediately-obvious community-supported tool to integrate and use for this purpose at this time (see: toml-lang/toml#532). Wait and hope for astral-sh/ruff#10738 to be implemented soon...?

Note

Ruff also supports linting Python scripts via ruff check. This is not integrated into any scripts at this time. However, as a drive-by improvement, all currently-outstanding warnings emitted by ruff check are fixed by this PR using both --fix and manual edits. Adding pyright was considered but postponed to keep this PR focused on formatting only.

---

For Bash scripts, this PR proposes using shfmt. Unlike Ruff, there is no convenient auto-detection of relevant project files, so etc/shfmt-format-all.sh uses a simple find command to find and filter .sh scripts under .evergreen, etc, and examples. For consistency with recent Bash code, indentation is set to two spaces with -i 2. Although shfmt also supports --simplify , but this flag is omitted for now. This flag can be included if preferable.

Note

Linting Bash scripts can be done using shellcheck via the shellcheck-py package, but this too is postponed to keep this PR focused on formatting only. However, this can be done manually using uv run --with shellcheck-py shellcheck -x <files>.

[eramongodb]

Moved ruff and shfmt-py into an optional "format-scripts" dependency group to avoid impacting the lint EVG task with the following build requirement and error on the rhel80 distro:

x Failed to build `shfmt-py==3.12.0.2`
├─▶ The build backend returned an error
╰─▶ Call to `setuptools.build_meta:__legacy__.build_wheel` failed (exit
    status: 1)
    [stdout]
    running bdist_wheel
    running build
    running fetch_binaries
    https://github.com/mvdan/sh/releases/download/v3.12.0/shfmt_v3.12.0_linux_amd64
    [stderr]
    error: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate
    verify failed: unable to get local issuer certificate (_ssl.c:1035)>
    hint: This usually indicates a problem with the package or the build
    environment.
help: `shfmt-py` (v3.12.0.2) was included because `mongo-cxx-driver:dev`
      (v0.1.0) depends on `shfmt-py`

The *-format-all.sh scripts have been updated accordingly to explicitly request the optional "format-scripts" dependency group.

[vector-of-bool]

I've been meaning to add Ruff format to the C repo for a while, so this is good precedent. LGTM

[vector-of-bool]

This function is not used (it's not used in the C repo either, I just haven't gotten around to deleting it yet)

[eramongodb]

Posted mongodb/mongo-c-driver#2102.

[connorsmacd]

LGTM