Use BATS for integration tests

[jandubois]

I've been using Bash Automated Testing System aka BATS for a few years in several projects and really like it. I propose to use it for Lima as well. Addresses #3678.

Just a quick example to show the level of diagnostics you get from failing tests.

demo.bats:

load "../helpers/load"

@test 'any failing command fails the test' {
    false
    true
}

@test 'run can check the exit status' {
    run -1 true
}

@test 'run captures output (and status)' {
    run echo "foo"
    assert_output "bar"
}

@test 'asserts/refutes can use regular expressions' {
    run echo $'one\ntwo\nthree'
    refute_line --regexp "w"
}

Running it with the directory structure from this PR:

❯ bats/lib/bats-core/bin/bats bats/tests/demo.bats
demo.bats
 ✗ any failing command fails the test
   (in test file bats/tests/demo.bats, line 4)
     `false' failed
 ✗ run can check the exit status
   (in test file bats/tests/demo.bats, line 9)
     `run -1 true' failed, expected exit code 1, got 0
 ✗ run captures output (and status)
   (from function `__assert_stream' in file bats/lib/bats-assert/src/assert_output.bash, line 246,
    from function `assert_output' in file bats/lib/bats-assert/src/assert_output.bash, line 125,
    in test file bats/tests/demo.bats, line 14)
     `assert_output "bar"' failed

   -- output differs --
   expected : bar
   actual   : foo
   --

 ✗ asserts/refutes can use regular expressions
   (from function `__refute_stream_line' in file bats/lib/bats-assert/src/refute_line.bash, line 272,
    from function `refute_line' in file bats/lib/bats-assert/src/refute_line.bash, line 128,
    in test file bats/tests/demo.bats, line 19)
     `refute_line --regexp "w"' failed

   -- no line should match the regular expression --
   regexp : w
   index  : 1
   output (3 lines):
     one
   > two
     three
   --

4 tests, 4 failures

There are per-file and per-test setup and teardown hooks to avoid repetitive code.

I've written some sample tests in this PR to show how this would look for actual Lima tests:

• limactl protect
• limactl shell --preserve-env

The latter uses an environment variable to keep the instance running, so you can quickly run the same tests over and over while working on a feature that does not effect the guest or hostagent:

❯ export LIMA_BATS_REUSE_INSTANCE=1
...
❯ bats/lib/bats-core/bin/bats -T bats/tests/preserve-env.bats
preserve-env.bats
 ✓ there are no FOO*, BAR*, or SSH_FOO* variables defined in the VM [139]
 ✓ environment is not preserved by default [115]
 ✓ environment is preserved with --preserve-env [125]
 ✓ profile settings inside the VM take precedence over preserved variables [173]
 ✓ builtin block list is used when LIMA_SHELLENV_BLOCK is not set [148]
 ✓ custom block list replaces builtin block list [209]
 ✓ custom block list starting with + appends to builtin block list [190]
 ✓ block list entries can use * wildcard at the end [213]
 ✓ wildcard does only work at the end of the pattern [185]
 ✓ block list can use a , separated list with whitespace ignored [283]
 ✓ allow list overrides block list but blocks everything else [139]
 ✓ allow list can use a , separated list with whitespace ignored [141]
 ✓ setting both allow list and block list generates a warning [117]
 ✓ limactl info includes the default block list [121]

14 tests, 0 failures in 3 seconds

You can specify a filter to run only a subset of the tests (only works if all tests are independent of each other):

❯ bats/lib/bats-core/bin/bats -T bats/tests/preserve-env.bats -f allow
preserve-env.bats
 ✓ allow list overrides block list but blocks everything else [133]
 ✓ allow list can use a , separated list with whitespace ignored [141]
 ✓ setting both allow list and block list generates a warning [113]

3 tests, 0 failures in 1 seconds

Making it easy to write (and read) test cases is really key to having tests that cover all the edge cases.

There is obviously a lot to do to flesh out helper functions, fully integrate it with CI, and make it work on Windows etc., but I first wanted to see if we can agree on adopting this framework for integration tests.

---

Edit: Each @test runs in its own subshell, so any variable assignments/exported variables are local to the block and don't need to be cleaned up.

[AkihiroSuda]

Can't we just apt/brew install it?
Or use install.sh
https://github.com/opencontainers/runc/blob/v1.3.1/.cirrus.yml#L67-L72

[jandubois]

That doesn't install the helper libraries, like bats-assert, bats-file, etc.

And it doesn't guarantee that you get the right version, so I would even go one step further, and clone the repos into the lima-vm org, and submodule them from there. That way we can easily redirect a submodule to local changes we may want to make. Because https://github.com/bats-core/bats-core/releases doesn't have frequent releases, so pinning to a commit is the best mechanism we have to specify the version we need.

E.g. I would prefer to include bats-core/bats-core#1118 to get the --errexit support that avoids missing failures in helper functions. I'm sure it will eventually get merged, but who knows how long it will take.

Using .gitmodule is not really any different to using go.mod for managing dependencies. And as long as you don't change the repos of any existing submodules, all it takes is git submodule update to make sure you are up-to-date with the current branch.

[jandubois]

I've forked the 4 modules into the lima-vm org now, and updated the .gitmodules to point to our forks. I've also pushed by errexit PR to our bats-core fork, and updated the submodule to use that commit.

I've also added a make bats target, and call it from the macOS integration test.

And I also added some wrapper logic to setup/teardown functions, so we can have a global function, which will in turn call the functions inside each test file. Currently mostly unused except to print the test filename when running in CI.

I think this can maybe be merged now, and improved in future updates, so will mark it ready for review.

[norio-nomura]

I've also added a make bats target, and call it from the macOS integration test.

I tried make bats on local mac, and it fails with error:

 ✗ create dummy instance [176]
   (in test file hack/bats/tests/protect.bats, line 18)
     `run -0 create_dummy_instance "$NAME" '.disk = "1M"'' failed, expected exit code 0, got 1
   time="2025-10-06T11:28:17+09:00" level=info msg="Terminal is not available, proceeding without opening an editor"
   time="2025-10-06T11:28:17+09:00" level=info msg="Attempting to download the image" arch=aarch64 digest= location=/etc/profile
   time="2025-10-06T11:28:17+09:00" level=info msg="Downloaded the image from \"/etc/profile\""
   time="2025-10-06T11:28:17+09:00" level=info msg="Converting \"/Users/norio/.lima-bats/dummy/basedisk\" (raw) to a raw disk \"/Users/norio/.lima-bats/dummy/diffdisk\""
   time="2025-10-06T11:28:17+09:00" level=info msg="Expanding to 1MiB"
   time="2025-10-06T11:28:17+09:00" level=fatal msg="failed to convert \"/Users/norio/.lima-bats/dummy/basedisk\" to a raw disk \"/Users/norio/.lima-bats/dummy/diffdisk\": open /Users/norio/.lima-bats/dummy/diffdisk: permission denied"

This doesn't look like it's been tested in macOS CI.

[jandubois]

It used to run on macOS, but @AkihiroSuda moved it to Linux, I think.

Anyways, make bats works for me on macOS, and so does this:

❯ limactl create --name dummy - <<<"{images: [location: /etc/profile]}"
INFO[0000] Terminal is not available, proceeding without opening an editor
INFO[0000] Attempting to download the image              arch=aarch64 digest= location=/etc/profile
INFO[0000] Downloaded the image from "/etc/profile"
INFO[0000] Attempting to download the nerdctl archive    arch=aarch64 digest="sha256:2f98346ed3dcaf47cfd6461a5685e582284329a1ece1d952ade881b9fe7491e8" location="https://github.com/containerd/nerdctl/releases/download/v2.1.6/nerdctl-full-2.1.6-linux-arm64.tar.gz"
INFO[0000] Using cache "/Users/jan/Library/Caches/lima/download/by-url-sha256/d82077f1308e8d063805c4c1f761e1c3fd636e18cab7e25919a18e7a0838afa2/data"
INFO[0000] Run `limactl start dummy` to start the instance.

The failure seems to be due to qemu-img missing on your machine, and the internal conversion failing.

I'll create an issue for this, but for now a workaround is making sure you have qemu-img on the PATH. The command never gets called; I just did this:

❯ cp /bin/cat ~/bin/qemu-img
❯ chmod +x ~/bin/qemu-img
❯ PATH=… limactl create --name dummy - <<<"{images: [location: /etc/profile]}"
INFO[0000] Terminal is not available, proceeding without opening an editor
INFO[0000] Attempting to download the image              arch=aarch64 digest= location=/etc/profile
INFO[0000] Downloaded the image from "/etc/profile"

[AkihiroSuda]

Thanks, do we plan to migrate the template tests to BATS too?

[jandubois]

Thanks, do we plan to migrate the template tests to BATS too?

I think so, eventually. But I would first want to add tests for other bits that are currently mostly untested.