`cabal test` sees the executable, but `cabal run` does not.

[kindaro]

There is a package called fourmolu that is going to be my example here. The relevant issue is fourmolu/fourmolu#231. The appropriate commit is 12af899611cea08c5d9a1f547d432edfbce38be4 — the current main.

See for yourself:

1. Remove all instances of fourmolu on the path. You should have this:

   % where fourmolu
   fourmolu not found
   

2. See that the tests pass:

   % cabal test
   Build profile: -w ghc-8.10.7 -O1
   In order, the following will be built (use -v for more details):
    - fourmolu-0.8.1.0 (test:fixity-tests) (ephemeral targets)
    - fourmolu-0.8.1.0 (test:region-tests) (ephemeral targets)
    - fourmolu-0.8.1.0 (test:tests) (ephemeral targets)
   Preprocessing test suite 'fixity-tests' for fourmolu-0.8.1.0..
   Building test suite 'fixity-tests' for fourmolu-0.8.1.0..
   Preprocessing test suite 'region-tests' for fourmolu-0.8.1.0..
   Building test suite 'region-tests' for fourmolu-0.8.1.0..
   Preprocessing test suite 'tests' for fourmolu-0.8.1.0..
   Building test suite 'tests' for fourmolu-0.8.1.0..
   Running 1 test suites...
   Running 1 test suites...
   Test suite region-tests: RUNNING...
   Test suite fixity-tests: RUNNING...
   Test suite region-tests: PASS
   Test suite logged to:
   /home/kindaro/projects/haskell/fourmolu/dist-newstyle/build/x86_64-linux/ghc-8.10.7/fourmolu-0.8.1.0/t/region-tests/test/fourmolu-0.8.1.0-region-tests.log
   1 of 1 test suites (1 of 1 test cases) passed.
   Test suite fixity-tests: PASS
   Test suite logged to:
   /home/kindaro/projects/haskell/fourmolu/dist-newstyle/build/x86_64-linux/ghc-8.10.7/fourmolu-0.8.1.0/t/fixity-tests/test/fourmolu-0.8.1.0-fixity-tests.log
   1 of 1 test suites (1 of 1 test cases) passed.
   Running 1 test suites...
   Test suite tests: RUNNING...
   Test suite tests: PASS
   Test suite logged to:
   /home/kindaro/projects/haskell/fourmolu/dist-newstyle/build/x86_64-linux/ghc-8.10.7/fourmolu-0.8.1.0/t/tests/test/fourmolu-0.8.1.0-tests.log
   1 of 1 test suites (1 of 1 test cases) passed.
   

3. See that the tests fail if run with cabal run:

   % grep 'test-suite' fourmolu.cabal | cut -d ' ' -f 2 | while read name; do cabal run test:"$name"; done
   Build profile: -w ghc-8.10.7 -O1
   In order, the following will be built (use -v for more details):
    - fourmolu-0.8.1.0 (test:tests) (additional components to build)
   Preprocessing test suite 'tests' for fourmolu-0.8.1.0..
   Building test suite 'tests' for fourmolu-0.8.1.0..
   …
   Finished in 4.1124 seconds
   985 examples, 1 failure, 2 pending
   Build profile: -w ghc-8.10.7 -O1
   In order, the following will be built (use -v for more details):
    - fourmolu-0.8.1.0 (test:region-tests) (additional components to build)
   Preprocessing test suite 'region-tests' for fourmolu-0.8.1.0..
   Building test suite 'region-tests' for fourmolu-0.8.1.0..
   …
   Finished in 0.0011 seconds
   8 examples, 1 failure, 7 pending
   Build profile: -w ghc-8.10.7 -O1
   In order, the following will be built (use -v for more details):
    - fourmolu-0.8.1.0 (test:fixity-tests) (additional components to build)
   Preprocessing test suite 'fixity-tests' for fourmolu-0.8.1.0..
   Building test suite 'fixity-tests' for fourmolu-0.8.1.0..
   …
   Finished in 0.0015 seconds
   6 examples, 1 failure, 5 pending
   

4. See that the tests pass if run with cabal run after cabal install:

   % cabal install
   …
   Symlinking 'fourmolu' to '/home/kindaro/.cabal/bin/fourmolu'
   % grep 'test-suite' fourmolu.cabal | cut -d ' ' -f 2 | while read name; do cabal run test:"$name"; done
   Resolving dependencies...
   Up to date
   …
   Finished in 6.4715 seconds
   985 examples, 0 failures
   Up to date
   …
   Finished in 0.3642 seconds
   8 examples, 0 failures
   Up to date
   …
   Finished in 0.7731 seconds
   6 examples, 0 failures
   

Expected:

Either the tests fail, or the tests pass. It does not matter which way I run them. In particular, Cabal makes the same executables available in the same way: either the fourmolu executable is made available to both cabal run and cabal test or to neither.

Actual:

The executable fourmolu is made available to the test programs when they are run with cabal test but not with cabal run.

System information:

% uname --all
Linux kindrom-silverheart 5.18.13-arch1-1 #1 SMP PREEMPT_DYNAMIC Fri, 22 Jul 2022 13:05:04 +0000 x86_64 GNU/Linux
% cabal --version
cabal-install version 3.8.1.0
compiled using version 3.8.1.0 of the Cabal library 
% ghc --version
The Glorious Glasgow Haskell Compilation System, version 8.10.7

[Mikolaj]

So is the underlying bug that build-tool-depends: fourmolu:fourmolu is not honoured by cabal run test:foo? What's the semantics of build-tool-depends? That the tool should only be made available for building the code, or also for running the artefact?

[Mikolaj]

Or, from another angle, is cabal run foo supposed to be equivalent to cabal build foo; path_to_foo/foo or not? How about cabal test foo?

[georgefst]

Or, from another angle, is cabal run foo supposed to be equivalent to cabal build foo; path_to_foo/foo or not?

This is an excellent point. To me, it would be surprising if it wasn't.

What's the semantics of build-tool-depends?

I'd always assumed from the name that it's just for build tools. But the user guide states that it also makes executables available at run time. And does not make any qualifications about cabal run vs. cabal test, or test-suite stanzas vs. executable.

It's too late to change the behaviour of adding to PATH for cabal test without very good reason, since people rely on the behaviour. Besides, some new test-tool-depends or similar mechanism would need to be introduced to cover the old use cases.

Perhaps the best fix here is to:

• Fix the user's guide to make it clear that build-tool-depends makes executables available on PATH for cabal test (and cabal bench? maybe even cabal repl?), but not cabal run.
• Improve the cabal test CLI, so that all the reasons for currently preferring to run cabal run test:foo no longer apply. --test-options are split wrong #8090 is one. Coloured output is another. There may be more.

[gbaz]

Why not just make it available in the PATH in cabal run as well? That seems most in keeping with the spirit of the field.

[georgefst]

Why not just make it available in the PATH in cabal run as well? That seems most in keeping with the spirit of the field.

I'm not totally against it. But as @Mikolaj alluded to, it seems desirable to have cabal run target be equivalent to cabal build target && $(cabal list-bin target). Unless there's some reason why this is already not strictly the case?

[Mikolaj]

IIRC cabal run can also be used for running interpreted Haskell scripts (as opposed to binaries), so perhaps it sets up the environments specially for them. If so, we are not complicating the semantics so much by unifying how the environments are set up for exes and scripts. Not sure how test-tool-depends works with scripts, though, nor how it should work.

[georgefst]

IIRC cabal run can also be used for running interpreted Haskell scripts (as opposed to binaries),

I thought those were still compiled (even after #7851)? I've always wanted them to (at least have the option to) go through the interpreter, but I don't remember seeing anything about it in release notes.

[Mikolaj]

I thought those were still compiled

You may be right. There were recent great contributions preventing them from being compiled anew each time they are executed, etc., etc., so I thought they may have been compiled to bytecode or interpreted beforehand. I'm not sure about any of that, but what I'm interested in in this context is whether they are executed in a somehow modified environment or not. There's also cabal exec. Basically, I'm looking for a convincing story we may write in the manual, as opposed to "everything works with its own special quirks, which are too numerous to mention".

[ulysses4ever]

My understanding is that scripts are still compiled to native, but the build artifacts are cached now (just like with "normal" packages). GHC recently added a --run option that may look like a good alternative to use for cabal run (e.g. cabal run --byte-code), but I didn't see a feature request for it. So far, Ben only proposed (#7393) it for usage in custom setups (which no one wants to touch with a six-feet pole).

[Mikolaj]

My understanding is that scripts are still compiled to native, but the build artifacts are cached now (just like with "normal" packages). GHC recently added a --run option that may look like a good alternative to use for cabal run (e.g. cabal run --byte-code), but I didn't see a feature request for it. So far, Ben only proposed (#7393) it for usage in custom setups (which no one wants to touch with a six-feet pole).

Thank you. That would imply that cabal run never changes the env in which the exe is run in any way. OTOH, cabal exec seems to do something close to what people want to use cabal run for:

https://cabal.readthedocs.io/en/latest/cabal-commands.html?highlight=exec#cabal-exec

Am I reading this right? Brings exes into scope? Would cabal exec test:foo (or cabal exec run test:foo? how is it called exactly?) work properly? If so, we could leave cabal run be and not mix it with cabal exec.

[fgaz]

cabal run foo supposed to be equivalent to cabal build foo; path_to_foo/foo or not

it isn't equivalent. A package with data files will not run with cabal build foo; path_to_foo/foo because Paths_pkg.hs points to the cabal store, so cabal run sets an environment variable that overrides that path (and so does cabal exec, you can try running cabal exec env).

If we want to support #6919 then the variable will have to stay, otherwise we could compile directly with local paths

[fgaz]

Why not just make it available in the PATH in cabal run as well? That seems most in keeping with the spirit of the field.

makes sense to me. after all cabal run test is just a convenience to avoid passing --test-arguments (see #4861)