New `--set-exit-if-fail` option

[FMorschel]

I'd like to propose a --set-exit-if-fail option for the dart test.

The name is coming from the dart format that already has an --set-exit-if-changed option.

This option would simply exit if any test fails. My use case for this would be running it inside a git hook, where I simply want to warn the development team if any test fails so they can run all and fix what is needed.

If #2161 could be accomplished as well or similar issues, it would be really helpful to only show the exact test that failed first, no need for all the rest.

[natebosch]

If I understand the request correctly, you should be able to get this behavior today with --reporter silent. The exit code will be set, and no other test runner output should be emitted. We do not prevent the tests themselves from outputting.

Does dart test -r silent do what you'd hope?

[FMorschel]

Yes it does, so I have some suggestions on the dart test --help text output.

dart --version:

Dart SDK version: 3.2.3 (stable) (Tue Dec 5 17:58:33 2023 +0000) on "windows_x64"

dart test --help

Runs tests in this package.

Usage: dart test [files or directories...]

-h, --help                            Show this usage information.
    --version                         Show the package:test version.

Selecting Tests:
-n, --name                            A substring of the name of the test to run.
                                      Regular expression syntax is supported.
                                      If passed multiple times, tests must match all substrings.
-N, --plain-name                      A plain-text substring of the name of the test to run.
                                      If passed multiple times, tests must match all substrings.
-t, --tags                            Run only tests with all of the specified tags.
                                      Supports boolean selector syntax.
-x, --exclude-tags                    Don't run tests with any of the specified tags.
                                      Supports boolean selector syntax.
    --[no-]run-skipped                Run skipped tests instead of skipping them.

Running Tests:
-p, --platform                        The platform(s) on which to run the tests.
                                      [vm (default), chrome, firefox, ie, edge, node, experimental-chrome-wasm].
                                      Each platform supports the following compilers:
                                      [vm]: kernel (default), source, exe
                                      [chrome]: dart2js (default)
                                      [firefox]: dart2js (default)
                                      [ie]: dart2js (default)
                                      [edge]: dart2js (default)
                                      [node]: dart2js (default)
                                      [experimental-chrome-wasm]: dart2wasm (default)
-c, --compiler                        The compiler(s) to use to run tests, supported compilers are [dart2js, dart2wasm, exe, kernel, source].
                                      Each platform has a default compiler but may support other compilers.
                                      You can target a compiler to a specific platform using arguments of the following form [<platform-selector>:]<compiler>.
                                      If a platform is specified but no given compiler is supported for that platform, then it will use its default compiler.
-P, --preset                          The configuration preset(s) to use.
-j, --concurrency=<threads>           The number of concurrent test suites run.
                                      (defaults to "4")
    --total-shards                    The total number of invocations of the test runner being run.
    --shard-index                     The index of this test runner invocation (of --total-shards).
    --pub-serve=<port>                The port of a pub serve instance serving "test/".
    --timeout                         The default test timeout. For example: 15s, 2x, none
                                      (defaults to "30s")
    --ignore-timeouts                 Ignore all timeouts (useful if debugging)
    --pause-after-load                Pause for debugging before any tests execute.
                                      Implies --concurrency=1, --debug, and --ignore-timeouts.
                                      Currently only supported for browser tests.
    --debug                           Run the VM and Chrome tests in debug mode.
    --coverage=<directory>            Gather coverage and output it to the specified directory.
                                      Implies --debug.
    --[no-]chain-stack-traces         Use chained stack traces to provide greater exception details
                                      especially for asynchronous code. It may be useful to disable
                                      to provide improved test performance but at the cost of
                                      debuggability.
    --no-retry                        Don't rerun tests that have retry set.
    --test-randomize-ordering-seed    Use the specified seed to randomize the execution order of test cases.
                                      Must be a 32bit unsigned integer or "random".
                                      If "random", pick a random seed to use.
                                      If not passed, do not randomize test case execution order.
    --[no-]fail-fast                  Stop running tests after the first failure.

Output:
-r, --reporter=<option>               Set how to print test results.

          [compact] (default)         A single line, updated continuously.
          [expanded]                  A separate line for each update.
          [github]                    A custom reporter for GitHub Actions (the default reporter when running on GitHub Actions).
          [json]                      A machine-readable format (see https://dart.dev/go/test-docs/json_reporter.md).

    --file-reporter                   Enable an additional reporter writing test results to a file.
                                      Should be in the form <reporter>:<filepath>, Example: "json:reports/tests.json"
    --verbose-trace                   Emit stack traces with core library frames.
    --js-trace                        Emit raw JavaScript stack traces for browser tests.
    --[no-]color                      Use terminal colors.
                                      (auto-detected by default)

• I assume that [silent] is missing down here at the bottom on reporter.

Also, I tested what I thought would do a part of that: dart test --fail-fast, but I found no difference when running dart test --no-fail-fast. Is this intended behaviour? I simply have one test at the top of all my tests where the value is true but it expects false. Everything else passes just fine when I remove that test. With it on the first file listed on the vsCode Testing tab, no other tests seem to run with the CLI.

• I suggest you make it more clear what is the default option between --fail-fast and --no-fail-fast.

[FMorschel]

Sorry if I wasn't clear enough with my doubt.

Is it intended that I see no difference in running dart test --no-fail-fast and dart test --fail-fast? What is intended for this option to do?

I ran the tests for this project: https://github.com/fmorschel/due_date/ in the main branch. If you go to /test/constants_test.dart and add:

test('Failed', () => expect(true, isFalse));

In line 6 (just above the first group), then on the vsCode Testing tab and click to run all, everything runs with no problem besides that exact test. Now, when I run the commands mentioned above, they stop at that intentionally failing test, no difference between the options.

[natebosch]

• I assume that [silent] is missing down here at the bottom on reporter.

It was omitted intentionally since we didn't expect it to have uses outside of our infra.

With it on the first file listed on the vsCode Testing tab, no other tests seem to run with the CLI.

No other tests running is expected with --fail-fast

Is it intended that I see no difference in running dart test --no-fail-fast and dart test --fail-fast? What is intended for this option to do?

If you are running with -r silent the only difference you see would be in the amount of time before the process exits. The option means that no other tests will be started once the first failure occurs. The benefit to --fail-fast is reducing the runtime of the process when it will fail anyway.

Now, when I run the commands mentioned above, they stop at that intentionally failing test, no difference between the options.

I would expect a different number of tests to run between --fail-fast and --no-fail-fast. Can you run with the default reporter -r compact and see if there is a difference between the numbers in the +n -n: Some tests failed when using the option?

[FMorschel]

Running ls test on my project directory:

    Directory: C:\Users\felip_0vh5fa6\Documents\Projects\dart\due_date\test

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        22/12/2023     07:53           6560 constants_test.dart
-a----        12/12/2023     09:30           9193 date_validator_test.dart
-a----        12/12/2023     09:30          14912 due_date_test.dart
-a----        12/12/2023     09:30          21977 enums_test.dart
-a----        21/12/2023     16:40          20192 every_modifier_test.dart
-a----        12/12/2023     09:30          45356 every_test.dart
-a----        12/12/2023     09:30           7781 extensions_test.dart
-a----        12/12/2023     09:30          55743 period_generator_test.dart
-a----        12/12/2023     09:30          39221 period_test.dart

dart --version:

Dart SDK version: 3.2.3 (stable) (Tue Dec 5 17:58:33 2023 +0000) on "windows_x64"

If you clone the repo I mentioned above, and simply do this change on constants_test.dart:

add:

test('Failed', () => expect(true, isFalse));

Running dart test --fail-fast on my project directory:

00:00 +18 -1: test\constants_test.dart: Failed [E]
  Expected: false
    Actual: <true>
  
  package:matcher                expect
  test\constants_test.dart 6:24  main.<fn>
  
00:00 +19 -1: test\due_date_test.dart: Constructors: Unnamed: Month start
To run this test again: C:\src\flutter\bin\cache\dart-sdk\bin\dart.exe test test\constants_test.dart -p vm --plain-name "Failed"
00:01 +20 -1: test\date_validator_test.dart: DateValidatorWeekdayCountInMonth Last Friday Is not valid - did not complete
 [E]
00:01 +20 -1: Some tests failed.

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.

Running dart test --no-fail-fast on my project directory:

00:00 +10 -1: test\constants_test.dart: Failed [E]
  Expected: false
    Actual: <true>
  
  package:matcher                expect
  test\constants_test.dart 6:24  main.<fn>
  
00:00 +11 -1: test\date_validator_test.dart: DateValidatorWeekdayCountInMonth First Monday Is valid
To run this test again: C:\src\flutter\bin\cache\dart-sdk\bin\dart.exe test test\constants_test.dart -p vm --plain-name "Failed"
00:01 +855 -1: Some tests failed.

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.

VsCode "Run tests" (two stacked play symbols) button:

Now, when I revert the change mentioned on the screenshot above.

Running dart test --fail-fast on my project directory:

00:01 +855: All tests passed!

Running dart test --no-fail-fast on my project directory:

00:01 +855: All tests passed!

VsCode "Run tests" (two stacked play symbols) button:

Edit

I tested adding -r compact explicitly on my tests, no change.

[natebosch]

Filtering down the output to the most relevant line:

Running dart test --fail-fast on my project directory:

00:01 +20 -1: Some tests failed.

Running dart test --no-fail-fast on my project directory:

00:01 +855 -1: Some tests failed.

The --fail-fast case skipped 835 tests, only 21 total tests were ran. The --no-fail-fast case ran all 856 tests.

[FMorschel]

I see. Sorry I missed those lines. Since both results were basically the same size I missed that those lines had different numbers.

That is almost what I asked at #2161 as well. I'll use this for now.

Thank you a lot for your time and help!