Running `swift test --xunit-output` with `--disable-xctest` doesn't generate test output file

[xtremekforever]

Is it reproducible with SwiftPM command-line tools: swift build, swift test, swift package etc?

• Confirmed reproduction steps with SwiftPM CLI. The description text must include reproduction steps with either of command-line SwiftPM commands, swift build, swift test, swift package etc.

Description

I had been noticing this in the Swift 6.0 snapshots but thought it would be fixed for Swift 6.0 final release. However, I still see this behavior in the official Swift 6.0.0 toolchain.

I am using swift-testing in all of my projects. I noticed, however, that if I use the --disable-xctest flag with swift test, then --xunit-output does not actually output a file. For example:

$ swift test --xunit-output output.xml --disable-xctest
Building for debugging...
[1/1] Write swift-version-223EC37E04925308.txt
Build complete! (0.13s)
◇ Test run started.
↳ Testing Library Version: 6.0 (9aa8076dff01b66)
◇ Test myTestFunctionReturns() started.
✔ Test myTestFunctionReturns() passed after 0.001 seconds.
✔ Test run with 1 test passed after 0.001 seconds.
$ ls
Package.swift  Sources  Tests

If I remove --disable-xctest, it creates an output file, although it also doesn't have the name I expect:

$ swift test --xunit-output output.xml
Building for debugging...
[1/1] Write swift-version-223EC37E04925308.txt
Build complete! (0.16s)
Test Suite 'All tests' started at 2024-09-26 16:32:52.103
Test Suite 'debug.xctest' started at 2024-09-26 16:32:52.104
Test Suite 'debug.xctest' passed at 2024-09-26 16:32:52.104
         Executed 0 tests, with 0 failures (0 unexpected) in 0.0 (0.0) seconds
Test Suite 'All tests' passed at 2024-09-26 16:32:52.104
         Executed 0 tests, with 0 failures (0 unexpected) in 0.0 (0.0) seconds
◇ Test run started.
↳ Testing Library Version: 6.0 (9aa8076dff01b66)
◇ Test myTestFunctionReturns() started.
✔ Test myTestFunctionReturns() passed after 0.001 seconds.
✔ Test run with 1 test passed after 0.001 seconds
$ ls
output-swift-testing.xml  Package.swift  Sources  Tests

Is this a known issue or something that is intended to be fixed in the future? I want to be able to completely disable XUnit in my test suites and have test results output generated correctly.

Expected behavior

I expect that if I run swift test with --disable-xctest and --xunit-output together, that an output XML file is generated using the specified name with the test results.

Actual behavior

I see that if I run swift test with --disable-xctest along with --xunit-output, no output file is generated at all.

Steps to reproduce

1. Take any Swift 6.0 project that has tests that use swift-testing only (no XCTest cases).
2. Run swift test --xunit-output output.xml --disable-xctest.
3. Observe that no xunit output file is actually generated, although the test cases are run according to the logs.

Swift Package Manager version/commit hash

Not sure, it's the one that is shipped with the Swift 6.0 toolchain for Ubuntu 22.04.

Swift & OS version (output of swift --version ; uname -a)

Swift version 6.0 (swift-6.0-RELEASE)
Target: x86_64-unknown-linux-gnu
Linux Ruby 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

[grynspan]

XCTest only supports XML output when you also pass --parallel. See #4752. So it's expected that you won't get the file you asked for here. The reason you get output-swift-testing.xml is because Swift Testing currently also outputs an XML file and we haven't built logic to merge the two, so we put it in a separate file to avoid destroying what's emitted by XCTest. In the future, we'll want to emit a single file, but we need to solve #4752 first I think.

The other problem here—that you're not getting output from XCTest when you pass --disable-xctest—is covered by #7849.

[grynspan]

I'm going to mark this issue as a duplicate of #7849, but if I've missed something please comment and we can reopen. Thanks!

[grynspan]

Duplicate of #7849