[BUG] Empty cobertura.xml in coverlet.collector 6.0.3 that is not reproduced in 6.0.2

[kotsaris]

Describe the bug
I recently upgraded to 6.0.3 and I noticed the pipelines failing. It seems that cobertura.xml comes out empty. I am using a runsettings file with exclude/include. I notice that if I remove these sections the cobertura gets populated.

That said, coverage works fine and cobertura gets entries on 6.0.2 of coverlet.collector nuget package.

To Reproduce
Check out https://github.com/kotsaris/repro-coverlet-empty-cobertura cd into ClassLibrary1 folder and run tests via

dotnet test --collect:"XPlat Code Coverage" --settings .\coverage.runsettings

Expected behavior
It should populate cobertura.xml with something.

Actual behavior
Cobertura.xml is empty.

<?xml version="1.0" encoding="utf-8"?>
<coverage line-rate="0" branch-rate="0" version="1.9" timestamp="1735730899" lines-covered="0" lines-valid="0" branches-covered="0" branches-valid="0">
  <sources />
  <packages />
</coverage>

Restore complete (0.3s)
  ClassLibrary1 succeeded (0.1s) → ClassLibrary1\bin\Debug\net9.0\ClassLibrary1.dll
  ClassLibrary1.Tests succeeded (0.0s) → ClassLibrary1.Tests\bin\Debug\net9.0\ClassLibrary1.Tests.dll
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v3.0.0+e341b939fe (64-bit .NET 9.0.0)
[xUnit.net 00:00:00.03]   Discovering: ClassLibrary1.Tests
[xUnit.net 00:00:00.05]   Discovered:  ClassLibrary1.Tests
[xUnit.net 00:00:00.06]   Starting:    ClassLibrary1.Tests
[xUnit.net 00:00:00.08]   Finished:    ClassLibrary1.Tests
  ClassLibrary1.Tests test succeeded (0.7s)

Test summary: total: 1, failed: 0, succeeded: 1, skipped: 0, duration: 0.7s
Build succeeded in 1.2s

Workload updates are available. Run `dotnet workload list` for more information.

Attachments:
  C:\dev\kotsaris\repro-coverlet-empty-cobertura\ClassLibrary1\ClassLibrary1.Tests\TestResults\5e5adf4e-c517-478b-a81e-ccff2099c832\coverage.cobertura.xml

Configuration (please complete the following information):
Run Settings:

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
    <DataCollectionRunSettings>
        <DataCollectors>
            <DataCollector friendlyName="XPlat code coverage">
                <Configuration>
                    <Format>cobertura</Format>
                    <Exclude>
                        [*]ClassLibrary1.Tests.*
                    </Exclude>
                    <Include>[*]ClassLibrary1*</Include>
                    <ExcludeByAttribute>Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute>
                    <SingleHit>false</SingleHit>
                    <UseSourceLink>false</UseSourceLink>
                    <IncludeTestAssembly>false</IncludeTestAssembly>
                    <SkipAutoProps>true</SkipAutoProps>
                    <DeterministicReport>false</DeterministicReport>
                    <ExcludeAssembliesWithoutSources>MissingAll,MissingAny,None</ExcludeAssembliesWithoutSources>
                </Configuration>
            </DataCollector>
        </DataCollectors>
    </DataCollectionRunSettings>
</RunSettings>

Please provide more information on your .NET configuration:
* Which coverlet package and version was used? 6.0.3
* Which version of .NET is the code running on? 9
* What OS and version, and what distro if applicable? Windows 11
* What is the architecture (x64, x86, ARM, ARM64)? x64
* Do you know whether it is specific to that configuration? 6.0.2 does not exhibit this

Additional context
Downgrade to 6.0.2 and the problem goes away.

❗ Please also read Known Issues

[sboulema]

I am running into the same problem ☹️

[daveMueller]

Thanks a lot for reporting and for the repro @kotsaris 🙏. I did some quick analysis and think the problem came in here #1645. To be even more precise I think this condition here was missed:
https://github.com/coverlet-coverage/coverlet/pull/1645/files#diff-be914c2ef40b4434ccb0927c453da5f7f2a4f828d601367f585bd7284349ba85L354-L357

I locally added it back and it seems to work. But maybe we should consider putting in a bit more work because I think the whole Include and Exclude filter logic is a bit difficult to understand? cc @Bertk @MarcoRossignoli

(btw @kotsaris in your repro it is also sufficient if you only remove the Exclude filter from the runsettings file)

[mu88]

I'm facing the same issue with coverlet.msbuild instead of coverlet.collector: when using v6.0.3 and running dotnet test <all the other things> /p:CoverletOutputFormat=opencover, the output file is empty. Downgrading the package to v6.0.2 resolves the issue.

[argium]

+1.

Version 6.0.3 produces a coverage.cobertura.xml without any results. Reverting to 6.0.2 without other changes resolves the issue.

[JohnYoungers]

Thanks a lot for reporting and for the repro @kotsaris 🙏. I did some quick analysis and think the problem came in here #1645. To be even more precise I think this condition here was missed: https://github.com/coverlet-coverage/coverlet/pull/1645/files#diff-be914c2ef40b4434ccb0927c453da5f7f2a4f828d601367f585bd7284349ba85L354-L357

I locally added it back and it seems to work. But maybe we should consider putting in a bit more work because I think the whole Include and Exclude filter logic is a bit difficult to understand? cc @Bertk @MarcoRossignoli

(btw @kotsaris in your repro it is also sufficient if you only remove the Exclude filter from the runsettings file)

would it make sense to add that condition back as a 6.0.4 patch? and address the filter logic in a minor bump?

[daveMueller]

Hi guys, could you help out by providing a bit more input on how you are using the Include and Exclude filters in combination in your projects and what is the result you expect. Just something like [module-names, include-filters, exclude-filters, expected-modules]. We already have the repro @kotsaris provided but some more references would be cool. We just want to make sure that we don't miss anything again that isn't covered by unit test.

@JohnYoungers not sure about a new release just for this, I need to discuss this with the other maintainers. There will be the possibility to consume the fix in our nightly once we have merged it.

[argium]

Hi guys, could you help out by providing a bit more input on how you are using the Include and Exclude filters in combination in your projects and what is the result you expect.

Here's what I've got:

          <Exclude>[coverlet.*.tests?]*,[*]Coverlet.Core*</Exclude> 
          <Include>*C.D*</Include>

The intention is to include assemblies such as:

A.C.D.E
B.C.D.E
A.C.D.E.F

and obviously exclude the corverlet assemblies. I honestly don't recall why the exclude looks the way it does other than to say that it took longer than I expected and I couldn't spend any more time on it. Without the exclude, coverlet assemblies were throwing off the code coverage percentage.

[mu88]

And we're doing something like this:

dotnet test 
  --configuration Release 
  --no-build 
  --no-restore 
  --logger:trx 
  --verbosity minimal 
  --filter RunOnBuildServer!=false&Category=Unit 
  --results-directory test-outputs/component 
  /p:CollectCoverage=true 
  /p:ExcludeByAttribute="GeneratedCodeAttribute,ExcludeFromUnitTestCoverageAttribute" 
  /p:Exclude=[*]*.Migrations.* 
  /p:CoverletOutputFormat=opencover 
  /p:CoverletOutput=coverage.opencover.Unit.xml 
  Tests/Tests.csproj

[JohnYoungers]

@JohnYoungers not sure about a new release just for this, I need to discuss this with the other maintainers. There will be the possibility to consume the fix in our nightly once we have merged it.

I understand the hesitation, but I think as time goes on, this will catch more and more consumers with automated nuget update processes for patch releases.

This would be a sample of our usage:

  /p:CollectCoverage=true \
  /p:CoverletOutputFormat=cobertura%2copencover \
  /p:CoverletOutput=${{ inputs.output_path }}/${{ inputs.test_run_name }} \
  /p:Exclude="Company.App/Subnamespace/**/*,Company.App.Api/Program.cs,database/**" \

[JSkimming]

Hi guys, could you help out by providing a bit more input on how you are using the Include and Exclude filters in combination in your projects and what is the result you expect.

We run our test builds from a bash script. Our problem is not specifically cobertura output but any output. Where we previous received coverage, some of our builds get no coverage.

Here's the GitHub actions output where we no longer receive results.

Broken coverage

$ dotnet test --no-restore --no-build -c Release \
  /home/runner/work/my-product/my-product/my-service/test/Abc.MyService.Tests/Abc.MyService.Tests.csproj \
  --results-directory /home/runner/work/my-product/my-product/my-service/test/TestResults/output/ \
  --logger trx;LogFileName=Abc.MyService.Tests.trx \
  --logger html;LogFileName=Abc.MyService.Tests.html \
  -p:CollectCoverage=true \
  -p:Include=[Abc.MyService*]* \
  -p:Exclude="[*.Tests]*,[Abc.*]*.Migrations.*,[Abc.*]*.SourceGenerationContext" \
  -p:CoverletOutput=/home/runner/work/my-product/my-product/my-service/test/TestResults/output/ \
  -p:CoverletOutputFormat="json,opencover,cobertura,lcov"

  [coverlet] _mapping file name: 'CoverletSourceRootsMapping_Abc.MyService.Tests'
Test run for /home/runner/work/my-product/my-product/my-service/test/Abc.MyService.Tests/bin/Release/net9.0/Abc.MyService.Tests.dll (.NETCoreApp,Version=v9.0)
VSTest version 17.12.0 (x64)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Results File: /home/runner/work/my-product/my-product/my-service/test/TestResults/output/Abc.MyService.Tests.trx
Html test results file : /home/runner/work/my-product/my-product/my-service/test/TestResults/output/Abc.MyService.Tests.html

Passed!  - Failed:     0, Passed:   687, Skipped:     0, Total:   687, Duration: 1 m 20 s - Abc.MyService.Tests.dll (net9.0)
  [coverlet] 
  Calculating coverage result...
   Generating report '/home/runner/work/my-product/my-product/my-service/test/TestResults/output/coverage.json'
   Generating report '/home/runner/work/my-product/my-product/my-service/test/TestResults/output/coverage.opencover.xml'
   Generating report '/home/runner/work/my-product/my-product/my-service/test/TestResults/output/coverage.cobertura.xml'
   Generating report '/home/runner/work/my-product/my-product/my-service/test/TestResults/output/coverage.info'

+--------+------+--------+--------+
| Module | Line | Branch | Method |
+--------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 0%   | 0%     | 0%     |
+---------+------+--------+--------+
| Average | 0%   | 0%     | 0%     |
+---------+------+--------+--------+

Though we have another service where we still receive the output.

Still working coverage

$ dotnet test --no-restore --no-build -c Release \
  /home/runner/work/my-product/my-product/other-service/test/Abc.OtherService.Tests/Abc.OtherService.Tests.csproj \
  --results-directory /home/runner/work/my-product/my-product/other-service/test/TestResults/output/ \
  --logger trx;LogFileName=Abc.OtherService.Tests.trx \
  --logger html;LogFileName=Abc.OtherService.Tests.html \
  -p:CollectCoverage=true \
  -p:Include=[Abc.*]* \
  -p:Exclude="[*.Tests]*,[Abc.Testing.*]*,[Abc.Platform.Testing.*]*" \
  -p:CoverletOutput=/home/runner/work/my-product/my-product/other-service/test/TestResults/output/Abc.OtherService.Tests.coverage.json

  [coverlet] _mapping file name: 'CoverletSourceRootsMapping_Abc.OtherService.Tests'
Test run for /home/runner/work/my-product/my-product/other-service/test/Abc.OtherService.Tests/bin/Release/net9.0/Abc.OtherService.Tests.dll (.NETCoreApp,Version=v9.0)
VSTest version 17.12.0 (x64)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Results File: /home/runner/work/my-product/my-product/other-service/test/TestResults/output/Abc.OtherService.Tests.trx
Html test results file : /home/runner/work/my-product/my-product/other-service/test/TestResults/output/Abc.OtherService.Tests.html

Passed!  - Failed:     0, Passed:   776, Skipped:     0, Total:   776, Duration: 50 s - Abc.OtherService.Tests.dll (net9.0)
  [coverlet] 
  Calculating coverage result...
   Generating report '/home/runner/work/my-product/my-product/other-service/test/TestResults/output/Abc.OtherService.Tests.coverage.json'

+-----------------------+--------+--------+--------+
| Module                | Line   | Branch | Method |
+-----------------------+--------+--------+--------+
| Abc.OtherService      | 100%   | 100%   | 100%   |
+-----------------------+--------+--------+--------+
| Abc.OtherService.Apps | 99.32% | 94.58% | 100%   |
+-----------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 99.36% | 94.76% | 100%   |
+---------+--------+--------+--------+
| Average | 99.66% | 97.29% | 100%   |
+---------+--------+--------+--------+

[LeszekKalibrate]

I am running into the same problem ☹️
How this 6.0.3 was release without checking that it still works?