Commit 1ffda75
authored
[tests] Fix test semantics, JniValueMarshalerContractTests (#474)
Commit ffbb2dc introduced a unit test failure, as it overlooked
updating an "expected" string within the test
`Java.InteropTests.JniValueMarshaler_IJavaPeerable_ContractTests.CreateReturnValueFromManagedExpression()`.
The more important question is this: why was that test failure missed?
Turns Out™ that our unit test infrastructure is "dodgy": it was
possible for tests to fail while the overall `make run-all-tests`
target reported success!
For example, as of commit ffbb2dc we *know* that
`Java.Interop-Tests.dll` will report a test failure. Yet, if we
run `make run-tests` such that a different (successful) unit test
assembly follows it, no error is reported:
$ make run-tests TESTS="bin/TestDebug/Java.Interop-Tests.dll bin/TestDebug/Java.Interop.Export-Tests.dll"
...
…/Java.Interop/build-tools/scripts/RunNUnitTests.targets(35,5): error MSB3073: The command "mono --debug packages/NUnit.ConsoleRunner.3.9.0/tools/nunit3-console.exe bin/TestDebug/Java.Interop-Tests.dll --result="TestResult-Java.Interop-Tests.xml;format=nunit2" --output="bin/TestDebug/TestOutput-Java.Interop-Tests.txt"" exited with code 1.
0 Warning(s)
1 Error(s)
...
$ echo $?
0
*This* is why the [DevOps build for PR #472][0] was green and the PR
was merged. It's also why one of the [subsequent master builds][1]
was green, even though it [reported unit test failures][2].
Our tests are unreliable. What will we do about it?
1. Update `make run-all-tests` to explicitly invoke targets.
2. Update the other `run-*` targets to exit with a non-zero value
if *any* command exits with a non-zero value.
`make run-all-tests` cannot use prerequisites, because once a
prerequisites fails, all following prerequisites are skipped, and we
want *all* of the unit tests to run, so that we collect as many
errors as possible from a test run. As such, this rule:
run-all-tests: run-tests run-ptests
must be converted into:
run-all-tests:
r=0; \
$(MAKE) run-tests || r=1; \
$(MAKE) run-ptests || r=1; \
exit $$r
The other `run-*` targets must be fixed in a similar way, to ensure
that if *any* command fails, the entire rule *also* fails, while also
executing as many tests as possible.
With these changes in place, `make run-all-tests` will now report an
error if any of the executed tests fail:
$ make run-tests TESTS="bin/TestDebug/Java.Interop-Tests.dll bin/TestDebug/Java.Interop.Export-Tests.dll"
...
$ echo $?
2
Finally, fix the error reported from
`Java.InteropTests.JniValueMarshaler_IJavaPeerable_ContractTests`.
[0]: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=2950259&view=results
[1]: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=2950430&view=results
[2]: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=2950430&view=ms.vss-test-web.build-test-results-tab1 parent 5fe28cd commit 1ffda75
File tree
2 files changed
+15
-5
lines changed- src/Java.Interop/Tests/Java.Interop
2 files changed
+15
-5
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
49 | | - | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
50 | 56 | | |
51 | 57 | | |
52 | 58 | | |
| |||
145 | 151 | | |
146 | 152 | | |
147 | 153 | | |
148 | | - | |
| 154 | + | |
149 | 155 | | |
150 | 156 | | |
151 | 157 | | |
152 | | - | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
153 | 161 | | |
154 | 162 | | |
155 | | - | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
156 | 166 | | |
157 | 167 | | |
158 | 168 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
601 | 601 | | |
602 | 602 | | |
603 | 603 | | |
604 | | - | |
| 604 | + | |
605 | 605 | | |
606 | 606 | | |
607 | 607 | | |
| |||
0 commit comments