-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Description
Rest extending AbstractBuilderTestCase (essential all Query- and AggregationBuilder unit test) are not reproducible with the info that we log on test failure. The random values generated in each test method change depending on whether the method is run separately (e.g. with -Dtest.method) or the whole test suite is run. This happens even when fixing the full seed and not just the master seed.
To demonstrate this, add the following test method to e.g. RangeQueryBuilderTests:
public void testRandomness() {
assertEquals(-7481642973173394925L, randomLong());
}
The whole test suite passes with 97B45898EB8C54FC as the master seed:
./gradlew :server:test -Dtests.seed=97B45898EB8C54FC -Dtests.class=org.elasticsearch.index.query.RangeQueryBuilderTests
However, only running with -Dtests.method="testRandomness", this fails with a different random long value than expected:
./gradlew :server:test -Dtests.seed=97B45898EB8C54FC -Dtests.class=org.elasticsearch.index.query.RangeQueryBuilderTests -Dtests.method="testRandomness"
> Throwable #1: java.lang.AssertionError: expected:<-7481642973173394925> but was:<-6611281964877420649>
> at __randomizedtesting.SeedInfo.seed([97B45898EB8C54FC:F0156D024791DA56]:0)
Using the full seed as reported above when running the full suite works, so this passes:
./gradlew :server:test -Dtests.seed=97B45898EB8C54FC:F0156D024791DA56 -Dtests.class=org.elasticsearch.index.query.RangeQueryBuilderTests
But fails when using the REPRODUCE log line that includes the "-Dtests.method" parameter:
./gradlew :server:test -Dtests.seed=97B45898EB8C54FC:F0156D024791DA56 \
-Dtests.class=org.elasticsearch.index.query.RangeQueryBuilderTests -Dtests.method="testRandomness"