Skip to content

[Tests] AbstractBuilderTestCase reproducibility is broken #32400

@cbuescher

Description

@cbuescher

Rest extending AbstractBuilderTestCase (essential all Query- and AggregationBuilder unit test) are not reproducible with the info that we log on test failure. The random values generated in each test method change depending on whether the method is run separately (e.g. with -Dtest.method) or the whole test suite is run. This happens even when fixing the full seed and not just the master seed.

To demonstrate this, add the following test method to e.g. RangeQueryBuilderTests:

    public void testRandomness() {
        assertEquals(-7481642973173394925L, randomLong());
    }

The whole test suite passes with 97B45898EB8C54FC as the master seed:

./gradlew :server:test -Dtests.seed=97B45898EB8C54FC -Dtests.class=org.elasticsearch.index.query.RangeQueryBuilderTests

However, only running with -Dtests.method="testRandomness", this fails with a different random long value than expected:

./gradlew :server:test -Dtests.seed=97B45898EB8C54FC -Dtests.class=org.elasticsearch.index.query.RangeQueryBuilderTests -Dtests.method="testRandomness"
   > Throwable #1: java.lang.AssertionError: expected:<-7481642973173394925> but was:<-6611281964877420649>
   >    at __randomizedtesting.SeedInfo.seed([97B45898EB8C54FC:F0156D024791DA56]:0)

Using the full seed as reported above when running the full suite works, so this passes:

./gradlew :server:test -Dtests.seed=97B45898EB8C54FC:F0156D024791DA56  -Dtests.class=org.elasticsearch.index.query.RangeQueryBuilderTests

But fails when using the REPRODUCE log line that includes the "-Dtests.method" parameter:

./gradlew :server:test -Dtests.seed=97B45898EB8C54FC:F0156D024791DA56 \
 -Dtests.class=org.elasticsearch.index.query.RangeQueryBuilderTests -Dtests.method="testRandomness"

Metadata

Metadata

Assignees

Labels

:Search/SearchSearch-related issues that do not fall into other categories>testIssues or PRs that are addressing/adding testsv7.0.0-beta1

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions