[CI] InternalAutoDateHistogramTests::testReduceRandom fails reproducable for certain seed

[hendrikmuhs]

The following test failed in a PR-build and is reproducible for me on my local box:

./gradlew :server:unitTest -Dtests.seed=24D81295C2796C6C -Dtests.class=org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests -Dtests.method="testReduceRandom" -Dtests.security.manager=true -Dtests.locale=el-CY -Dtests.timezone=Europe/Nicosia -Dcompiler.java=11 -Druntime.java=8

fails with:

  1> [2019-02-28T10:25:36,507][INFO ][o.e.s.a.b.h.InternalAutoDateHistogramTests] [testReduceRandom] before test
  1> [2019-02-28T10:25:36,723][INFO ][o.e.s.a.b.h.InternalAutoDateHistogramTests] [testReduceRandom] after test
  2> REPRODUCE WITH: ./gradlew :server:unitTest -Dtests.seed=24D81295C2796C6C -Dtests.class=org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests -Dtests.method="testReduceRandom" -Dtests.security.manager=true -Dtests.locale=el-CY -Dtests.timezone=Europe/Nicosia -Dcompiler.java=11 -Druntime.java=8
FAILURE 1.13s | InternalAutoDateHistogramTests.testReduceRandom <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: expected:<{1548979200000=1700}> but was:<{1548979200000=1569, 1551398400000=131}>
   >    at __randomizedtesting.SeedInfo.seed([24D81295C2796C6C:E4E5BE84E424F53A]:0)
   >    at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests.assertReduced(InternalAutoDateHistogramTests.java:200)
   >    at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests.assertReduced(InternalAutoDateHistogramTests.java:50)
   >    at org.elasticsearch.test.InternalAggregationTestCase.testReduceRandom(InternalAggregationTestCase.java:294)
   >    at java.lang.Thread.run(Thread.java:748)
  2> NOTE: leaving temporary files on disk at: /home/hendrik/work/git-elastic/elasticsearch/server/build/testrun/unitTest/J0/temp/org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests_24D81295C2796C6C-001
  2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=1501, maxMBSortInHeap=7.215678116490036, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@5f67521f), locale=el-CY, timezone=Europe/Nicosia
  2> NOTE: Linux 4.18.0-15-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=449796568,total=514850816
  2> NOTE: All tests run in this JVM: [InternalAutoDateHistogramTests]
Completed [1/1] in 2.83s, 1 test, 1 failure <<< FAILURES!


> Task :server:unitTest FAILED

Tested only on master but likely applies to other versions.

[elasticmachine]

Pinging @elastic/es-analytics-geo

[original-brownbear]

Also fails for

./gradlew :server:unitTest -Dtests.seed=7DBCFCFC5EFB7BF8 -Dtests.class=org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests -Dtests.method="testReduceRandom" -Dtests.security.manager=true -Dtests.locale=fil-PH -Dtests.timezone=Asia/Krasnoyarsk -Dcompiler.java=11 -Druntime.java=11

➜  elasticsearch git:(master) ./gradlew :server:unitTest -Dtests.seed=7DBCFCFC5EFB7BF8 -Dtests.class=org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests -Dtests.method="testReduceRandom" -Dtests.security.manager=true -Dtests.locale=fil-PH -Dtests.timezone=Asia/Krasnoyarsk -Dcompiler.java=11 -Druntime.java=11

> Configure project :benchmarks
=======================================
Elasticsearch Build Hamster says Hello!
  Gradle Version        : 5.2.1
  OS Info               : Linux 4.18.0-15-generic (amd64)
  JDK Version           : 11 (Oracle Corporation 11.0.1 [OpenJDK 64-Bit Server VM 11.0.1+13-Ubuntu-3ubuntu3.18.10.1])
  JAVA_HOME             : /usr/lib/jvm/java-11-openjdk-amd64
  Random Testing Seed   : 7DBCFCFC5EFB7BF8
=======================================
Suite: org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests
  1> [2019-03-01T01:57:18,664][INFO ][o.e.s.a.b.h.InternalAutoDateHistogramTests] [testReduceRandom] before test
  1> [2019-03-01T01:57:18,762][INFO ][o.e.s.a.b.h.InternalAutoDateHistogramTests] [testReduceRandom] after test
  2> REPRODUCE WITH: ./gradlew :server:unitTest -Dtests.seed=7DBCFCFC5EFB7BF8 -Dtests.class=org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests -Dtests.method="testReduceRandom" -Dtests.security.manager=true -Dtests.locale=fil-PH -Dtests.timezone=Asia/Krasnoyarsk -Dcompiler.java=11 -Druntime.java=11
FAILURE 0.41s | InternalAutoDateHistogramTests.testReduceRandom <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: expected:<{1548979200000=989}> but was:<{1548979200000=928, 1551398400000=61}>
   >    at __randomizedtesting.SeedInfo.seed([7DBCFCFC5EFB7BF8:BD8150ED78A6E2AE]:0)
   >    at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests.assertReduced(InternalAutoDateHistogramTests.java:200)
   >    at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests.assertReduced(InternalAutoDateHistogramTests.java:50)
   >    at org.elasticsearch.test.InternalAggregationTestCase.testReduceRandom(InternalAggregationTestCase.java:294)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   >    at java.base/java.lang.Thread.run(Thread.java:834)
  2> NOTE: leaving temporary files on disk at: /home/brownbear/src/elasticsearch/server/build/testrun/unitTest/J0/temp/org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests_7DBCFCFC5EFB7BF8-001
  2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=1350, maxMBSortInHeap=7.054482784277555, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@73807316), locale=fil-PH, timezone=Asia/Krasnoyarsk
  2> NOTE: Linux 4.18.0-15-generic amd64/Oracle Corporation 11.0.1 (64-bit)/cpus=16,threads=1,free=502433536,total=536870912
  2> NOTE: All tests run in this JVM: [InternalAutoDateHistogramTests]
Completed [1/1] in 1.18s, 1 test, 1 failure <<< FAILURES!


> Task :server:unitTest FAILED
   [junit4] <JUnit4> says g'day! Master seed: 7DBCFCFC5EFB7BF8
==> Test Info: seed=7DBCFCFC5EFB7BF8; jvm=1; suite=1
Tests with failures:
  - org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests.testReduceRandom

   [junit4] JVM J0:     0.26 ..     1.76 =     1.51s
   [junit4] Execution time total: 1.77 sec.
   [junit4] Tests summary: 1 suite, 1 test, 1 failure

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':server:unitTest'.
> There were test failures: 1 suite, 1 test, 1 failure [seed: 7DBCFCFC5EFB7BF8]

➜ elasticsearch git:(master) ./gradlew :server:unitTest -Dtests.seed=7DBCFCFC5EFB7BF8 -Dtests.class=org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests -Dtests.method="testReduceRandom" -Dtests.security.manager=true -Dtests.locale=fil-PH -Dtests.timezone=Asia/Krasnoyarsk -Dcompiler.java=11 -Druntime.java=11

Configure project :benchmarks
=======================================
Elasticsearch Build Hamster says Hello!
Gradle Version : 5.2.1
OS Info : Linux 4.18.0-15-generic (amd64)
JDK Version : 11 (Oracle Corporation 11.0.1 [OpenJDK 64-Bit Server VM 11.0.1+13-Ubuntu-3ubuntu3.18.10.1])
JAVA_HOME : /usr/lib/jvm/java-11-openjdk-amd64
Random Testing Seed : 7DBCFCFC5EFB7BF8
=======================================
Suite: org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests
1> [2019-03-01T01:57:18,664][INFO ][o.e.s.a.b.h.InternalAutoDateHistogramTests] [testReduceRandom] before test
1> [2019-03-01T01:57:18,762][INFO ][o.e.s.a.b.h.InternalAutoDateHistogramTests] [testReduceRandom] after test
2> REPRODUCE WITH: ./gradlew :server:unitTest -Dtests.seed=7DBCFCFC5EFB7BF8 -Dtests.class=org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests -Dtests.method="testReduceRandom" -Dtests.security.manager=true -Dtests.locale=fil-PH -Dtests.timezone=Asia/Krasnoyarsk -Dcompiler.java=11 -Druntime.java=11
FAILURE 0.41s | InternalAutoDateHistogramTests.testReduceRandom <<< FAILURES!
Throwable #1: java.lang.AssertionError: expected:<{1548979200000=989}> but was:<{1548979200000=928, 1551398400000=61}>
at __randomizedtesting.SeedInfo.seed([7DBCFCFC5EFB7BF8:BD8150ED78A6E2AE]:0)
at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests.assertReduced(InternalAutoDateHistogramTests.java:200)
at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests.assertReduced(InternalAutoDateHistogramTests.java:50)
at org.elasticsearch.test.InternalAggregationTestCase.testReduceRandom(InternalAggregationTestCase.java:294)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.base/java.lang.Thread.run(Thread.java:834)
2> NOTE: leaving temporary files on disk at: /home/brownbear/src/elasticsearch/server/build/testrun/unitTest/J0/temp/org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests_7DBCFCFC5EFB7BF8-001
2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=1350, maxMBSortInHeap=7.054482784277555, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@73807316), locale=fil-PH, timezone=Asia/Krasnoyarsk
2> NOTE: Linux 4.18.0-15-generic amd64/Oracle Corporation 11.0.1 (64-bit)/cpus=16,threads=1,free=502433536,total=536870912
2> NOTE: All tests run in this JVM: [InternalAutoDateHistogramTests]
Completed [1/1] in 1.18s, 1 test, 1 failure <<< FAILURES!

Task :server:unitTest FAILED
[junit4] says g'day! Master seed: 7DBCFCFC5EFB7BF8
==> Test Info: seed=7DBCFCFC5EFB7BF8; jvm=1; suite=1
Tests with failures:

• org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests.testReduceRandom

[junit4] JVM J0: 0.26 .. 1.76 = 1.51s
[junit4] Execution time total: 1.77 sec.
[junit4] Tests summary: 1 suite, 1 test, 1 failure

FAILURE: Build failed with an exception.

• What went wrong:
  Execution failed for task ':server:unitTest'.

There were test failures: 1 suite, 1 test, 1 failure [seed: 7DBCFCFC5EFB7BF8]

[jkakavas]

This failed in a PR too #38382. This should be muted

[not-napoleon]

Found a case of this failing in 7.x as well: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+intake/360/

REPRODUCE WITH: ./gradlew :server:unitTest \
  -Dtests.seed=79976EC1854DEAAF \
  -Dtests.class=org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests \
  -Dtests.method="testReduceRandom" \
  -Dtests.security.manager=true \
  -Dtests.locale=th \
  -Dtests.timezone=America/Winnipeg \
  -Dcompiler.java=11 \
  -Druntime.java=8

I'm going to backport the mute to the 7.x branch as well, and add the tag to this ticket.

[pcsanwald]

Thanks for the mute, I'll take a look at this one.