[CI] IndicesRequestCacheIT.testQueryRewriteDates failure

[ywangd]

Build scan:
https://gradle-enterprise.elastic.co/s/g2zdlxmibxf54

Repro line:

REPRODUCE WITH: ./gradlew ':server:integTest' \
  -Dtests.seed=CD7E7F3C3084555E \
  -Dtests.class=org.elasticsearch.indices.IndicesRequestCacheIT \
  -Dtests.method="testQueryRewriteDates" \
  -Dtests.security.manager=true \
  -Dtests.locale=de-CH \
  -Dtests.timezone=ROK \
  -Dcompiler.java=12 \
  -Druntime.java=8

Reproduces locally?:
No

Applicable branches:
It seems to be 6.8 only.

Failure history:

Only failed 3 times for 6.8 in past 60 days based on build-stats. But the error message looks pretty valid.
We had an issue (#32827) that is similar to this. It has been fixed by #54071. But the fix is for 7.7+. I wonder whether it is related and needs to be backported to 6.8.

Failure excerpt:

FAILURE 2.99s J15 | IndicesRequestCacheIT.testQueryRewriteDates <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: expected:<[1, 1]> but was:<[0, 2]>
   > 	at __randomizedtesting.SeedInfo.seed([CD7E7F3C3084555E:3A52EEA99B8D2B5B]:0)
   > 	at org.elasticsearch.indices.IndicesRequestCacheIT.assertCacheState(IndicesRequestCacheIT.java:418)
   > 	at org.elasticsearch.indices.IndicesRequestCacheIT.testQueryRewriteDates(IndicesRequestCacheIT.java:225)
   > 	at java.lang.Thread.run(Thread.java:748)

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[javanna]

I have checked the fix #54071 and it does not apply to 6.8, as it modifies serialization so that newly added fields are not considered part of the cache key. Those newly added fields are not present in 6.8, so it must be a different problem.

[davidkyle]

Another failure: https://gradle-enterprise.elastic.co/s/diklknl2excoi

java.lang.AssertionError: 
Expected: <8L>
     but: was <7L>
	at __randomizedtesting.SeedInfo.seed([224FA70B6604A6A4:964DAEABD1514A4B]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.indices.IndicesRequestCacheIT.testQueryRewriteDatesWithNow(IndicesRequestCacheIT.java:274)

[DaveCTurner]

Another: https://gradle-enterprise.elastic.co/s/lzgal3sjdvmx6

[mayya-sharipova]

2 more failures in the last 60 days on 6.8. The last failure on July 5th:

FAILURE 1.56s J15 \| IndicesRequestCacheIT.testQueryRewriteDatesWithNow <<< FAILURES! |  
-- | --
  | > Throwable #1: java.lang.AssertionError: expected:<[1, 1]> but was:<[0, 2]> |  
  | > 	at __randomizedtesting.SeedInfo.seed([A038FE2ED1BE81BA:143AF78E66EB6D55]:0) |  
  | > 	at org.elasticsearch.indices.IndicesRequestCacheIT.assertCacheState(IndicesRequestCacheIT.java:418) |  
  | > 	at org.elasticsearch.indices.IndicesRequestCacheIT.testQueryRewriteDatesWithNow(IndicesRequestCacheIT.java:286)

[ywelsch]

I think we should close this one as NOFIX. ES 6.8 will be EOL in a couple of weeks.