[CI] Failure in MixedClusterClientYamlTestSuiteIT search.aggregation/10_histogram/date_histogram on range tests

[droberts195]

Two tests have been failing in the mixed cluster tests where the mixed cluster contains 7.7 with 7.2 or 7.3:

1. search.aggregation/10_histogram/date_histogram on range with offset
2. search.aggregation/10_histogram/date_histogram on range

Some example failures are:

• 7.7.0 with 7.2.1: https://gradle-enterprise.elastic.co/s/qjbffp22ul7zy
• 7.7.0 with 7.3.0: https://gradle-enterprise.elastic.co/s/rjhtqo76kuscw
• 7.7.0 with 7.3.2: https://gradle-enterprise.elastic.co/s/wiqyvibsqtvva

But there are other combinations too.

Example repro commands that consistently fail:

./gradlew ':qa:mixed-cluster:v7.2.1#mixedClusterTest' --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" \
  -Dtests.method="test {p0=search.aggregation/10_histogram/date_histogram on range}" \
  -Dtests.seed=9966AE98B59516C2 \
  -Dtests.security.manager=true \
  -Dtests.locale=es-EC \
  -Dtests.timezone=Canada/Atlantic \
  -Dcompiler.java=13 \
  -Druntime.java=11

./gradlew ':qa:mixed-cluster:v7.2.1#mixedClusterTest' --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" \
  -Dtests.method="test {p0=search.aggregation/10_histogram/date_histogram on range with offset}" \
  -Dtests.seed=9966AE98B59516C2 \
  -Dtests.security.manager=true \
  -Dtests.locale=es-EC \
  -Dtests.timezone=Canada/Atlantic \
  -Dcompiler.java=13 \
  -Druntime.java=11

Errors are of the form:

hits.total.value didn't match expected value:
              hits.total.value: expected Integer [4] but was Integer [1]

Sometimes the but was is 2 or 3 instead of 1.

[elasticmachine]

Pinging @elastic/es-analytics-geo (:Analytics/Aggregations)

[droberts195]

Also search.aggregation/10_histogram/date_histogram on range sometimes fails in the same way.

[droberts195]

All 3 tests are muted on the 7.x branch (only) in 6283648

[nik9000]

My fault. I'll have a look.