[CI] CcsCommonYamlTestSuiteIT test {p0=search/510_range_query_out_of_bounds/Test range query for float field with out of bounds lower limit} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #10454 / encryption-at-rest
• elasticsearch-pull-request #89851 / part-6
• elasticsearch-pull-request #89555 / part-6

Reproduction Line:

./gradlew ":qa:ccs-common-rest:yamlRestTest" --tests "org.elasticsearch.test.rest.yaml.CcsCommonYamlTestSuiteIT.test {p0=search/510_range_query_out_of_bounds/Test range query for float field with out of bounds lower limit}" -Dtests.seed=9A76BDA5AAFD820E -Dtests.locale=be-TARASK -Dtests.timezone=Asia/Thimphu -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.Exception: Test abandoned because suite timeout was reached.

Issue Reasons:

• [main] 3 failures in test test {p0=search/510_range_query_out_of_bounds/Test range query for float field with out of bounds lower limit} (0.6% fail rate in 489 executions)
• [main] 2 failures in step part-6 (0.8% fail rate in 255 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (0.8% fail rate in 255 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 failures in test test {p0=search/510_range_query_out_of_bounds/Test range query for float field with out of bounds lower limit} (0.6% fail rate in 489 executions)
• [main] 2 failures in step part-6 (0.8% fail rate in 255 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (0.8% fail rate in 255 executions)

Build Scans:

• elasticsearch-periodic #10454 / encryption-at-rest
• elasticsearch-pull-request #89851 / part-6
• elasticsearch-pull-request #89555 / part-6

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[cbuescher]

Another issue related to #133409. These tests fail on encryption-at-rest occasionally with timeouts. That had only been the case once there though, and seems much less frequent since we increased the timeouts of that suite a bit.

Only three failures here really it seems, two of which are from PRs that apparently messed up something in transport (I saw some NPEs at a quick glance) and for that reason several hundreds of the CcsCommonYamlTest tests failed a few times, possibly other stuff too.

In my opinion PRs should not contribute to the signal for muting tests that much, I think this is something that we need to look at. I will unmute the test here since there's nothing particularly wrong with it but would like to keep this ticket open a bit to discuss how we can re-tune our CI issue automation / muting logic here a bit.

[cbuescher]

@elastic/es-delivery could you take a look here on how we can tune the test automation / muting for cases like this?

We have two larger yaml test suites that if something is broken in e.g. PRs seem to add to the signal in raising a CI issue and muting the test. For example, this issue has been opened apparently based on only three failures. The two PRs that contributed to it have apparently broken many yaml tests wholesale and since that can likely happen any time in PRs, I'd like to ask if we can remove that effect somehow.