[CI] ClusterInfoServiceIT testMaxQueueLatenciesInClusterInfo failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #10025 / windows-2022_checkpart1_platform-support-windows
• elasticsearch-pull-request #86036 / part-1
• elasticsearch-periodic-platform-support #10025 / almalinux-9_platform-support-unix
• elasticsearch-periodic-platform-support #10025 / debian-12_platform-support-unix
• elasticsearch-pull-request #86008 / part-1
• elasticsearch-pull-request #86000 / part-1

Reproduction Line:

gradlew ":server:internalClusterTest" --tests "org.elasticsearch.cluster.ClusterInfoServiceIT.testMaxQueueLatenciesInClusterInfo" -Dtests.seed=F5440B5D5A6B1B4C -Dtests.locale=kxv-Orya -Dtests.timezone=SystemV/MST7 -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Unexpectedly found a task queued for the write thread pool. Write thread pool dump: TaskExecutionTimeTrackingEsThreadPoolExecutor[name = node_t1/write, queue capacity = 10000, task execution EWMA = 17.7ms, total task execution time = 1s, org.elasticsearch.common.util.concurrent.TaskExecutionTimeTrackingEsThreadPoolExecutor@12a864d[Running, pool size = 1, active threads = 1, queued tasks = 3, completed tasks = 10]]
Expected: <0L>
     but: was <1L>

Issue Reasons:

• [main] 6 failures in test testMaxQueueLatenciesInClusterInfo (13.3% fail rate in 45 executions)
• [main] 3 failures in step part-1 (11.5% fail rate in 26 executions)
• [main] 3 failures in pipeline elasticsearch-pull-request (12.0% fail rate in 25 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 failures in test testMaxQueueLatenciesInClusterInfo (9.1% fail rate in 33 executions)
• [main] 2 failures in step part-1 (8.7% fail rate in 23 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (9.1% fail rate in 22 executions)

Build Scans:

• elasticsearch-periodic-platform-support #10025 / debian-12_platform-support-unix
• elasticsearch-pull-request #86008 / part-1
• elasticsearch-pull-request #86000 / part-1

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[DiannaHohensee]

There appears to be another stray task running on the write thread pool. @henningandersen I'm not sure that this is worth tracking down. This goes back to the PR thread about ensuring silence on the write thread pool or using an assertBusy that the queue empties.

The way I debugged it last time was running the debugger locally and peeking at the queue task. Maybe I could debug more easily by adding a log message to peekMaxQueueLatencyInQueueMillis() to dump the task found at the top of the queue, and then rerun the test until it presumably reproduces.

I can dig into this some more, or I can add back the assertBusy. I would naively consider some index background noise reasonable, but I'm not familiar with our particular indexing stack and whether it's reasonable for it. Preference?

[henningandersen]

I would prefer to track it down. I think adding an assertion like:

assertThat(trackingWriteExecutor.getQueue().peek(), is(nullValue()));

would help you track it down, showing you the problematic task.