[CI] ClusterStateRestCancellationIT testClusterStateRestCancellation failing

[dimitris-athanasiou]

Build scan:
https://gradle-enterprise.elastic.co/s/swrn2vccdizu6/tests/:qa:smoke-test-http:javaRestTest/org.elasticsearch.http.ClusterStateRestCancellationIT/testClusterStateRestCancellation

Reproduction line:
./gradlew ':qa:smoke-test-http:javaRestTest' --tests "org.elasticsearch.http.ClusterStateRestCancellationIT.testClusterStateRestCancellation" -Dtests.seed=2034D67B43AAFEF0 -Dtests.locale=ca -Dtests.timezone=Europe/Madrid -Druntime.java=17

Applicable branches:
main

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.http.ClusterStateRestCancellationIT&tests.test=testClusterStateRestCancellation

Failure excerpt:

java.lang.IllegalStateException: unable to launch a new watch service

  at __randomizedtesting.SeedInfo.seed([2034D67B43AAFEF0:A2A0FB19FD8E07BE]:0)
  at org.elasticsearch.reservedstate.service.FileSettingsService.startWatcher(FileSettingsService.java:271)
  at org.elasticsearch.reservedstate.service.FileSettingsService.startIfMaster(FileSettingsService.java:142)
  at org.elasticsearch.reservedstate.service.FileSettingsService.doStart(FileSettingsService.java:116)
  at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:43)
  at org.elasticsearch.node.Node.start(Node.java:1372)
  at org.elasticsearch.test.InternalTestCluster$NodeAndClient.startNode(InternalTestCluster.java:982)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
  at java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:820)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  at java.lang.Thread.run(Thread.java:833)

  Caused by: java.io.IOException: User limit of inotify watches reached

    at sun.nio.fs.LinuxWatchService$Poller.implRegister(LinuxWatchService.java:264)
    at sun.nio.fs.AbstractPoller.processRequests(AbstractPoller.java:266)
    at sun.nio.fs.LinuxWatchService$Poller.run(LinuxWatchService.java:364)
    at java.lang.Thread.run(Thread.java:833)

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[dimitris-athanasiou]

Not sure if this is a code issue or something with the CI worker setup.

[idegtiarenko]

This looks like another case of #89231

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[slobodanadamovic]

Happened again for NetworkDisruptionIT ->
https://gradle-enterprise.elastic.co/s/s3tqzw2477a46/tests/:test:framework:integTest/org.elasticsearch.test.disruption.NetworkDisruptionIT/testClearDisruptionSchemeWhenNodeIsDown?expanded-stacktrace=WyIwLTEiXQ&top-execution=1

[thecoop]

Just merged a fix for another race condition in this class, #90302. If this happens again we can reopen & take another look.