[CI] DocsClientYamlTestSuiteIT timed out after 30 minutes

[andreidan]

Build failure https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=corretto11,nodes=general-purpose/362/console

Suite timeout exceeded (>= 1800000 msec)

Build scan available at https://gradle-enterprise.elastic.co/s/lp2vd4vqmj25o

[elasticmachine]

Pinging @elastic/es-docs (:Docs)

[mark-vieira]

This is happening increasingly often over the past couple of days due to the scope of the suite simply slowly increasing over time. I think we may need to bump this timeout in the interim until we sort out a better solution like splitting it up.

[mark-vieira]

Now I am indeed starting to believe we've introduced some kind of regression. This is failing consistently across many OSes and JREs. Here's the runtime graph for one such CI build over the last several builds. You can see the sudden spike here.

[mark-vieira]

So something has clearly changed to make this test suite suddenly run so much slower. The reason I don't believe we are seeing this failure in intake and pull request checks is that those are split up into multiple smaller parallel builds and therefore this is much less contention on the worker then for the full periodic builds.

There's no single test that's running long. It simply looks as though all test cases are just a bit slower, adding up to a pretty substantial runtime hit.

Another interesting data point is this is only happening in master and 7.x. We are not seeing the increased runtimes in 7.5. My closest guess at a potential culprit for a large change that might impact performance here is perhaps #50907.

@ywelsch @DaveCTurner Any thoughts? Any chance that change might have an adverse effect on performance on a system with lot of other CPU and IO contention (lots of other concurrent tests).

@danielmitterdorfer Have you seen any recent changes in performance benchmarks that might indicate some kind regression might have been introduced?

[hendrikmuhs]

For me this look like a duplicate of #49579

However, since 20/01/13 the number of failures have increased, which support the investigations above.

[DaveCTurner]

I suspect #50907 and #50928. Although these changes save potentially hundreds of fsync calls from some cluster state updates in large clusters they will add an fsync or two to cluster state updates in tiny clusters, and these tests use a tiny cluster which does a lot of cluster state updates.

Do these tests run using a ramdisk? If not, can we move them to use a ramdisk? That would make fsync calls free again.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Build)

[hendrikmuhs]

While investigations are running which hopefully result in a proper fix, I will increase the timeout as a temporary solution, because too many builds are failing at the moment.