Netty4HttpRequestSizeLimitIT testDoesNotLimitExcludedRequests fails reproducibly

[jaymode]

I hit the following failure in Netty4HttpRequestSizeLimitIT#testDoesNotLimitExcludedRequests locally and it consistently reproduces on master as of writing (commit 084e06e).

Failure:

FAILURE 40.5s | Netty4HttpRequestSizeLimitIT.testDoesNotLimitExcludedRequests <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: Failed to get all expected responses.
   >    at __randomizedtesting.SeedInfo.seed([898AB80ABB9F5DAD:73FB0193B66107ED]:0)
   >    at org.elasticsearch.http.netty4.Netty4HttpClient.sendRequests(Netty4HttpClient.java:150)
   >    at org.elasticsearch.http.netty4.Netty4HttpClient.processRequestsWithBody(Netty4HttpClient.java:130)
   >    at org.elasticsearch.http.netty4.Netty4HttpClient.put(Netty4HttpClient.java:116)
   >    at org.elasticsearch.http.netty4.Netty4HttpRequestSizeLimitIT.testDoesNotLimitExcludedRequests(Netty4HttpRequestSizeLimitIT.java:124)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   >    at java.base/java.lang.Thread.run(Thread.java:834)

Reproduce line:

./gradlew :modules:transport-netty4:integTestRunner -Dtests.seed=898AB80ABB9F5DAD -Dtests.class=org.elasticsearch.http.netty4.Netty4HttpRequestSizeLimitIT -Dtests.method="testDoesNotLimitExcludedRequests" -Dtests.security.manager=true -Dtests.locale=en-PK -Dtests.timezone=SystemV/YST9YDT -Dcompiler.java=11 -Druntime.java=11

[elasticmachine]

Pinging @elastic/es-distributed

[original-brownbear]

The problem here is that we're running 1.5k updates to the cluster state which takes on average ~20s on my machine and has to happen within a 30s timeout to not run into org.elasticsearch.cluster.service.MasterService.Batcher#onTimeout and hence fail to answer the test http client in time.
The obvious fix would be to not run 1.5k requests but less, but I wonder why this is so slow and if it's a new issue. Looking into that now.

[original-brownbear]

Also found a way to speed up the underlying setting update logic in #36560 I think.

[original-brownbear]

Fixed by preventing unnecessary state updates via #36560 now.