[CI] ForecastIT.testOverflowToDisk is timing out the test suite

[jbaiera]

Seeing a good number of failures from ForcastIT today on the 7.x branch.

org.elasticsearch.xpack.ml.integration.ForecastIT > testOverflowToDisk FAILED
--
java.lang.Exception: Test abandoned because suite timeout was reached.
REPRODUCE WITH: ./gradlew :x-pack:plugin:ml:qa:native-multi-node-tests:integTestRunner --tests "org.elasticsearch.xpack.ml.integration.ForecastIT.testOverflowToDisk" -Dtests.seed=9E0824A9F8810D30 -Dtests.security.manager=true -Dtests.locale=lt-LT -Dtests.timezone=SystemV/PST8 -Dcompiler.java=12 -Druntime.java=8

According to the thread dump, it looks like the test case is blocked waiting on a future:

2> 	"TEST-ForecastIT.testOverflowToDisk-seed#[9E0824A9F8810D30]" ID=359 WAITING on org.elasticsearch.common.util.concurrent.BaseFuture$Sync@75126b11
2> 	at sun.misc.Unsafe.park(Native Method)
2> 	- waiting on org.elasticsearch.common.util.concurrent.BaseFuture$Sync@75126b11
2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
2> 	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:252)
2> 	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:87)
2> 	at org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:57)
2> 	at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:34)
2> 	at org.elasticsearch.xpack.ml.integration.MlNativeAutodetectIntegTestCase.postData(MlNativeAutodetectIntegTestCase.java:259)
2> 	at org.elasticsearch.xpack.ml.integration.ForecastIT.createDataWithLotsOfClientIps(ForecastIT.java:399)
2> 	at org.elasticsearch.xpack.ml.integration.ForecastIT.testOverflowToDisk(ForecastIT.java:233)
2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2> 	at java.lang.reflect.Method.invoke(Method.java:498)
2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
2> 	at java.lang.Thread.run(Thread.java:748)

Some Gradle Scans:
https://scans.gradle.com/s/hkqitizrjhsrq
https://scans.gradle.com/s/p4b6hmnl7crce
https://scans.gradle.com/s/o4bsd7fq4prgo

[elasticmachine]

Pinging @elastic/ml-core

[mark-vieira]

Also seeing this test fail due to odd SSL connection errors: https://gradle-enterprise.elastic.co/s/k65qn2vrqiqqy/tests/zcquf3hc3eoda-p36ippgnsdeic

There are a few instances of this test failing due to networking issues:
https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now%2Fd,mode:quick,to:now%2Fd))&_a=(columns:!(build-id,message),index:e58bf320-7efd-11e8-bf69-63c8ef516157,interval:auto,query:(language:kuery,query:'test:%20%22testOverflowToDisk%22%20and%20%22NodeDisconnectedException%22'),sort:!(time,desc))

In fact other tests in ForecastIT seems to have failed a number of recent times for similar reasons:
https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now%2Fd,mode:quick,to:now%2Fd))&_a=(columns:!(build-id,message,test,class),index:e58bf320-7efd-11e8-bf69-63c8ef516157,interval:auto,query:(language:kuery,query:%22NodeDisconnectedException%22),sort:!(time,desc))

[davidkyle]

Reopening, looks like it was accidentally closed in the mute PR

[droberts195]

I strongly suspect this is a side effect of #44837.

Sometimes it fails because of a suite timeout and sometimes it fails because of a networking problem. However, I think the suite timeouts are caused because a network message has been lost without causing an exception to be logged. So in reality all the failures are caused by networking problems.

Looking at the history of this failure, the very first time it failed on was on 8th August in a PR build of #44837, which made a change to networking. It did not start failing frequently until that PR was merged. The second failure was in the intake build for 740d58f which is the commit immediately after af908ef which is the backport of #44837.

The ML native-multi-node-tests suite is an example of a test suite where real node to node communications are exercised in non-trivial ways with security enabled.

The SSL errors that have been observed are javax.net.ssl.SSLException: Received fatal alert: bad_record_mac and javax.crypto.AEADBadTagException: Tag mismatch! and these have not been reported before in any other issue in this repo. They sound like the sort of thing that could happen if 1 or 2 bytes were lost in the networking stack.

@tbrooks8 can you think of any way #44837 might sometimes lose some data? Possibly it might be a combination of the changes in #44837 in conjunction with TLS or some other aspect of Elasticsearch security.

Also, although ForecastIT.testOverflowToDisk is the test that's shown up most often in failure reports so far I suspect that now it's muted we'll see other tests in the native-multi-node-tests suite failing in the same way. If other tests in the suite take over as the first cause of failure then we should remove the individual muting on ForecastIT.testOverflowToDisk and mute the entire test suite. But the network communications problem needs investigating for 7.4 because I'm pretty sure customers will hit it if it's released in its current form.