CI Failure in UpgradeClusterClientYamlTestSuiteIT 80_data_frame_jobs_crud

[jdconrad]

Seeing a number of failures here for this

Reproduce with:

REPRODUCE WITH: ./gradlew ':x-pack:qa:rolling-upgrade:v7.2.1#twoThirdsUpgradedTest' --tests "org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT.test {p0=mixed_cluster/80_data_frame_jobs_crud/Test put continuous data frame transform on mixed cluster}" \
  -Dtests.seed=5B615469BC55690B \
  -Dtests.security.manager=true \
  -Dtests.locale=es-BO \
  -Dtests.timezone=Pacific/Yap \
  -Dtests.distribution=default \
  -Dcompiler.java=12 \
  -Druntime.java=8 \
  -Dtests.rest.suite=mixed_cluster \
  -Dtests.rest.blacklist=mixed_cluster/80_data_frame_jobs_crud/Test GET, stop, start, old continuous transforms

REPRODUCE WITH: ./gradlew ':x-pack:qa:rolling-upgrade:v7.2.1#twoThirdsUpgradedTest' --tests "org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT.test {p0=mixed_cluster/80_data_frame_jobs_crud/Test put continuous data frame transform on mixed cluster}" \
  -Dtests.seed=5B615469BC55690B \
  -Dtests.security.manager=true \
  -Dtests.locale=es-BO \
  -Dtests.timezone=Pacific/Yap \
  -Dtests.distribution=default \
  -Dcompiler.java=12 \
  -Druntime.java=8 \
  -Dtests.rest.suite=mixed_cluster \
  -Dtests.rest.blacklist=mixed_cluster/80_data_frame_jobs_crud/Test GET, stop, start, old continuous transforms

from (https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+bwc/BWC_VERSION=7.3.1,nodes=centos-7&&immutable/189/console)

Stack trace:

11:01:39 » WARN ][o.e.c.NodeConnectionsService] [v7.3.1-2] failed to connect to {v7.3.1-0}{YzPEUjsxRtyNkSi9YO_j6Q}{Q4R7DVBEQR-gC_R9_aA0Hg}{127.0.0.1}{127.0.0.1:35564}{dim}{testattr=test, ml.machine_memory=101204291584, ml.max_open_jobs=20, xpack.installed=true} (tried [1] times)
11:01:39 »  org.elasticsearch.transport.ConnectTransportException: [v7.3.1-0][127.0.0.1:35564] connect_exception
11:01:39 »  	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:957) ~[elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$3(ActionListener.java:161) ~[elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.3.1.jar:7.3.1]
11:01:39 »  	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_221]
11:01:39 »  	at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) ~[?:1.8.0_221]
11:01:39 »  	at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140) ~[?:1.8.0_221]
11:01:39 »  	at org.elasticsearch.common.concurrent.CompletableContext.addListener(CompletableContext.java:45) ~[elasticsearch-core-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.transport.netty4.Netty4TcpChannel.addConnectListener(Netty4TcpChannel.java:121) ~[?:?]
11:01:39 »  	at org.elasticsearch.transport.TcpTransport.initiateConnection(TcpTransport.java:299) ~[elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:266) ~[elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.transport.ConnectionManager.internalOpenConnection(ConnectionManager.java:206) ~[elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.transport.ConnectionManager.connectToNode(ConnectionManager.java:104) ~[elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:346) ~[elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:333) ~[elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.cluster.NodeConnectionsService$ConnectionTarget$1.doRun(NodeConnectionsService.java:304) [elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_221]
11:01:39 »  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_221]
11:01:39 »  	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]
11:01:39 »  Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 127.0.0.1/127.0.0.1:35564
11:01:39 »  	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
11:01:39 »  	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
11:01:39 »  	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:670) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?]
11:01:39 »  	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) ~[?:?]
11:01:39 »  	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
11:01:39 »  	... 1 more
11:01:39 »  Caused by: java.net.ConnectException: Connection refused
11:01:39 »  	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
11:01:39 »  	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
11:01:39 »  	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:670) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?]
11:01:39 »  	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) ~[?:?]
11:01:39 »  	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
11:01:39 »  	... 1 more
11:01:39 » WARN ][o.e.c.NodeConnectionsService] [v7.3.1-2] failed to connect to {v7.3.1-0}{YzPEUjsxRtyNkSi9YO_j6Q}{MfnZeaUrQvyMsQeS3-s9Xg}{127.0.0.1}{127.0.0.1:34126}{dilm}{testattr=test, ml.machine_memory=101204291584, upgraded=true, ml.max_open_jobs=20, xpack.installed=true} (tried [1] times)
11:01:39 »  org.elasticsearch.transport.ConnectTransportException: [v7.3.1-0][127.0.0.1:34126] connect_exception
11:01:39 »  	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:957) ~[elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$3(ActionListener.java:161) ~[elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.3.1.jar:7.3.1]
11:01:39 »  	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_221]
11:01:39 »  	at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) ~[?:1.8.0_221]
11:01:39 »  	at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140) ~[?:1.8.0_221]
11:01:39 »  	at org.elasticsearch.common.concurrent.CompletableContext.addListener(CompletableContext.java:45) ~[elasticsearch-core-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.transport.netty4.Netty4TcpChannel.addConnectListener(Netty4TcpChannel.java:121) ~[?:?]
11:01:39 »  	at org.elasticsearch.transport.TcpTransport.initiateConnection(TcpTransport.java:299) ~[elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:266) ~[elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.transport.ConnectionManager.internalOpenConnection(ConnectionManager.java:206) ~[elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.transport.ConnectionManager.connectToNode(ConnectionManager.java:104) ~[elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:346) ~[elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:333) ~[elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.cluster.NodeConnectionsService$ConnectionTarget$1.doRun(NodeConnectionsService.java:304) [elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]
11:01:39 »  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_221]
11:01:39 »  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_221]
11:01:39 »  	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]
11:01:39 »  Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 127.0.0.1/127.0.0.1:34126
11:01:39 »  	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
11:01:39 »  	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
11:01:39 »  	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:670) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?]
11:01:39 »  	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) ~[?:?]
11:01:39 »  	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
11:01:39 »  	... 1 more
11:01:39 »  Caused by: java.net.ConnectException: Connection refused
11:01:39 »  	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
11:01:39 »  	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
11:01:39 »  	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:670) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) ~[?:?]
11:01:39 »  	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?]
11:01:39 »  	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) ~[?:?]
11:01:39 »  	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
11:01:39 »  	... 1 more

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[jdconrad]

More:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+bwc/BWC_VERSION=7.3.1,nodes=centos-7&&immutable/189/console

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+bwc/BWC_VERSION=7.3.0,nodes=centos-7&&immutable/189/console

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+bwc/BWC_VERSION=7.3.2,nodes=centos-7&&immutable/189/console

[droberts195]

I think the ConnectTransportExceptions are irrelevant and you always get them in the rolling upgrade tests when nodes are shut down.

The problem that causes this failure is:

18:01:38 org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT > test {p0=mixed_cluster/80_data_frame_jobs_crud/Test GET, stop, start, old continuous transforms} FAILED
18:01:38     java.lang.AssertionError: Failure at [mixed_cluster/80_data_frame_jobs_crud:298]: transforms.0.state didn't match expected value:
18:01:38                 transforms.0.state: expected String [stopped] but was LinkedHashMap [{task_state=stopped, indexer_state=stopped, checkpoint=1}]
18:01:38 
18:01:38         Caused by:
18:01:38         java.lang.AssertionError: transforms.0.state didn't match expected value:
18:01:38                     transforms.0.state: expected String [stopped] but was LinkedHashMap [{task_state=stopped, indexer_state=stopped, checkpoint=1}]

Not sure why this has only started failing recently as that incompatibility was introduced a long time ago.

[droberts195]

@hendrikmuhs in the 7.x version of 80_data_frame_jobs_crud.yml all the assertions on transforms.0.state used to be commented out with this comment:

   # Since we are breaking the stats format between 7.3 and 7.4 (allowed because we're beta) we cannot
   # assert on state in the mixed cluster as it could be state at the top level or state.task_state

But in e1842c0#diff-a8566ec4c3b7275e96a4686786b88f0d you added some more assertions on transforms.0.state that are not commented out in 7.x. Please could you comment these like the others in the file? (Only on 7.x - in master we can assert on transforms.0.state.)

[mark-vieira]

Friendly reminder this test is still consistently failing in the 7.x BWC jobs.

[hendrikmuhs]

on it, I somehow missed the gh notification

[hendrikmuhs]

fixed by #47898