SnapshotIT testCreateSnapshot fails possibly due to simultaneous deletions #37581
Description
I hit this failure locally multiple times but it does not reproduce.
Reproduce line:
./gradlew :client:rest-high-level:integTestRunner -Dtests.seed=DEF00E9C2189727D -Dtests.class=org.elasticsearch.client.SnapshotIT -Dtests.method="testCreateSnapshot" -Dtests.security.manager=true -Dtests.locale=sr-Cyrl-XK -Dtests.timezone=Asia/Yekaterinburg -Dcompiler.java=11 -Druntime.java=11
ERROR 0.12s | SnapshotIT.testCreateSnapshot <<< FAILURES!
> Throwable #1: ElasticsearchStatusException[Elasticsearch exception [type=repository_exception, reason=[test_repository] concurrent modification of the index-N file, expected current generation [13], actual current generation [14] - possibly due to simultaneous snapshot deletion requests]]
> at __randomizedtesting.SeedInfo.seed([DEF00E9C2189727D:20969B2F173A8064]:0)
> at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
> at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1703)
> at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1679)
> at org.elasticsearch.client.RestHighLevelClient$1.onFailure(RestHighLevelClient.java:1598)
> at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:614)
> at org.elasticsearch.client.RestClient$1.completed(RestClient.java:317)
> at org.elasticsearch.client.RestClient$1.completed(RestClient.java:294)
> at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
> at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
> at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
> at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)
> at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
> at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
> at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
> at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Suppressed: org.elasticsearch.client.ResponseException: method [DELETE], host [http://127.0.0.1:35141], URI [/_snapshot/test_repository/test_snapshot?master_timeout=30s], status line [HTTP/1.1 500 Internal Server Error]
> {"error":{"root_cause":[{"type":"repository_exception","reason":"[test_repository] concurrent modification of the index-N file, expected current generation [13], actual current generation [14] - possibly due to simultaneous snapshot deletion requests"}],"type":"repository_exception","reason":"[test_repository] concurrent modification of the index-N file, expected current generation [13], actual current generation [14] - possibly due to simultaneous snapshot deletion requests"},"status":500}
> at org.elasticsearch.client.RestClient$1.completed(RestClient.java:309)
> ... 16 more
Test runner output with test ordering:
Suite: org.elasticsearch.client.SnapshotIT
1> [2019-01-17T20:42:16,972][INFO ][o.e.c.SnapshotIT ] [testSnapshotGetRepositoriesUsingParams] before test
1> [2019-01-17T20:42:16,974][INFO ][o.e.c.SnapshotIT ] [testSnapshotGetRepositoriesUsingParams] initializing REST clients against [http://127.0.0.1:35141]
1> [2019-01-17T20:42:17,074][INFO ][o.e.c.SnapshotIT ] [testSnapshotGetRepositoriesUsingParams] after test
1> [2019-01-17T20:42:17,087][INFO ][o.e.c.SnapshotIT ] [testSnapshotDeleteRepository] before test
1> [2019-01-17T20:42:17,148][INFO ][o.e.c.SnapshotIT ] [testSnapshotDeleteRepository] after test
1> [2019-01-17T20:42:17,160][INFO ][o.e.c.SnapshotIT ] [testSnapshotsStatus] before test
1> [2019-01-17T20:42:17,316][INFO ][o.e.c.SnapshotIT ] [testSnapshotsStatus] after test
1> [2019-01-17T20:42:17,328][INFO ][o.e.c.SnapshotIT ] [testRestoreSnapshot] before test
1> [2019-01-17T20:42:17,553][INFO ][o.e.c.SnapshotIT ] [testRestoreSnapshot] after test
1> [2019-01-17T20:42:17,568][INFO ][o.e.c.SnapshotIT ] [testSnapshotGetRepositoriesNonExistent] before test
1> [2019-01-17T20:42:17,600][INFO ][o.e.c.SnapshotIT ] [testSnapshotGetRepositoriesNonExistent] after test
1> [2019-01-17T20:42:17,615][INFO ][o.e.c.SnapshotIT ] [testCreateSnapshot] before test
1> [2019-01-17T20:42:17,710][INFO ][o.e.c.SnapshotIT ] [testCreateSnapshot] after test
2> REPRODUCE WITH: ./gradlew :client:rest-high-level:integTestRunner -Dtests.seed=DEF00E9C2189727D -Dtests.class=org.elasticsearch.client.SnapshotIT -Dtests.method="testCreateSnapshot" -Dtests.security.manager=true -Dtests.locale=sr-Cyrl-XK -Dtests.timezone=Asia/Yekaterinburg -Dcompiler.java=11 -Druntime.java=11
ERROR 0.12s | SnapshotIT.testCreateSnapshot <<< FAILURES!
> Throwable #1: ElasticsearchStatusException[Elasticsearch exception [type=repository_exception, reason=[test_repository] concurrent modification of the index-N file, expected current generation [13], actual current generation [14] - possibly due to simultaneous snapshot deletion requests]]
> at __randomizedtesting.SeedInfo.seed([DEF00E9C2189727D:20969B2F173A8064]:0)
> at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
> at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1703)
> at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1679)
> at org.elasticsearch.client.RestHighLevelClient$1.onFailure(RestHighLevelClient.java:1598)
> at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:614)
> at org.elasticsearch.client.RestClient$1.completed(RestClient.java:317)
> at org.elasticsearch.client.RestClient$1.completed(RestClient.java:294)
> at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
> at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
> at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
> at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)
> at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
> at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
> at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
> at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Suppressed: org.elasticsearch.client.ResponseException: method [DELETE], host [http://127.0.0.1:35141], URI [/_snapshot/test_repository/test_snapshot?master_timeout=30s], status line [HTTP/1.1 500 Internal Server Error]
> {"error":{"root_cause":[{"type":"repository_exception","reason":"[test_repository] concurrent modification of the index-N file, expected current generation [13], actual current generation [14] - possibly due to simultaneous snapshot deletion requests"}],"type":"repository_exception","reason":"[test_repository] concurrent modification of the index-N file, expected current generation [13], actual current generation [14] - possibly due to simultaneous snapshot deletion requests"},"status":500}
> at org.elasticsearch.client.RestClient$1.completed(RestClient.java:309)
> ... 16 more
1> [2019-01-17T20:42:17,729][INFO ][o.e.c.SnapshotIT ] [testSnapshotGetDefaultRepositories] before test
1> [2019-01-17T20:42:17,813][INFO ][o.e.c.SnapshotIT ] [testSnapshotGetDefaultRepositories] after test
1> [2019-01-17T20:42:17,827][INFO ][o.e.c.SnapshotIT ] [testGetSnapshots] before test
1> [2019-01-17T20:42:17,938][INFO ][o.e.c.SnapshotIT ] [testGetSnapshots] after test
1> [2019-01-17T20:42:17,951][INFO ][o.e.c.SnapshotIT ] [testDeleteSnapshot] before test
1> [2019-01-17T20:42:18,029][INFO ][o.e.c.SnapshotIT ] [testDeleteSnapshot] after test
1> [2019-01-17T20:42:18,041][INFO ][o.e.c.SnapshotIT ] [testVerifyRepository] before test
1> [2019-01-17T20:42:18,108][INFO ][o.e.c.SnapshotIT ] [testVerifyRepository] after test
1> [2019-01-17T20:42:18,121][INFO ][o.e.c.SnapshotIT ] [testCreateRepository] before test
1> [2019-01-17T20:42:18,175][INFO ][o.e.c.SnapshotIT ] [testCreateRepository] after test
2> NOTE: leaving temporary files on disk at: /var/jenkins_home/jobs/elasticsearch-pipeline/workspace/elasticsearch/client/rest-high-level/build/testrun/integTestRunner/J0/temp/org.elasticsearch.client.SnapshotIT_DEF00E9C2189727D-001
2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=1237, maxMBSortInHeap=7.33712303577021, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@123f275b), locale=sr-Cyrl-XK, timezone=Asia/Yekaterinburg
2> NOTE: Linux 4.18.0-13-generic amd64/Oracle Corporation 11 (64-bit)/cpus=32,threads=1,free=430176280,total=536870912
2> NOTE: All tests run in this JVM: [CCRDocumentationIT, WatcherDocumentationIT, BulkRequestWithGlobalParametersIT, StoredScriptsDocumentationIT, SecurityIT, RankEvalIT, MigrationIT, BulkProcessorRetryIT, MigrationDocumentationIT, ClusterClientIT, ClusterClientDocumentationIT, BulkProcessorIT, ReindexIT, IngestClientIT, MachineLearningIT, IndicesClientIT, GraphIT, CrudIT, MigrationClientDocumentationIT, LicensingDocumentationIT, SnapshotClientDocumentationIT, CRUDDocumentationIT, WatcherIT, IndexLifecycleIT, SearchDocumentationIT, PingAndInfoIT, CCRIT, RollupDocumentationIT, SearchIT, MachineLearningGetResultsIT, RollupIT, MlClientDocumentationIT, ILMDocumentationIT, LicenseIT, SnapshotIT]
Completed [35/43] in 1.22s, 11 tests, 1 error <<< FAILURES!
Log from integ test cluster:
Cluster client_rest-high-level_integTestCluster - node 0 log excerpt:
(full log at /var/jenkins_home/jobs/elasticsearch-pipeline/workspace/elasticsearch/client/rest-high-level/build/cluster/integTestCluster node0/elasticsearch-7.0.0-SNAPSHOT/logs/client_rest-high-level_integTestCluster.log)
-----------------------------------------
[2019-01-17T15:38:44,034][INFO ][o.e.e.NodeEnvironment ] [node-0] using [1] data paths, mounts [[/var/jenkins_home (/dev/nvme0n1p1)]], net usable_space [397.3gb], net total_space [468.4gb], types [ext4]
[2019-01-17T15:38:44,037][INFO ][o.e.e.NodeEnvironment ] [node-0] heap size [494.9mb], compressed ordinary object pointers [true]
[2019-01-17T15:38:44,039][INFO ][o.e.n.Node ] [node-0] node name [node-0], node ID [71qEwof_Rp2r8YOBlWNorw]
[2019-01-17T15:38:44,039][INFO ][o.e.n.Node ] [node-0] version[7.0.0-SNAPSHOT], pid[49947], build[default/zip/0b849e3/2019-01-17T14:55:42.754849Z], OS[Linux/4.18.0-13-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11/11+28]
[2019-01-17T15:38:44,040][INFO ][o.e.n.Node ] [node-0] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-1700522338155642831, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Dio.netty.allocator.type=unpooled, -Des.scripting.update.ctx_in_params=false, -Xms512m, -Xmx512m, -ea, -esa, -Des.path.home=/var/jenkins_home/jobs/elasticsearch-pipeline/workspace/elasticsearch/client/rest-high-level/build/cluster/integTestCluster node0/elasticsearch-7.0.0-SNAPSHOT, -Des.path.conf=/var/jenkins_home/jobs/elasticsearch-pipeline/workspace/elasticsearch/client/rest-high-level/build/cluster/integTestCluster node0/elasticsearch-7.0.0-SNAPSHOT/config, -Des.distribution.flavor=default, -Des.distribution.type=zip]
[2019-01-17T15:38:44,040][WARN ][o.e.n.Node ] [node-0] version [7.0.0-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
[2019-01-17T15:38:45,628][INFO ][o.e.p.PluginsService ] [node-0] loaded module [aggs-matrix-stats]
[2019-01-17T15:38:45,629][INFO ][o.e.p.PluginsService ] [node-0] loaded module [analysis-common]
[2019-01-17T15:38:45,629][INFO ][o.e.p.PluginsService ] [node-0] loaded module [ingest-common]
[2019-01-17T15:38:45,629][INFO ][o.e.p.PluginsService ] [node-0] loaded module [ingest-geoip]
[2019-01-17T15:38:45,629][INFO ][o.e.p.PluginsService ] [node-0] loaded module [ingest-user-agent]
[2019-01-17T15:38:45,629][INFO ][o.e.p.PluginsService ] [node-0] loaded module [lang-expression]
[2019-01-17T15:38:45,629][INFO ][o.e.p.PluginsService ] [node-0] loaded module [lang-mustache]
[2019-01-17T15:38:45,629][INFO ][o.e.p.PluginsService ] [node-0] loaded module [lang-painless]
[2019-01-17T15:38:45,629][INFO ][o.e.p.PluginsService ] [node-0] loaded module [mapper-extras]
[2019-01-17T15:38:45,629][INFO ][o.e.p.PluginsService ] [node-0] loaded module [parent-join]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService ] [node-0] loaded module [percolator]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService ] [node-0] loaded module [rank-eval]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService ] [node-0] loaded module [reindex]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService ] [node-0] loaded module [repository-url]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService ] [node-0] loaded module [transport-netty4]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService ] [node-0] loaded module [x-pack-ccr]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService ] [node-0] loaded module [x-pack-core]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService ] [node-0] loaded module [x-pack-deprecation]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService ] [node-0] loaded module [x-pack-graph]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService ] [node-0] loaded module [x-pack-ilm]
[2019-01-17T15:38:45,631][INFO ][o.e.p.PluginsService ] [node-0] loaded module [x-pack-logstash]
[2019-01-17T15:38:45,631][INFO ][o.e.p.PluginsService ] [node-0] loaded module [x-pack-ml]
[2019-01-17T15:38:45,631][INFO ][o.e.p.PluginsService ] [node-0] loaded module [x-pack-monitoring]
[2019-01-17T15:38:45,631][INFO ][o.e.p.PluginsService ] [node-0] loaded module [x-pack-rollup]
[2019-01-17T15:38:45,631][INFO ][o.e.p.PluginsService ] [node-0] loaded module [x-pack-security]
[2019-01-17T15:38:45,631][INFO ][o.e.p.PluginsService ] [node-0] loaded module [x-pack-sql]
[2019-01-17T15:38:45,631][INFO ][o.e.p.PluginsService ] [node-0] loaded module [x-pack-upgrade]
[2019-01-17T15:38:45,631][INFO ][o.e.p.PluginsService ] [node-0] loaded module [x-pack-watcher]
[2019-01-17T15:38:45,633][INFO ][o.e.p.PluginsService ] [node-0] no plugins loaded
[2019-01-17T15:38:55,216][INFO ][o.e.x.s.a.s.FileRolesStore] [node-0] parsed [0] roles from file [/var/jenkins_home/jobs/elasticsearch-pipeline/workspace/elasticsearch/client/rest-high-level/build/cluster/integTestCluster node0/elasticsearch-7.0.0-SNAPSHOT/config/roles.yml]
[2019-01-17T15:38:58,538][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-0] [controller/50200] [Main.cc@109] controller (64 bit): Version 7.0.0-SNAPSHOT (Build cac31ab73e71c1) Copyright (c) 2019 Elasticsearch BV
[2019-01-17T15:38:59,816][DEBUG][o.e.a.ActionModule ] [node-0] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-01-17T15:39:00,087][INFO ][o.e.d.DiscoveryModule ] [node-0] using discovery type [zen2] and host providers [settings, file]
[2019-01-17T15:39:00,666][INFO ][o.e.n.Node ] [node-0] initialized
[2019-01-17T15:39:00,666][INFO ][o.e.n.Node ] [node-0] starting ...
[2019-01-17T15:39:00,811][INFO ][o.e.t.TransportService ] [node-0] publish_address {127.0.0.1:38429}, bound_addresses {127.0.0.1:38429}
[2019-01-17T15:39:00,822][WARN ][o.e.b.BootstrapChecks ] [node-0] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2019-01-17T15:39:00,822][WARN ][o.e.b.BootstrapChecks ] [node-0] HTTPS is required in order to use the token service; please enable HTTPS using the [xpack.security.http.ssl.enabled] setting or disable the token service using the [xpack.security.authc.token.enabled] setting
[2019-01-17T15:39:00,832][WARN ][o.e.d.z.FileBasedUnicastHostsProvider] [node-0] expected, but did not find, a dynamic hosts list at [/var/jenkins_home/jobs/elasticsearch-pipeline/workspace/elasticsearch/client/rest-high-level/build/cluster/integTestCluster node0/elasticsearch-7.0.0-SNAPSHOT/config/unicast_hosts.txt]
[2019-01-17T15:39:00,858][INFO ][o.e.c.c.Coordinator ] [node-0] setting initial configuration to VotingConfiguration{71qEwof_Rp2r8YOBlWNorw}
[2019-01-17T15:39:00,970][INFO ][o.e.c.s.MasterService ] [node-0] elected-as-master ([1] nodes joined)[{node-0}{71qEwof_Rp2r8YOBlWNorw}{qPGV8i6PQsuMbWnX3a2KyQ}{127.0.0.1}{127.0.0.1:38429}{ml.machine_memory=67473756160, xpack.installed=true, testattr=test, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, reason: master node changed {previous [], current [{node-0}{71qEwof_Rp2r8YOBlWNorw}{qPGV8i6PQsuMbWnX3a2KyQ}{127.0.0.1}{127.0.0.1:38429}{ml.machine_memory=67473756160, xpack.installed=true, testattr=test, ml.max_open_jobs=20}]}
[2019-01-17T15:39:00,994][INFO ][o.e.c.s.ClusterApplierService] [node-0] master node changed {previous [], current [{node-0}{71qEwof_Rp2r8YOBlWNorw}{qPGV8i6PQsuMbWnX3a2KyQ}{127.0.0.1}{127.0.0.1:38429}{ml.machine_memory=67473756160, xpack.installed=true, testattr=test, ml.max_open_jobs=20}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2019-01-17T15:39:01,067][INFO ][o.e.h.AbstractHttpServerTransport] [node-0] publish_address {127.0.0.1:35141}, bound_addresses {127.0.0.1:35141}
[2019-01-17T15:39:01,069][INFO ][o.e.n.Node ] [node-0] started
[2019-01-17T15:39:01,119][INFO ][o.e.g.GatewayService ] [node-0] recovered [0] indices into cluster_state
... SKIPPED 34 LINES ...
[2019-01-17T15:40:12,719][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=yfUFRnNwTFOOltzVl94t0A, pollTimeout=1m, maxBatchSize=32mb}]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2019-01-17T15:40:12,719][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=yfUFRnNwTFOOltzVl94t0A, pollTimeout=1m, maxBatchSize=32mb}]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2019-01-17T15:40:12,719][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=yfUFRnNwTFOOltzVl94t0A, pollTimeout=1m, maxBatchSize=32mb}]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
... SKIPPED 32 LINES ...
[2019-01-17T15:40:13,328][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=0PJlSPs7RgSddZsb7_xGDQ, pollTimeout=1m, maxBatchSize=32mb}]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2019-01-17T15:40:13,328][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=0PJlSPs7RgSddZsb7_xGDQ, pollTimeout=1m, maxBatchSize=32mb}]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
... SKIPPED 38 LINES ...
[2019-01-17T15:40:14,054][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=3w2wEnMFQse4DmhDi4OnCg, pollTimeout=1m, maxBatchSize=32mb}]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2019-01-17T15:40:14,054][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=3w2wEnMFQse4DmhDi4OnCg, pollTimeout=1m, maxBatchSize=32mb}]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
... SKIPPED 71 LINES ...
[2019-01-17T15:40:14,915][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=gYfHgM-FQ2eWidG7sUFXYg, pollTimeout=1m, maxBatchSize=32mb}]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2019-01-17T15:40:14,915][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=gYfHgM-FQ2eWidG7sUFXYg, pollTimeout=1m, maxBatchSize=32mb}]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
... SKIPPED 75 LINES ...
[2019-01-17T15:40:15,976][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_id_1c2f9cc6-90a1-4168-819d-d2cf4e7c51f1-2019-01-17T15:40:15.808Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]]
... SKIPPED 14 LINES ...
[2019-01-17T15:40:16,464][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_id_3ade6bac-354c-4bbb-ac81-437da2668d91-2019-01-17T15:40:16.352Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]]
[2019-01-17T15:40:16,512][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_id_9e658efa-3913-4bf9-9689-c61d46a9ff30-2019-01-17T15:40:16.510Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]]
... SKIPPED 198 LINES ...
[2019-01-17T15:40:23,212][WARN ][r.suppressed ] [node-0] path: /_migration/upgrade/test, params: {index=test, wait_for_completion=true}
java.lang.IllegalStateException: Index [test] cannot be upgraded
at org.elasticsearch.xpack.upgrade.IndexUpgradeService.upgrade(IndexUpgradeService.java:115) ~[?:?]
at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:59) ~[?:?]
at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:25) ~[?:?]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:208) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 8 LINES ...
[2019-01-17T15:40:23,395][WARN ][o.e.t.LoggingTaskListener] [node-0] 1869 failed with exception
java.lang.IllegalStateException: Index [test] cannot be upgraded
at org.elasticsearch.xpack.upgrade.IndexUpgradeService.upgrade(IndexUpgradeService.java:115) ~[?:?]
at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:59) ~[?:?]
at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:25) ~[?:?]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:208) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 560 LINES ...
[2019-01-17T15:40:42,592][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [node-0] [test-start-datafeed] [autodetect/56095] [CBucketGatherer.cc@466] No counts available at 1547739580, current bucket = [1547739585,1547739590)
... SKIPPED 6 LINES ...
[2019-01-17T15:40:42,717][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [node-0] [test-start-datafeed] [autodetect/56095] [CBucketGatherer.cc@466] No counts available at 1547739580, current bucket = [1547739585,1547739590)
... SKIPPED 1422 LINES ...
[2019-01-17T15:41:14,572][WARN ][r.suppressed ] [node-0] path: /_migration/upgrade/test, params: {index=test, wait_for_completion=true}
java.lang.IllegalStateException: Index [test] cannot be upgraded
at org.elasticsearch.xpack.upgrade.IndexUpgradeService.upgrade(IndexUpgradeService.java:115) ~[?:?]
at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:59) ~[?:?]
at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:25) ~[?:?]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:208) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 17 LINES ...
[2019-01-17T15:41:14,783][WARN ][o.e.t.LoggingTaskListener] [node-0] 17987 failed with exception
java.lang.IllegalStateException: Index [test] cannot be upgraded
at org.elasticsearch.xpack.upgrade.IndexUpgradeService.upgrade(IndexUpgradeService.java:115) ~[?:?]
at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:59) ~[?:?]
at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:25) ~[?:?]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:208) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 9 LINES ...
[2019-01-17T15:41:14,989][WARN ][r.suppressed ] [node-0] path: /_migration/upgrade/test, params: {index=test, wait_for_completion=true}
java.lang.IllegalStateException: Index [test] cannot be upgraded
at org.elasticsearch.xpack.upgrade.IndexUpgradeService.upgrade(IndexUpgradeService.java:115) ~[?:?]
at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:59) ~[?:?]
at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:25) ~[?:?]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:208) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 76 LINES ...
[2019-01-17T15:41:16,323][WARN ][o.e.r.b.FileRestoreContext] [node-0] [[restored_index][0]] [test_snapshot/Ti2vomt7QSqQK1p3ZCiIrw] Can't read metadata from store, will not reuse local files during restore
org.apache.lucene.index.IndexNotFoundException: no segments* file found in store(ByteSizeCachingDirectory(HybridDirectory@/var/jenkins_home/jobs/elasticsearch-pipeline/workspace/elasticsearch/client/rest-high-level/build/cluster/integTestCluster node0/elasticsearch-7.0.0-SNAPSHOT/data/nodes/0/indices/e9g2MZX_R52ynn7CqrkW0w/0/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@3bf9292)): files: [write.lock]
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:675) ~[lucene-core-8.0.0-snapshot-a1c6e642aa.jar:8.0.0-snapshot-a1c6e642aa a1c6e642aad90d3615b4c71bf261a5aad7e32369 - nknize - 2019-01-02 14:49:37]
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0-snapshot-a1c6e642aa.jar:8.0.0-snapshot-a1c6e642aa a1c6e642aad90d3615b4c71bf261a5aad7e32369 - nknize - 2019-01-02 14:49:37]
at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0-snapshot-a1c6e642aa.jar:8.0.0-snapshot-a1c6e642aa a1c6e642aad90d3615b4c71bf261a5aad7e32369 - nknize - 2019-01-02 14:49:37]
at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:200) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.store.Store.access$200(Store.java:128) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:839) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.store.Store$MetadataSnapshot.<init>(Store.java:772) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.store.Store.getMetadata(Store.java:286) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1173) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.repositories.blobstore.FileRestoreContext.restore(FileRestoreContext.java:121) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:861) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:464) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:279) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:302) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:277) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1683) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$7(IndexShard.java:2207) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 211 LINES ...
[2019-01-17T15:41:19,723][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [YWplWKWVsJ_15aad032-751e-4927-8aa0-9a37d0fbbcae-2019-01-17T15:41:19.722Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]]
... SKIPPED 61 LINES ...
[2019-01-17T15:41:20,791][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [opJZLzKeuO_d35244fa-f80b-4946-ba37-016360092086-2019-01-17T15:41:20.712Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]]
... SKIPPED 252 LINES ...
[2019-01-17T15:41:27,524][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[logs-20200101][0], expectedHistoryUUID=MckyWE5cR6iDYFUhpeSaaA, pollTimeout=1m, maxBatchSize=32mb}]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
... SKIPPED 37 LINES ...
[2019-01-17T15:41:29,982][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=2, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=BIrVQ7NATVunPovqvTyZHQ, pollTimeout=1m, maxBatchSize=32mb}]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
... SKIPPED 1937 LINES ...
[2019-01-17T15:42:04,191][ERROR][o.e.x.m.j.p.a.w.JsonDataToProcessWriter] [node-0] Cannot parse timestamp '' as epoch value
[2019-01-17T15:42:04,191][ERROR][o.e.x.m.j.p.a.w.JsonDataToProcessWriter] [node-0] Cannot parse timestamp '' as epoch value
[2019-01-17T15:42:04,196][ERROR][o.e.x.m.j.p.a.w.JsonDataToProcessWriter] [node-0] Cannot parse timestamp '' as epoch value
... SKIPPED 97 LINES ...
[2019-01-17T15:42:06,193][WARN ][o.e.x.m.d.DatafeedManager] [node-0] [start-datafeed-job] Datafeed lookback retrieved no data
... SKIPPED 437 LINES ...
[2019-01-17T15:42:15,512][ERROR][o.e.x.i.IndexLifecycleRunner] [node-0] policy [my_policy] for index [my_index] failed on step [{"phase":"warm","action":"shrink","name":"shrink"}]. Moving to ERROR step
java.lang.IllegalArgumentException: can't shrink an index with only one shard
at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validateShrinkIndex(MetaDataCreateIndexService.java:664) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.prepareResizeIndexSettings(MetaDataCreateIndexService.java:744) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$IndexCreationTask.execute(MetaDataCreateIndexService.java:408) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:685) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:310) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:210) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2019-01-17T15:42:15,555][ERROR][o.e.x.i.IndexLifecycleRunner] [node-0] policy [my_policy] for index [my_index] failed on step [{"phase":"warm","action":"shrink","name":"shrink"}]. Moving to ERROR step
java.lang.IllegalArgumentException: can't shrink an index with only one shard
at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validateShrinkIndex(MetaDataCreateIndexService.java:664) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.prepareResizeIndexSettings(MetaDataCreateIndexService.java:744) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$IndexCreationTask.execute(MetaDataCreateIndexService.java:408) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:685) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:310) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:210) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 32 LINES ...
[2019-01-17T15:42:16,313][WARN ][o.e.l.LicenseService ] [node-0] LICENSE [EXPIRED] ON [THURSDAY, JANUARY 01, 1970].
# IF YOU HAVE A NEW LICENSE, PLEASE UPDATE IT. OTHERWISE, PLEASE REACH OUT TO
# YOUR SUPPORT CONTACT.
#
# COMMERCIAL PLUGINS OPERATING WITH REDUCED FUNCTIONALITY
# - security
# - Cluster health, cluster stats and indices stats operations are blocked
# - All data operations (read and write) continue to work
# - watcher
# - PUT / GET watch APIs are disabled, DELETE watch API continues to work
# - Watches execute and write to the history
# - The actions of the watches don't execute
# - monitoring
# - The agent will stop collecting cluster and indices metrics
# - The agent will stop automatically cleaning indices older than [xpack.monitoring.history.duration]
# - graph
# - Graph explore APIs are disabled
# - ml
# - Machine learning APIs are disabled
# - logstash
# - Logstash will continue to poll centrally-managed pipelines
# - beats
# - Beats will continue to poll centrally-managed configuration
# - deprecation
# - Deprecation APIs are disabled
# - upgrade
# - Upgrade API is disabled
# - sql
# - SQL support is disabled
# - rollup
# - Creating and Starting rollup jobs will no longer be allowed.
# - Stopping/Deleting existing jobs, RollupCaps API and RollupSearch continue to function.
[2019-01-17T15:42:16,313][WARN ][o.e.l.LicenseService ] [node-0] LICENSE [EXPIRED] ON [THURSDAY, JANUARY 01, 1970].
# IF YOU HAVE A NEW LICENSE, PLEASE UPDATE IT. OTHERWISE, PLEASE REACH OUT TO
# YOUR SUPPORT CONTACT.
#
# COMMERCIAL PLUGINS OPERATING WITH REDUCED FUNCTIONALITY
# - security
# - Cluster health, cluster stats and indices stats operations are blocked
# - All data operations (read and write) continue to work
# - watcher
# - PUT / GET watch APIs are disabled, DELETE watch API continues to work
# - Watches execute and write to the history
# - The actions of the watches don't execute
# - monitoring
# - The agent will stop collecting cluster and indices metrics
# - The agent will stop automatically cleaning indices older than [xpack.monitoring.history.duration]
# - graph
# - Graph explore APIs are disabled
# - ml
# - Machine learning APIs are disabled
# - logstash
# - Logstash will continue to poll centrally-managed pipelines
# - beats
# - Beats will continue to poll centrally-managed configuration
# - deprecation
# - Deprecation APIs are disabled
# - upgrade
# - Upgrade API is disabled
# - sql
# - SQL support is disabled
# - rollup
# - Creating and Starting rollup jobs will no longer be allowed.
# - Stopping/Deleting existing jobs, RollupCaps API and RollupSearch continue to function.
[2019-01-17T15:42:16,323][WARN ][o.e.x.c.a.AutoFollowCoordinator] [node-0] skipping auto-follower coordination
org.elasticsearch.ElasticsearchSecurityException: current license is non-compliant for [ccr]
at org.elasticsearch.license.LicenseUtils.newComplianceException(LicenseUtils.java:27) ~[x-pack-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator.updateAutoFollowers(AutoFollowCoordinator.java:178) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator.clusterChanged(AutoFollowCoordinator.java:291) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateListeners$7(ClusterApplierService.java:498) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) [?:?]
at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) [?:?]
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) [?:?]
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:495) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:478) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 6 LINES ...
[2019-01-17T15:42:16,454][WARN ][o.e.x.c.a.AutoFollowCoordinator] [node-0] skipping auto-follower coordination
org.elasticsearch.ElasticsearchSecurityException: current license is non-compliant for [ccr]
at org.elasticsearch.license.LicenseUtils.newComplianceException(LicenseUtils.java:27) ~[x-pack-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator.updateAutoFollowers(AutoFollowCoordinator.java:178) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator.clusterChanged(AutoFollowCoordinator.java:291) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateListeners$7(ClusterApplierService.java:498) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) [?:?]
at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) [?:?]
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) [?:?]
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:495) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:478) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 51 LINES ...
[2019-01-17T15:42:17,447][WARN ][o.e.r.b.FileRestoreContext] [node-0] [[test_index_restored][0]] [snapshot_1/V9FwD0UnSiGM6p5AFEe3hA] Can't read metadata from store, will not reuse local files during restore
org.apache.lucene.index.IndexNotFoundException: no segments* file found in store(ByteSizeCachingDirectory(HybridDirectory@/var/jenkins_home/jobs/elasticsearch-pipeline/workspace/elasticsearch/client/rest-high-level/build/cluster/integTestCluster node0/elasticsearch-7.0.0-SNAPSHOT/data/nodes/0/indices/z7f-71byQm6UyXJLR67gtg/0/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@3bf9292)): files: [write.lock]
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:675) ~[lucene-core-8.0.0-snapshot-a1c6e642aa.jar:8.0.0-snapshot-a1c6e642aa a1c6e642aad90d3615b4c71bf261a5aad7e32369 - nknize - 2019-01-02 14:49:37]
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0-snapshot-a1c6e642aa.jar:8.0.0-snapshot-a1c6e642aa a1c6e642aad90d3615b4c71bf261a5aad7e32369 - nknize - 2019-01-02 14:49:37]
at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0-snapshot-a1c6e642aa.jar:8.0.0-snapshot-a1c6e642aa a1c6e642aad90d3615b4c71bf261a5aad7e32369 - nknize - 2019-01-02 14:49:37]
at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:200) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.store.Store.access$200(Store.java:128) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:839) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.store.Store$MetadataSnapshot.<init>(Store.java:772) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.store.Store.getMetadata(Store.java:286) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1173) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.repositories.blobstore.FileRestoreContext.restore(FileRestoreContext.java:121) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:861) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:464) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:279) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:302) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:277) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1683) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$7(IndexShard.java:2207) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 15 LINES ...
[2019-01-17T15:42:17,649][WARN ][r.suppressed ] [node-0] path: /_snapshot/test_repository/test_snapshot, params: {master_timeout=30s, repository=test_repository, snapshot=test_snapshot}
org.elasticsearch.repositories.RepositoryException: [test_repository] concurrent modification of the index-N file, expected current generation [13], actual current generation [14] - possibly due to simultaneous snapshot deletion requests
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeIndexGen(BlobStoreRepository.java:712) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.deleteSnapshot(BlobStoreRepository.java:420) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.snapshots.SnapshotsService.lambda$deleteSnapshotFromRepository$5(SnapshotsService.java:1337) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
=========================================
> Task :client:rest-high-level:integTestRunner FAILED
> Task :client:rest-high-level:integTestCluster#stop