Skip to content

Conversation

@jakelandis
Copy link
Contributor

7.5+ for SLM requires [stats] object to exist in the cluster state.
When doing an in-place upgrade from 7.4 to 7.5+ [stats] does not exist
in cluster state, result in an exception on startup [1].

This commit moves the [stats] to be an optional object in the parser
and if not found will default to an empty stats object.

[1] Caused by: java.lang.IllegalArgumentException: Required [stats]


Note - this is was not caught by normal full cluster state restart tests since by default there is no SLM data in cluster state.

Below is the full error when the test is run without the fix:

./gradlew :x-pack:qa:full-cluster-restart:v7.4.0#bwc -Dtests.class=org.elasticsearch.xpack.restart.FullClusterRestartIT -Dtests.method="testSlmStats"

> Task :printGlobalBuildInfo UP-TO-DATE
=======================================
Elasticsearch Build Hamster says Hello!
  Gradle Version        : 5.6.2
  OS Info               : Mac OS X 10.14.6 (x86_64)
  JDK Version           : 12 (Oracle Corporation 12.0.1 [Java HotSpot(TM) 64-Bit Server VM 12.0.1+12])
  JAVA_HOME             : /Users/jakelandis/workspace/java/jdk-12.0.1.jdk/Contents/Home
  Random Testing Seed   : 2B2BAE156D0301A9
=======================================

> Task :x-pack:plugin:core:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :x-pack:qa:full-cluster-restart:compileTestJava
Note: /Users/jakelandis/workspace/7x/elasticsearch/x-pack/qa/full-cluster-restart/src/test/java/org/elasticsearch/xpack/restart/MlMigrationFullClusterRestartIT.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :x-pack:qa:full-cluster-restart:v7.4.0#upgradedClusterTest

=== Standard output of node `node{:x-pack:qa:full-cluster-restart:v7.4.0-0}` ===

»    ↓ errors and warnings from /Users/jakelandis/workspace/7x/elasticsearch/x-pack/qa/full-cluster-restart/build/testclusters/v7.4.0-0/logs/es.stdout.log ↓
» WARN ][o.e.d.FileBasedSeedHostsProvider] [v7.4.0-0] expected, but did not find, a dynamic hosts list at [/Users/jakelandis/workspace/7x/elasticsearch/x-pack/qa/full-cluster-restart/build/testclusters/v7.4.0-0/config/unicast_hosts.txt]
»   ↑ repeated 4 times ↑
» ERROR][o.e.g.GatewayMetaState   ] [v7.4.0-0] failed to read or upgrade local state, exiting...
»  org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read /Users/jakelandis/workspace/7x/elasticsearch/x-pack/qa/full-cluster-restart/build/testclusters/v7.4.0-0/data/_state/global-22.st
»       at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:138) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:414) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:80) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:147) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:89) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.node.Node.start(Node.java:690) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:271) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:348) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) [elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  Caused by: java.io.IOException: failed to read /Users/jakelandis/workspace/7x/elasticsearch/x-pack/qa/full-cluster-restart/build/testclusters/v7.4.0-0/data/_state/global-22.st
»       at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:408) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       ... 13 more
»  Caused by: java.lang.IllegalArgumentException: Required [stats]
»       at org.elasticsearch.common.xcontent.ConstructingObjectParser$Target.finish(ConstructingObjectParser.java:446) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.common.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:169) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.xpack.ilm.IndexLifecycle.lambda$getNamedXContent$3(IndexLifecycle.java:205) ~[?:?]
»       at org.elasticsearch.common.xcontent.NamedXContentRegistry$Entry.lambda$new$0(NamedXContentRegistry.java:63) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.common.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:141) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.common.xcontent.support.AbstractXContentParser.namedObject(AbstractXContentParser.java:385) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1326) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1371) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1362) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:302) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:404) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       ... 13 more
» WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [v7.4.0-0] uncaught exception in thread [main]
»  org.elasticsearch.bootstrap.StartupException: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read /Users/jakelandis/workspace/7x/elasticsearch/x-pack/qa/full-cluster-restart/build/testclusters/v7.4.0-0/data/_state/global-22.st
»       at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read /Users/jakelandis/workspace/7x/elasticsearch/x-pack/qa/full-cluster-restart/build/testclusters/v7.4.0-0/data/_state/global-22.st
»       at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:138) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:414) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:80) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:147) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:89) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.node.Node.start(Node.java:690) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:271) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:348) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       ... 6 more
»  Caused by: java.io.IOException: failed to read /Users/jakelandis/workspace/7x/elasticsearch/x-pack/qa/full-cluster-restart/build/testclusters/v7.4.0-0/data/_state/global-22.st
»       at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:408) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:80) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:147) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:89) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.node.Node.start(Node.java:690) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:271) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:348) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       ... 6 more
»  Caused by: java.lang.IllegalArgumentException: Required [stats]
»       at org.elasticsearch.common.xcontent.ConstructingObjectParser$Target.finish(ConstructingObjectParser.java:446) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.common.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:169) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.xpack.ilm.IndexLifecycle.lambda$getNamedXContent$3(IndexLifecycle.java:205) ~[?:?]
»       at org.elasticsearch.common.xcontent.NamedXContentRegistry$Entry.lambda$new$0(NamedXContentRegistry.java:63) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.common.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:141) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.common.xcontent.support.AbstractXContentParser.namedObject(AbstractXContentParser.java:385) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1326) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1371) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1362) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:302) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:404) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:80) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:147) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:89) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.node.Node.start(Node.java:690) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:271) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:348) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       ... 6 more

7.5+ for SLM requires [stats] object to exist in the cluster state.
When doing an in-place upgrade from 7.4 to 7.5+ [stats] does not exist
in cluster state, result in an exception on startup [1].

This commit moves the [stats] to be an optional object in the parser
and if not found will default to an empty stats object.

[1] Caused by: java.lang.IllegalArgumentException: Required [stats]
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

Copy link
Contributor

@gwbrown gwbrown left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM once CI is happy, thanks again for taking care of this. We'll likely want to expand on the restart test at some point, but that can be done in another PR.

@jakelandis
Copy link
Contributor Author

@elasticmachine run elasticsearch-ci/1
@elasticmachine run elasticsearch-ci/2
@elasticmachine run elasticsearch-ci/bwc
@elasticmachine run elasticsearch-ci/default-distro
@elasticmachine run elasticsearch-ci/packaging-sample-matrix

@jakelandis
Copy link
Contributor Author

@elasticmachine update branch

Copy link
Member

@dakrone dakrone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for fixing this Jake!

@jakelandis jakelandis merged commit f691daa into elastic:master Oct 23, 2019
@jakelandis jakelandis deleted the slm_stats_fix_master branch October 23, 2019 15:24
jakelandis added a commit to jakelandis/elasticsearch that referenced this pull request Oct 23, 2019
…#48367)

7.5+ for SLM requires [stats] object to exist in the cluster state.
When doing an in-place upgrade from 7.4 to 7.5+ [stats] does not exist
in cluster state, result in an exception on startup [1].

This commit moves the [stats] to be an optional object in the parser
and if not found will default to an empty stats object.

[1] Caused by: java.lang.IllegalArgumentException: Required [stats]

Co-authored-by: Elastic Machine <[email protected]>
jakelandis added a commit to jakelandis/elasticsearch that referenced this pull request Oct 23, 2019
…#48367)

7.5+ for SLM requires [stats] object to exist in the cluster state.
When doing an in-place upgrade from 7.4 to 7.5+ [stats] does not exist
in cluster state, result in an exception on startup [1].

This commit moves the [stats] to be an optional object in the parser
and if not found will default to an empty stats object.

[1] Caused by: java.lang.IllegalArgumentException: Required [stats]

Co-authored-by: Elastic Machine <[email protected]>
@jakelandis
Copy link
Contributor Author

backported via #48411 and #48412

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants