ES: Failed to read local state, exiting...

[showalski]

Describe the feature:

Elasticsearch version :6.2.4

Plugins installed: ingest-geoip, ingest-user-agent

JVM version :

openjdk version "1.8.0_161"
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)

OS version :

Linux elasticsearch-0 4.18.0-21-generic #22~18.04.1-Ubuntu SMP Thu May 16 15:07:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
Problem: elasticsearch-0 kept crashing due to Failed to read local state, exiting....
Expected: elasticsearch could bootup successfully.
Description: We are running a one ES instance in k8s pod. It kept reporting below error(Please refer to more detailed error and config at the end of this issue). According to the link, metadata seems corrupted. We removed the corrputed data and ES could bootup. Can someone help to give the hit why this happened? What should be done to avoid this issue. Thanks in advance.

failed to read [id:121, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/indices/lN_rrlwwSTmqxGn2Vi_2Tg/_state/state-121.st]

Steps to reproduce:

1. Leave old ES data
2. Delete crashed ES pod
3. Restart ES pod

Provide logs (if relevant):

elasticsearch.yml

Data
====
elasticsearch.yml:
----
# Cassandra storage config YAML
cluster.name: "docker-cluster"
network.host: 0.0.0.0

# minimum_master_nodes need to be explicitly set when bound on a public IP
# set to 1 to allow single node clusters
# Details: https://github.com/elastic/elasticsearch/pull/17288
discovery.zen.ping.unicast.hosts: elasticsearch

discovery.zen.minimum_master_nodes: 1
#xpack.security.enabled: false
#xpack.monitoring.enabled: false
Events:  <none>

jvm.options

Data
====
jvm.options:
----
## JVM configuration

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms8g
-Xmx8g

## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly

## optimizations

# pre-touch memory pages used by the JVM during initialization
-XX:+AlwaysPreTouch

## basic

# force the server VM
-server

# explicitly set the stack size
-Xss1m

# set to headless, just in case
-Djava.awt.headless=true

# ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8

# use our provided JNA always versus the system one
-Djna.nosys=true

# turn off a JDK optimization that throws away stack traces for common
# exceptions because stack traces are important for debugging
-XX:-OmitStackTraceInFastThrow

# flags to configure Netty
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0

# log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true

## heap dumps
# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError
Events:  <none>

ES error log

[2019-09-02T07:06:17,275][INFO ][o.e.n.Node               ] [] initializing ...
[2019-09-02T07:06:17,346][INFO ][o.e.e.NodeEnvironment    ] [W5QPW1e] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sdb1)]], net usable_space [939.4gb], net total_space [1006.9gb], types [ext4]
[2019-09-02T07:06:17,346][INFO ][o.e.e.NodeEnvironment    ] [W5QPW1e] heap size [7.8gb], compressed ordinary object pointers [true]
[2019-09-02T07:06:17,418][INFO ][o.e.n.Node               ] node name [W5QPW1e] derived from node ID [W5QPW1esSxaGZCzL40iVwA]; set [node.name] to override
[2019-09-02T07:06:17,418][INFO ][o.e.n.Node               ] version[6.2.4], pid[1], build[ccec39f/2018-04-12T20:37:28.497551Z], OS[Linux/4.18.0-21-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2019-09-02T07:06:17,418][INFO ][o.e.n.Node               ] JVM arguments [-Xms8g, -Xmx8g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
[2019-09-02T07:06:18,431][INFO ][o.e.p.PluginsService     ] [W5QPW1e] loaded module [aggs-matrix-stats]
[2019-09-02T07:06:18,432][INFO ][o.e.p.PluginsService     ] [W5QPW1e] loaded module [analysis-common]
[2019-09-02T07:06:18,432][INFO ][o.e.p.PluginsService     ] [W5QPW1e] loaded module [ingest-common]
[2019-09-02T07:06:18,432][INFO ][o.e.p.PluginsService     ] [W5QPW1e] loaded module [lang-expression]
[2019-09-02T07:06:18,432][INFO ][o.e.p.PluginsService     ] [W5QPW1e] loaded module [lang-mustache]
[2019-09-02T07:06:18,432][INFO ][o.e.p.PluginsService     ] [W5QPW1e] loaded module [lang-painless]
[2019-09-02T07:06:18,432][INFO ][o.e.p.PluginsService     ] [W5QPW1e] loaded module [mapper-extras]
[2019-09-02T07:06:18,432][INFO ][o.e.p.PluginsService     ] [W5QPW1e] loaded module [parent-join]
[2019-09-02T07:06:18,432][INFO ][o.e.p.PluginsService     ] [W5QPW1e] loaded module [percolator]
[2019-09-02T07:06:18,432][INFO ][o.e.p.PluginsService     ] [W5QPW1e] loaded module [rank-eval]
[2019-09-02T07:06:18,432][INFO ][o.e.p.PluginsService     ] [W5QPW1e] loaded module [reindex]
[2019-09-02T07:06:18,433][INFO ][o.e.p.PluginsService     ] [W5QPW1e] loaded module [repository-url]
[2019-09-02T07:06:18,433][INFO ][o.e.p.PluginsService     ] [W5QPW1e] loaded module [transport-netty4]
[2019-09-02T07:06:18,433][INFO ][o.e.p.PluginsService     ] [W5QPW1e] loaded module [tribe]
[2019-09-02T07:06:18,433][INFO ][o.e.p.PluginsService     ] [W5QPW1e] loaded plugin [ingest-geoip]
[2019-09-02T07:06:18,433][INFO ][o.e.p.PluginsService     ] [W5QPW1e] loaded plugin [ingest-user-agent]
[2019-09-02T07:06:21,869][ERROR][o.e.g.GatewayMetaState   ] [W5QPW1e] failed to read local state, exiting...
org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:121, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/indices/lN_rrlwwSTmqxGn2Vi_2Tg/_state/state-121.st]
	at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:190) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:334) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:90) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.common.util.IndexFolderUpgrader.upgradeIndicesIfNeeded(IndexFolderUpgrader.java:128) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:87) [elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.node.Node.<init>(Node.java:424) [elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.node.Node.<init>(Node.java:246) [elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) [elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) [elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) [elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.2.4.jar:6.2.4]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.2.4.jar:6.2.4]
Caused by: java.io.IOException: failed to read [id:121, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/indices/lN_rrlwwSTmqxGn2Vi_2Tg/_state/state-121.st]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:327) ~[elasticsearch-6.2.4.jar:6.2.4]
	... 15 more
Caused by: java.lang.IllegalArgumentException: Unexpected field [mapping_version]
	at org.elasticsearch.cluster.metadata.IndexMetaData$Builder.fromXContent(IndexMetaData.java:1259) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.cluster.metadata.IndexMetaData$2.fromXContent(IndexMetaData.java:1305) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.cluster.metadata.IndexMetaData$2.fromXContent(IndexMetaData.java:1296) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:202) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:322) ~[elasticsearch-6.2.4.jar:6.2.4]
	... 15 more
[2019-09-02T07:06:21,881][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[java.io.IOException: failed to read [id:121, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/indices/lN_rrlwwSTmqxGn2Vi_2Tg/_state/state-121.st]]; nested: IOException[failed to read [id:121, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/indices/lN_rrlwwSTmqxGn2Vi_2Tg/_state/state-121.st]]; nested: IllegalArgumentException[Unexpected field [mapping_version]];
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.4.jar:6.2.4]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.4.jar:6.2.4]
Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:121, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/indices/lN_rrlwwSTmqxGn2Vi_2Tg/_state/state-121.st]
	at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:190) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:334) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:90) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.common.util.IndexFolderUpgrader.upgradeIndicesIfNeeded(IndexFolderUpgrader.java:128) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:87) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.node.Node.<init>(Node.java:424) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]
	... 6 more
Caused by: java.io.IOException: failed to read [id:121, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/indices/lN_rrlwwSTmqxGn2Vi_2Tg/_state/state-121.st]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:327) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:90) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.common.util.IndexFolderUpgrader.upgradeIndicesIfNeeded(IndexFolderUpgrader.java:128) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:87) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.node.Node.<init>(Node.java:424) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]
	... 6 more
Caused by: java.lang.IllegalArgumentException: Unexpected field [mapping_version]
	at org.elasticsearch.cluster.metadata.IndexMetaData$Builder.fromXContent(IndexMetaData.java:1259) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.cluster.metadata.IndexMetaData$2.fromXContent(IndexMetaData.java:1305) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.cluster.metadata.IndexMetaData$2.fromXContent(IndexMetaData.java:1296) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:202) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:322) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:90) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.common.util.IndexFolderUpgrader.upgradeIndicesIfNeeded(IndexFolderUpgrader.java:128) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:87) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.node.Node.<init>(Node.java:424) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]
	... 6 more

Hexdump of state file

00000000  3f d7 6c 17 05 73 74 61  74 65 00 00 00 01 00 00  |?.l..state......|
00000010  00 01 3a 29 0a 05 fa 92  6c 6f 67 73 74 61 73 68  |..:)....logstash|
00000020  2d 32 30 31 39 2e 30 38  2e 33 31 fa 86 76 65 72  |-2019.08.31..ver|
00000030  73 69 6f 6e 24 03 88 8e  6d 61 70 70 69 6e 67 5f  |sion$...mapping_|
00000040  76 65 72 73 69 6f 6e c2  8f 73 65 74 74 69 6e 67  |version..setting|
00000050  73 5f 76 65 72 73 69 6f  6e c2 91 72 6f 75 74 69  |s_version..routi|
00000060  6e 67 5f 6e 75 6d 5f 73  68 61 72 64 73 ca 84 73  |ng_num_shards..s|
00000070  74 61 74 65 43 6f 70 65  6e 87 73 65 74 74 69 6e  |tateCopen.settin|
00000080  67 73 fa 92 69 6e 64 65  78 2e 63 72 65 61 74 69  |gs..index.creati|
00000090  6f 6e 5f 64 61 74 65 4c  31 35 36 37 32 32 30 36  |on_dateL15672206|
000000a0  33 38 36 34 33 97 69 6e  64 65 78 2e 6e 75 6d 62  |38643.index.numb|
000000b0  65 72 5f 6f 66 5f 72 65  70 6c 69 63 61 73 40 31  |er_of_replicas@1|
000000c0  95 69 6e 64 65 78 2e 6e  75 6d 62 65 72 5f 6f 66  |.index.number_of|
000000d0  5f 73 68 61 72 64 73 40  35 92 69 6e 64 65 78 2e  |_shards@5.index.|
000000e0  70 72 6f 76 69 64 65 64  5f 6e 61 6d 65 52 6c 6f  |provided_nameRlo|
000000f0  67 73 74 61 73 68 2d 32  30 31 39 2e 30 38 2e 33  |gstash-2019.08.3|
00000100  31 95 69 6e 64 65 78 2e  72 65 66 72 65 73 68 5f  |1.index.refresh_|
00000110  69 6e 74 65 72 76 61 6c  41 35 73 89 69 6e 64 65  |intervalA5s.inde|
00000120  78 2e 75 75 69 64 55 6c  4e 5f 72 72 6c 77 77 53  |x.uuidUlN_rrlwwS|
00000130  54 6d 71 78 47 6e 32 56  69 5f 32 54 67 94 69 6e  |TmqxGn2Vi_2Tg.in|
00000140  64 65 78 2e 76 65 72 73  69 6f 6e 2e 63 72 65 61  |dex.version.crea|
00000150  74 65 64 46 36 30 32 30  34 39 39 95 69 6e 64 65  |tedF6020499.inde|
00000160  78 2e 76 65 72 73 69 6f  6e 2e 75 70 67 72 61 64  |x.version.upgrad|
00000170  65 64 46 36 30 37 30 32  39 39 fb 87 6d 61 70 70  |edF6070299..mapp|
00000180  69 6e 67 73 f8 fd 04 83  44 46 4c 00 94 51 cb 4e  |ings....DFL..Q.N|
00000190  c3 40 0c fc 17 1f 51 4e  48 70 e8 a9 ff 81 d0 ca  |.@....QNHp......|
000001a0  24 4e b2 62 5f da 75 0a  51 94 7f 67 92 6e 03 bd  |$N.b_.u.Q..g.n..|
000001b0  54 e2 68 cf d8 33 1e 2f  64 3a e9 79 72 6a e8 b4  |T.h..3./d:.yrj..|
000001c0  50 37 07 f6 b6 35 2a 3e  39 56 29 74 7a 5b c8 4b  |P7...5*>9V)tz[.K|
000001d0  29 3c 88 e9 ad b8 6e a3  25 d6 d1 78 d6 76 a4 d3  |)<....n.%..x.v..|
000001e0  0d a5 86 f6 0e fa 29 d9  30 18 9d 93 00 2e 9a 51  |......).0......Q|
000001f0  ed e8 de df e6 43 cc 1e  ab 7b 76 45 1a aa 44 95  |.....C...{vE..D.|
00000200  6f a5 75 5d 9b a5 ce 5c  f5 c0 83 83 aa f5 f4 1f  |o.u]...\........|
00000210  95 df f1 4f 99 bf 62 de  ad db 01 e2 62 f8 23 5e  |...O..b.....b.#^|
00000220  e0 ee f9 e5 f5 d0 bf 91  e0 e0 b1 c1 f7 86 52 8e  |..............R.|
00000230  49 b2 da 2d a0 85 ce 6a  11 91 b2 4f 5b 55 cf e9  |I..-...j...O[U..|
00000240  10 1f 61 d5 f9 22 b9 d8  18 fe 40 87 52 43 83 44  |..a.."....@.RC.D|
00000250  bb 4f d5 e4 11 98 e6 49  70 e6 bd c4 95 54 57 a3  |.O.....Ip....TW.|
00000260  c0 62 fc c7 ea d4 e1 8a  03 18 d9 f5 a6 77 91 11  |.b...........w..|
00000270  24 08 b1 05 e5 4e 19 72  26 45 1b 2a 1e 86 47 1b  |$....N.r&E.*..G.|
00000280  f0 8b 75 fd 01 00 00 ff  ff 03 00 fd 05 a4 44 46  |..u...........DF|
00000290  4c 00 b4 53 c1 6e c3 20  0c fd 17 1f a7 9c 26 6d  |L..S.n. ......&m|
000002a0  87 9e fa 1f d3 84 58 e2  24 a8 04 90 71 da 45 51  |......X.$...q.EQ|
000002b0  ff 7d 86 90 2e 5d a7 4a  93 56 6e f0 8c df f3 b3  |.}...].J.Vn.....|
000002c0  3d 43 6b 47 74 dc c0 6e  86 66 72 7a 30 b5 62 1c  |=CkGt..n.frz0.b.|
000002d0  82 d5 8c 11 76 6f 33 0c  18 a3 ee 50 b5 06 6d 0e  |....vo3....P..m.|
000002e0  0b 9a 7b 35 68 ae 7b d8  ad 28 54 90 5f e4 3d 04  |..{5h.{..(T._.=.|
000002f0  e3 3a c5 53 40 81 23 93  dc 32 9a df d3 7f e7 69  |.:.S@.#..2.....i|
00000300  90 d4 ad b6 11 2b 28 81  8c 9f 0c e7 f3 b9 9a cb  |.....+(.........|
00000310  9f 85 4f e2 44 41 e1 7a  fa 0b cb f7 f7 03 4e 27  |..O.DA.z......N'|
00000320  4f 59 ba e9 84 1c 95 fe  f0 47 51 f7 fc f2 7a e1  |OY.......GQ...z.|
00000330  5f 83 44 c1 7d 81 ef 15  04 f2 01 89 4d 32 68 86  |_.D.}.......M2h.|
00000340  3d 1b b1 88 f5 10 d2 ad  94 d3 88 7d 20 a9 f6 47  |=..........} ..G|
00000350  a4 68 bc db 40 17 a6 0a  ea 7e 74 87 0d 94 4d f8  |.h..@....~t...M.|
00000360  c9 ff 6b 21 85 66 cd 55  dd 56 96 bc 5c 08 94 c9  |..k!.f.U.V..\...|
00000370  b5 6f 8d fe 4f 0e 24 f2  f4 48 82 0e bd c9 de 96  |.o..O.$..H......|
00000380  f9 94 b1 62 1a 51 8c ba  6e c4 12 54 ca 94 8b d4  |...b.Q..n..T....|
00000390  2f 53 6c 78 6c a4 d7 17  a0 d7 b6 55 ad f5 5a c6  |/Slxl......U..Z.|
000003a0  4d 02 7c 2d 21 57 fd 11  3a 15 bc 71 05 77 dd bd  |M.|-!W..:..q.w..|
000003b0  0c c9 e4 b2 20 f7 2d 48  63 25 dd 55 84 4c 93 8a  |.... .-Hc%.U.L..|
000003c0  58 7b b7 0c f7 a3 da 42  42 41 57 14 d6 cb fe 89  |X{.....BBAW.....|
000003d0  8e 45 42 9a da 0d ff 2d  b8 55 b7 a2 ac f3 06 3f  |.EB....-.U.....?|
000003e0  4a b3 ac e9 01 b7 b3 b4  f0 ca f9 02 00 00 ff ff  |J...............|
000003f0  03 00 f9 86 61 6c 69 61  73 65 73 fa fb 8c 70 72  |....aliases...pr|
00000400  69 6d 61 72 79 5f 74 65  72 6d 73 f8 24 ae 24 ae  |imary_terms.$.$.|
00000410  24 ae 24 ae 24 ae f9 92  69 6e 5f 73 79 6e 63 5f  |$.$.$...in_sync_|
00000420  61 6c 6c 6f 63 61 74 69  6f 6e 73 fa 80 33 f8 55  |allocations..3.U|
00000430  69 44 43 44 76 4b 48 43  51 7a 36 6d 67 68 7a 79  |iDCDvKHCQz6mghzy|
00000440  4f 30 4f 78 4d 67 f9 80  31 f8 55 72 57 76 67 4c  |O0OxMg..1.UrWvgL|
00000450  6d 4c 49 51 71 2d 49 35  78 62 33 5a 32 55 42 4a  |mLIQq-I5xb3Z2UBJ|
00000460  41 f9 80 32 f8 55 4c 31  51 6f 31 41 4e 68 53 66  |A..2.UL1Qo1ANhSf|
00000470  71 71 73 77 32 4c 39 37  79 47 75 51 f9 80 34 f8  |qqsw2L97yGuQ..4.|
00000480  55 70 79 52 62 68 33 78  66 53 2d 79 68 6f 6a 59  |UpyRbh3xfS-yhojY|
00000490  71 63 37 36 76 67 67 f9  80 30 f8 55 5a 69 4a 65  |qc76vgg..0.UZiJe|
000004a0  34 56 4c 55 53 6a 4f 78  74 70 5a 31 71 6a 46 62  |4VLUSjOxtpZ1qjFb|
000004b0  59 51 f9 fb 8c 72 6f 6c  6c 6f 76 65 72 5f 69 6e  |YQ...rollover_in|
000004c0  66 6f fa fb fb fb c0 28  93 e8 00 00 00 00 00 00  |fo.....(........|
000004d0  00 00 18 55 56 16                                 |...UV.|
000004d6

[martijnvg]

Looks the ES 6.2.4 es node is reading a data directory that was used by an es node with a higher version. Looking at the stack trace, the mapping_version field was introduced in #33147, which was included in the 6.5 release. Did you do an upgrade to a newer version and then downgraded to 6.2.4 using the same data directory?

[showalski]

@martijnvg Thanks for your help. Yes, we tested different versions on same virtual machine without cleaning old ES data. The main versions being used are 6.7.2 and 6.2.4.

So when we test 6.2.4 version, we need to remove data generated by ES that has version higher than 6.5 release? This is the only way to let version 6.2.4 ES work as expected, right?

[martijnvg]

So when we test 6.2.4 version, we need to remove data generated by ES that has version higher than 6.5 release? This is the only way to let version 6.2.4 ES work as expected, right?

Elasticsearch supports upgrading to a newer version with the same data directory. However Elasticsearch doesn't support downgrading to an older version with the same data directory. In that case the data directory should be removed. In any case when upgrading please consult the upgrade docs: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html

[martijnvg]

I'm going to close this issue. Actually we use GitHub for verified bug reports, feature requests, and pull requests. Next time we would really appreciate if you go to the forums first before opening an issue here. If you do have a followup question then open a thread in the Elasticsearch forum and then I'm happy to help you there.

[showalski]

Thanks for your help.

[stefanhorning]

Happened to me on a test instance, which I tested a newer ES version on and then forgot, upon rerunning our infra provisioning the version was downgraded to 6.8 again.

For me the simple fix was just to remove all states like this:

rm -r /data/elasticsearch/nodes/0/indices/*/_state/
rm -r /data/elasticsearch/nodes/0/_state/

(where /data/elasticsearch is my data dir)

So I could at least run elasticsearch again. This however might lead to data loss! So no guarantees here.

[mithlajkn]

I tried to downgrade Es from 7 > 6.8.22. I got the same exceptions like org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:19, file:/var/lib/elasticsearch/nodes/0/_state/node-19.st] etc.... I removed $rm -rf /var/lib/elasticsearch (this folder remained in the system after the removal of es 7. Then reinstalled Es 6.8 and its worked. thanks to @martijnvg @showalski