[Elasticsearch Docs] Add clarification that `.security` index needs to be upgraded to `.security-6` index before upgrade the Elastic Stack from 5.6 to 6.x

[kunisen]

Current Status

Currently, we have the following statement to describe about upgrading internal indices.

[1]
https://www.elastic.co/guide/en/elastic-stack/6.5/upgrading-elastic-stack.html#upgrade-internal-indices

If you are upgrading from a version prior to 5.6, you must upgrade them after after installing Elasticsearch 6.5.4.

It only indicates the version prior to 5.6, but it didn't tell this includes 5.6 or not.
To say it clearly, it makes people think the version before (which means less than <) 5.6 needs upgraded, but the version 5.6 is safe and not need upgraded before upgrading Elastic Stack.

[2]
https://www.elastic.co/guide/en/elastic-stack-overview/6.5/security-auth-failure-upgrade.html

You must upgrade the .security index to the 6.x format. For instructions, see Upgrading internal indices.

There are two situations where it’s necessary to manually upgrade the .security index:

・After upgrading directly to 6.5.4 from 5.5 or earlier.
・After restoring a snapshot from 5.5 or earlier that contains a .security index in the old format to a 6.0 cluster.

It only indicates that .security index of version 5.5 needs upgraded manually and it seems like .security index of version 5.6 does not need upgraded manually.

The problem we encountered

If we do not upgrade .security to .security-6 before we upgrade the elasticsearch package itself, we will get the following error message and the new-version node cannot join the cluster.
We searched the document but didn't find anything officially documented.

[2018-12-20T03:51:57,435][INFO ][o.e.d.z.ZenDiscovery     ] [node3] failed to send join request to master [{node1}{nuKEosHfQiSEfY_oek5z_w}{OEYFHhINTZWzISxbHkSuIg}{10.146.0.10}{10.146.0.10:9300}{ml.max_open_jobs=10, ml.enabled=true}], reason [RemoteTransportException[[node1][10.146.0.10:9300][internal:discovery/zen/join]]; nested: IllegalStateException[failure when sending a validation request to node]; nested: RemoteTransportException[[node3][10.146.0.4:9300][internal:discovery/zen/join/validate]]; nested: IllegalStateException[Security index is not on the current version [6] - The Upgrade API must be run for 6.x nodes to join the cluster]; ]

What we concerned as the problem/cause

[a] The documentation about .security index upgrade is not clear, and can make confusion on the version different
[b] There is no documentation described the above-mentioned error, which may lead confusion.

Feature Request

With the above said, could you please

[i] Add some explicit description such as .security index of any 5.x version needs to be upgraded before Elastic Stack upgrade
[ii] Add a page about the above mentioned failed to send join request to master ... error to Troubleshooting security page?

[elasticmachine]

Pinging @elastic/es-security

[albertzaharovits]

Hi @kunisen ,

ES <= 5.5 only understands the old .security index format. ES 5.6 understands both .security index formats, i.e. the old (multiple types) and the new (a single doc type). ES >= 6.0 understands only the new format.

If you upgrade straight from <= 5.5 to >= 6.0 a "manual" format upgrade is required (after the full-cluster restart into the new version). The "manual"" format upgrade happens after the upgrade. It is not possible to do rolling upgrade from 5.5 to 6.0 .
The rolling upgrade procedure from 5.6 to >= 6.0 includes the format upgrade step. This is why a "manual" upgrade, after the update, is not required (though it is still "manual", it is just embedded in the procedure). The gist of this is stated in the stack upgrade docs pt 5.

It only indicates the version prior to 5.6, but it didn't tell this includes 5.6 or not.
To say it clearly, it makes people think the version before (which means less than <) 5.6 needs upgraded, but the version 5.6 is safe and not need upgraded before upgrading Elastic Stack.

This is correct. "before" does not include 5.6 . The important word here is after ("If you are upgrading from a version prior to 5.6, you must upgrade them after after installing Elasticsearch 6.5.4.") - after is maybe duplicated for emphasis purposes 😛 . Again, no "after upgrade" is required on 5.6 because the format upgrade is embedded in the cluster rolling upgrade procedure.

I agree "manual" upgrade is a stretch term and after is not emphasized. Do you have other docs improvement suggestions?

[eedugon]

Hi, @albertzaharovits , @kunisen :

the format upgrade is embedded in the cluster rolling upgrade procedure.
I believe it does not, but please correct me if i'm wrong:
https://www.elastic.co/guide/en/elasticsearch/reference/current/rolling-upgrades.html

The format upgrade (manual, to be done before the real upgrade, as you have explained perfectly fine), is NOT mentioned or described in the rolling upgrade procedure, and that's causing a lot of people to believe that 5.6 can be upgraded to 6.x smoothly via rolling upgrade, when it's not possible if Xpack or Kibana are used.

This is perfecly explained in the "Upgrading Elastic Stack" guide, but not in the rolling upgrade guide I guess...

Anyway I agree with both of your views!

Maybe just adding a link and note to "Preparing Upgrade step 5" or "Upgrading from 5.6" sections of the Elastic Stack upgrade document in the Rolling Upgrade document would be enough (in order to not duplicate info).