7.0 upgrade (#242)

* [DOCS] First pass at 7.0 upgrade update.

* [DOCS] Fixed formatting issues & flagged the Upgrade Assistant as xpack.

* Update docs/en/install-upgrade/upgrading-stack.asciidoc

Co-Authored-By: debadair <[email protected]>

* [DOCS] Incorporated feedback from lcawl.

* [DOCS] Fixes broken link

* [DOCS] Removed tip about temporary superuser when upgrading the security index.

Author: debadair

docs/en/install-upgrade/upgrading-stack.asciidoc

@@ -1,27 +1,13 @@
 [[upgrading-elastic-stack]]
 == Upgrading the Elastic Stack
 
-When upgrading to a new version of Elasticsearch, you need to upgrade
-each of the products in your Elastic Stack. Beats and Logstash 5.6 are
-compatible with Elasticsearch {version} to give you flexibility in scheduling the
-upgrade.
+When upgrading to a new version of {es}, you need to upgrade
+each of the products in your Elastic Stack. Beats and Logstash 7.n are
+compatible with {es} {version} to give you flexibility in scheduling
+the upgrade.
 
-****
-The steps you need to take to upgrade differ depending on which products you
-are using. Want a list that's tailored to your stack? Try out our
-{upgrade_guide}[Interactive Upgrade Guide].
-****
-
-If you are running a pre-6.0 version, we recommend upgrading to the most
-recent 5.6 before upgrading to {version}. {xpack} 5.6
-provides a free Upgrade Assistant that identifies issues you need to address
-before upgrading and simplifies migrating indices that need to be reindexed
-before you upgrade. The Upgrade Assistant is enabled with both Trial and
-Basic licenses. You can install {xpack} solely for the purpose of upgrading.
-
-Rolling upgrades are supported when upgrading from Elasticsearch 5.6 and
-Elasticsearch 6.0-6.2 to {version}. Upgrading from any
-version prior to 5.6 requires a full cluster restart.
+{es} supports rolling upgrades between minor versions, from {es} 5.6 to 6.7,
+from 6.7 to 7.x, and from 7.n to {version}.
 
 IMPORTANT: 6.x and older indices are not compatible with {version}. You must
 remove or reindex them on your 7.x cluster before upgrading to {version}. The internal
@@ -32,241 +18,99 @@ also need to be updated to work with {version}.
 
 Before upgrading the Elastic Stack to {version}:
 
-. Back up your data. You **cannot roll back** to an earlier version unless
-you have a backup of your data. For information about creating snapshots, see
-{ref}/modules-snapshots.html[Snapshot and Restore].
-
-. Check the Elasticsearch {ref}/settings.html[deprecation log] to see if
-you're using any deprecated features and update your code accordingly.
-By default, deprecation log messages are enabled at the `WARN` level.
-
-. Review the breaking changes for each product you use
-and make the necessary changes so your code is compatible with {version}. See
-<<elastic-stack-breaking-changes>>.
+. Check the {es} {ref}/logging.html#deprecation-logging[deprecation log]
+to see if you're using any deprecated features and update your code accordingly.
+By default, deprecation warnings are logged when the log level is set to `WARN`.
 
-. {ref}/docs-reindex.html[Reindex] or delete any indices created on 2.n. We recommend
-upgrading to the most recent 5.6 and using the {xpack} Reindex Helper to reindex 2.n indices.
+. Review the <<elastic-stack-breaking-changes>> and upgrade your code to work
+with {version}.
 
-. If Kibana and {xpack} are part of your stack, upgrade the internal Kibana
-and {xpack} indices. We recommend using the {xpack} 5.6 Reindex Helper to
-upgrade the internal indices. If you're performing a full cluster restart upgrade
-from an earlier version, you can also
-<<upgrade-internal-indices,use the `_migration/upgrade` API>> directly to
-upgrade the internal indices after you install Elasticsearch {version}.
-
-. If you use {stack} {security-features} to secure your cluster:
-.. Make sure TLS is enabled to encrypt communications between nodes. TLS must
-be enabled to upgrade to {version}. For more information, see
-{stack-ov}/encrypting-communications.html[Encrypting communications].
+. Upgrade to 6.7 and use the {kibana-ref}/upgrade-assistant.html[{kib} Upgrade Assistant] to {ref}/docs-reindex.html[reindex]
+any indices that are not compatible with {version}.
 +
-NOTE: Enabling TLS requires a full cluster restart. Nodes that have TLS
-enabled cannot communicate with nodes that do not have TLS enabled. You must
-restart all nodes to maintain communication across the cluster.
-
-.. Make sure real passwords are configured for the built-in `elasticsearch`,
-`kibana`, and `logstash_system` users. They cannot use the 5.n default
-password (`changeme`). For more information, see
-{stack-ov}/built-in-users.html[Built-in users].
-
-. Consider closing {ml} jobs before you start the upgrade process. It is not
-required, but there are pros and cons to leaving the jobs running. These
-considerations are described in the steps related to 
-{ref}/setup-upgrade.html[upgrading {es}].
+[role="xpack"]
+.Upgrade Assistant
+******
+The Upgrade Assistant and migration APIs are enabled with both the Basic and
+Trial licenses. You can install the default distribution of 6.7 to use the
+Upgrade Assistant to prepare to upgrade even if you are upgrading to the OSS
+distribution of {version}.
+******
 
-IMPORTANT: Test upgrades in a dev environment before upgrading your
-production cluster.
+. Use the Upgrade Assistant to identify any changes you need to make to your
+cluster configuration.
 
 [[upgrade-order-elastic-stack]]
-=== Upgrade order
-
-Upgrade the Elastic Stack products you use in the following order:
+=== Upgrade process
 
-. Elasticsearch Hadoop: {hadoop-ref}/install.html[install instructions]
-. Elasticsearch: {ref}/setup-upgrade.html[upgrade instructions]
-. Kibana: {kibana-ref}/upgrade.html[upgrade instructions]
-. Logstash: {logstash-ref}/upgrading-logstash.html[upgrade instructions]
-. Beats: {beats-ref}/upgrading.html[upgrade instructions]
-. APM Server: {apm-server-ref}/upgrading.html[upgrade instructions]
-
-NOTE: Logstash 5.6 and 6.n and Beats 5.6 and 6.n are compatible with all 6.n versions of
-Elasticsearch. This provides flexibility in when you schedule the upgrades
-for your Logstash instances and Beats agents. We recommend upgrading Logstash
-and Beats as soon as possible to take advantage of performance improvements
-and other enhancements.
-
-=== Upgrading to 6.3
-Starting in 6.3, the default distributions of {es}, {ls}, and {kib}
-include {xpack} and a free Basic license that never expires.
-
-You can perform rolling upgrades to 6.3 from OSS-only clusters running 5.6
-or 6.0-6.2. Basic features are operational once the cluster is fully
-upgraded. If you are already using {xpack}, your settings are preserved when
-you upgrade.
-
-IMPORTANT: If you use {xpack} you must explicitly remove the old {xpack} plugin
-before restarting: `bin/elasticsearch-plugin remove x-pack`. The node will fail
-to start if the {xpack} plugin is present.
-
-You must explicitly enable data collection after the upgrade to use monitoring.
-Set `xpack.monitoring.collection.enabled` to `true` with the `_cluster/settings`
-API:
-
-[source,json]
-----------------------------------------------------------
-PUT /_cluster/settings
-{
-    "persistent" : {
-        "xpack.monitoring.collection.enabled" : "true"
-    }
-}
-----------------------------------------------------------
-// CONSOLE
-
-To take more of the {stack} features for a spin, you can start a 30-day trial
-from Kibana or with the Start Trial API:
-
-[source,json]
-----------------------------------------------------------
-POST _license/start_trial
-----------------------------------------------------------
-// CONSOLE
-
-The 30-day trial enables you to try out the full set of platinum features,
-including security, machine learning, alerting, graph capabilities, and more.
-
-[role="xpack"]
-[[xpack-stack-upgrade]]
-=== Upgrading from 5.6
+When you've made the necessary changes and are ready to upgrade from 6.7 to
+{version}:
 
-{xpack} 5.6 provides migration and upgrade APIs for Elasticsearch and a
-Upgrade Assistant UI for Kibana. These tools are included with the trial
-license and the free basic license.
-
-To upgrade to {version} from 5.6:
+. Test the upgrade in a dev environment *before* upgrading your
+production cluster.
 
-. {ref}/setup-upgrade.html[Upgrade Elasticsearch] to the most recent 5.6 and
-install {xpack} on all nodes in your cluster. If you are upgrading from an
-earlier 5.x release, you can perform a rolling upgrade. To upgrade from older
-versions you must perform a full cluster restart.
-+
-If your trial license expires,
-https://register.elastic.co/[register for a free Basic license]. To apply the
-license, upload the license file with the `license` API:
-+
-[source,json]
-----------------------------------------------------------
-license -d @license.json
-----------------------------------------------------------
+. Back up your data. You **cannot roll back** to an earlier version unless
+you have a snapshot of your data. For information about creating snapshots, see
+{ref}/modules-snapshots.html[Snapshot and Restore].
 
-. If {xpack} **IS NOT** normally a part of your {stack}, disable the
-{es} {security-features} in `elasticsearch.yml`:
-+
-[source,yaml]
-----------------------------------------------------------
-xpack.security.enabled: false
-----------------------------------------------------------
+. Consider closing {ml} jobs before you start the upgrade process. While {ml}
+jobs can continue to run during a rolling upgrade, it increases the overhead
+on the cluster during the upgrade process. For more information, see
+{ref}/rolling-upgrades.html[Rolling upgrades].
 
-. Upgrade {kib} to the most recent 5.6 and install {xpack}.
+. Upgrade the components of your Elastic Stack in the following order:
 
-. If you disabled the {es} {security-features}, also disable the {kib}
-{security-features} in `kibana.yml`:
-+
-[source,yaml]
-----------------------------------------------------------
-xpack.security.enabled: false
-----------------------------------------------------------
+.. {es} Hadoop: {hadoop-ref}/install.html[install instructions]
+.. {es}: {ref}/setup-upgrade.html[upgrade instructions]
+.. Kibana: {kibana-ref}/upgrade.html[upgrade instructions]
+.. Logstash: {logstash-ref}/upgrading-logstash.html[upgrade instructions]
+.. Beats: {beats-ref}/upgrading.html[upgrade instructions]
+.. APM Server: {apm-server-ref}/upgrading.html[upgrade instructions]
 
-. Use the Upgrade Assistant in Kibana to
-view incompatibilities that you need to fix, identify any 2.x indices that
-need to be migrated or deleted, and upgrade the internal indices to the
-{major-version} index format.
-+
-You can also call the Elasticsearch migration APIs directly:
-+
-`/_xpack/migration/deprecations`:: Retrieves information about cluster-, node-,
-and index-level settings that use deprecated features.
-+
-`/_xpack/migration/assistance`:: deprecated[6.7.0] Returns a list of indices
-that need to be reindexed before you can upgrade to {version}.
-+
-`/_xpack/migration/upgrade`:: deprecated[6.7.0] Upgrades the indices for the
-{watcher} and {security-features} to a single-type format compatible with
-Elasticsearch 6.x.
-
-. Once you've resolved all of the migration issues, perform
-a {ref}/rolling-upgrades.html[rolling upgrade] from Elasticsearch 5.6 to {version}.
+NOTE: Logstash 6.7 and Beats 6.7 are compatible with all 7.x versions of
+{es}. This provides flexibility in when you schedule the upgrades
+for your Logstash instances and Beats agents, but we recommend upgrading as
+soon as possible to take advantage of performance improvements
+and other enhancements.
 
 [[oss-stack-upgrade]]
-=== Upgrading from a pre-5.6 installation
-
-It is possible to upgrade directly to {major-version} from a pre-5.6 installation,
-but it requires a {ref}/restart-upgrade.html[full cluster restart] and you must
-manually reindex any 2.x indices you need to carry forward to {major-version}.
-
-IMPORTANT: If you use Kibana or {xpack}, you also need to upgrade the
-internal Kibana and {xpack} indices. For information about upgrading them
-after you install Elasticsearch {version}, see
-<<upgrade-internal-indices, Upgrading internal indices>>.
-
-To manually reindex a 2.x index:
-
-. Create an index with 6.x compatible mappings.
-. Use the {ref}/docs-reindex.html[reindex API] to copy documents from the
-2.x index into the new index. You can use a script to perform any necessary
-modifications to the document data and metadata during reindexing.
-. Use the {ref}/indices-aliases.html[_aliases] API to add the name of the 2.x
-index as alias for the new index and delete the 2.x index.
-
-[[upgrade-internal-indices]]
-==== Upgrading internal indices for {major-version}
-
-The format used for the internal indices used by Kibana and {xpack} has
-changed in {major-version}. Before you can run Kibana and {xpack} in {version},
-these indices must be upgraded to the new format. If you are upgrading from a
-version prior to 5.6, you must upgrade them after after installing
-Elasticsearch {version}.
-
-To get a list of the indices that need to be upgraded, use the
-{ref}/migration-api-deprecation.html[deprecation info API]:
-
-[source,json]
-----------------------------------------------------------
-GET /_xpack/migration/deprecations
-----------------------------------------------------------
-// CONSOLE
-
-To upgrade the `.security` index:
+=== Upgrading from 6.6 or earlier
 
-. On a single node, add a temporary superuser account to the `file` realm.
-. Use the {kib} Upgrade Assistant to upgrade the security index, submitting the
-request with the credentials for the temporary superuser. Alternatively, you can
-{ref}/reindex-upgrade.html[reindex manually].
+To upgrade directly to {es} {version} from versions 6.0-6.6, you must
+{ref}/reindex-upgrade.html[manually reindex] any 5.x indices you need to
+carry forward, and perform a {ref}/restart-upgrade.html[full cluster restart].
+This includes any internal indices created in 5.x, such as the `.kibana` and
+`.security*` indices.
 
-. Delete the temporary superuser account from the file realm.
+Make sure all 5.x indices have been deleted before upgrading to {version}. {es}
+{version} will fail to start if any 5.x indices are present.
 
-You can use your regular administration credentials to upgrade the other
-internal indices.
+If you are running a version prior to 6.0,
+https://www.elastic.co/guide/en/elastic-stack/6.7/upgrading-elastic-stack.html[upgrade to 6.7]
+and reindex your old indices or bring up a new {version} cluster and
+{ref}/reindex-upgrade-remote.html[reindex from remote].
 
-TIP: Once you upgrade the `.kibana` index, you can run Kibana and use the
-{xpack} Reindex Helper UI to upgrade the other indices.
+The recommended path is to upgrade to 6.7 before upgrading to {version}. This
+makes it easier to identify the changes you need to make to upgrade and enables
+you to perform a rolling upgrade with no downtime.
 
 [[upgrade-elastic-stack-for-elastic-cloud]]
 === Upgrading on Elastic Cloud
 
 A single click in the Elastic Cloud console can upgrade a cluster to a newer
 version, add more processing capacity, change plugins, and enable or disable
 high availability, all at the same time. During the upgrade process,
-Elasticsearch, Kibana, {xpack} and the officially included plugins are
+{es}, Kibana, {xpack} and the officially included plugins are
 upgraded simultaneously.
 
 Although upgrading your Elastic Cloud clusters is easy, you still need to
 address breaking changes that affect your application. Minor version upgrades,
-upgrades from 5.6 to {major-version}, and all other cluster configuration
+upgrades from 6.7 to {version}, and all other cluster configuration
 changes can be performed with no downtime.
 
 To avoid downtime when a full cluster restart is required:
 
-. Provision an additional cluster with the new Elasticsearch version, reindex
+. Provision an additional cluster with the new {es} version, reindex
 your data, and send index requests to both clusters temporarily.
 
 . Verify that the new cluster performs as expected, fix any problems, and then
@@ -276,7 +120,7 @@ permanently swap in the new cluster.
 only for the time that the new cluster runs in parallel with your old cluster.
 Usage is billed on an hourly basis.
 
-To learn more about the upgrade process on Elastic Cloud, see 
+To learn more about the upgrade process on Elastic Cloud, see
 {cloud}/ec-upgrade-cluster.html[Upgrade versions].
 
 NOTE: Elastic Cloud only supports upgrades to released versions. Preview