Node repurpose tool docs

Improved documentation for node repurpose tool.

Author: henningandersen

docs/reference/commands/node-tool.asciidoc

@@ -1,23 +1,85 @@
 [[node-tool]]
 == elasticsearch-node
 
-The `elasticsearch-node` command enables you to perform unsafe operations that
-risk data loss but which may help to recover some data in a disaster and
-repurpose nodes into new roles.
+The `elasticsearch-node` command enables you to perform certain unsafe
+operations on a node that are only possible while it is shut down. This command
+allows you to adjust the <<modules-node,role>> of a node and may be able to
+recover some data after a disaster.
 
 [float]
 === Synopsis
 
 [source,shell]
 --------------------------------------------------
-bin/elasticsearch-node unsafe-bootstrap|detach-cluster|repurpose
+bin/elasticsearch-node repurpose|unsafe-bootstrap|detach-cluster
   [--ordinal <Integer>] [-E <KeyValuePair>]
   [-h, --help] ([-s, --silent] | [-v, --verbose])
 --------------------------------------------------
 
 [float]
 === Description
 
+This tool has three modes, described in the following sections:
+
+* `elasticsearch-node repurpose` reports on and performs necessary index meta-data
+  and shard data cleanup when a node is repurposed to no longer hold <<data-node,data>> or be
+  <<master-node,master-eligible>>.
+
+* `elasticsearch-node unsafe-bootstrap` can be used to perform _unsafe cluster bootstrapping_.
+  It forces one of the nodes to form a brand-new cluster on its own, using its local copy of
+  the cluster metadata.
+
+* `elasticsearch-node detach-cluster` enables you to move nodes from one cluster to another.
+  This can be used to move nodes into a new cluster created with the `elasticsearch-node unsafe-bootstap`
+  command. If unsafe cluster bootstrapping was not possible, it also enables you to move nodes
+  into a brand-new cluster.
+
+[[node-tool-repurpose]]
+[float]
+==== Changing the role of a node
+
+Changing the role of a node can lead to excess on-disk data that will prevent a node from starting. The
+`elasticsearch-node` tool should be used to cleanup excess data after certain role changes, detailed in the following.
+
+Two types of on-disk index data can need cleanup:
+
+* Index metadata: information about indices like name, mapping and more.
+* Shard data: primary or replica shards allocated to this node.
+
+Repurposing an existing node by changing node.master or node.data to false can leave lingering on-disk metadata and
+shard data around, which will not be accessible by the node's new role. Beside storing non-accessible data, this could lead
+to situations where <<modules-gateway-dangling-indices,dangling indices>> are imported even though the node might not
+be able to host any shards, leading to a red cluster health.  To avoid this,
+
+* nodes with on-disk shard data and `node.data` set to false will refuse to start
+* nodes with on-disk index/shard data and both `node.master` and `node.data set` to false will refuse to start
+
+The `elasticsearch-node repurpose` tool allows cleaning up such excess data, enabling nodes to start
+after being repurposed. The cleanup that will be done are:
+
+* `node.data=false` and `node.master=false`: both index metadata and shard data will be cleaned up.
+* `node.data=false` and `node.master=true` : only shard data will be cleaned up.
+* Any other combinations: no cleanup necessary (will be reported by the tool).
+
+[WARNING]
+Running this command can lead to data loss for the indices mentioned if the data
+contained is not available on other nodes in the cluster. Only run this tool if you
+understand and accept the possible consequences.
+
+The intended use is:
+
+. Stop the node
+. Update node settings (node.master/node.data) to desired new role
+. Run `elasticsearch-node repurpose` on the node
+. Start the node
+
+The tool provides a summary of the the cleanup necessary and asks for confirmation before
+making any changes. Use the verbose (-v) option to get detailed information on indices and
+shards affected.
+
+[float]
+==== Recovering data after a disaster
+
 Sometimes {es} nodes are temporarily stopped, perhaps because of the need to
 perform some maintenance activity or perhaps because of a hardware failure.
 After you resolve the temporary condition and restart the node,
@@ -54,31 +116,10 @@ way forward that does not risk data loss, but it may be possible to use the
 `elasticsearch-node` tool to construct a new cluster that contains some of the
 data from the failed cluster.
 
-If a node that previously held data or were master-eligible is repurposed into
-being an either no-master or no-data node, cleanup of data (e.g. shard data) that
-is no longer needed on the node is necessary to avoid future problems. The
-`elasticsearch-node` tool should be used to do such cleanup.
-
-This tool has three modes:
-
-* `elasticsearch-node unsafe-bootstap` can be used if there is at least one
-  remaining master-eligible node. It forces one of the remaining nodes to form
-  a brand-new cluster on its own, using its local copy of the cluster metadata.
-  This is known as _unsafe cluster bootstrapping_.
-
-* `elasticsearch-node detach-cluster` enables you to move nodes from one cluster
-  to another. This can be used to move nodes into the new cluster created with
-  the `elastisearch-node unsafe-bootstap` command. If unsafe cluster bootstrapping was not
-  possible, it also enables you to
-  move nodes into a brand-new cluster.
-
-* `elasticsearch-node repurpose` reports on and performs necessary index meta-data
-  and shard data cleanup when a node is repurposed to not hold data or be
-  master-eligible.
 
 [[node-tool-unsafe-bootstrap]]
 [float]
-==== Unsafe cluster bootstrapping
+===== Unsafe cluster bootstrapping
 
 If there is at least one remaining master-eligible node, but it is not possible
 to restart a majority of them, then the `elasticsearch-node unsafe-bootstrap`
@@ -153,7 +194,7 @@ job.
 
 [[node-tool-detach-cluster]]
 [float]
-==== Detaching nodes from their cluster
+===== Detaching nodes from their cluster
 
 It is unsafe for nodes to move between clusters, because different clusters
 have completely different cluster metadata. There is no way to safely merge the
@@ -217,54 +258,17 @@ that there has been no data loss, it just means that tool was able to complete
 its job.
 
 
-[[node-tool-repurpose]]
-[float]
-==== Repurposing nodes
-
-A master-eligible and data carrying node holds on-disk index data:
-
-* Index metadata: information about indices like name, mapping and more.
-* Shard data: primary or replica shards allocated to this node.
-
-Repurposing an existing node by changing node.master or node.data to false can leave lingering on-disk metadata and
-data around, which will not be accessible by the node's new role. Beside storing non-accessible data, this could lead
-to situations where <<modules-gateway-dangling-indices,dangling indices>> are imported even though the node might not
-be able to host any shards, leading to a red cluster health.  To avoid this,
-
-* nodes with on-disk shard data and node.data set to false will refuse to start
-* nodes with on-disk index/shard data and both node.master and node.data set to false will refuse to start
-
-The `elasticsearch-node repurpose` tool allows cleaning up such excess data, enabling nodes to start
-after being repurposed.
-
-[WARNING]
-Running this command can lead to data loss for the indices mentioned if the data
-contained is not available on other nodes in the cluster. Only run this tool if you
-understand and accept the possible consequences.
-
-The intended use is:
-
-. Stop the node
-. Update node settings (node.master/node.data) to desired new role
-. Run `elasticsearch-node repurpose` on the node
-. Start the node
-
-The tool provides a summary of the the cleanup necessary and asks for confirmation before
-making any changes. Use the verbose (-v) option to get detailed information on indices and
-shards affected.
-
-
 [float]
 === Parameters
 
+`repurpose`:: Report on and cleanup excess persisted data.
+
 `unsafe-bootstrap`:: Specifies to unsafely bootstrap this node as a new
 one-node cluster.
 
 `detach-cluster`:: Specifies to unsafely detach this node from its cluster so
 it can join a different cluster.
 
-`repurpose`:: Report on and cleanup excess persisted data.
-
 `--ordinal <Integer>`:: If there is <<max-local-storage-nodes,more than one
 node sharing a data path>> then this specifies which node to target. Defaults
 to `0`, meaning to use the first node in the data path.
@@ -280,6 +284,49 @@ to `0`, meaning to use the first node in the data path.
 [float]
 === Examples
 
+[float]
+==== Repurposing node as master, no-data
+
+In this example, a node that previously held data is repurposed as master, no-data.
+First update the node settings to `node.master=true` and `node.data=false`. Then run the
+`elasticsearch-node repurpose` command to check for and remove excess shard data:
+
+[source,txt]
+----
+node$ ./bin/elasticsearch-node repurpose
+
+    WARNING: Elasticsearch MUST be stopped before running this tool.
+
+Found 2 shards in 2 indices to clean up
+Use -v to see list of paths and indices affected
+Node is being re-purposed as master and no-data. Clean-up of shard data will be performed.
+Do you want to proceed?
+Confirm [y/N] y
+Node successfully repurposed to master and no-data.
+----
+
+[float]
+==== Repurposing node as no-master, no-data
+
+In this example, a node that previously held data is repurposed as no-master, no-data.
+First update the node settings to `node.master=true` and `node.data=false`. Then run the
+`elasticsearch-node repurpose` command to check for and remove excess index meta-data and
+shard data:
+
+[source,txt]
+----
+node$./bin/elasticsearch-node repurpose
+
+    WARNING: Elasticsearch MUST be stopped before running this tool.
+
+Found 2 indices (2 shards and 2 index meta data) to clean up
+Use -v to see list of paths and indices affected
+Node is being re-purposed as no-master and no-data. Clean-up of index data will be performed.
+Do you want to proceed?
+Confirm [y/N] y
+Node successfully repurposed to no-master and no-data.
+----
+
 [float]
 ==== Unsafe cluster bootstrapping
 
@@ -381,46 +428,3 @@ Do you want to proceed?
 Confirm [y/N] y
 Node was successfully detached from the cluster
 ----
-
-[float]
-==== Repurposing node as master, no-data
-
-In this example, a node that previously held data is repurposed as master, no-data.
-First update the node settings to `node.master=true` and `node.data=false`. Then run the
-`elasticsearch-node repurpose` command to check for and remove excess shard data:
-
-[source,txt]
-----
-node$ ./bin/elasticsearch-node repurpose
-
-    WARNING: Elasticsearch MUST be stopped before running this tool.
-
-Found 2 shards in 2 indices to clean up
-Use -v to see list of paths and indices affected
-Node is being re-purposed as master and no-data. Clean-up of shard data will be performed.
-Do you want to proceed?
-Confirm [y/N] y
-Node successfully repurposed to master and no-data.
-----
-
-[float]
-==== Repurposing node as no-master, no-data
-
-In this example, a node that previously held data is repurposed as no-master, no-data.
-First update the node settings to `node.master=true` and `node.data=false`. Then run the
-`elasticsearch-node repurpose` command to check for and remove excess index meta-data and
-shard data:
-
-[source,txt]
-----
-node$./bin/elasticsearch-node repurpose
-
-    WARNING: Elasticsearch MUST be stopped before running this tool.
-
-Found 2 indices (2 shards and 2 index meta data) to clean up
-Use -v to see list of paths and indices affected
-Node is being re-purposed as no-master and no-data. Clean-up of index data will be performed.
-Do you want to proceed?
-Confirm [y/N] y
-Node successfully repurposed to no-master and no-data.
-----

docs/reference/modules/node.asciidoc

@@ -204,6 +204,15 @@ NOTE: These settings apply only when {xpack} is not installed. To create a
 dedicated coordinating node when {xpack} is installed, see <<modules-node-xpack,{xpack} node settings>>.
 endif::include-xpack[]
 
+[float]
+[[change-node-role]]
+=== Changing the role of a node
+
+Changing the role of a node will in some cases require cleanup of on disk data. A node will refuse to start if it has
+excess shard data or index metadata. The <<node-tool-repurpose,`elasticsearch-node repurpose`>> tool
+should be used to perform the necessary cleanup. For more details on the cleanups necessary and how to use the tool, see
+<<node-tool-repurpose,`elasticsearch-node repurpose`>>.
+
 [float]
 == Node data path settings
 

server/src/main/java/org/elasticsearch/cluster/coordination/NodeToolCli.java

@@ -36,9 +36,9 @@ public NodeToolCli() {
         super("A CLI tool to do unsafe cluster and index manipulations on current node",
             ()->{});
         CommandLoggingConfigurator.configureLoggingWithoutConfig();
+        subcommands.put("repurpose", new NodeRepurposeCommand());
         subcommands.put("unsafe-bootstrap", new UnsafeBootstrapMasterCommand());
         subcommands.put("detach-cluster", new DetachClusterCommand());
-        subcommands.put("repurpose", new NodeRepurposeCommand());
     }
 
     public static void main(String[] args) throws Exception {