From 02dd1d3f5cd5c1323ac13f06c27356e5cb12456f Mon Sep 17 00:00:00 2001 From: Henning Andersen Date: Wed, 27 Mar 2019 14:31:50 +0100 Subject: [PATCH 1/7] Node repurpose tool docs Added documentation for node repurpose tool. --- docs/reference/commands/node-tool.asciidoc | 102 ++++++++++++++++++++- 1 file changed, 97 insertions(+), 5 deletions(-) diff --git a/docs/reference/commands/node-tool.asciidoc b/docs/reference/commands/node-tool.asciidoc index d7ed054f47599..97ad1cde6bd25 100644 --- a/docs/reference/commands/node-tool.asciidoc +++ b/docs/reference/commands/node-tool.asciidoc @@ -2,14 +2,15 @@ == elasticsearch-node The `elasticsearch-node` command enables you to perform unsafe operations that -risk data loss but which may help to recover some data in a disaster. +risk data loss but which may help to recover some data in a disaster and +repurpose nodes into new roles. [float] === Synopsis [source,shell] -------------------------------------------------- -bin/elasticsearch-node unsafe-bootstrap|detach-cluster +bin/elasticsearch-node unsafe-bootstrap|detach-cluster|repurpose [--ordinal ] [-E ] [-h, --help] ([-s, --silent] | [-v, --verbose]) -------------------------------------------------- @@ -53,19 +54,28 @@ way forward that does not risk data loss, but it may be possible to use the `elasticsearch-node` tool to construct a new cluster that contains some of the data from the failed cluster. -This tool has two modes: +If a node that previously held data or were master-eligible is repurposed into +being an either no-master or no-data node, cleanup of data (e.g. shard data) that +is no longer needed on the node is necessary to avoid future problems. The +`elasticsearch-node` tool should be used to do such cleanup. -* `elastisearch-node unsafe-bootstap` can be used if there is at least one +This tool has three modes: + +* `elasticsearch-node unsafe-bootstap` can be used if there is at least one remaining master-eligible node. It forces one of the remaining nodes to form a brand-new cluster on its own, using its local copy of the cluster metadata. This is known as _unsafe cluster bootstrapping_. -* `elastisearch-node detach-cluster` enables you to move nodes from one cluster +* `elasticsearch-node detach-cluster` enables you to move nodes from one cluster to another. This can be used to move nodes into the new cluster created with the `elastisearch-node unsafe-bootstap` command. If unsafe cluster bootstrapping was not possible, it also enables you to move nodes into a brand-new cluster. +* `elasticsearch-node repurpose` reports on and performs necessary index meta-data + and shard data cleanup when a node is repurposed to not hold data or be + master-eligible. + [[node-tool-unsafe-bootstrap]] [float] ==== Unsafe cluster bootstrapping @@ -206,6 +216,44 @@ The message `Node was successfully detached from the cluster` does not mean that there has been no data loss, it just means that tool was able to complete its job. + +[[node-tool-repurpose]] +[float] +==== Repurposing nodes + +A master-eligible and data carrying node holds on-disk index data: + +* Index metadata: information about indices like name, mapping and more. +* Shard data: primary or replica shards allocated to this node. + +Repurposing an existing node by changing node.master or node.data to false can leave lingering on-disk metadata and +data around, which will not be accessible by the node's new role. Beside storing non-accessible data, this could lead +to situations where <> are imported even though the node might not +be able to host any shards, leading to a red cluster health. To avoid this, + +* nodes with on-disk shard data and node.data set to false will refuse to start +* nodes with on-disk index/shard data and both node.master and node.data set to false will refuse to start + +The `elasticsearch-node repurpose` tool allows cleaning up such excess data, enabling nodes to start +after being repurposed. + +[WARNING] +Running this command can lead to data loss for the indices mentioned if the data +contained is not available on other nodes in the cluster. Only run this tool if you +understand and accept the possible consequences. + +The intended use is: + +. Stop the node +. Update node settings (node.master/node.data) to desired new role +. Run `elasticsearch-node repurpose` on the node +. Start the node + +The tool provides a summary of the the cleanup necessary and asks for confirmation before +making any changes. Use the verbose (-v) option to get detailed information on indices and +shards affected. + + [float] === Parameters @@ -215,6 +263,8 @@ one-node cluster. `detach-cluster`:: Specifies to unsafely detach this node from its cluster so it can join a different cluster. +`repurpose`:: Report on and cleanup excess persisted data. + `--ordinal `:: If there is <> then this specifies which node to target. Defaults to `0`, meaning to use the first node in the data path. @@ -332,3 +382,45 @@ Confirm [y/N] y Node was successfully detached from the cluster ---- +[float] +==== Repurposing node as master, no-data + +In this example, a node that previously held data is repurposed as master, no-data. +First update the node settings to `node.master=true` and `node.data=false`. Then run the +`elasticsearch-node repurpose` command to check for and remove excess shard data: + +[source,txt] +---- +node$ ./bin/elasticsearch-node repurpose + + WARNING: Elasticsearch MUST be stopped before running this tool. + +Found 2 shards in 2 indices to clean up +Use -v to see list of paths and indices affected +Node is being re-purposed as master and no-data. Clean-up of shard data will be performed. +Do you want to proceed? +Confirm [y/N] y +Node successfully repurposed to master and no-data. +---- + +[float] +==== Repurposing node as no-master, no-data + +In this example, a node that previously held data is repurposed as no-master, no-data. +First update the node settings to `node.master=true` and `node.data=false`. Then run the +`elasticsearch-node repurpose` command to check for and remove excess index meta-data and +shard data: + +[source,txt] +---- +node$./bin/elasticsearch-node repurpose + + WARNING: Elasticsearch MUST be stopped before running this tool. + +Found 2 indices (2 shards and 2 index meta data) to clean up +Use -v to see list of paths and indices affected +Node is being re-purposed as no-master and no-data. Clean-up of index data will be performed. +Do you want to proceed? +Confirm [y/N] y +Node successfully repurposed to no-master and no-data. +---- From a09c826586175ed64472316af5fa5264795282d7 Mon Sep 17 00:00:00 2001 From: Henning Andersen Date: Fri, 29 Mar 2019 17:30:50 +0100 Subject: [PATCH 2/7] Node repurpose tool docs Improved documentation for node repurpose tool. --- docs/reference/commands/node-tool.asciidoc | 222 +++++++++--------- docs/reference/modules/node.asciidoc | 9 + .../cluster/coordination/NodeToolCli.java | 2 +- 3 files changed, 123 insertions(+), 110 deletions(-) diff --git a/docs/reference/commands/node-tool.asciidoc b/docs/reference/commands/node-tool.asciidoc index 97ad1cde6bd25..5030ea6156bd5 100644 --- a/docs/reference/commands/node-tool.asciidoc +++ b/docs/reference/commands/node-tool.asciidoc @@ -1,16 +1,17 @@ [[node-tool]] == elasticsearch-node -The `elasticsearch-node` command enables you to perform unsafe operations that -risk data loss but which may help to recover some data in a disaster and -repurpose nodes into new roles. +The `elasticsearch-node` command enables you to perform certain unsafe +operations on a node that are only possible while it is shut down. This command +allows you to adjust the <> of a node and may be able to +recover some data after a disaster. [float] === Synopsis [source,shell] -------------------------------------------------- -bin/elasticsearch-node unsafe-bootstrap|detach-cluster|repurpose +bin/elasticsearch-node repurpose|unsafe-bootstrap|detach-cluster [--ordinal ] [-E ] [-h, --help] ([-s, --silent] | [-v, --verbose]) -------------------------------------------------- @@ -18,6 +19,67 @@ bin/elasticsearch-node unsafe-bootstrap|detach-cluster|repurpose [float] === Description +This tool has three modes, described in the following sections: + +* `elasticsearch-node repurpose` reports on and performs necessary index meta-data + and shard data cleanup when a node is repurposed to no longer hold <> or be + <>. + +* `elasticsearch-node unsafe-bootstrap` can be used to perform _unsafe cluster bootstrapping_. + It forces one of the nodes to form a brand-new cluster on its own, using its local copy of + the cluster metadata. + +* `elasticsearch-node detach-cluster` enables you to move nodes from one cluster to another. + This can be used to move nodes into a new cluster created with the `elasticsearch-node unsafe-bootstap` + command. If unsafe cluster bootstrapping was not possible, it also enables you to move nodes + into a brand-new cluster. + +[[node-tool-repurpose]] +[float] +==== Changing the role of a node + +Changing the role of a node can lead to excess on-disk data that will prevent a node from starting. The +`elasticsearch-node` tool should be used to cleanup excess data after certain role changes, detailed in the following. + +Two types of on-disk index data can need cleanup: + +* Index metadata: information about indices like name, mapping and more. +* Shard data: primary or replica shards allocated to this node. + +Repurposing an existing node by changing node.master or node.data to false can leave lingering on-disk metadata and +shard data around, which will not be accessible by the node's new role. Beside storing non-accessible data, this could lead +to situations where <> are imported even though the node might not +be able to host any shards, leading to a red cluster health. To avoid this, + +* nodes with on-disk shard data and `node.data` set to false will refuse to start +* nodes with on-disk index/shard data and both `node.master` and `node.data set` to false will refuse to start + +The `elasticsearch-node repurpose` tool allows cleaning up such excess data, enabling nodes to start +after being repurposed. The cleanup that will be done are: + +* `node.data=false` and `node.master=false`: both index metadata and shard data will be cleaned up. +* `node.data=false` and `node.master=true` : only shard data will be cleaned up. +* Any other combinations: no cleanup necessary (will be reported by the tool). + +[WARNING] +Running this command can lead to data loss for the indices mentioned if the data +contained is not available on other nodes in the cluster. Only run this tool if you +understand and accept the possible consequences. + +The intended use is: + +. Stop the node +. Update node settings (node.master/node.data) to desired new role +. Run `elasticsearch-node repurpose` on the node +. Start the node + +The tool provides a summary of the the cleanup necessary and asks for confirmation before +making any changes. Use the verbose (-v) option to get detailed information on indices and +shards affected. + +[float] +==== Recovering data after a disaster + Sometimes {es} nodes are temporarily stopped, perhaps because of the need to perform some maintenance activity or perhaps because of a hardware failure. After you resolve the temporary condition and restart the node, @@ -54,31 +116,10 @@ way forward that does not risk data loss, but it may be possible to use the `elasticsearch-node` tool to construct a new cluster that contains some of the data from the failed cluster. -If a node that previously held data or were master-eligible is repurposed into -being an either no-master or no-data node, cleanup of data (e.g. shard data) that -is no longer needed on the node is necessary to avoid future problems. The -`elasticsearch-node` tool should be used to do such cleanup. - -This tool has three modes: - -* `elasticsearch-node unsafe-bootstap` can be used if there is at least one - remaining master-eligible node. It forces one of the remaining nodes to form - a brand-new cluster on its own, using its local copy of the cluster metadata. - This is known as _unsafe cluster bootstrapping_. - -* `elasticsearch-node detach-cluster` enables you to move nodes from one cluster - to another. This can be used to move nodes into the new cluster created with - the `elastisearch-node unsafe-bootstap` command. If unsafe cluster bootstrapping was not - possible, it also enables you to - move nodes into a brand-new cluster. - -* `elasticsearch-node repurpose` reports on and performs necessary index meta-data - and shard data cleanup when a node is repurposed to not hold data or be - master-eligible. [[node-tool-unsafe-bootstrap]] [float] -==== Unsafe cluster bootstrapping +===== Unsafe cluster bootstrapping If there is at least one remaining master-eligible node, but it is not possible to restart a majority of them, then the `elasticsearch-node unsafe-bootstrap` @@ -153,7 +194,7 @@ job. [[node-tool-detach-cluster]] [float] -==== Detaching nodes from their cluster +===== Detaching nodes from their cluster It is unsafe for nodes to move between clusters, because different clusters have completely different cluster metadata. There is no way to safely merge the @@ -217,54 +258,17 @@ that there has been no data loss, it just means that tool was able to complete its job. -[[node-tool-repurpose]] -[float] -==== Repurposing nodes - -A master-eligible and data carrying node holds on-disk index data: - -* Index metadata: information about indices like name, mapping and more. -* Shard data: primary or replica shards allocated to this node. - -Repurposing an existing node by changing node.master or node.data to false can leave lingering on-disk metadata and -data around, which will not be accessible by the node's new role. Beside storing non-accessible data, this could lead -to situations where <> are imported even though the node might not -be able to host any shards, leading to a red cluster health. To avoid this, - -* nodes with on-disk shard data and node.data set to false will refuse to start -* nodes with on-disk index/shard data and both node.master and node.data set to false will refuse to start - -The `elasticsearch-node repurpose` tool allows cleaning up such excess data, enabling nodes to start -after being repurposed. - -[WARNING] -Running this command can lead to data loss for the indices mentioned if the data -contained is not available on other nodes in the cluster. Only run this tool if you -understand and accept the possible consequences. - -The intended use is: - -. Stop the node -. Update node settings (node.master/node.data) to desired new role -. Run `elasticsearch-node repurpose` on the node -. Start the node - -The tool provides a summary of the the cleanup necessary and asks for confirmation before -making any changes. Use the verbose (-v) option to get detailed information on indices and -shards affected. - - [float] === Parameters +`repurpose`:: Report on and cleanup excess persisted data. + `unsafe-bootstrap`:: Specifies to unsafely bootstrap this node as a new one-node cluster. `detach-cluster`:: Specifies to unsafely detach this node from its cluster so it can join a different cluster. -`repurpose`:: Report on and cleanup excess persisted data. - `--ordinal `:: If there is <> then this specifies which node to target. Defaults to `0`, meaning to use the first node in the data path. @@ -280,6 +284,49 @@ to `0`, meaning to use the first node in the data path. [float] === Examples +[float] +==== Repurposing node as master, no-data + +In this example, a node that previously held data is repurposed as master, no-data. +First update the node settings to `node.master=true` and `node.data=false`. Then run the +`elasticsearch-node repurpose` command to check for and remove excess shard data: + +[source,txt] +---- +node$ ./bin/elasticsearch-node repurpose + + WARNING: Elasticsearch MUST be stopped before running this tool. + +Found 2 shards in 2 indices to clean up +Use -v to see list of paths and indices affected +Node is being re-purposed as master and no-data. Clean-up of shard data will be performed. +Do you want to proceed? +Confirm [y/N] y +Node successfully repurposed to master and no-data. +---- + +[float] +==== Repurposing node as no-master, no-data + +In this example, a node that previously held data is repurposed as no-master, no-data. +First update the node settings to `node.master=true` and `node.data=false`. Then run the +`elasticsearch-node repurpose` command to check for and remove excess index meta-data and +shard data: + +[source,txt] +---- +node$./bin/elasticsearch-node repurpose + + WARNING: Elasticsearch MUST be stopped before running this tool. + +Found 2 indices (2 shards and 2 index meta data) to clean up +Use -v to see list of paths and indices affected +Node is being re-purposed as no-master and no-data. Clean-up of index data will be performed. +Do you want to proceed? +Confirm [y/N] y +Node successfully repurposed to no-master and no-data. +---- + [float] ==== Unsafe cluster bootstrapping @@ -381,46 +428,3 @@ Do you want to proceed? Confirm [y/N] y Node was successfully detached from the cluster ---- - -[float] -==== Repurposing node as master, no-data - -In this example, a node that previously held data is repurposed as master, no-data. -First update the node settings to `node.master=true` and `node.data=false`. Then run the -`elasticsearch-node repurpose` command to check for and remove excess shard data: - -[source,txt] ----- -node$ ./bin/elasticsearch-node repurpose - - WARNING: Elasticsearch MUST be stopped before running this tool. - -Found 2 shards in 2 indices to clean up -Use -v to see list of paths and indices affected -Node is being re-purposed as master and no-data. Clean-up of shard data will be performed. -Do you want to proceed? -Confirm [y/N] y -Node successfully repurposed to master and no-data. ----- - -[float] -==== Repurposing node as no-master, no-data - -In this example, a node that previously held data is repurposed as no-master, no-data. -First update the node settings to `node.master=true` and `node.data=false`. Then run the -`elasticsearch-node repurpose` command to check for and remove excess index meta-data and -shard data: - -[source,txt] ----- -node$./bin/elasticsearch-node repurpose - - WARNING: Elasticsearch MUST be stopped before running this tool. - -Found 2 indices (2 shards and 2 index meta data) to clean up -Use -v to see list of paths and indices affected -Node is being re-purposed as no-master and no-data. Clean-up of index data will be performed. -Do you want to proceed? -Confirm [y/N] y -Node successfully repurposed to no-master and no-data. ----- diff --git a/docs/reference/modules/node.asciidoc b/docs/reference/modules/node.asciidoc index e9f171cce8b4a..6c98aeb2f5dd3 100644 --- a/docs/reference/modules/node.asciidoc +++ b/docs/reference/modules/node.asciidoc @@ -204,6 +204,15 @@ NOTE: These settings apply only when {xpack} is not installed. To create a dedicated coordinating node when {xpack} is installed, see <>. endif::include-xpack[] +[float] +[[change-node-role]] +=== Changing the role of a node + +Changing the role of a node will in some cases require cleanup of on disk data. A node will refuse to start if it has +excess shard data or index metadata. The <> tool +should be used to perform the necessary cleanup. For more details on the cleanups necessary and how to use the tool, see +<>. + [float] == Node data path settings diff --git a/server/src/main/java/org/elasticsearch/cluster/coordination/NodeToolCli.java b/server/src/main/java/org/elasticsearch/cluster/coordination/NodeToolCli.java index e9f42062a7ada..d6bd22bcd76fd 100644 --- a/server/src/main/java/org/elasticsearch/cluster/coordination/NodeToolCli.java +++ b/server/src/main/java/org/elasticsearch/cluster/coordination/NodeToolCli.java @@ -36,9 +36,9 @@ public NodeToolCli() { super("A CLI tool to do unsafe cluster and index manipulations on current node", ()->{}); CommandLoggingConfigurator.configureLoggingWithoutConfig(); + subcommands.put("repurpose", new NodeRepurposeCommand()); subcommands.put("unsafe-bootstrap", new UnsafeBootstrapMasterCommand()); subcommands.put("detach-cluster", new DetachClusterCommand()); - subcommands.put("repurpose", new NodeRepurposeCommand()); } public static void main(String[] args) throws Exception { From cda16dbf39719a1fa6ba6cb55e912d55eb264265 Mon Sep 17 00:00:00 2001 From: David Turner Date: Sun, 31 Mar 2019 20:26:05 +0200 Subject: [PATCH 3/7] Node repurpose tool docs Clarifty that changes happen in .yml file. Co-Authored-By: henningandersen <33268011+henningandersen@users.noreply.github.com> --- docs/reference/commands/node-tool.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commands/node-tool.asciidoc b/docs/reference/commands/node-tool.asciidoc index 5030ea6156bd5..fcf5d660d71e9 100644 --- a/docs/reference/commands/node-tool.asciidoc +++ b/docs/reference/commands/node-tool.asciidoc @@ -309,7 +309,7 @@ Node successfully repurposed to master and no-data. ==== Repurposing node as no-master, no-data In this example, a node that previously held data is repurposed as no-master, no-data. -First update the node settings to `node.master=true` and `node.data=false`. Then run the +First update the node's settings to `node.master: false` and `node.data: false` in its `elasticsearch.yml` config file. Then run the `elasticsearch-node repurpose` command to check for and remove excess index meta-data and shard data: From 4d37e7999a798b26ca55dd2f0a50943c43311b4a Mon Sep 17 00:00:00 2001 From: David Turner Date: Sun, 31 Mar 2019 20:26:44 +0200 Subject: [PATCH 4/7] Node repurpose tool docs Clarify that changes happen in .yml file. Co-Authored-By: henningandersen <33268011+henningandersen@users.noreply.github.com> --- docs/reference/commands/node-tool.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commands/node-tool.asciidoc b/docs/reference/commands/node-tool.asciidoc index fcf5d660d71e9..7434f30dc6edd 100644 --- a/docs/reference/commands/node-tool.asciidoc +++ b/docs/reference/commands/node-tool.asciidoc @@ -288,7 +288,7 @@ to `0`, meaning to use the first node in the data path. ==== Repurposing node as master, no-data In this example, a node that previously held data is repurposed as master, no-data. -First update the node settings to `node.master=true` and `node.data=false`. Then run the +First update the node's settings to `node.master: true` and `node.data: false` in its `elasticsearch.yml` config file. Then run the `elasticsearch-node repurpose` command to check for and remove excess shard data: [source,txt] From 57511a54c12510a3110bb6816df8471c8bf7ccc6 Mon Sep 17 00:00:00 2001 From: Henning Andersen Date: Sun, 31 Mar 2019 20:33:29 +0200 Subject: [PATCH 5/7] Node repurpose tool docs Changed terminology a bit to use coordinating-only and dedicated master. Though not entirely true, it gives a slightly easier read. --- docs/reference/commands/node-tool.asciidoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/reference/commands/node-tool.asciidoc b/docs/reference/commands/node-tool.asciidoc index 7434f30dc6edd..4e3816254698b 100644 --- a/docs/reference/commands/node-tool.asciidoc +++ b/docs/reference/commands/node-tool.asciidoc @@ -285,9 +285,9 @@ to `0`, meaning to use the first node in the data path. === Examples [float] -==== Repurposing node as master, no-data +==== Repurposing node as dedicated master (master: true, data: false) -In this example, a node that previously held data is repurposed as master, no-data. +In this example, a node that previously held data is repurposed as a dedicated master node. First update the node's settings to `node.master: true` and `node.data: false` in its `elasticsearch.yml` config file. Then run the `elasticsearch-node repurpose` command to check for and remove excess shard data: @@ -306,9 +306,9 @@ Node successfully repurposed to master and no-data. ---- [float] -==== Repurposing node as no-master, no-data +==== Repurposing node as coordinating-only (master: false, data: false) -In this example, a node that previously held data is repurposed as no-master, no-data. +In this example, a node that previously held data is repurposed as a coordinating-only node. First update the node's settings to `node.master: false` and `node.data: false` in its `elasticsearch.yml` config file. Then run the `elasticsearch-node repurpose` command to check for and remove excess index meta-data and shard data: From 7fefa44523812de15939846cedd3e63bc3bd7e25 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 8 Apr 2019 15:16:15 +0100 Subject: [PATCH 6/7] Rewording --- docs/reference/commands/node-tool.asciidoc | 99 ++++++++++------------ docs/reference/modules/node.asciidoc | 43 +++++++++- 2 files changed, 86 insertions(+), 56 deletions(-) diff --git a/docs/reference/commands/node-tool.asciidoc b/docs/reference/commands/node-tool.asciidoc index 4e3816254698b..68b96116997ad 100644 --- a/docs/reference/commands/node-tool.asciidoc +++ b/docs/reference/commands/node-tool.asciidoc @@ -19,63 +19,56 @@ bin/elasticsearch-node repurpose|unsafe-bootstrap|detach-cluster [float] === Description -This tool has three modes, described in the following sections: +This tool has three modes: -* `elasticsearch-node repurpose` reports on and performs necessary index meta-data - and shard data cleanup when a node is repurposed to no longer hold <> or be - <>. +* `elasticsearch-node repurpose` can be used to delete unwanted metadata from a + node if it used to be a <> or a + <> but has been repurposed not to have one + or other of these roles. -* `elasticsearch-node unsafe-bootstrap` can be used to perform _unsafe cluster bootstrapping_. - It forces one of the nodes to form a brand-new cluster on its own, using its local copy of - the cluster metadata. +* `elasticsearch-node unsafe-bootstrap` can be used to perform _unsafe cluster + bootstrapping_. It forces one of the nodes to form a brand-new cluster on + its own, using its local copy of the cluster metadata. -* `elasticsearch-node detach-cluster` enables you to move nodes from one cluster to another. - This can be used to move nodes into a new cluster created with the `elasticsearch-node unsafe-bootstap` - command. If unsafe cluster bootstrapping was not possible, it also enables you to move nodes +* `elasticsearch-node detach-cluster` enables you to move nodes from one + cluster to another. This can be used to move nodes into a new cluster + created with the `elasticsearch-node unsafe-bootstap` command. If unsafe + cluster bootstrapping was not possible, it also enables you to move nodes into a brand-new cluster. [[node-tool-repurpose]] [float] ==== Changing the role of a node -Changing the role of a node can lead to excess on-disk data that will prevent a node from starting. The -`elasticsearch-node` tool should be used to cleanup excess data after certain role changes, detailed in the following. +There may be situations where you want to repurpose a node without following +the <>. The `elasticsearch-node +repurpose` tool allows you to delete any excess on-disk data and start a node +after repurposing it. -Two types of on-disk index data can need cleanup: - -* Index metadata: information about indices like name, mapping and more. -* Shard data: primary or replica shards allocated to this node. - -Repurposing an existing node by changing node.master or node.data to false can leave lingering on-disk metadata and -shard data around, which will not be accessible by the node's new role. Beside storing non-accessible data, this could lead -to situations where <> are imported even though the node might not -be able to host any shards, leading to a red cluster health. To avoid this, - -* nodes with on-disk shard data and `node.data` set to false will refuse to start -* nodes with on-disk index/shard data and both `node.master` and `node.data set` to false will refuse to start +The intended use is: -The `elasticsearch-node repurpose` tool allows cleaning up such excess data, enabling nodes to start -after being repurposed. The cleanup that will be done are: +* Stop the node +* Update `elasticsearch.yml` by setting `node.master` and `node.data` as + desired. +* Run `elasticsearch-node repurpose` on the node +* Start the node -* `node.data=false` and `node.master=false`: both index metadata and shard data will be cleaned up. -* `node.data=false` and `node.master=true` : only shard data will be cleaned up. -* Any other combinations: no cleanup necessary (will be reported by the tool). +If you run `elasticsearch-node repurpose` on a node with `node.data: false` and +`node.master: true` then it will delete any remaining shard data on that node, +but it will leave the index and cluster metadata alone. If you run +`elasticsearch-node repurpose` on a node with `node.data: false` and +`node.master: false` then it will delete any remaining shard data and index +metadata, but it will leave the cluster metadata alone. [WARNING] -Running this command can lead to data loss for the indices mentioned if the data -contained is not available on other nodes in the cluster. Only run this tool if you -understand and accept the possible consequences. - -The intended use is: - -. Stop the node -. Update node settings (node.master/node.data) to desired new role -. Run `elasticsearch-node repurpose` on the node -. Start the node +Running this command can lead to data loss for the indices mentioned if the +data contained is not available on other nodes in the cluster. Only run this +tool if you understand and accept the possible consequences, and only after +determining that the node cannot be repurposed cleanly. -The tool provides a summary of the the cleanup necessary and asks for confirmation before -making any changes. Use the verbose (-v) option to get detailed information on indices and -shards affected. +The tool provides a summary of the data to be deleted and asks for confirmation +before making any changes. You can get detailed information about the affected +indices and shards by passing the verbose (`-v`) option. [float] ==== Recovering data after a disaster @@ -261,7 +254,7 @@ its job. [float] === Parameters -`repurpose`:: Report on and cleanup excess persisted data. +`repurpose`:: Delete excess data when a node's roles are changed. `unsafe-bootstrap`:: Specifies to unsafely bootstrap this node as a new one-node cluster. @@ -285,11 +278,12 @@ to `0`, meaning to use the first node in the data path. === Examples [float] -==== Repurposing node as dedicated master (master: true, data: false) +==== Repurposing a node as a dedicated master node (master: true, data: false) -In this example, a node that previously held data is repurposed as a dedicated master node. -First update the node's settings to `node.master: true` and `node.data: false` in its `elasticsearch.yml` config file. Then run the -`elasticsearch-node repurpose` command to check for and remove excess shard data: +In this example, a former data node is repurposed as a dedicated master node. +First update the node's settings to `node.master: true` and `node.data: false` +in its `elasticsearch.yml` config file. Then run the `elasticsearch-node +repurpose` command to find and remove excess shard data: [source,txt] ---- @@ -306,12 +300,13 @@ Node successfully repurposed to master and no-data. ---- [float] -==== Repurposing node as coordinating-only (master: false, data: false) +==== Repurposing a node as a coordinating-only node (master: false, data: false) -In this example, a node that previously held data is repurposed as a coordinating-only node. -First update the node's settings to `node.master: false` and `node.data: false` in its `elasticsearch.yml` config file. Then run the -`elasticsearch-node repurpose` command to check for and remove excess index meta-data and -shard data: +In this example, a node that previously held data is repurposed as a +coordinating-only node. First update the node's settings to `node.master: +false` and `node.data: false` in its `elasticsearch.yml` config file. Then run +the `elasticsearch-node repurpose` command to find and remove excess shard data +and index metadata: [source,txt] ---- diff --git a/docs/reference/modules/node.asciidoc b/docs/reference/modules/node.asciidoc index 6c98aeb2f5dd3..c2496ba270ba3 100644 --- a/docs/reference/modules/node.asciidoc +++ b/docs/reference/modules/node.asciidoc @@ -208,10 +208,45 @@ endif::include-xpack[] [[change-node-role]] === Changing the role of a node -Changing the role of a node will in some cases require cleanup of on disk data. A node will refuse to start if it has -excess shard data or index metadata. The <> tool -should be used to perform the necessary cleanup. For more details on the cleanups necessary and how to use the tool, see -<>. +Each data node maintains the following data on disk: + +* the shard data for every shard allocated to that node, +* the index metadata corresponding with every shard allocated to that node, and +* the cluster-wide metadata, such as settings and index templates. + +Similarly, each master-eligible node maintains the following data on disk: + +* the index metadata for every index in the cluster, and +* the cluster-wide metadata, such as settings and index templates. + +Each node checks the contents of its data path at startup. If it discovers +unexpected data then it will refuse to start. This is to avoid importing +unwanted <> which can lead +to a red cluster health. To be more precise, nodes with `node.data: false` will +refuse to start if they find any shard data on disk at startup, and nodes with +both `node.master: false` and `node.data: false` will refuse to start if they +have any index metadata on disk at startup. +// TODO what if there's cluster metadata on disk in this second case? + +It is possible to change the roles of a node by adjusting its +`elasticsearch.yml` file and restarting it. This is known as _repurposing_ a +node. In order to satisfy the checks for unexpected data described above, you +must perform some extra steps to prepare a node for repurposing when setting +its `node.data` or `node.master` roles to `false`: + +* If you want to repurpose a data node by changing `node.data` to `false` then + you should first use an <> to safely + migrate all the shard data onto other nodes in the cluster. + +* If you want to repurpose a node to have both `node.master: false` and + `node.data: false` then it is simplest to start a brand-new node with an + empty data path and the desired roles. You may find it safest to use an + <> to migrate the shard data + elsewhere in the cluster first. + +If it is not possible to follow these extra steps then you may be able to use +the <> tool to delete any +excess data that prevents a node from starting. [float] == Node data path settings From 514831dfa6556f3f8435166c5537a3057ae6a962 Mon Sep 17 00:00:00 2001 From: Henning Andersen Date: Tue, 9 Apr 2019 10:13:37 +0200 Subject: [PATCH 7/7] Node repurpose tool docs Removed todo and changed metadata to data to also capture shard data. --- docs/reference/commands/node-tool.asciidoc | 2 +- docs/reference/modules/node.asciidoc | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/reference/commands/node-tool.asciidoc b/docs/reference/commands/node-tool.asciidoc index 68b96116997ad..f070d11aa8fb0 100644 --- a/docs/reference/commands/node-tool.asciidoc +++ b/docs/reference/commands/node-tool.asciidoc @@ -21,7 +21,7 @@ bin/elasticsearch-node repurpose|unsafe-bootstrap|detach-cluster This tool has three modes: -* `elasticsearch-node repurpose` can be used to delete unwanted metadata from a +* `elasticsearch-node repurpose` can be used to delete unwanted data from a node if it used to be a <> or a <> but has been repurposed not to have one or other of these roles. diff --git a/docs/reference/modules/node.asciidoc b/docs/reference/modules/node.asciidoc index c2496ba270ba3..f988e97ef553c 100644 --- a/docs/reference/modules/node.asciidoc +++ b/docs/reference/modules/node.asciidoc @@ -226,7 +226,6 @@ to a red cluster health. To be more precise, nodes with `node.data: false` will refuse to start if they find any shard data on disk at startup, and nodes with both `node.master: false` and `node.data: false` will refuse to start if they have any index metadata on disk at startup. -// TODO what if there's cluster metadata on disk in this second case? It is possible to change the roles of a node by adjusting its `elasticsearch.yml` file and restarting it. This is known as _repurposing_ a