From f11948b3bb01465754a17f49e2489f3e6fe75e7b Mon Sep 17 00:00:00 2001 From: Henning Andersen Date: Thu, 18 Mar 2021 10:19:21 +0100 Subject: [PATCH 1/2] [DOCS] Frozen tier dedicated The frozen tier is now dedicated for searchable snapshots mounted with the `shared_cache` option. This commit adjusts docs accordingly. --- docs/reference/datatiers.asciidoc | 12 ++++--- .../ilm/actions/ilm-allocate.asciidoc | 28 +++++++-------- .../reference/ilm/actions/ilm-freeze.asciidoc | 2 +- .../ilm/actions/ilm-migrate.asciidoc | 9 ++--- .../ilm/actions/ilm-readonly.asciidoc | 2 +- .../ilm/actions/ilm-set-priority.asciidoc | 12 +++---- .../ilm/ilm-index-lifecycle.asciidoc | 35 ++++++++----------- .../allocation/data_tier_allocation.asciidoc | 4 ++- docs/reference/modules/node.asciidoc | 6 ++-- 9 files changed, 54 insertions(+), 56 deletions(-) diff --git a/docs/reference/datatiers.asciidoc b/docs/reference/datatiers.asciidoc index 8daa3538ab578..05928fde27c46 100644 --- a/docs/reference/datatiers.asciidoc +++ b/docs/reference/datatiers.asciidoc @@ -11,7 +11,7 @@ and hold your most recent, most-frequently-accessed data. * <> nodes hold time series data that is accessed less-frequently and rarely needs to be updated. * <> nodes hold time series data that is accessed infrequently and not normally updated. -* <> nodes hold time series data that is accessed rarely and never updated. +* <> nodes hold time series data that is accessed rarely and never updated, kept in searchable snapshots. When you index documents directly to a specific index, they remain on content tier nodes indefinitely. @@ -85,12 +85,14 @@ For resiliency, indices in the cold tier can rely on [[frozen-tier]] === Frozen tier +experimental::[] + Once data is no longer being queried, or being queried rarely, it may move from the cold tier to the frozen tier where it stays for the rest of its life. -The frozen tier is a less responsive query tier than the cold tier, and data in the frozen tier is -not normally updated. As data transitions into the frozen tier it can be compressed and shrunken. -For resiliency, indices in the frozen tier can rely on <>, eliminating the need for replicas or even a local copy. +The frozen tier is a less responsive query tier than the cold tier, and data in the frozen tier +cannot be updated. The frozen tier holds searchable snapshots mounted using the +`shared_cache` storage option exclusively. The <> automatically converts data +transitioning into the frozen tier into a searchable snapshot, eliminating the need for replicas or even a local copy. [discrete] [[data-tier-allocation]] diff --git a/docs/reference/ilm/actions/ilm-allocate.asciidoc b/docs/reference/ilm/actions/ilm-allocate.asciidoc index 16ac105bfae73..fbb9a7e075786 100644 --- a/docs/reference/ilm/actions/ilm-allocate.asciidoc +++ b/docs/reference/ilm/actions/ilm-allocate.asciidoc @@ -2,17 +2,17 @@ [[ilm-allocate]] === Allocate -Phases allowed: warm, cold, frozen. +Phases allowed: warm, cold. Updates the index settings to change which nodes are allowed to host the index shards and change the number of replicas. -The allocate action is not allowed in the hot phase. -The initial allocation for the index must be done manually or via +The allocate action is not allowed in the hot phase. +The initial allocation for the index must be done manually or via <>. -You can configure this action to modify both the allocation rules and number of replicas, -only the allocation rules, or only the number of replicas. +You can configure this action to modify both the allocation rules and number of replicas, +only the allocation rules, or only the number of replicas. For more information about how {es} uses replicas for scaling, see <>. See <> for more information about controlling where {es} allocates shards of a particular index. @@ -21,11 +21,11 @@ controlling where {es} allocates shards of a particular index. [[ilm-allocate-options]] ==== Options -You must specify the number of replicas or at least one -`include`, `exclude`, or `require` option. +You must specify the number of replicas or at least one +`include`, `exclude`, or `require` option. An empty allocate action is invalid. -For more information about using custom attributes for shard allocation, +For more information about using custom attributes for shard allocation, see <>. `number_of_replicas`:: @@ -47,7 +47,7 @@ Assigns an index to nodes that have _all_ of the specified custom attributes. [[ilm-allocate-ex]] ==== Example -The allocate action in the following policy changes the index's number of replicas to `2`. +The allocate action in the following policy changes the index's number of replicas to `2`. The index allocation rules are not changed. [source,console] @@ -71,11 +71,11 @@ PUT _ilm/policy/my_policy [[ilm-allocate-assign-index-attribute-ex]] ===== Assign index to nodes using a custom attribute -The allocate action in the following policy assigns the index to nodes +The allocate action in the following policy assigns the index to nodes that have a `box_type` of _hot_ or _warm_. To designate a node's `box_type`, you set a custom attribute in the node configuration. -For example, set `node.attr.box_type: hot` in `elasticsearch.yml`. +For example, set `node.attr.box_type: hot` in `elasticsearch.yml`. For more information, see <>. [source,console] @@ -129,11 +129,11 @@ PUT _ilm/policy/my_policy [[ilm-allocate-assign-index-node-ex]] ===== Assign index to a specific node and update replica settings -The allocate action in the following policy updates the index to have one replica per shard -and be allocated to nodes that have a `box_type` of _cold_. +The allocate action in the following policy updates the index to have one replica per shard +and be allocated to nodes that have a `box_type` of _cold_. To designate a node's `box_type`, you set a custom attribute in the node configuration. -For example, set `node.attr.box_type: cold` in `elasticsearch.yml`. +For example, set `node.attr.box_type: cold` in `elasticsearch.yml`. For more information, see <>. [source,console] diff --git a/docs/reference/ilm/actions/ilm-freeze.asciidoc b/docs/reference/ilm/actions/ilm-freeze.asciidoc index d3da5f3ccd0f0..abc5b4ce4a6cc 100644 --- a/docs/reference/ilm/actions/ilm-freeze.asciidoc +++ b/docs/reference/ilm/actions/ilm-freeze.asciidoc @@ -2,7 +2,7 @@ [[ilm-freeze]] === Freeze -Phases allowed: cold, frozen. +Phases allowed: cold. <> an index to minimize its memory footprint. diff --git a/docs/reference/ilm/actions/ilm-migrate.asciidoc b/docs/reference/ilm/actions/ilm-migrate.asciidoc index b5793aa9904d5..f096fd98815c2 100644 --- a/docs/reference/ilm/actions/ilm-migrate.asciidoc +++ b/docs/reference/ilm/actions/ilm-migrate.asciidoc @@ -2,12 +2,12 @@ [[ilm-migrate]] === Migrate -Phases allowed: warm, cold, frozen. +Phases allowed: warm, cold. Moves the index to the <> that corresponds to the current phase by updating the <> index setting. -{ilm-init} automatically injects the migrate action in the warm, cold, and frozen +{ilm-init} automatically injects the migrate action in the warm and cold phases if no allocation options are specified with the <> action. If you specify an allocate action that only modifies the number of index replicas, {ilm-init} reduces the number of replicas before migrating the index. @@ -30,9 +30,10 @@ to `data_cold,data_warm,data_hot`. This moves the index to nodes in the <>. If there are no nodes in the cold tier, it falls back to the <> tier, or the <> tier if there are no warm nodes available. -In the frozen phase, the `migrate` action sets +The migrate action is not allowed in the frozen phase. The frozen phase directly +mounts the searchable snapshot using a <> -to `data_frozen,data_cold,data_warm,data_hot`. This moves the index to nodes in the +of `data_frozen,data_cold,data_warm,data_hot`. This moves the index to nodes in the <>. If there are no nodes in the frozen tier, it falls back to the <> tier, then the <> tier, then finally the <> tier. diff --git a/docs/reference/ilm/actions/ilm-readonly.asciidoc b/docs/reference/ilm/actions/ilm-readonly.asciidoc index 2b21b62f09e2b..e1997813557bb 100644 --- a/docs/reference/ilm/actions/ilm-readonly.asciidoc +++ b/docs/reference/ilm/actions/ilm-readonly.asciidoc @@ -2,7 +2,7 @@ [[ilm-readonly]] === Read only -Phases allowed: hot, warm, cold, frozen. +Phases allowed: hot, warm, cold. Makes the index <>. diff --git a/docs/reference/ilm/actions/ilm-set-priority.asciidoc b/docs/reference/ilm/actions/ilm-set-priority.asciidoc index 369dc818d5a5d..08f2eec1c599c 100644 --- a/docs/reference/ilm/actions/ilm-set-priority.asciidoc +++ b/docs/reference/ilm/actions/ilm-set-priority.asciidoc @@ -2,14 +2,14 @@ [[ilm-set-priority]] === Set priority -Phases allowed: hot, warm, cold, frozen. +Phases allowed: hot, warm, cold. Sets the <> of the index as -soon as the policy enters the hot, warm, cold, or frozen phase. -Higher priority indices are recovered before indices with lower priorities following a node restart. +soon as the policy enters the hot, warm, or cold phase. +Higher priority indices are recovered before indices with lower priorities following a node restart. Generally, indexes in the hot phase should have the highest value and -indexes in the cold phase should have the lowest values. +indexes in the cold phase should have the lowest values. For example: 100 for the hot phase, 50 for the warm phase, and 0 for the cold phase. Indices that don't set this value have a default priority of 1. @@ -17,8 +17,8 @@ Indices that don't set this value have a default priority of 1. ==== Options `priority`:: -(Required, integer) -The priority for the index. +(Required, integer) +The priority for the index. Must be 0 or greater. Set to `null` to remove the priority. diff --git a/docs/reference/ilm/ilm-index-lifecycle.asciidoc b/docs/reference/ilm/ilm-index-lifecycle.asciidoc index 32cb04e8447d5..37a1287425616 100644 --- a/docs/reference/ilm/ilm-index-lifecycle.asciidoc +++ b/docs/reference/ilm/ilm-index-lifecycle.asciidoc @@ -16,13 +16,13 @@ needs to be searchable, but it's okay if those queries are slower. needs to be searchable, but it's okay if those queries are extremely slow. * **Delete**: The index is no longer needed and can safely be removed. -An index's _lifecycle policy_ specifies which phases +An index's _lifecycle policy_ specifies which phases are applicable, what actions are performed in each phase, and when it transitions between phases. -You can manually apply a lifecycle policy when you create an index. +You can manually apply a lifecycle policy when you create an index. For time series indices, you need to associate the lifecycle policy with -the index template used to create new indices in the series. +the index template used to create new indices in the series. When an index rolls over, a manually-applied policy isn't automatically applied to the new index. If you use {es}'s security features, {ilm-init} performs operations as the user @@ -34,7 +34,7 @@ update. [[ilm-phase-transitions]] === Phase transitions -{ilm-init} moves indices through the lifecycle according to their age. +{ilm-init} moves indices through the lifecycle according to their age. To control the timing of these transitions, you set a _minimum age_ for each phase. For an index to move to the next phase, all actions in the current phase must be complete and the index must be older than the minimum age of the next phase. Configured minimum ages must increase between @@ -42,12 +42,12 @@ subsequent phases, for example, a "warm" phase with a minimum age of 10 days can a "cold" phase with a minimum age either unset, or >= 10 days. The minimum age defaults to zero, which causes {ilm-init} to move indices to the next phase -as soon as all actions in the current phase complete. +as soon as all actions in the current phase complete. -If an index has unallocated shards and the <> is yellow, +If an index has unallocated shards and the <> is yellow, the index can still transition to the next phase according to its {ilm} policy. However, because {es} can only perform certain clean up tasks on a green -cluster, there might be unexpected side effects. +cluster, there might be unexpected side effects. To avoid increased disk usage and reliability issues, address any cluster health problems in a timely fashion. @@ -63,18 +63,18 @@ what _steps_ are executed to perform the necessary index operations for each act When an index enters a phase, {ilm-init} caches the phase definition in the index metadata. This ensures that policy updates don't put the index into a state where it can never exit the phase. If changes can be safely applied, {ilm-init} updates the cached phase definition. -If they cannot, phase execution continues using the cached definition. +If they cannot, phase execution continues using the cached definition. -{ilm-init} runs periodically, checks to see if an index meets policy criteria, -and executes whatever steps are needed. +{ilm-init} runs periodically, checks to see if an index meets policy criteria, +and executes whatever steps are needed. To avoid race conditions, {ilm-init} might need to run more than once to execute all of the steps required to complete an action. -For example, if {ilm-init} determines that an index has met the rollover criteria, -it begins executing the steps required to complete the rollover action. -If it reaches a point where it is not safe to advance to the next step, execution stops. -The next time {ilm-init} runs, {ilm-init} picks up execution where it left off. +For example, if {ilm-init} determines that an index has met the rollover criteria, +it begins executing the steps required to complete the rollover action. +If it reaches a point where it is not safe to advance to the next step, execution stops. +The next time {ilm-init} runs, {ilm-init} picks up execution where it left off. This means that even if `indices.lifecycle.poll_interval` is set to 10 minutes and an index meets -the rollover criteria, it could be 20 minutes before the rollover is complete. +the rollover criteria, it could be 20 minutes before the rollover is complete. [discrete] [[ilm-phase-actions]] @@ -111,11 +111,6 @@ ifdef::permanently-unreleased-branch[] endif::[] - <> * Frozen - - <> - - <> - - <> - - <> - - <> - <> * Delete - <> diff --git a/docs/reference/index-modules/allocation/data_tier_allocation.asciidoc b/docs/reference/index-modules/allocation/data_tier_allocation.asciidoc index ea5aa3c567806..aa606a92514ed 100644 --- a/docs/reference/index-modules/allocation/data_tier_allocation.asciidoc +++ b/docs/reference/index-modules/allocation/data_tier_allocation.asciidoc @@ -13,9 +13,11 @@ These tier attributes are set using the data node roles: * <> * <> * <> +* <> NOTE: The <> role is not a valid data tier and cannot be used -for data tier filtering. +for data tier filtering. The <> role can only be +used for searchable snapshots mounted with the `shared_cache` option. [discrete] [[data-tier-allocation-filters]] diff --git a/docs/reference/modules/node.asciidoc b/docs/reference/modules/node.asciidoc index 61e91cad2b15f..d759e3f694c3d 100644 --- a/docs/reference/modules/node.asciidoc +++ b/docs/reference/modules/node.asciidoc @@ -270,10 +270,8 @@ node.roles: [ data_cold ] [[data-frozen-node]] ==== [x-pack]#Frozen data node# -Frozen data nodes store read-only indices that are accessed rarely. Nodes in the -frozen tier use less performant hardware than the cold tier. To minimize -resources, indices in the frozen tier may rely on searchable snapshots for -resiliency. +Frozen data nodes store searchable snapshots mounted with the `shared_cache` +option exclusively. To create a dedicated frozen node, set: [source,yaml] From 2c885a060dc1475e85dfbcf01706c5ffed94e2e1 Mon Sep 17 00:00:00 2001 From: Henning Andersen <33268011+henningandersen@users.noreply.github.com> Date: Thu, 18 Mar 2021 19:39:27 +0100 Subject: [PATCH 2/2] Lee's suggestion Co-authored-by: Lee Hinman --- docs/reference/datatiers.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/datatiers.asciidoc b/docs/reference/datatiers.asciidoc index 05928fde27c46..2d6369b46bc37 100644 --- a/docs/reference/datatiers.asciidoc +++ b/docs/reference/datatiers.asciidoc @@ -91,7 +91,7 @@ Once data is no longer being queried, or being queried rarely, it may move from to the frozen tier where it stays for the rest of its life. The frozen tier is a less responsive query tier than the cold tier, and data in the frozen tier cannot be updated. The frozen tier holds searchable snapshots mounted using the -`shared_cache` storage option exclusively. The <> automatically converts data +`shared_cache` storage option exclusively. The <> converts data transitioning into the frozen tier into a searchable snapshot, eliminating the need for replicas or even a local copy. [discrete]