Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions docs/reference/datatiers.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ and hold your most recent, most-frequently-accessed data.
* <<warm-tier, Warm tier>> nodes hold time series data that is accessed less-frequently
and rarely needs to be updated.
* <<cold-tier, Cold tier>> nodes hold time series data that is accessed infrequently and not normally updated.
* <<frozen-tier, Frozen tier>> nodes hold time series data that is accessed rarely and never updated.
* <<frozen-tier, Frozen tier>> nodes hold time series data that is accessed rarely and never updated, kept in searchable snapshots.

When you index documents directly to a specific index, they remain on content tier nodes indefinitely.

Expand Down Expand Up @@ -85,12 +85,14 @@ For resiliency, indices in the cold tier can rely on
[[frozen-tier]]
=== Frozen tier

experimental::[]

Once data is no longer being queried, or being queried rarely, it may move from the cold tier
to the frozen tier where it stays for the rest of its life.
The frozen tier is a less responsive query tier than the cold tier, and data in the frozen tier is
not normally updated. As data transitions into the frozen tier it can be compressed and shrunken.
For resiliency, indices in the frozen tier can rely on <<ilm-searchable-snapshot, searchable
snapshots>>, eliminating the need for replicas or even a local copy.
The frozen tier is a less responsive query tier than the cold tier, and data in the frozen tier
cannot be updated. The frozen tier holds searchable snapshots mounted using the
`shared_cache` storage option exclusively. The <<ilm-index-lifecycle, frozen phase>> converts data
transitioning into the frozen tier into a searchable snapshot, eliminating the need for replicas or even a local copy.

[discrete]
[[data-tier-allocation]]
Expand Down
28 changes: 14 additions & 14 deletions docs/reference/ilm/actions/ilm-allocate.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,17 @@
[[ilm-allocate]]
=== Allocate

Phases allowed: warm, cold, frozen.
Phases allowed: warm, cold.

Updates the index settings to change which nodes are allowed to host the index shards
and change the number of replicas.

The allocate action is not allowed in the hot phase.
The initial allocation for the index must be done manually or via
The allocate action is not allowed in the hot phase.
The initial allocation for the index must be done manually or via
<<index-templates, index templates>>.

You can configure this action to modify both the allocation rules and number of replicas,
only the allocation rules, or only the number of replicas.
You can configure this action to modify both the allocation rules and number of replicas,
only the allocation rules, or only the number of replicas.
For more information about how {es} uses replicas for scaling, see
<<scalability>>. See <<shard-allocation-filtering>> for more information about
controlling where {es} allocates shards of a particular index.
Expand All @@ -21,11 +21,11 @@ controlling where {es} allocates shards of a particular index.
[[ilm-allocate-options]]
==== Options

You must specify the number of replicas or at least one
`include`, `exclude`, or `require` option.
You must specify the number of replicas or at least one
`include`, `exclude`, or `require` option.
An empty allocate action is invalid.

For more information about using custom attributes for shard allocation,
For more information about using custom attributes for shard allocation,
see <<shard-allocation-filtering>>.

`number_of_replicas`::
Expand All @@ -47,7 +47,7 @@ Assigns an index to nodes that have _all_ of the specified custom attributes.
[[ilm-allocate-ex]]
==== Example

The allocate action in the following policy changes the index's number of replicas to `2`.
The allocate action in the following policy changes the index's number of replicas to `2`.
The index allocation rules are not changed.

[source,console]
Expand All @@ -71,11 +71,11 @@ PUT _ilm/policy/my_policy
[[ilm-allocate-assign-index-attribute-ex]]
===== Assign index to nodes using a custom attribute

The allocate action in the following policy assigns the index to nodes
The allocate action in the following policy assigns the index to nodes
that have a `box_type` of _hot_ or _warm_.

To designate a node's `box_type`, you set a custom attribute in the node configuration.
For example, set `node.attr.box_type: hot` in `elasticsearch.yml`.
For example, set `node.attr.box_type: hot` in `elasticsearch.yml`.
For more information, see <<index-allocation-filters>>.

[source,console]
Expand Down Expand Up @@ -129,11 +129,11 @@ PUT _ilm/policy/my_policy
[[ilm-allocate-assign-index-node-ex]]
===== Assign index to a specific node and update replica settings

The allocate action in the following policy updates the index to have one replica per shard
and be allocated to nodes that have a `box_type` of _cold_.
The allocate action in the following policy updates the index to have one replica per shard
and be allocated to nodes that have a `box_type` of _cold_.

To designate a node's `box_type`, you set a custom attribute in the node configuration.
For example, set `node.attr.box_type: cold` in `elasticsearch.yml`.
For example, set `node.attr.box_type: cold` in `elasticsearch.yml`.
For more information, see <<index-allocation-filters>>.

[source,console]
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/ilm/actions/ilm-freeze.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
[[ilm-freeze]]
=== Freeze

Phases allowed: cold, frozen.
Phases allowed: cold.

<<frozen-indices, Freezes>> an index to minimize its memory footprint.

Expand Down
9 changes: 5 additions & 4 deletions docs/reference/ilm/actions/ilm-migrate.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
[[ilm-migrate]]
=== Migrate

Phases allowed: warm, cold, frozen.
Phases allowed: warm, cold.

Moves the index to the <<data-tiers, data tier>> that corresponds
to the current phase by updating the <<tier-preference-allocation-filter, `index.routing.allocation.include._tier_preference`>>
index setting.
{ilm-init} automatically injects the migrate action in the warm, cold, and frozen
{ilm-init} automatically injects the migrate action in the warm and cold
phases if no allocation options are specified with the <<ilm-allocate, allocate>> action.
If you specify an allocate action that only modifies the number of index
replicas, {ilm-init} reduces the number of replicas before migrating the index.
Expand All @@ -30,9 +30,10 @@ to `data_cold,data_warm,data_hot`. This moves the index to nodes in the
<<cold-tier, cold tier>>. If there are no nodes in the cold tier, it falls back to the
<<warm-tier, warm>> tier, or the <<hot-tier, hot>> tier if there are no warm nodes available.

In the frozen phase, the `migrate` action sets
The migrate action is not allowed in the frozen phase. The frozen phase directly
mounts the searchable snapshot using a
<<tier-preference-allocation-filter, `index.routing.allocation.include._tier_preference`>>
to `data_frozen,data_cold,data_warm,data_hot`. This moves the index to nodes in the
of `data_frozen,data_cold,data_warm,data_hot`. This moves the index to nodes in the
<<frozen-tier, frozen tier>>. If there are no nodes in the frozen tier, it falls back to the
<<cold-tier, cold>> tier, then the <<warm-tier, warm>> tier, then finally the <<hot-tier, hot>>
tier.
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/ilm/actions/ilm-readonly.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
[[ilm-readonly]]
=== Read only

Phases allowed: hot, warm, cold, frozen.
Phases allowed: hot, warm, cold.

Makes the index <<index-blocks-read-only,read-only>>.

Expand Down
12 changes: 6 additions & 6 deletions docs/reference/ilm/actions/ilm-set-priority.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,23 @@
[[ilm-set-priority]]
=== Set priority

Phases allowed: hot, warm, cold, frozen.
Phases allowed: hot, warm, cold.

Sets the <<recovery-prioritization, priority>> of the index as
soon as the policy enters the hot, warm, cold, or frozen phase.
Higher priority indices are recovered before indices with lower priorities following a node restart.
soon as the policy enters the hot, warm, or cold phase.
Higher priority indices are recovered before indices with lower priorities following a node restart.

Generally, indexes in the hot phase should have the highest value and
indexes in the cold phase should have the lowest values.
indexes in the cold phase should have the lowest values.
For example: 100 for the hot phase, 50 for the warm phase, and 0 for the cold phase.
Indices that don't set this value have a default priority of 1.

[[ilm-set-priority-options]]
==== Options

`priority`::
(Required, integer)
The priority for the index.
(Required, integer)
The priority for the index.
Must be 0 or greater.
Set to `null` to remove the priority.

Expand Down
35 changes: 15 additions & 20 deletions docs/reference/ilm/ilm-index-lifecycle.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,13 @@ needs to be searchable, but it's okay if those queries are slower.
needs to be searchable, but it's okay if those queries are extremely slow.
* **Delete**: The index is no longer needed and can safely be removed.

An index's _lifecycle policy_ specifies which phases
An index's _lifecycle policy_ specifies which phases
are applicable, what actions are performed in each phase,
and when it transitions between phases.

You can manually apply a lifecycle policy when you create an index.
You can manually apply a lifecycle policy when you create an index.
For time series indices, you need to associate the lifecycle policy with
the index template used to create new indices in the series.
the index template used to create new indices in the series.
When an index rolls over, a manually-applied policy isn't automatically applied to the new index.

If you use {es}'s security features, {ilm-init} performs operations as the user
Expand All @@ -34,20 +34,20 @@ update.
[[ilm-phase-transitions]]
=== Phase transitions

{ilm-init} moves indices through the lifecycle according to their age.
{ilm-init} moves indices through the lifecycle according to their age.
To control the timing of these transitions, you set a _minimum age_ for each phase. For an index to
move to the next phase, all actions in the current phase must be complete and the index must be
older than the minimum age of the next phase. Configured minimum ages must increase between
subsequent phases, for example, a "warm" phase with a minimum age of 10 days can only be followed by
a "cold" phase with a minimum age either unset, or >= 10 days.

The minimum age defaults to zero, which causes {ilm-init} to move indices to the next phase
as soon as all actions in the current phase complete.
as soon as all actions in the current phase complete.

If an index has unallocated shards and the <<cluster-health,cluster health status>> is yellow,
If an index has unallocated shards and the <<cluster-health,cluster health status>> is yellow,
the index can still transition to the next phase according to its {ilm} policy.
However, because {es} can only perform certain clean up tasks on a green
cluster, there might be unexpected side effects.
cluster, there might be unexpected side effects.

To avoid increased disk usage and reliability issues,
address any cluster health problems in a timely fashion.
Expand All @@ -63,18 +63,18 @@ what _steps_ are executed to perform the necessary index operations for each act
When an index enters a phase, {ilm-init} caches the phase definition in the index metadata.
This ensures that policy updates don't put the index into a state where it can never exit the phase.
If changes can be safely applied, {ilm-init} updates the cached phase definition.
If they cannot, phase execution continues using the cached definition.
If they cannot, phase execution continues using the cached definition.

{ilm-init} runs periodically, checks to see if an index meets policy criteria,
and executes whatever steps are needed.
{ilm-init} runs periodically, checks to see if an index meets policy criteria,
and executes whatever steps are needed.
To avoid race conditions, {ilm-init} might need to run more than once to execute all of the steps
required to complete an action.
For example, if {ilm-init} determines that an index has met the rollover criteria,
it begins executing the steps required to complete the rollover action.
If it reaches a point where it is not safe to advance to the next step, execution stops.
The next time {ilm-init} runs, {ilm-init} picks up execution where it left off.
For example, if {ilm-init} determines that an index has met the rollover criteria,
it begins executing the steps required to complete the rollover action.
If it reaches a point where it is not safe to advance to the next step, execution stops.
The next time {ilm-init} runs, {ilm-init} picks up execution where it left off.
This means that even if `indices.lifecycle.poll_interval` is set to 10 minutes and an index meets
the rollover criteria, it could be 20 minutes before the rollover is complete.
the rollover criteria, it could be 20 minutes before the rollover is complete.

[discrete]
[[ilm-phase-actions]]
Expand Down Expand Up @@ -111,11 +111,6 @@ ifdef::permanently-unreleased-branch[]
endif::[]
- <<ilm-searchable-snapshot, Searchable Snapshot>>
* Frozen
- <<ilm-set-priority,Set Priority>>
- <<ilm-unfollow,Unfollow>>
- <<ilm-allocate,Allocate>>
- <<ilm-migrate,Migrate>>
- <<ilm-freeze,Freeze>>
- <<ilm-searchable-snapshot, Searchable Snapshot>>
* Delete
- <<ilm-wait-for-snapshot,Wait For Snapshot>>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,11 @@ These tier attributes are set using the data node roles:
* <<data-hot-node, data_hot>>
* <<data-warm-node, data_warm>>
* <<data-cold-node, data_cold>>
* <<data-frozen-node, data_frozen>>

NOTE: The <<data-node, data>> role is not a valid data tier and cannot be used
for data tier filtering.
for data tier filtering. The <<data-frozen-node, data_frozen>> role can only be
used for searchable snapshots mounted with the `shared_cache` option.

[discrete]
[[data-tier-allocation-filters]]
Expand Down
6 changes: 2 additions & 4 deletions docs/reference/modules/node.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -270,10 +270,8 @@ node.roles: [ data_cold ]
[[data-frozen-node]]
==== [x-pack]#Frozen data node#

Frozen data nodes store read-only indices that are accessed rarely. Nodes in the
frozen tier use less performant hardware than the cold tier. To minimize
resources, indices in the frozen tier may rely on searchable snapshots for
resiliency.
Frozen data nodes store searchable snapshots mounted with the `shared_cache`
option exclusively.

To create a dedicated frozen node, set:
[source,yaml]
Expand Down