From da5446455df137e18dc5d7125dac4952942f3274 Mon Sep 17 00:00:00 2001 From: Deb Adair Date: Tue, 21 Jan 2020 21:17:38 -0800 Subject: [PATCH 1/4] [DOCS} Split off overview to a separate topic. --- docs/reference/ilm/index.asciidoc | 66 +++--------------------- docs/reference/ilm/overview-ilm.asciidoc | 58 +++++++++++++++++++++ 2 files changed, 64 insertions(+), 60 deletions(-) create mode 100644 docs/reference/ilm/overview-ilm.asciidoc diff --git a/docs/reference/ilm/index.asciidoc b/docs/reference/ilm/index.asciidoc index c26e698efda25..21b4312a34dcd 100644 --- a/docs/reference/ilm/index.asciidoc +++ b/docs/reference/ilm/index.asciidoc @@ -5,68 +5,14 @@ [partintro] -- - -The <> enable you to -automate how you want to manage your indices over time. Rather than simply -performing management actions on your indices on a set schedule, you can base -actions on other factors such as shard size and performance requirements. - -You control how indices are handled as they age by attaching a -lifecycle policy to the index template used to create them. You can update -the policy to modify the lifecycle of both new and existing indices. - -For time series indices, there are four stages in the index lifecycle: - -* Hot--the index is actively being updated and queried. -* Warm--the index is no longer being updated, but is still being queried. -* Cold--the index is no longer being updated and is seldom queried. The -information still needs to be searchable, but it's okay if those queries are -slower. -* Delete--the index is no longer needed and can safely be deleted. - -The lifecycle policy governs how the index transitions through these stages and -the actions that are performed on the index at each stage. The policy can -specify: - -* The maximum size or age at which you want to roll over to a new index. -* The point at which the index is no longer being updated and the number of -primary shards can be reduced. -* When to force a merge to permanently delete documents marked for deletion. -* The point at which the index can be moved to less performant hardware. -* The point at which the availability is not as critical and the number of -replicas can be reduced. -* When the index can be safely deleted. - -For example, if you are indexing metrics data from a fleet of ATMs into -Elasticsearch, you might define a policy that says: - -. When the index reaches 50GB, roll over to a new index. -. Move the old index into the warm stage, mark it read only, and shrink it down -to a single shard. -. After 7 days, move the index into the cold stage and move it to less expensive -hardware. -. Delete the index once the required 30 day retention period is reached. - -*Snapshot Lifecycle Management* - -ILM itself does allow managing indices, however, managing snapshots for a set of -indices is outside of the scope of an index-level policy. Instead, there are -separate APIs for managing snapshot lifecycles. Please see the -<> -documentation for information about configuring snapshots. - -See <>. - -[IMPORTANT] -=========================== -{ilm} does not support mixed-version cluster usage. Although it -may be possible to create such new policies against -newer-versioned nodes, there is no guarantee they will -work as intended. New policies using new actions that -do not exist in the oldest versioned node will cause errors. -=========================== +You can configure index lifecycle policies to automatically manage indices as they age. +Index lifecycle policies can trigger actions such as rollover, shrink, force merge, freeze, and delete +based on your performance, resiliency, and retention requirements. +To automatically back up your indices and manage snapshots, +you use <>. -- +include::overview-ilm.asciidoc[] include::getting-started-ilm.asciidoc[] diff --git a/docs/reference/ilm/overview-ilm.asciidoc b/docs/reference/ilm/overview-ilm.asciidoc new file mode 100644 index 0000000000000..bbfdb5f3a94ef --- /dev/null +++ b/docs/reference/ilm/overview-ilm.asciidoc @@ -0,0 +1,58 @@ +[role="xpack"] +[testenv="basic"] +[[index-lifecycle-management-overview]] +== Index lifecycle management overview + +++++ +Overview +++++ + +You can create and apply {ilm-cap} ({ilm-init}) policies to automatically manage your indices +according to your performance, resiliency, and retention requirements. +Index lifecycle policies can trigger actions such as rollover, shrink, freeze, and delete. + +Typically, you associate a lifecycle policy with an index template so it is applied +whenever an index is created from the template. +You can also apply a policy manually when you create an index. + +Index lifecycle policies are most often used to manage time-series indices. +Time-series indices have a four stage lifecycle: + +* Hot--the index is actively being updated and queried. +* Warm--the index is no longer being updated, but is still being queried. +* Cold--the index is no longer being updated and is seldom queried. The +information still needs to be searchable, but it's okay if those queries are +slower. +* Delete--the index is no longer needed and can safely be deleted. + +A lifecycle policy governs how the index transitions through these stages and +the actions that are performed on the index at each stage. +The policy can specify: + +* The maximum size or age at which you want to roll over to a new index. +* The point at which the index is no longer being updated and the number of +primary shards can be reduced. +* When to force a merge to permanently delete documents marked for deletion. +* The point at which the index can be moved to less performant hardware. +* The point at which the availability is not as critical and the number of +replicas can be reduced. +* When the index can be safely deleted. + +For example, if you are indexing metrics data from a fleet of ATMs into +Elasticsearch, you might define a policy that says: + +. When the index reaches 50GB, roll over to a new index. +. Move the old index into the warm stage, mark it read only, and shrink it down +to a single shard. +. After 7 days, move the index into the cold stage and move it to less expensive +hardware. +. Delete the index once the required 30 day retention period is reached. + +[IMPORTANT] +=========================== +{ilm} does not support mixed-version clusters. +Although it might be possible to create and apply policies in a mixed cluster, +there is no guarantee they will work as intended. +Attempting to use a policy that contains actions that aren't +supported on all nodes in a cluster will cause errors. +=========================== \ No newline at end of file From 0c3f0fa818640d01d06ffb81beb6661cd5498450 Mon Sep 17 00:00:00 2001 From: Deb Adair Date: Tue, 21 Jan 2020 22:31:47 -0800 Subject: [PATCH 2/4] Edits --- docs/reference/ilm/index.asciidoc | 13 ++++++++++++- docs/reference/ilm/overview-ilm.asciidoc | 2 +- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/docs/reference/ilm/index.asciidoc b/docs/reference/ilm/index.asciidoc index 21b4312a34dcd..9ac08680dcde3 100644 --- a/docs/reference/ilm/index.asciidoc +++ b/docs/reference/ilm/index.asciidoc @@ -9,7 +9,18 @@ You can configure index lifecycle policies to automatically manage indices as th Index lifecycle policies can trigger actions such as rollover, shrink, force merge, freeze, and delete based on your performance, resiliency, and retention requirements. To automatically back up your indices and manage snapshots, -you use <>. +you use <>. + +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> -- include::overview-ilm.asciidoc[] diff --git a/docs/reference/ilm/overview-ilm.asciidoc b/docs/reference/ilm/overview-ilm.asciidoc index bbfdb5f3a94ef..f0f26c4e1745d 100644 --- a/docs/reference/ilm/overview-ilm.asciidoc +++ b/docs/reference/ilm/overview-ilm.asciidoc @@ -1,6 +1,6 @@ [role="xpack"] [testenv="basic"] -[[index-lifecycle-management-overview]] +[[overview-index-lifecycle-management]] == Index lifecycle management overview ++++ From c45993e68371d0c4b678e0e631d091f35a2c7847 Mon Sep 17 00:00:00 2001 From: Deb Adair Date: Tue, 21 Jan 2020 23:51:44 -0800 Subject: [PATCH 3/4] Fixed xref --- docs/reference/ilm/index.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/ilm/index.asciidoc b/docs/reference/ilm/index.asciidoc index 9ac08680dcde3..734e948e3aa44 100644 --- a/docs/reference/ilm/index.asciidoc +++ b/docs/reference/ilm/index.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] [[index-lifecycle-management]] -= Managing the index lifecycle += Manage the index lifecycle [partintro] -- @@ -9,7 +9,7 @@ You can configure index lifecycle policies to automatically manage indices as th Index lifecycle policies can trigger actions such as rollover, shrink, force merge, freeze, and delete based on your performance, resiliency, and retention requirements. To automatically back up your indices and manage snapshots, -you use <>. +use <>. * <> * <> From 6acc42cb5c9c95cb3cdc4e42b3cfb42e18d3f711 Mon Sep 17 00:00:00 2001 From: Deb Adair Date: Mon, 27 Jan 2020 17:07:47 -0800 Subject: [PATCH 4/4] [DOCS] Incorporated feedback from @jrodewig. --- docs/reference/glossary.asciidoc | 55 ++++++++++++++++++- .../ilm/getting-started-ilm.asciidoc | 4 +- docs/reference/ilm/ilm-and-snapshots.asciidoc | 2 +- .../ilm/ilm-with-existing-indices.asciidoc | 2 +- docs/reference/ilm/index.asciidoc | 39 +++++++------ docs/reference/ilm/overview-ilm.asciidoc | 31 +++++++---- .../ilm/set-up-lifecycle-policy.asciidoc | 2 +- .../ilm/using-policies-rollover.asciidoc | 2 +- 8 files changed, 104 insertions(+), 33 deletions(-) diff --git a/docs/reference/glossary.asciidoc b/docs/reference/glossary.asciidoc index 07270228bace9..40ff0828afe2d 100644 --- a/docs/reference/glossary.asciidoc +++ b/docs/reference/glossary.asciidoc @@ -99,6 +99,30 @@ Follower indices are the target indices for <>. They exist in your local cluster and replicate <>. // end::follower-index-def[] +[[glossary-force-merge]] force merge :: +// tag::force-merge-def[] +// tag::force-merge-def-short[] +Manually trigger a merge to reduce the number of segments in each shard of an index +and free up the space used by deleted documents. +// end::force-merge-def-short[] +You should not force merge indices that are actively being written to. +Merging is normally performed automatically, but you can use force merge after +<> to reduce the shards in the old index to a single segment. +See the {ref}/indices-forcemerge.html[force merge API]. +// end::force-merge-def[] + +[[glossary-freeze]] freeze :: +// tag::freeze-def[] +// tag::freeze-def-short[] +Make an index read-only and minimize its memory footprint. +// end::freeze-def-short[] +Frozen indices can be searched without incurring the overhead of of re-opening a closed index, +but searches are throttled and might be slower. +You can freeze indices to reduce the overhead of keeping older indices searchable +before you are ready to archive or delete them. +See the {ref}/freeze-index-api.html[freeze API]. +// end::freeze-def[] + [[glossary-id]] id :: // tag::id-def[] The ID of a <> identifies a document. The @@ -136,6 +160,18 @@ See {ref}/indices-add-alias.html[Add index alias]. // end::index-alias-def[] -- +[[glossary-index-template]] index template :: ++ +-- +// tag::index-template-def[] +// tag::index-template-def-short[] +Defines settings and mappings to apply to new indexes that match a simple naming pattern, such as _logs-*_. +// end::index-template-def-short[] +An index template can also attach a lifecycle policy to the new index. +Index templates are used to automatically configure indices created during <>. +// end::index-template-def[] +-- + [[glossary-leader-index]] leader index :: // tag::leader-index-def[] Leader indices are the source indices for <>. They exist @@ -253,6 +289,19 @@ shard will never be started on the same node as its primary shard. // end::replica-shard-def[] -- +[[glossary-rollover]] rollover :: ++ +-- +// tag::rollover-def[] +// tag::rollover-def-short[] +Redirect an alias to begin writing to a new index when the existing index reaches a certain age, number of docs, or size. +// end::rollover-def-short[] +The new index is automatically configured according to any matching <>. +For example, if you're indexing log data, you might use rollover to create daily or weekly indices. +See the {ref}/indices-rollover-index.html[rollover index API]. +// end::rollover-def[] +-- + [[glossary-routing]] routing :: + -- @@ -292,7 +341,11 @@ nodes. [[glossary-shrink]] shrink :: // tag::shrink-def[] -To reduce the amount of shards in an index. +// tag::shrink-def-short[] +Reduce the number of primary shards in an index. +// end::shrink-def-short[] +You can shrink an index to reduce its overhead when the request volume drops. +For example, you might opt to shrink an index once it is no longer the write index. See the {ref}/indices-shrink-index.html[shrink index API]. // end::shrink-def[] diff --git a/docs/reference/ilm/getting-started-ilm.asciidoc b/docs/reference/ilm/getting-started-ilm.asciidoc index f7ca3f4d216fc..6ff32cdd2a50c 100644 --- a/docs/reference/ilm/getting-started-ilm.asciidoc +++ b/docs/reference/ilm/getting-started-ilm.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] [[getting-started-index-lifecycle-management]] -== Getting started with {ilm} +== Get started with {ilm} Let's jump into {ilm} ({ilm-init}) by working through a hands-on scenario. This section will leverage many new concepts unique to {ilm-init} that @@ -160,7 +160,7 @@ GET datastream-*/_ilm/explain The above request will retrieve {ilm-init} execution information for all our managed indices. - +[[36818c6d9f434d387819c30bd9addb14]] [source,console-result] -------------------------------------------------- { diff --git a/docs/reference/ilm/ilm-and-snapshots.asciidoc b/docs/reference/ilm/ilm-and-snapshots.asciidoc index c71fbe0471da2..5de30f11382b4 100644 --- a/docs/reference/ilm/ilm-and-snapshots.asciidoc +++ b/docs/reference/ilm/ilm-and-snapshots.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] [[index-lifecycle-and-snapshots]] -== Restoring snapshots of managed indices +== Restore a managed index When restoring a snapshot that contains indices managed by Index Lifecycle Management, the lifecycle will automatically continue to execute after the diff --git a/docs/reference/ilm/ilm-with-existing-indices.asciidoc b/docs/reference/ilm/ilm-with-existing-indices.asciidoc index 96c1e4589de9a..8def57c0bc346 100644 --- a/docs/reference/ilm/ilm-with-existing-indices.asciidoc +++ b/docs/reference/ilm/ilm-with-existing-indices.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] [[ilm-with-existing-indices]] -== Using {ilm-init} with existing indices +== Manage existing indices While it is recommended to use {ilm-init} to manage the index lifecycle from start to finish, it may be useful to use {ilm-init} with existing indices, diff --git a/docs/reference/ilm/index.asciidoc b/docs/reference/ilm/index.asciidoc index 734e948e3aa44..a398f56d1ce9a 100644 --- a/docs/reference/ilm/index.asciidoc +++ b/docs/reference/ilm/index.asciidoc @@ -5,22 +5,28 @@ [partintro] -- -You can configure index lifecycle policies to automatically manage indices as they age. -Index lifecycle policies can trigger actions such as rollover, shrink, force merge, freeze, and delete -based on your performance, resiliency, and retention requirements. +You can configure {ilm} ({ilm-init}) policies to automatically manage indices according to +your performance, resiliency, and retention requirements. +For example, you could use {ilm-init} to: + +* Create a new index each day, week, or month and archive previous ones +* Spin up a new index when an index reaches a certain size +* Delete stale indices to enforce data retention standards + +[TIP] To automatically back up your indices and manage snapshots, use <>. -* <> -* <> -* <> -* <> -* <> -* <> -* <> -* <> -* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> * <> +* <> -- include::overview-ilm.asciidoc[] @@ -31,14 +37,15 @@ include::policy-definitions.asciidoc[] include::set-up-lifecycle-policy.asciidoc[] -include::using-policies-rollover.asciidoc[] - include::update-lifecycle-policy.asciidoc[] include::error-handling.asciidoc[] -include::ilm-and-snapshots.asciidoc[] - include::start-stop-ilm.asciidoc[] +include::using-policies-rollover.asciidoc[] + include::ilm-with-existing-indices.asciidoc[] + +include::ilm-and-snapshots.asciidoc[] + diff --git a/docs/reference/ilm/overview-ilm.asciidoc b/docs/reference/ilm/overview-ilm.asciidoc index f0f26c4e1745d..af3797587fb69 100644 --- a/docs/reference/ilm/overview-ilm.asciidoc +++ b/docs/reference/ilm/overview-ilm.asciidoc @@ -9,14 +9,26 @@ You can create and apply {ilm-cap} ({ilm-init}) policies to automatically manage your indices according to your performance, resiliency, and retention requirements. -Index lifecycle policies can trigger actions such as rollover, shrink, freeze, and delete. +Index lifecycle policies can trigger actions such as: -Typically, you associate a lifecycle policy with an index template so it is applied -whenever an index is created from the template. +* **Rollover** - +include::../glossary.asciidoc[tag=rollover-def-short] +* **Shrink** - +include::../glossary.asciidoc[tag=shrink-def-short] +* **Force merge** - +include::../glossary.asciidoc[tag=force-merge-def-short] +* **Freeze** - +include::../glossary.asciidoc[tag=freeze-def-short] +* **Delete** - Permanently remove an index, including all of its data and metadata. + +Typically, you associate a lifecycle policy with an index template so it is automatically applied +to new indices. You can also apply a policy manually when you create an index. -Index lifecycle policies are most often used to manage time-series indices. -Time-series indices have a four stage lifecycle: +{ilm-init} simplifies managing indices in hot-warm-cold architectures, +which are common when you're working with time-series data such as logs and metrics. + +As an index ages, it moves through four possible phases: * Hot--the index is actively being updated and queried. * Warm--the index is no longer being updated, but is still being queried. @@ -25,9 +37,8 @@ information still needs to be searchable, but it's okay if those queries are slower. * Delete--the index is no longer needed and can safely be deleted. -A lifecycle policy governs how the index transitions through these stages and -the actions that are performed on the index at each stage. -The policy can specify: +A lifecycle policy controls how an index moves between phases and +what actions to perform during each phase. You can specify: * The maximum size or age at which you want to roll over to a new index. * The point at which the index is no longer being updated and the number of @@ -50,8 +61,8 @@ hardware. [IMPORTANT] =========================== -{ilm} does not support mixed-version clusters. -Although it might be possible to create and apply policies in a mixed cluster, +To use {ilm-init}, all nodes in a cluster must run the same version. +Although it might be possible to create and apply policies in a mixed-version cluster, there is no guarantee they will work as intended. Attempting to use a policy that contains actions that aren't supported on all nodes in a cluster will cause errors. diff --git a/docs/reference/ilm/set-up-lifecycle-policy.asciidoc b/docs/reference/ilm/set-up-lifecycle-policy.asciidoc index e430ae9479a5a..18ed9c63d8c66 100644 --- a/docs/reference/ilm/set-up-lifecycle-policy.asciidoc +++ b/docs/reference/ilm/set-up-lifecycle-policy.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] [[set-up-lifecycle-policy]] -== Set up {ilm} policy +== Create lifecycle policy In order for an index to use an {ilm} policy to manage its lifecycle we must first define a lifecycle policy for it to use. The following request creates a diff --git a/docs/reference/ilm/using-policies-rollover.asciidoc b/docs/reference/ilm/using-policies-rollover.asciidoc index 04b084a0e8e59..ff41c29f4bc21 100644 --- a/docs/reference/ilm/using-policies-rollover.asciidoc +++ b/docs/reference/ilm/using-policies-rollover.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] [[using-policies-rollover]] -== Using policies to manage index rollover +== Roll over automatically The rollover action enables you to automatically roll over to a new index based on the index size, document count, or age. When a rollover is triggered, a new