Skip to content

[DOCS] Edit ILM GS tutorial #51513

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Feb 4, 2020
Merged

[DOCS] Edit ILM GS tutorial #51513

Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
f18427b
[DOCS] Edit ILM GS tutorial
debadair Jan 28, 2020
b812d90
More edits
debadair Jan 30, 2020
0245b34
[DOCS] Incorporated review feedback from @andreidan.
debadair Jan 30, 2020
78b73af
[DOCS] Removed test link & fixed anchor & title.
debadair Jan 30, 2020
0abe008
Edits
debadair Jan 30, 2020
61ae641
Merge branch 'master' into ilm-gs-edit
debadair Jan 30, 2020
99f219e
Reverted explicit actions anchor.
debadair Jan 31, 2020
0cc6a8c
Update docs/reference/ilm/getting-started-ilm.asciidoc
debadair Feb 4, 2020
6db949a
Update docs/reference/ilm/getting-started-ilm.asciidoc
debadair Feb 4, 2020
52374ce
Incorporated feedback from @jrodewig
debadair Feb 4, 2020
cdb2829
Update docs/reference/ilm/getting-started-ilm.asciidoc
debadair Feb 4, 2020
6f8e53b
Update docs/reference/ilm/getting-started-ilm.asciidoc
debadair Feb 4, 2020
643b606
Update docs/reference/ilm/getting-started-ilm.asciidoc
debadair Feb 4, 2020
d89292f
Update docs/reference/ilm/getting-started-ilm.asciidoc
debadair Feb 4, 2020
8996f29
Update docs/reference/ilm/getting-started-ilm.asciidoc
debadair Feb 4, 2020
2d2dd82
Update docs/reference/ilm/getting-started-ilm.asciidoc
debadair Feb 4, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
227 changes: 106 additions & 121 deletions docs/reference/ilm/getting-started-ilm.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,79 +1,93 @@
[role="xpack"]
[testenv="basic"]

[[getting-started-index-lifecycle-management]]
== Get started with {ilm}
== Get started: Automate rollover with {ilm-init}

++++
<titleabbrev>Automate rollover</titleabbrev>
++++

This tutorial demonstrates how to use {ilm} ({ilm-init})
to manage indices that contain time-series data.

When you continuously index timestamped documents into {es} using
Filebeat, Logstash, or some other mechanism,
you typically use an index alias so you can periodically roll over to a new index.
Copy link
Contributor

@jrodewig jrodewig Feb 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Link for index alias?

This enables you to implement a hot-warm-cold architecture to meet your performance
requirements for your newest data, control costs over time, enforce retention policies,
and still get the most out of your data.

Let's jump into {ilm} ({ilm-init}) by working through a hands-on scenario.
This section will leverage many new concepts unique to {ilm-init} that
you may not be familiar with. The following sections will explore
these in more details.
To automate rollover and management of time-series indices with {ilm-init}, you:

The goal of this example is to set up a set of indices that will encapsulate
the data from a time series data source. We can imagine there is a system
like {filebeat-ref}[Filebeat] that continuously indexes documents into
our writing index. We wish to roll over the index after it reaches a size
of 50 gigabytes, or has been created 30 days ago, and then delete the index
after 90 days.
. <<ilm-gs-create-policy, Create a lifecycle policy>> with the {ilm-init} put policy API.
. <<ilm-gs-apply-policy, Create an index template>> to apply the policy to each new index.
. <<ilm-gs-bootstrap, Bootstrap an index>> as the initial write index.
Copy link
Contributor

@jrodewig jrodewig Feb 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
. <<ilm-gs-bootstrap, Bootstrap an index>> as the initial write index.
. <<ilm-gs-bootstrap, Create an initial write index>>.

. <<ilm-gs-check-progress, Verify indices are moving through the lifecycle phases>>
as expected with the {ilm-init} explain API.

[float]
[[ilm-gs-create-policy]]
=== Setting up a policy
=== Create a lifecycle policy

There are many new features introduced by {ilm-init}, but we will only focus on
a few that are needed for our example. For starters, we will use the
<<ilm-put-lifecycle,Put Policy>> API to define our first policy. Lifecycle
policies are defined in JSON and include specific
<<ilm-policy-definition,phases and actions>>.
A lifecycle policy specifies the phases in the index lifecycle
and the actions to perform in each phase. A lifecycle can have up to four phases:
`hot`, `warm`, `cold`, and `delete`. Policies are defined in JSON
and added through the {ilm-init} put policy API.

For example, the following request creates a `datastream_policy` with two phases:

* The `hot` phase defines a `rollover` action to specify that an index rolls over when it
reaches either a `max_size` of 50 gigabytes or a `max_age` of 30 days.
* The `delete` phase uses `min_age` to remove the index 90 days after rollover.
Note that this value is relative to the rollover time, not the index creation time.

[source,console]
------------------------
PUT _ilm/policy/datastream_policy <1>
PUT _ilm/policy/datastream_policy
{
"policy": { <2>
"policy": {
"phases": {
"hot": { <3>
"hot": { <1>
"actions": {
"rollover": { <4>
"max_size": "50GB",
"rollover": {
"max_size": "50GB", <2>
"max_age": "30d"
}
}
},
"delete": {
"min_age": "90d", <5>
"min_age": "90d", <3>
"actions": {
"delete": {} <6>
"delete": {} <4>
}
}
}
}
}
------------------------
<1> The `min_age` defaults to `0ms`, so new indices enter the `hot` phase immediately.
<2> Trigger the `rollover` action when either of the conditions are met.
<3> Move the index into the `delete` phase 90 days after rollover.
<4> Trigger the `delete` action when the index enters the delete phase.

<1> call to the <<ilm-put-lifecycle,put lifecycle API>> endpoint to create
a new policy named "datastream_policy"
<2> policy definition sub-object
<3> the hot phase defined in the "phases" section. Optional `min_age` field
not defined -- defaults to `0ms`
<4> rollover action definition
<5> delete phase begins after 90 days
<6> delete action definition


Here we created the policy called `datastream_policy` which rolls over
the index being written to after it reaches 50 gigabytes, or it is 30
days old. The rollover will occur when either of these conditions is true.
The index will be deleted 90 days after it is rolled over.
See <<_actions>> for the complete list of actions available in each phase.

[float]
[[ilm-gs-apply-policy]]
=== Applying a policy to our index
=== Create an index template to apply the lifecycle policy

To automaticaly apply a lifecycle policy to the new write index on rollover,
specify the policy in the index template used to create new indices.

There are <<set-up-lifecycle-policy,a few ways>> to associate a
policy to an index. Since we wish specific settings to be applied to
the new index created from Rollover, we will set the policy via
index templates.
For example, the following request creates a `datastream_template` that is applied to new indices
whose names match the `datastream-*` index pattern.
The template configures two {ilm-init} settings:

* `index.lifecycle.name` specifies the name of the lifecycle policy to apply to all new indices that match
the index pattern.
* `index.lifecycle.rollover_alias` specifies the index alias to be rolled over
when the rollover action is triggered for an index.

[source,console]
-----------------------
Expand All @@ -90,6 +104,11 @@ PUT _template/datastream_template
-----------------------
// TEST[continued]

<1> Apply the template to a new index if its name starts with `datastream-`.
<2> The name of the lifecycle policy to apply to each new index.
<3> The name of the alias used to reference these indices.
Required for policies that use the rollover action.

//////////////////////////

[source,console]
Expand All @@ -100,23 +119,17 @@ DELETE /_template/datastream_template

//////////////////////////

<1> match all indices starting with "datastream-". These will include all
newly created indices from actions like rollover
<2> the name of the lifecycle policy managing the index
<3> alias to use for the rollover action, required since a rollover action is
defined in the policy.

The above index template introduces a few new settings specific to {ilm-init}.
The first being `index.lifecycle.name`. This setting will configure
the "datastream_policy" to the index applying this template. This means
that all newly created indices prefixed "datastream-" will be managed by
our policy. The other setting used here is `index.lifecycle.rollover_alias`.
This setting is required when using a policy containing the rollover
action and specifies which alias to rollover on behalf of this index.
The intention here is that the rollover alias is also defined on the index.
[float]
[[ilm-gs-bootstrap]]
=== Bootstrap the initial time-series index
Copy link
Contributor

@jrodewig jrodewig Feb 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
=== Bootstrap the initial time-series index
=== Create an initial time-series index


To begin, we will want to bootstrap our first index to write to.
To get things started, you need to bootstrap an initial index and
Copy link
Contributor

@jrodewig jrodewig Feb 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To get things started, you need to bootstrap an initial index and
To get things started, you need to create an initial index and

designate it as the write index for the rollover alias specified in your index template.
The name of this index must match the template's index pattern and end with a number.
On rollover, this value is incremented to generate a name for the new index.

For example, the following request creates an index called `datastream-000001`
and makes it the write index for the `datastream` alias.

[source,console]
-----------------------
Expand All @@ -131,44 +144,40 @@ PUT datastream-000001
-----------------------
// TEST[continued]

When creating our index, we have to consider a few important configurations
that tie our index and our policy together correctly. We need to make sure
that our index name matches our index template pattern of "datastream-*",
which it does. We are using the <<ilm-rollover-action, Rollover Action>> in our policy, which
requires that our index name ends with a number. In our case, we used
`000001`. This is important so that Rollover can increment this number when
naming the new index created from rolling over.

Our index creation request leverages its template to apply our settings,
but we must also configure our rollover alias: "datastream". To do this,
we take advantage of <<aliases-write-index,write indices>>. This is a way
to define an alias to be used for both reading and writing, with only one
index being the index that is being written to at a time. Rollover swaps
the write index to be the new index created from rollover, and sets the
alias to be read-only for the source index.
When the rollover conditions are met, the `rollover` action:

* Creates a new index called `datastream-000002`.
This matches the `datastream-*` pattern, so the settings from `datastream_template` are applied to the new index.
* Designates the new index as the write index and makes the bootstrap index read-only.
Copy link
Contributor

@jrodewig jrodewig Feb 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Designates the new index as the write index and makes the bootstrap index read-only.
* Designates the new index as the write index and makes `datastream-000001` read-only.


This process repeats each time rollover conditions are met.
You can search across all of the indices managed by the `datastream_policy` with the `datastream` alias.
Write operations are routed to the current write index.

For more information about write indices and rollover, see the <<rollover-index-api-desc, rollover API>>.

[float]
[[ilm-gs-check-progress]]
=== Checking progress

Now that we have an index managed by our policy, how do we tell what is going
on? Which phase are we in? Is something broken? This section will go over a
few APIs and their responses to help us inspect our indices with respect
to {ilm-init}.
To get status information for managed indices, you use the {ilm-init} explain API.
This lets you find out things like:

With the help of the <<ilm-explain-lifecycle,Explain API>>, we can know
things like which phase we're in and when we entered that phase. The API
will also provide further info if errors occurred, or if we are blocked on
certain checks within actions.
* What phase an index is in and when it entered that phase.
* The current action and what step is being performed.
* If any errors have occurred or progress is blocked.

For example, the following request gets information about the `datastream` indices:

[source,console]
--------------------------------------------------
GET datastream-*/_ilm/explain
--------------------------------------------------
// TEST[continued]

The above request will retrieve {ilm-init} execution information for all our
managed indices.
The response below shows that the bootstrap index is waiting in the `hot` phase's `rollover` action.
Copy link
Contributor

@jrodewig jrodewig Feb 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The response below shows that the bootstrap index is waiting in the `hot` phase's `rollover` action.
The response below shows that the `datastream-000001` index is waiting in the `hot` phase's `rollover` action.

It remains in this state and {ilm-init} continues to call `attempt-rollover`
until the rollover conditions are met.

[[36818c6d9f434d387819c30bd9addb14]]
[source,console-result]
Expand All @@ -177,19 +186,19 @@ managed indices.
"indices": {
"datastream-000001": {
"index": "datastream-000001",
"managed": true, <1>
"policy": "datastream_policy", <2>
"managed": true,
"policy": "datastream_policy", <1>
"lifecycle_date_millis": 1538475653281,
"age": "30s", <3>
"phase": "hot", <4>
"age": "30s", <2>
"phase": "hot",
"phase_time_millis": 1538475653317,
"action": "rollover", <5>
"action": "rollover",
"action_time_millis": 1538475653317,
"step": "attempt-rollover", <6>
"step": "attempt-rollover", <3>
"step_time_millis": 1538475653317,
"phase_execution": {
"policy": "datastream_policy",
"phase_definition": { <7>
"phase_definition": { <4>
"min_age": "0ms",
"actions": {
"rollover": {
Expand All @@ -198,7 +207,7 @@ managed indices.
}
}
},
"version": 1, <8>
"version": 1,
"modified_date_in_millis": 1539609701576
}
}
Expand All @@ -207,33 +216,9 @@ managed indices.
--------------------------------------------------
// TESTRESPONSE[skip:no way to know if we will get this response immediately]

<1> this index is managed by ILM
<2> the policy in question, in this case, "datastream_policy"
<3> the current age of the index
<4> what phase the index is currently in
<5> what action the index is currently on
<6> what step the index is currently on
<7> the definition of the phase
(in this case, the "hot" phase) that the index is currently on
<8> the version of the policy being used to execute the current phase

You can read about the full details of this response in the
<<ilm-explain-lifecycle, explain API docs>>. For now, let's focus on how
the response details which phase, action, and step we're in. We are in the
"hot" phase, and "rollover" action. Rollover will continue to be called
by {ilm-init} until its conditions are met and it rolls over the index.
Afterwards, the original index will stay in the hot phase until 90 more
days pass and it is deleted in the delete phase.
As time goes on, new indices will be created and deleted.
With `datastream-000002` being created when the index mets the rollover
conditions and `datastream-000003` created after that. We will be able
to search across all of our managed indices using the "datastream" alias,
and we will be able to write to our to-be-rolled-over write indices using
that same alias.



That's it! We have our first use-case managed by {ilm-init}.

To learn more about all our APIs,
check out <<index-lifecycle-management-api,ILM APIs>>.
<1> The policy used to manage the index
<2> The age of the index
<3> The step {ilm-init} is performing on the index
<4> The definition of the current phase (the `hot` phase)

See the <<index-lifecycle-management-api,ILM APIs>> for more information.