Add Rollup ILM Action #65633

talevy · 2020-11-30T19:53:11Z

this commit introduces a new Rollup ILM Action that allows indices
to be rolled up according to a specific rollup config. The
action also allows for the new rolled up index to be associated with
a different policy than the original/source index.

Relates #42720.

Closes #48003.

elasticmachine · 2020-12-07T23:34:26Z

Pinging @elastic/es-analytics-geo (Team:Analytics)

this commit introduces a new Rollup ILM Action that allows indices to be rolled up according to a specific rollup config. The action also allows for the new rolled up index to be associated with a different policy than the original/source index. Optionally, the original index can be deleted. Relates elastic#42720. Closes elastic#48003.

not-napoleon

This looks good to me, but probably worth having someone from ILM review as well.

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/RollupILMAction.java

talevy · 2020-12-09T19:06:01Z

run elasticsearch-ci/default-distro

csoulios

This is great work and it looks a lot simpler than I had expected.
I left some comments wrt to rollup specific points for discussion.
Also, I am no expert in ILM. So, perhaps someone with more experience in this area could help

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/RollupStep.java

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/RollupILMAction.java

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/TimeseriesLifecycleType.java

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/RollupILMAction.java

docs/reference/ilm/ilm-actions.asciidoc

jrodewig

Docs LGTM. Thanks @talevy!

docs/reference/ilm/actions/ilm-rollup.asciidoc

Co-authored-by: James Rodewig <[email protected]>

talevy · 2021-01-27T20:35:29Z

thanks all! ready for another review!

dakrone

This LGTM, thanks for iterating on this Tal!

I left one more comment, but it can be addressed either here or in a subsequent PR (if we decide a change is desired), up to you.

dakrone · 2021-01-28T22:10:50Z

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/TimeseriesLifecycleType.java

+    static {
+        if (RollupV2.isEnabled()) {
+            ORDERED_VALID_HOT_ACTIONS = Arrays.asList(SetPriorityAction.NAME, UnfollowAction.NAME, RolloverAction.NAME,
+                ReadOnlyAction.NAME, ShrinkAction.NAME, ForceMergeAction.NAME, RollupILMAction.NAME, SearchableSnapshotAction.NAME);


I missed this before, but if we're going to rollup right after a shrink and force merge, what's the point of shrinking and force merging? Should we just rollup first and then do the shrink/forcemerge?

(I think this is okay to leave as is for now, but we might want to investigate making them mutually exclusive in the future so we don't waste resources doing a shrink and force merge right before we rollup and create a new index anyway. At the very least I think maybe force merge should move to after the Rollup?)

these are fair points. I figured these things would occur anyways, so order should not matter at the end of the day. That being said, waiting to do these operations before rolling up is not necessary. it might be interesting to benchmark and see if fewer shards ends up resulting in a faster rollup or not. but this should not be of concern to the user anyways.

I will move Rollup to happen before Shrink, but will bring this up with the team to see if there are any other opinions

this commit introduces a new Rollup ILM Action that allows indices to be rolled up according to a specific rollup config. The action also allows for the new rolled up index to be associated with a different policy than the original/source index. Relates elastic#42720. Closes elastic#48003.

ruflin · 2021-01-29T07:49:06Z

docs/reference/ilm/actions/ilm-rollup.asciidoc

+
+For more information about rollup, see the <<rollup-api, rollup action documentation>>
+
+The name of the rolled up index will be the original index name of the managed index prefixed


@talevy How does this work in the context of data streams? Does it work in the context of data streams? What are the index names there?

ILM works on the index level, not directly on data streams. You can roll up a backing index for a data stream but not a data stream itself.

If the index is a backing index for a data stream, the rollup index is a backing index for the same stream. The name of the new rollup index is based on the name of the backing index. For example, the name of a rollup index for .ds-my-data-stream-2021.01.29-000001 would be rollup-.ds-my-data-stream-2021.01.29-000001.

I hope that helps. I'll open up a follow-up PR to better clarify this in the docs.

Hey @ruflin and @jrodewig,

this is exactly how it works today, but due to some failure-scenarios we discussed recently, we're going to change this behavior.

In the context of datastreams, we will likely randomize the name of the rollup index to prevent collision and conflict with any past or rogue rollup actions. The belief is that the underlying name of the index should not be of concern to users since they will be added into the datastream and managed in ILM. Do you see any problems with this on your end @ruflin?

For this reason, it might make sense to hold off just a little bit on your docs change PR @jrodewig, but much appreciated!

I've opened #68225 to track it. Will likely change next week

Thanks for the update, @talevy!

The belief is that the underlying name of the index should not be of concern to users since they will be added into the datastream and managed in ILM

If this is the case, it is great. I was under the assumption all indices belong to a data stream must use the .ds-* prefix and be hidden. So I expect index names like .ds-rollup-* or something similar. This would also ensure there are no conflicts and makes it clear that these rollups belong to the data streams.

The rollup- prefix is what put me on the wrong track here and made me assume it does not belong to the data stream.

it is actually common for indices managed by ILM to be renamed with a new prefix. For example, shrunken indices start with the shrunk- prefix.

independent of the name, the indices will be hidden and added to the backing datastream

@talevy Interesting. Why are these not prefixed by a . and not inside .ds-*-shrunk-* or similar to indicate they belong together? I think I'm derailing but could you share a link where I can read up on this? The part I'm concerned is that this not seem to fully fit into our naming scheme: https://www.elastic.co/blog/an-introduction-to-the-elastic-data-stream-naming-scheme

this commit introduces a new Rollup ILM Action that allows indices to be rolled up according to a specific rollup config. The action also allows for the new rolled up index to be associated with a different policy than the original/source index.

talevy added >non-issue WIP :StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data v8.0.0 Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v7.11.0 labels Nov 30, 2020

talevy force-pushed the rollup-ilm branch 5 times, most recently from 05169e9 to 33d61ad Compare November 30, 2020 23:15

talevy mentioned this pull request Nov 30, 2020

Refactor rollups meta (AKA Rollup V2) #42720

Closed

21 tasks

talevy force-pushed the rollup-ilm branch 2 times, most recently from 7a73bd4 to c09d9a7 Compare December 7, 2020 23:33

talevy removed the WIP label Dec 7, 2020

talevy marked this pull request as ready for review December 7, 2020 23:34

talevy requested review from csoulios and not-napoleon December 7, 2020 23:34

talevy force-pushed the rollup-ilm branch from c09d9a7 to 176af36 Compare December 8, 2020 05:48

talevy added 2 commits December 8, 2020 07:38

Merge remote-tracking branch 'elastic/master' into rollup-ilm

ca3e695

Merge remote-tracking branch 'elastic/master' into rollup-ilm

3631adc

not-napoleon approved these changes Dec 9, 2020

View reviewed changes

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/RollupILMAction.java Outdated Show resolved Hide resolved

talevy added 2 commits December 9, 2020 10:13

Merge remote-tracking branch 'elastic/master' into rollup-ilm

184c7ca

respond to review

434f49c

csoulios reviewed Dec 14, 2020

View reviewed changes

talevy added 2 commits December 14, 2020 10:39

Merge remote-tracking branch 'elastic/master' into rollup-ilm

fe1f124

fix for latest master

638cbe7

talevy added 4 commits January 26, 2021 13:36

Merge remote-tracking branch 'elastic/master' into rollup-ilm

96a1622

docs and cleanup

c9e31d5

add docs to ILM Actions page

1d1fc31

place ILM Rollup Docs behind unreleased branch conditional

49b5802

talevy commented Jan 26, 2021

View reviewed changes

docs/reference/ilm/ilm-actions.asciidoc Show resolved Hide resolved

jrodewig approved these changes Jan 27, 2021

View reviewed changes

docs/reference/ilm/actions/ilm-rollup.asciidoc Outdated Show resolved Hide resolved

docs/reference/ilm/actions/ilm-rollup.asciidoc Outdated Show resolved Hide resolved

jrodewig reviewed Jan 27, 2021

View reviewed changes

docs/reference/ilm/actions/ilm-rollup.asciidoc Outdated Show resolved Hide resolved

talevy and others added 3 commits January 27, 2021 12:34

Update docs/reference/ilm/actions/ilm-rollup.asciidoc

e340f07

Co-authored-by: James Rodewig <[email protected]>

Update docs/reference/ilm/actions/ilm-rollup.asciidoc

7efb1a3

Co-authored-by: James Rodewig <[email protected]>

Update docs/reference/ilm/actions/ilm-rollup.asciidoc

10487aa

Co-authored-by: James Rodewig <[email protected]>

talevy requested a review from dakrone January 27, 2021 20:35

Merge remote-tracking branch 'elastic/master' into rollup-ilm

382915d

jrodewig mentioned this pull request Jan 27, 2021

[DOCS] Rollup refactor docs #65515

Closed

26 tasks

dakrone approved these changes Jan 28, 2021

View reviewed changes

talevy added 2 commits January 28, 2021 15:19

Merge remote-tracking branch 'elastic/master' into rollup-ilm

e01d063

prioritize rollup before shrink/forcemerge in hot phase

ec30764

talevy merged commit 5811deb into elastic:master Jan 29, 2021

talevy deleted the rollup-ilm branch January 29, 2021 01:52

talevy added the backport pending label Jan 29, 2021

ruflin reviewed Jan 29, 2021

View reviewed changes

talevy mentioned this pull request Jan 29, 2021

Randomize Rollup Index name when one is not provided to Rollup Action #68225

Closed

williamrandolph removed the backport pending label Feb 19, 2021

talevy mentioned this pull request Feb 24, 2021

Generate random rollup index names for RollupILMAction #69237

Merged

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021


		For more information about rollup, see the <<rollup-api, rollup action documentation>>

		The name of the rolled up index will be the original index name of the managed index prefixed

Add Rollup ILM Action #65633

Add Rollup ILM Action #65633

Uh oh!

Conversation

talevy commented Nov 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Dec 7, 2020

Uh oh!

not-napoleon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

talevy commented Dec 9, 2020

Uh oh!

csoulios left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jrodewig left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

talevy commented Jan 27, 2021

Uh oh!

dakrone left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jrodewig Jan 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

talevy commented Nov 30, 2020 •

edited

Loading

jrodewig Jan 29, 2021 •

edited

Loading