[BACKPORT] Adds Index lifecycle feature #35194
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Also tweaks some of the code in the steps of the allocate action
Also adds tests for those steps and ReplicasAction.toSteps
The force-merge is an a TODO state due to the unresolved issue around best_compression. - updated ReadOnlyStep with tests - implemented an update to the ForceMergeAction - added UpdateBestCompressionSettingsStep - added tests for SegmentCountStep
Various classes had some code that was not used and is not going to be needed so this change cleans up those classes so we don’t have dead code hanging around
Also renames EnoughShardsWaitStep to ReplicasAllocatedStep, removes it from the allocate action and adds a check that th number of replicas in the cluster state is correct to it.
- added ShrinkStep/Tests - AsyncActionStep now passes in IndexMetaData instead of Index - Delete usage of ClusterStateActionStep - with ClusterStateActionStep gone, InitializePolicyContextStep is the only other ClusterState-nonWait step - Migrate setting-updates to UpdateSettingsStep
Adds some missing tests including checking the hashcode and equals methods of `DeleteStep`, `StepKey`, and `TerminalPolicyStep` as well as adding a test for `DeleteAction.toSteps()`
Specifically for the initialPolicyContextStep and the PhaseAfterStep
this resulted in a few new steps being introduced - AliasStep - EnoughShardsWaitStep - ShrinkStep - ShrunkenIndexCheckStep
- make DeleteActionTests pass - remove getType from LifecycleType
To make it in line with the fact that other equality tests use multiple iterations
Also fixes failing EnoughShardsWaitStep
- Renames EnoughShardsWaitStep to ShrunkShardsAllocatedStep - Changes ShrunkShardsAllocatedStep to check the shards of the shrunken index rather than the current one - shrink index prefix is now passed into the steps of the shrink aciton - Related Test Changes
This is needed so that clients have the ability to serialise and deserialise the objects it creates and gets back.
…4387) Previously, phase X's `after` step had `X` as its associated phase. This causes confusion because we have only entered phase `X` once the `after` step is complete. Therefore, this refactor pushes the after's phase to be associated with the previous phase. This first phase is an exception. The first phase's `after` step is associated with the first phase (not some non-existent prior phase).
This contains updates to the ILM Docs explaining how updates to policies work and affect policy execution on indices.
ILM would return a resource-not-found exception when requesting policies while the IndexLifecycleMetaData is not initialized. The behavior here should not be as extreme since it is not the user's fault. This commit changes the behavior so that it succeeds and returns no policies when no policy names are explicitely specified, otherwise keep the same behavior of throwing an exception
In some cases it wasn't mutating and caused test failures
we are restructuring the docs, this migrates ILM docs outside of the x-pack doc structure.
The Move To Step API now checks to see if the target step is an AsyncActionStep, and if so, runs it. Previously, AsyncActionSteps would only be run when they are entered by executing the previous step, so if an AsyncActionStep was entered via the Move To Step API, ILM would never touch that index again.
Previously, if ClusterStateActionSteps or ClusterStateWaitSteps threw an exception executing, the exception would only be caught and logged by the generic ClusterStateUpdateTask machinery and the index would become stuck on that step. Now, exceptions thrown in these steps will be caught and the index will be moved to the Error step.
This PR renames the CRUD APIS for ILM GET _ilm/<policy>, _ilm -> _ilm/policy/<policy>, _ilm/policy PUT _ilm/<policy> -> _ilm/policy/<policy> DELETE _ilm/<policy> -> _ilm/policy/<policy> closes #34929.
… alias (#35065) The ILM Rollover Step can execute on the incorrect index if the rollover alias exists on another valid index, but not the one the step is executing against. This is a problem and is now guarded against
…ef (#35070) * [DOCS] Fixed edit links for ILM APIs and added the APIs to the REST API section. * [DOCS] Fixed link to ILM APIs.
the settings variable was previously created by the AbstractComponent class inherited by IndexLifecycleService. this is no more.
This commit does a few things - moves ILM-specifc rest yaml tests into plugin/ilm/qa, and creates special :plugin:ilm:qa:rest module to test them - removes the with-security tests of the yaml tests since they are covered in the rest tests now - moves ChangePolicyforIndexIT into the qa/multi-node project since that test is not currently running in main ilm since integTest is disabled
ILM's Shrink Action was using a nodes "_name" attribute to allocate to prepare for the shrink step. Since the name is configurable by a user and may use the same name for multiple nodes on one machine, _id is safer since it is guaranteed to be unique. closes #35043.
colings86
added
:Data Management/ILM+SLM
Collaborator
|
Pinging @elastic/es-core-infra |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of #35193