Skip to content

Allow for setting the total shards per node in the Allocate ILM action #76134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 11 commits into from
Aug 11, 2021
Merged

Allow for setting the total shards per node in the Allocate ILM action #76134

merged 11 commits into from
Aug 11, 2021

Conversation

@masseyke
Copy link
Member

@masseyke masseyke commented Aug 4, 2021

This adds a new optional field to the allocate ILM action called "total_shards_per_node". If present, the value of this field
is set as the value of "index.routing.allocation.total_shards_per_node" before the allocation takes place.

@masseyke masseyke requested a review from dakrone August 4, 2021 21:19
@masseyke masseyke added the Team:Data Management Meta label for data/management team label Aug 4, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Aug 4, 2021

Pinging @elastic/es-core-features (Team:Core/Features)

@masseyke
Copy link
Member Author

masseyke commented Aug 4, 2021

@elasticmachine update branch

@masseyke masseyke marked this pull request as draft August 4, 2021 21:55
@masseyke masseyke marked this pull request as ready for review August 5, 2021 14:42
dakrone
dakrone requested changes Aug 11, 2021
View reviewed changes
Copy link
Member

@dakrone dakrone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this Keith, I left a few comments

@masseyke
Copy link
Member Author

masseyke commented Aug 11, 2021

@elasticmachine update branch

@masseyke
Copy link
Member Author

masseyke commented Aug 11, 2021

Note that one catch on this PR is that if you set the total_number_of_shards in your allocate action to less than the number of shards in the current index, then shrinking will just hang (because the step in shrinking that copies all shards to one node will be blocked). This was mentioned in #44070 but hasn't been addressed in this PR.

@masseyke masseyke linked an issue Aug 11, 2021 that may be closed by this pull request
Allow adjustment of index resource constraints in ILM phase transitions #44070
Closed
@masseyke masseyke requested a review from dakrone August 11, 2021 19:46
dakrone
dakrone approved these changes Aug 11, 2021
View reviewed changes
Copy link
Member

@dakrone dakrone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, though I think in the interest of user friendliness this should go into 8.0 and 7.16+ instead of 7.15, so that we have time to work on the changes to the shrink action that makes it not possible to wedge itself (which should be done in a followup PR I think)

@masseyke masseyke merged commit 8610db6 into elastic:master Aug 11, 2021
@masseyke masseyke deleted the feature/allow-changing-total_shards_per_node-in-ilm branch August 11, 2021 21:36
@ppf2
Copy link
Contributor

ppf2 commented Aug 11, 2021

Thx! def +++++ on the backport to 7.1x

@masseyke masseyke mentioned this pull request Aug 19, 2021
Merged
@masseyke
Copy link
Member Author

masseyke commented Aug 19, 2021

Note that one catch on this PR is that if you set the total_number_of_shards in your allocate action to less than the number of shards in the current index, then shrinking will just hang (because the step in shrinking that copies all shards to one node will be blocked). This was mentioned in #44070 but hasn't been addressed in this PR.

This PR fixes the above problem: #76732

masseyke added a commit to masseyke/elasticsearch that referenced this pull request Aug 20, 2021
@masseyke
Allow for setting the total shards per node in the Allocate ILM action (
f6b4df8 
elastic#76134)

This adds a new optional field to the allocate ILM action called "total_shards_per_node". If present, the
value of this field is set as the value of "index.routing.allocation.total_shards_per_node" before the allocation
takes place.
Relates to elastic#44070
@masseyke masseyke mentioned this pull request Aug 20, 2021
Merged
masseyke added a commit that referenced this pull request Aug 23, 2021
@masseyke
Allow for setting the total shards per node in the Allocate ILM action (
0f9da2e 
#76775)

Allow for setting the total shards per node in the Allocate ILM action (#76134)

This is a backport of #76134. It adds a new optional field to the allocate ILM action called
"total_shards_per_node". If present, the value of this field is set as the value of "index.routing.allocation.total_shards_per_node" before the allocation takes place.
Relates to #44070
@dakrone dakrone added :Data Management/ILM+SLM Index and Snapshot lifecycle management v7.16.0 labels Aug 23, 2021
@cjcenizal cjcenizal mentioned this pull request Jan 18, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dakrone dakrone dakrone approved these changes

Assignees

No one assigned

Labels

:Data Management/ILM+SLM Index and Snapshot lifecycle management >enhancement Team:Data Management Meta label for data/management team v7.16.0 v8.0.0-alpha2

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Allow adjustment of index resource constraints in ILM phase transitions

7 participants

@masseyke @elasticmachine @ppf2 @dakrone @jakelandis @danhermann @elasticsearchmachine