[ML] Switch ML internal index templates to composable templates #73232
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Legacy index templates are deprecated but ML was still using them for its hidden indices. This PR switches the legacy ML index templates to use the new composable index template framework. The composable index templates get installed once the master node is on a version that understands them. For templates that need to be up-to-date in mixed version clusters where the master might still be on a version that doesn't understand composable index templates we still ship the legacy template too, and install this if required in the mixed version cluster. (The notifications index template falls into this category.) This makes a couple of places in the code a little messy, as the new style template definitions don't contain a dummy _doc level (where the type used to be), but the legacy template definitions do - hopefully we can tidy this up in master once 8.0 is released. There is one more change of note in this PR that is not strictly related to switching to composable templates, but which was shown up during the testing. We used to wait for all templates to be installed by the master node before running tests in mixed version clusters. I do not believe we should have been doing this, as other upgrade orchestration systems, e.g. Cloud, will not be doing this. Our production code needs to install templates and/or mappings before any operation that requires them if there's a chance that the elected master won't have done this in time. Fixes elastic#65437
Collaborator
|
Pinging @elastic/ml-core (Team:ML) |
droberts195
pushed a commit
to droberts195/elasticsearch
that referenced
this pull request
May 24, 2021
Legacy index templates are deprecated but ML was still using them for its hidden indices. This PR switches the legacy ML index templates to use the new composable index template framework. The composable index templates get installed once the master node is on a version that understands them. For templates that need to be up-to-date in mixed version clusters where the master might still be on a version that doesn't understand composable index templates we still ship the legacy template too, and install this if required in the mixed version cluster. (The notifications index template falls into this category.) This makes a couple of places in the code a little messy, as the new style template definitions don't contain a dummy _doc level (where the type used to be), but the legacy template definitions do - hopefully we can tidy this up in master once 8.0 is released. There is one more change of note in this PR that is not strictly related to switching to composable templates, but which was shown up during the testing. We used to wait for all templates to be installed by the master node before running tests in mixed version clusters. I do not believe we should have been doing this, as other upgrade orchestration systems, e.g. Cloud, will not be doing this. Our production code needs to install templates and/or mappings before any operation that requires them if there's a chance that the elected master won't have done this in time. Backport of elastic#73232
droberts195
pushed a commit
that referenced
this pull request
May 25, 2021
Legacy index templates are deprecated but ML was still using them for its hidden indices. This PR switches the legacy ML index templates to use the new composable index template framework. The composable index templates get installed once the master node is on a version that understands them. For templates that need to be up-to-date in mixed version clusters where the master might still be on a version that doesn't understand composable index templates we still ship the legacy template too, and install this if required in the mixed version cluster. (The notifications index template falls into this category.) This makes a couple of places in the code a little messy, as the new style template definitions don't contain a dummy _doc level (where the type used to be), but the legacy template definitions do - hopefully we can tidy this up in master once 8.0 is released. There is one more change of note in this PR that is not strictly related to switching to composable templates, but which was shown up during the testing. We used to wait for all templates to be installed by the master node before running tests in mixed version clusters. I do not believe we should have been doing this, as other upgrade orchestration systems, e.g. Cloud, will not be doing this. Our production code needs to install templates and/or mappings before any operation that requires them if there's a chance that the elected master won't have done this in time. Backport of #73232
This was referenced Oct 13, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Legacy index templates are deprecated but ML was still using
them for its hidden indices.
This PR switches the legacy ML index templates to use the new
composable index template framework.
The composable index templates get installed once the master
node is on a version that understands them. For templates
that need to be up-to-date in mixed version clusters where the
master might still be on a version that doesn't understand
composable index templates we still ship the legacy template
too, and install this if required in the mixed version cluster.
(The notifications index template falls into this category.)
This makes a couple of places in the code a little messy, as
the new style template definitions don't contain a dummy _doc
level (where the type used to be), but the legacy template
definitions do - hopefully we can tidy this up in master once
8.0 is released.
There is one more change of note in this PR that is not
strictly related to switching to composable templates, but
which was shown up during the testing. We used to wait for
all templates to be installed by the master node before running
tests in mixed version clusters. I do not believe we should
have been doing this, as other upgrade orchestration systems,
e.g. Cloud, will not be doing this. Our production code needs
to install templates and/or mappings before any operation that
requires them if there's a chance that the elected master won't
have done this in time.
Fixes #65437