Allows multiple patterns to be specified for index templates

[a2lin]

Index templates now take an list for the template: field instead of a string.

For example:

{
  "index_patterns":  ["c*", "d*"],
}

Instead of the previous form

{
  "template": "c*"
}

Template metadata from older versions of ES are still openable with this code, although I don't know how important that is since it's probably assumed that you can't just open a new elasticsearch release on an old one's data directory and expect it to work.

Closes #20690

[nik9000]

@a2lin, Sorry to let this sit for a few days. I've added this to my work queue (browser tabs...) and I promise I'll get to it before too long. My queue is getting longer than I'd like these days.

[nik9000]

I think it is worth doing something like ((List<?>) entry.getValue()).stream().map(Object::toString).collect(toList()) so we are sure we have strings here. As you've written it now if someone sticks [0] in the template field then we'll cram an Integer in the list and we'll get ClassCastExceptions in weird places.

[nik9000]

Maybe "should be a string or a list of strings".

[nik9000]

OK! New fun things for you to learn about. The way we support mixed version cluster (like, during a rolling restart) is by adding branches to this code. So this needs to read like:

if (in.getVersion().onOrAfter(Version.5_1_0)) {
  template = in.readList(StreamInput::readString);
} else {
  template = singletonList(in.readString());
}

And the out needs to have the same branching. And you need to find some place to cram the test for this.

The specific version number that I chose is based on my expectation of which release will be the first to see this feature.

[nik9000]

It is good that you've added single string support - this is needed so we can add this feature without breaking backwards compatibility. We could chose to delay this to 6.0 and only support lists here but I don't think that is a good idea anyway.

[nik9000]

You could add a new method that takes a String does a singletoneList on the input for backwards compatibility but we don't tend to do that for these builders because we expect that users of the Java API can deal with these changes on upgrade because the compiler will catch it for them. In other words: we don't respect semver in the Java API.

[nik9000]

I'd just do "[" + String.join(", ", indexData.getTemplate()) + "]". I was a big fan of String.format for a while but I've fallen out of love. And the compiler is quite good at optimizing + for Strings.

[nik9000]

I do really with it were indexData.getTemplate*s*() or indexData.getPatterns() though.

[nik9000]

Personally I do a static import for singletonList but I know lots of folks don't. If you want to keep it as is that is fine with me, or if you want to do static import that is fine with me.

[nik9000]

a simple pattern or a list of simple patterns?

[nik9000]

I might make this one ["te*", "bar*"] but then leave the others as strings rather than lists.

[nik9000]

Can you fix the \s+ so it lines up with the \s+ above it?

[a2lin]

@nik9000 Sorry for the delayed response! I think I fixed all the points.

There's a bit of oddness left, but I can fix #18922 (rename template to index_pattern in the rest api) after this one and that should change everything.

I've added the setTemplate(String indexPattern) api back to the PutIndexTemplateRequestBuilder, with @deprecated attached, I thought it might be nice since the method got renamed. I can change it back if this is against convention; I was unsure about your earlier comment on this part :)

For the in/out tests I used:

    public void testToBytes() throws IOException{
        BytesStreamOutput output = new BytesStreamOutput();
        output.setVersion(Version.V_5_0_0_rc1);
                IndexTemplateMetaData data = IndexTemplateMetaData.builder("foo")
            .template("bar")
            .order(1)
            .settings(Settings.builder()
                .put("setting1", "value1")
                .put("setting2", "value2"))
            .putAlias(newAliasMetaDataBuilder("alias-bar1")).build();
        data.writeTo(output);
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        output.bytes().writeTo(baos);
        System.out.println(Base64.getEncoder().encodeToString(baos.toByteArray()));
    }

and a similar variant for the PutIndexTemplateRequest to generate the base64 strings used as constants.

Thanks again for reviewing!

[nik9000]

I've left a few very minor things. I think this is just about ready! Thanks for being so patient with me. I keep getting distracted and forgetting about this. Sorry!

Anyway, I mention rebasing in one of my minor points. If we didn't have any remaining issues I'd say to just squash and rebase and fix those points. But since we have some open stuff I'd wait on those points until later.

[nik9000]

We actually prefer == false or false == of !. Well, we have one core contributor who really prefers it and the rest of us are fine with it. false == is unlikely to be misread, whereas ! is small and can get lost if you scan the code quickly. So I'd change this back. It is weird, yes, but it is our kind of weird.

[nik9000]

👍

[nik9000]

If you rebase you can use V_5_0_0 which is more correct, I think. Both are fine, but we really expect wire compatibility with 5.0.0, not 5.0.0-rc1.

[nik9000]

Same comment about the version constant here.

[nik9000]

OK. I think we should make this index_patterns or patterns. We're already making a breaking change here (string -> list) so let's make the output more readable while we are here. I'm unsure if this means we can't merge this to the 5.x branch or not.... I hadn't thought this through until now. It is is a fairly out of the way section of the code....

[nik9000]

This is all the docs I could find on this API. So it is fairly vague about the return format.

[a2lin]

@nik9000 I've addressed the bit about the boolean and also the ClusterState.
edit: Was a bit too eager with the IndexTemplateMetaData; changing the 'template' there to 'index_patterns' would be a breaking REST api change, so I left that for a follow-up.

Thanks again for the reviews! I'll rebase on top of master and fix the versions (& the VersionTests thing) if these changes seem okay.

[nik9000]

edit: Was a bit too eager with the IndexTemplateMetaData; changing the 'template' there to 'index_patterns' would be a breaking REST api change, so I left that for a follow-up.

I think you are already breaking there because you change from a string to a list. Honestly I think you should just change it. Either we say "this has to go into 6.0 because of this" or we say "this part of the API is so deep we're willing to break it in 5.1". I don't think you can get away with not switching to a list though.

If you are willing to make the IndexTemplateMetaData change then I think it is fine to go ahead and squash and rebase to pick up the Version constant.

We should also add a note to the breaking changes docs about the IndexTemplateMetaData format but I'm unsure if it should be in the 6.0 or 5.1 docs.

[a2lin]

@nik9000 Okay, I rebased and fixed things! (This change solves #18922 as well now since the metadata causes index_patterns to be exposed on GET).

A summary of the changes:

For the put_template api:

curl -XPUT 'localhost:9200/_template/template_1?pretty' -d'
{
  "index_patterns": ["zzz*"],
  "order": 1,
  "version": 1,
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "type1": {
      "_source": {
        "enabled": false
      },
      "properties": {
        "field1": {
            "type": "text",
            "store": "false"
        },
        "field2": {
            "type": "keyword",
            "store": "true"
        }
      }
    }
  }
}'

put_template also works with (in the string case)

"index_patterns": "zzz*"

and also with

"template": "zzz*"

giving the same get_template in each case. Thus, the legacy template field only takes a string, and index_patterns takes both string and array.

Directly specifying the parameters for put_template also works:
curl -XPUT 'localhost:9200/_template/template_2?index_patterns=zzz*,eee*&order=5' -d {}
The legacy parameter template still works as well:
curl -XPUT 'localhost:9200/_template/template_3?template=zzz*&order=5' -d {}

Support for template (at least as part of the source for PutIndexTemplateRequest) seems to be necessary to pass the testBeatsTemplatesBWC and testLogstashTemplatesBWC tests.

Performing a get_template operation for any of the above should return:

{
  "template_1": {
    "order": 1,
    "version": 1,
    "index_patterns": [
      "zzz*"
    ],
    "settings": {
      "index": {
        "number_of_shards": "1"
      }
    },
    "mappings": {
      "type1": {
        "_source": {
          "enabled": false
        },
        "properties": {
          "field1": {
            "store": "false",
            "type": "text"
          },
          "field2": {
            "store": "true",
            "type": "keyword"
          }
        }
      }
    },
    "aliases": {}
  }
}

The cat_templates api returns:

name       index_patterns order version
template_1 [zzz*]         1     1

and for multiple patterns:

name       index_patterns order version
template_3 [zzz*]         5
template_6 [zzz*, eee*]   1     1

As for the Java PutIndexRequestBuilder API:
setTemplate(String) still remains (but marked with deprecated), but there is a new method
setPatterns(List<String>) for future use.

Going to and from pre-(this patch):
From 5.0.0 towards 5.1.0, legacy 'template' fields in the metadata are upgraded to a list and placed as singletonLists under 'index_patterns'.

If you do a backwards roll from 5.1.0 back to 5.0.0, all of the template patterns will be seen as null (as they have been rewritten in the metadata to use the index_patterns field instead of the template field). A follow-up forward roll will not fix the issue.
Therefore, this change is only be safe in the roll-forward direction.

Sorry for the essay, and thanks again for all the reviews and comments!

[clintongormley]

Hi @a2lin

Thanks for all of the effort you've put into this. It'd be good to have some deprecation logging when the user specifies template instead of index_pattern.

[nik9000]

It'd be good to have some deprecation logging when the user specifies template instead of index_pattern.

See ParseField#withAllDeprecated.

[nik9000]

If you do a backwards roll from 5.1.0 back to 5.0.0,

So so so so much stuff is likely to break if you try to downgrade.

On the other hand, in a mixed version cluster you might get this behavior you describe if you query a node on the old version.

[nik9000]

@clintongormley what is your opinion on the target branch for this? I'd love the change to go into 5.x but it breaks backwards compatibility on the return for _cat/templates and _cluster/state, renaming the template field to index_patterns and making it a list. I figured since we were already changing from a string to a list then the rename is worth doing too. I certainly see good arguments for targeting this for release with 5.1 and for targeting it as 6.0 only.

[nik9000]

I left two super minor things and @clintongormley left one a request to add deprecation logging when you use the old parameter.

[nik9000]

Could you copy the javadoc on the method below to this method as well? Including the @Deprecate javadoc annotation telling folks to use #setPatterns instead?

[nik9000]

I'd change the comment to something like "prior to 5.1 Elasticsearch only supported a single index pattern and called it template."

[a2lin]

@nik9000 I've completed the changes. I found it easier to just use DeprecationLogger directly, though. I'll leave swapping the ones in the source and fromXContent methods out for ParseField#withAllDeprecated for a follow-up (converting these to ObjectParser)?

[nik9000]

Looks like we have some failing documentation tests. You should be able to reproduce with gradle docs:check. Can you fix them?

[nik9000]

Can you fix them?

Debugging docs tests can be a bit of a trip. Have a look in docs/README.asciidoc for some hints on what is going on.

[nik9000]

I think that the docs failures were caused by the deprecation logging. Deprecation logging causes Warning HTTP headers to be returned with the response. The tests fail unless those headers match exactly what is declared in the text. I added that failing a while back so that if we deprecated something then the build would fail and we'd either convert the snippets to the new syntax or we'd mark the snippets as being deprecated. The build failure forces us to look at the document. We were missing a lot of things and this helped....

Anyway, I think you can fix these failures by using the new name.

[a2lin]

@nik9000 Sorry about that :(. gradle docs:check passes for me on this new version.

[nik9000]

@nik9000 Sorry about that :(. gradle docs:check passes for me on this new version.

No problem! It is sneaky. I'm retesting one last time locally to be super ultra paranoid but everything is looking good so far.

[nik9000]

Merged! Thanks again @a2lin!

[nik9000]

I still owe a breaking changes doc patch for this. I'll write one in the morning.

[a2lin]

Thanks for all the help and reviews!

[nik9000]

Breaking changes docs: 7c38867

[m9aertner]

Sorry for asking (and not changing this myself via pull request), it would be great to add a statement to the docs just what that "pattern" can be. I was hoping for a "regexp", but it looks like it is in fact a "glob". Most examples out there suggest it might be a "suffix asterisk" only, Thanks.

[ruflin]

@nik9000 I assume you mean here pre-6.0 templates. Or will there be support for indext-patterns in the 5.x releases?