Expand template "index_pattern" matching to allow negative matching #62681
Description
Following from this discussion: https://discuss.elastic.co/t/template-index-pattern-matching/249289
And this: https://discuss.elastic.co/t/defining-index-mappings-on-elasticsearch/248774/6
It would be helpful to add a way to define templates on all new indices in ES, except those matching a condition. Right now the only way to do this is to use wildcard matching, which is quite difficult to apply in real-case scenarios.
Let's say I want to store documents that represent movies in my ElasticSearch engine, and I want to store each movie in a different index for each production company. I would have indices for warnerbros, disney, 20thcentury, etc. I know I won't store any other kind of object in my engine, so I want all indices to share the same mapping. So it would make sense to define a template right after spinning up the ElasticSearch instance, so that any new index created when indexing a document has the mappings defined in my template.
Right now I have two ways of doing this:
- either I prepend a common string to all the index names, so that I store them in, say: m-warnerbros, m-disney, m-20thcentury, and then define the template with the field:
"index_patterns": ["m-*"] - or I list all the production companies in the template request, like
"index_patterns": ["warnerbros", "disney", "..."]
The first solution is not very practical, it forces me to manipulate the index name, and if I want to infer the index name automatically from the contents of my documents this won't work. The other solution is tedious and won't scale well if I need to add new productions companies that didn't previously exist.
Using a pattern matching with just the wildcard, i.e. "index_patterns": ["*"], causes problems with the .log* and .metrics* indices, and if I define an alias in the template then it applies to any new index created, including logs and metrics.
My suggestion is to allow negative matching, so that I can define a template on all new indices that do not match a certain pattern. This way I could exclude the logs and metrics indices from the template. Something along the lines of:
"index_patterns_match": ["*"],
"index_patterns_exclude": [".*", "-*",]