Alias types: read_only vs read_write

[elasticmachine]

Original comment by @clintongormley:

In Index Lifecycle Management we have come across a problem with rollover which we hope to solve with a change to aliases.

Problem 1: Rollover requires an alias

Rollovers creates a new index, then changes the alias from pointing to the old index to point to the new index.

The alias name is the only index-specific value that is needed by the ILM policy today, because of the rollover action. If we manage to remove this requirement, then the same policy could be applied to several different index patterns/templates. The rollover action in ILM needs a way to inspect the index to know which alias should be swapped. (eg you shouldn't swap an alias like all-logs which is intended for reading).

Problem 2: Coordinating multiple settings

We could add a setting like index.lifecycle.rollover_alias to know which alias to use, but we also need to add the alias to the first index to bootstrap the process. It would be more convenient to have a single way of specifying the alias.

Problem 3: A multi-index alias cannot be used for writing

The rollover alias cannot be applied to more than one index, otherwise it cannot be used for indexing.

Beats or Logstash need to be able to bootstrap ILM and indices by:

• Putting the ILM policy if it doesn't exist
• Putting the index template if it doesn't exist
• Creating the initial index (eg mysql-logs-2018.01.01-00001) with the appropriate alias: mysql-logs-write)

Imagine a thousand beats doing this near midnight: you could create two indices with different dates, both of which have the mysql-logs-write alias, so causing indexing to fail. Ideally, we need a way to stop index creation if the alias is already in use.

Proposal

• We add a type or mode parameter to aliases, which can be set to read_only or read_write.
• read_only aliases can point to multiple indices.
• read_write aliases can only point to a single index, and an exception will be thrown at index creation time or alias update time if the user tries to assign a read_write alias to more than one index.
• When rollover is called on an index (instead of an alias), then all read_write aliases will be switched to the new index, but read_only aliases will not be changed.
• For BWC, any aliases which currently point to multiple indices will automatically become read_only, but single-index aliases will remain unspecified. We add a deprecation warning to say that the type will become mandatory in 7.0

[elasticmachine]

Original comment by @MorrieAtElastic:

Please make sure that the exception associated with mapping multiple indexes to a read-write alias is clearly stated, because this is precisely the kind of error that is likely to be made.

Other than that I can see no problems with implementing read-write aliases.

[elasticmachine]

Original comment by @pickypg:

We add a deprecation warning to say that the type will become mandatory in 7.0

What happens for users that don't need to create / set any new aliases between now and then, thus they never see the deprecation? What happens in 7.0? They are automatically set to read_write?

[elasticmachine]

Original comment by @s1monw:

When rollover is called on an index (instead of an alias), then all read_write aliases will be switched to the new index, but read_only aliases will not be changed.

I don't understand that, you are switching all aliases or only the ones that point to this specific index? What if there are more than one write alias that point to it?

Also can you apply write ops to an alias with 1 index that is read_only? I think this is very very confusing if that is the case.

I also wonder what the implications are that we are going from an O(1) pattern to a O(n) pattern. I don't think we should do such a change without considering the implications (I am not sure what they are).

In Index Lifecycle Management we have come across a problem with rollover which we hope to solve with a change to aliases.

if we do this are we sure we won't need any specific parameters in the future and is that worth the trouble?

FWIW I really like specifying the alias from the index via a setting wich can be a one-off this is less ambiguous IMO. The proposal here has too many what-if parts...

[elasticmachine]

Original comment by @clintongormley:

I don't understand that, you are switching all aliases or only the ones that point to this specific index? What if there are more than one write alias that point to it?

Yes, we'd switch all write aliases to the new index. I agree that I'm not crazy about this. I'd prefer to have a rollover alias type of which there can only be one. The problem is that rollover is x-pack and so it seems weird to introduce such a concept into core.

Also can you apply write ops to an alias with 1 index that is read_only? I think this is very very confusing if that is the case.

No, we'd enforce that a read_only alias would not accept writes.

I also wonder what the implications are that we are going from an O(1) pattern to a O(n) pattern. I don't think we should do such a change without considering the implications (I am not sure what they are).

Agreed.

FWIW I really like specifying the alias from the index via a setting wich can be a one-off this is less ambiguous IMO. The proposal here has too many what-if parts...

This was the original idea. The reasons for not liking it:

• you need to specify the settings AND add the alias, as opposed to doing both in a single place
• it doesn't solve the need for atomic index+alias creation from beats

For the second point, beats could always try to create {index}-0001 as opposed to {index}-{YYYY.MM.dd}-0001, which would reduce the race condition to "highly unlikely".

[elasticmachine]

Original comment by @bleskes:

For the second point, beats could always try to create {index}-0001 as opposed to {index}-{YYYY.MM.dd}-0001, which would reduce the race condition to "highly unlikely"

How does having read-write alias solve the race condition if multiple beats use a daily pattern and disagree on the date? one of them will try to create an index on it's date and fail because there's already another index which has that alias. That index has the wrong date from the perspective of that specific beat so at this point, we either get stuck or accept the existing index as a target. However, that index will then receive data from a beat with a different idea of what day it is. This mean an index named 2018-02-01 can have data with 2018-01-31. At this point what's the added value of that date over a {index}-001 naming scheme?

[elasticmachine]

Original comment by @clintongormley:

That index has the wrong date from the perspective of that specific beat so at this point, we either get stuck or accept the existing index as a target. However, that index will then receive data from a beat with a different idea of what day it is. This mean an index named 2018-02-01 can have data with 2018-01-31. At this point what's the added value of that date over a {index}-001 naming scheme?

The beat would use the existing alias. The dates on the indices are a convenience, rather than a requirement. It just makes it easy to see when they were created which is useful when it comes to eg snapshot management etc. But as @colings86 said, we should rather expose the index creation date in more APIs.

[elasticmachine]

Original comment by @s1monw:

This was the original idea. The reasons for not liking it:

you need to specify the settings AND add the alias, as opposed to doing both in a single place
it doesn't solve the need for atomic index+alias creation from beats

I mean we can make a write alias creation a side-effect of the index creation and skip it if the alias already exists. I wonder if we should do this by default at some point. If you create an index we give it some pattern name starting with -00000 and just create an alias with the specified name. This index creation would then set the setting to associate the index with the alias. It's all part of the create index call or some template?

[elasticmachine]

Original comment by @clintongormley:

I like the idea, but there are some issues with it, eg the alias may be apache-logs-write. We would want to create the index apache-logs-000001 rather than apache-logs-write-000001.

There have been requests to add the ability to infer the index name from an alias,
eg LINK REDACTED

What about something like this?

PUT _template/logs
{
  "alias_patterns": {
    "apache-logs-write": "apache-logs-000001"
  },
  "aliases": {
    "apache-logs-read": {}
  }
}

Or with a regex to match more widely:

PUT _template/logs
{
  "alias_patterns": {
    "/(.*-logs)-write/": "\\1-000001"
  },
  "aliases": {
    "{index}-read": {}
  }
}

or if the user wanted the date in there as well:

PUT _template/logs
{
  "alias_patterns": {
    "/(.*-logs)-write/": "<\\1-{now/d}-000001>"
  },
  "aliases": {
    "{index}-read": {}
  }
}

[elasticmachine]

Original comment by @s1monw:

I am ok with this proposal! (I still hate regexes)

[elasticmachine]

Original comment by @bleskes:

@clintongormley I want to make sure I understand the entire flow:

1. A user tries to create an index named apache-logs-0001.
2. If the index exists, it fails. If it does we match template and you can add a new alias_patterns clause that adds aliases if the pattern matches. That section will add a *-write alias if the index name ends with 0001. (PS - I don't understand how the alias name is derived from "/(.*-logs)-write/").
3. When a rollover is called, it creates a new index named apache-logs-0002. This one is configured by the template, but the alias-patterns is ignored and the alias isn't added.
4. The rollover waits for the new index to be allocated and then transfer aliases. How does it know which one to move? I guess we configure another pattern on the lifecycle / API call?

Last regarding your date example - I think that still has the issue of a race condition when beats are not aligned on dates? now we have two date indices ending with 0001 and both have an alias with a -write suffix?

[elasticmachine]

Original comment by @clintongormley:

@bleskes No.

• A user indexes into apache-logs-write. The index doesn't exist. The template matches the alias against alias_patterns and uses it to create index apache-logs-000001 and adds the apache-logs-write alias to it.

• When rollover is called, it create apache-logs-000002 and uses the index.lifecycle.rollover_alias setting to choose which alias to rollover.

Last regarding your date example - I think that still has the issue of a race condition when beats are not aligned on dates? now we have two date indices ending with 0001 and both have an alias with a -write suffix?

No, because the beats never create the index. They just index directly into the alias.

[elasticmachine]

Original comment by @colings86:

When rollover is called, it create apache-logs-000002 and uses the index.lifecycle.rollover_alias setting to choose which alias to rollover.

I'm confused, I thought one of the goals of doing this was to avoid us needed to have a index.lifecycle.rollover_alias setting?

[elasticmachine]

Original comment by @clintongormley:

It was, but see LINK REDACTED - I think it is not worth it given all the complications.

This solution still requires you to configure things in two places, but it solves both issues.

[elasticmachine]

Original comment by @clintongormley:

After discussion with @talevy and @colings86, we've decided to simplify the alias_patterns proposal as follows:

1. Initially, don't accept patterns, just literal 1-to-1 mappings, eg:

PUT _template/logs
{
  "match_alias": {
    "apache-logs-write": "apache-logs-000001"
  },
  "aliases": {
    "apache-logs-read": {}
  }
}

(we can always revisit patterns later on if we need to)

2. match_alias takes preference over index_patterns
3. Once we have resolved the real index name from the match_alias, all templates with matching index_patterns will be checked and applied in order
4. A template which uses match_alias is not required to have an index_patterns section, but if it does, then the resolved index name must match one of the patterns.

[elasticmachine]

Original comment by @bleskes:

I spend some more days thinking about this and I'm still worried that the solution we came up with is too "messy":

1. It adds complexity to a place which is already complex - i.e., templates.
2. When an indexing requests comes and the string in the url doesn't resolve to an index or an alias, we translate it internally to a create index request where the string is set in the index name. Based on the solution, the master will then resolve the incoming index name parameter first against match_alias sections in all template and then will try to resolve it as an index name. This is trappy, so we should an extra parameter to the CreateIndexRequest called indexOrAlias or some other name (all I can think of sucks). I don't think it's OK to hijack an explicit create index request in a way that will result in an index with another name being created.
3. As stated, there is no guarantee that the index being created will actually end up having the alias that was used to create it. We can of course add that to the logic of "alias matched" templates - they should automatically add the alias to the create index.
4. There are still annoying esoteric distributed race conditions which can result in Bad Things. For example - an esoteric combination of indexing and rollover calls can cause a situation where a node which is behind on it's cluster state processing and didn't learn of the alias can translate an incoming indexing op to a create index request to the master. The master will go through the proposed flow and will create a new index named apache-logs-00001 (as it doesn't exist due to the rollover) with the apache-logs-write added to it. This will cause all indexing to stop as we now have a write alias pointing at two indices. This is maybe fixable by requiring the indexOrAlias parameter to be also checked against existing aliases - but that's IMO another messiness in the API.

I still believe that the "read/write" aliases is the cleanest solution we had so far but I agree it's a big change and we should all be comfortable with it if we go down that path.

Instead, I would like to suggest another solution. This one is also "messy" but at least it is way more contained, enforced by the master, and we can easily remove it when it's not needed. We can add a parameter to the create index api called "ensureNoExisitngAlias=apache-logs-write" . The master will check, under the cluster state thread that there is no existing alias with that name before creating the index. This will allow the beats to allows start by creating the apache-logs-000001 index (which will succeed or fail with an ResourceAlreadyExistsException due to an existing index/alias) and then starting to index.