red* (asterisk) index health

[cjcenizal]

Is this a bug on master? Cat indices reports an index as having red* health. This seems like a recent change because our client-side logic in Kibana expects health to be either green, yellow, or red.

[jasontedor]

I suspect this is related to #51456 which converted the ILM history index to a hidden index, from #50452. Hidden indices are not expanded, which means they shouldn't resolve in health/stats calls. When we can't get this information, we mark an index as red*, behavior that's been there since the beginning (092fd6f).

@gwbrown @jaymode Can you think about the proper handling of this?

[elasticmachine]

Pinging @elastic/es-ui (:ES-UI)

[gwbrown]

Still digging into this, but some notes:

• This reproduces only with Security enabled, hidden indices do not show up in the list at all with Security disabled.
• This reproduces only when the hidden index in question has an alias.
• I believe this is due to how aliases are handled in IndicesAndAliasResolver when resolving "all indices".
• This causes hidden indices with aliases show up in GET */_settings (which is also a bug), although I don't yet understand the logic as to why.
• However, hidden indices do not show up in the ClusterHealthResponse retrieved to build the _cat/indices response, which is the proximate cause of the red* health value.

@jaymode, this also caused me to realize that there's currently no way to view hidden indices in the _cat APIs (except for this bug), as the IndicesOptions used for them are hardcoded. We'll need to add a new request parameter specific to the _cat APIs, or always include hidden indices in the _cat APIs, or rework the _cat APIs to accept IndicesOptions (which I haven't thought through the consequences of but seems like a significant change). Do you have an opinion on which way would be better?

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Core)

[tvernum]

This reproduces only when the hidden index in question has an alias.

It sounds like yet another consequence of #32238 (and the connected issues).
The current behaviour around Security's expansion of wildcards doesn't work well with aliases.

My preference (but there's a variety of different views across the ES team) is actually to move wildcard resolution from being in a pre-action interceptor to being in a pluggable core component.

Right now security tries to replace wildcards with fixed names before the action runs. That means we replace a* with every index and alias that you have access to that starts with a. By the time the core action runs it doesn't have wildcards, it has real names.
But many actions have different handling for what they should do if an alias is explicitly included in the request, vs what they should do if it is covered by a wildcard. So when security is enabled you get the former behaviour, but without security you get the latter. (As I write this, I guess the same issue now applies to expanding wildcards to hidden index names).

If we move wildcard expansion into some pluggable component, then the actions always deal with wildcards, but security would be able to take responsibility for filtering that expansion to only include indices that the user has access to.

This causes hidden indices with aliases show up in GET */_settings (which is also a bug), although I don't yet understand the logic as to why

I think this is the same situation as above (per my "I guess..."). Good luck with that one @jaymode - I guess you'll never really get to escape this probem :)

This reproduces only when the hidden index in question has an alias

What semantics do we intend to have when there is an alias to a hidden index?
Is the alias also hidden? It sounds like that's what you're aiming for, but the whole sceario seems weird to me.

[jaymode]

Do you have an opinion on which way would be better?

I knew that the cat APIs had hardcoded their indices options from when I went through all of the REST APIs to see where we needed to update for hidden indices. My thought at the time was that hidden indices could be left out of the cat apis and I vaguely remember that this was discussed during the last EAH. Currently we have code like

elasticsearch/server/src/main/java/org/elasticsearch/rest/action/cat/RestIndicesAction.java

Line 93 in cecee07

final IndicesOptions indicesOptions = IndicesOptions.strictExpand();

My suggestion is to simply modify this to:

final IndicesOptions indicesOptions = IndicesOptions.fromRequest(request, IndicesOptions.strictExpand());

The current behaviour around Security's expansion of wildcards doesn't work well with aliases.

This is true and has always been problematic with security. Ultimately part of the problem is the feature where actions can be authorized against an alias without the user having permissions against the underlying index.

This causes hidden indices with aliases show up in GET */_settings (which is also a bug), although I don't yet understand the logic as to why.

It is the same issue where security doesn't introspect anything about an alias when doing expansion

elasticsearch/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolver.java

Lines 419 to 425 in a9afdd7

	AliasOrIndex aliasOrIndex = metaData.getAliasAndIndexLookup().get(index);
	if (aliasOrIndex.isAlias()) {
	//it's an alias, ignore expandWildcardsOpen and expandWildcardsClosed.
	//complicated to support those options with aliases pointing to multiple indices...
	//TODO investigate supporting expandWildcards option for aliases too, like es core does.
	return indicesOptions.ignoreAliases() == false;
	}

[jaymode]

What semantics do we intend to have when there is an alias to a hidden index?
Is the alias also hidden? It sounds like that's what you're aiming for, but the whole sceario seems weird to me.

In some cases, the alias should also be hidden in my opinion. The reason why I think that should be the case is that we have ILM, which write aliases are a part of that and if we think about some of our hidden indices, I think ILM makes sense to be used for those. So if expanding wildcards and an alias points to only hidden indices, then the alias should also be hidden. When hidden and non-hidden are mixed in an alias, that's when we start getting into the cases where it gets muddy. My suggestion:

1. By default an alias is visible
2. If an alias only points to hidden indices, it is also hidden.
3. If an alias points to a mix of hidden and non-hidden indices, it is visible.
4. Allow setting a property on an alias to enable it to be hidden ("is_hidden" : true)

Item 4 is the ugliest aspect especially when it comes to the API for managing aliases and I think we could get away with just doing items 1-3.

[rjernst]

Since aliases are supposed to act like indices on their own, why does what they point to matter? Could we decouple them so that they can point to any mix of hidden and non-hidden indices, and only the hidden property on the alias matters for visibility? This would match with the dot prefix behavior in linux filesystems we have modeled hidden indices on, where an alias is like a symlink.

[jaymode]

why does what they point to matter?

I dropped a sentence in my reply while editing; I was using the write index support and attempting to align this with that. In that case, there is the implicit write index on an alias when the alias only points to a single index. Hidden and write indices are different so they aren't going to align exactly, so we can avoid that implicit support.

[gwbrown]

@jaymode and I discussed this issue and the underlying issues today. We concluded:

• We're likely going to have to make some improvements to how Security handles aliases. @jaymode is going to look into this as he's very familiar with that code.
• I'm going to start on implementing "hidden aliases" using a property on the alias. This is going to come with some challenges due to how aliases are implemented, but hopefully we should be able to do it relatively cleanly.

We're also going to hold off on recommending other teams transition to using hidden indices until we've made some progress on resolving the problems with aliases and hidden indices, as most of the use cases for hidden indices involve an alias.