Reduce the need to reload roles from index

[jaymode]

If the roles.yml file is changed, the Security code takes the heavy handed approach of invalidating all cache entries rather than those that use changed roles. This should be a rare occurrence but in a busy cluster this is problematic since any roles stored in a index will need to be reloaded, which means a search (or get if only one name is requested) needs to be executed and shares the same threadpool/queue as other searches. If the search is rejected, then we use our negative lookup cache to prevent this role from being queried for again. This leads to poor behavior for the user as users do not have the expected permissions.

We should look at how we can improve this behavior through the need to reduce which roles get reloaded from an index. This may mean a change in our caching or how we deal with changes to the roles.yml and reloading.

[elasticmachine]

Pinging @elastic/es-security

[jaymode]

Another item to consider is to limit role combination loading to one per unique role combination. This would be similar to what was done for concurrent authentications of a user.

[jasontedor]

I have been thinking about this issue on and off since @s1monw opened the pull request to enable to route these requests to the generic thread pool. That led to a discussion about how we should avoid that because that thread pool is large and unbounded, and we do not want to be shuttling tasks from user-initiated actions to an unbounded thread pool. We reversed that portion of the change and I had proposed that we force put these requests on to the search thread pool, so that they can not be rejected. When @s1monw pinged me this morning that that is basically the same as allowing these to grow unbounded again, that triggered me to take another look at the underlying issue.

If the search is rejected, then we use our negative lookup cache to prevent this role from being queried for again.

I think this is the key point that we need to revisit. I do not think we should be entering role lookups that fail (not fail negatively, but fail exceptionally) into the negative lookup cache. Rather, an exceptional failure should be an indication that we do not know the result of the role lookup. So it's neither a positive hit, nor a negative hit, but we simply do not know and we should not be caching this result. In particular, rejections of role lookups on the search thread pool should cause the initial request to fail, we can notify the client who can retry the request.

Separately, I think that we can consider migrating these role lookups to a separate thread pool but still with a bounded queue so that they are not competing with search requests.

What do you think @jaymode and @s1monw?