.geoip_databases is included in _cat API responses and causes exception in GET /_alias

[stevejgordon]

Elasticsearch version: 7.14.0-SNAPSHOT
OS version Windows 10:

Description of the problem including expected versus actual behavior:

I've noticed while running YML tests against the .NET client some have started failing for 7.14.0-SNAPSHOT due to the inclusion of geoip_databases in results from the _cat/* APIs ( for example, _cat/indices and _cat/shards), which the assertions are not expecting. It looks like this was recently re-enabled in this PR and I wanted to see if this should be treated as a system index?

Slightly more problematic is that a request to GET /_alias receives an exception response:

{
    "error": {
        "root_cause": [
            {
                "type": "illegal_argument_exception",
                "reason": "Indices [.geoip_databases] use and access is reserved for system operations"
            }
        ],
        "type": "illegal_argument_exception",
        "reason": "Indices [.geoip_databases] use and access is reserved for system operations"
    },
    "status": 400
}

I would expect this to be excluded. The merge of this PR seems to be very close to the date when the tests started failing.

Steps to reproduce:

1. Start a new single-node cluster using the latest 7.14.0-SNAPSHOT (I pulled and ran the Docker image).
2. Send a GET /_alias request

Expect that this API should expand to open, closed and hidden indices but not system indices.

[martijnvg]

This reproduces for me as well when using the distribution that is created when running: ./gradlew -p distribution/archives/darwin-tar build.

Initially I tried to reproduce using ./gradlew run, but this didn't reproduce as for test clusters the geoip downloader is disabled by default and the run task sets up a node with admin user. After enabling geoip downloader by default in test cluster and tweaking the run task to not setup an user (the error doesn't occur when interacting with ES with an authenticated user) the reproduced for me via ./gradlew run.

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[gwbrown]

I've been working on investigating this and it appears this issue is limited to 7.x only. Please let me know ASAP @stevejgordon if we've observed this happening on master (8.0.0-SNAPSHOT) builds.

[stevejgordon]

Hi @gwbrown. I've just checked and I can see similar test failures (in our .NET client YML tests), including the exception above against 8.0.0-SNAPSHOT in the most recent Jenkins build.

[gwbrown]

@stevejgordon Thanks for the quick follow-up, I've determined that the difference was due to a mistake in my test environment - sorry for the noise. Now that I have this reproducing reliably I'm looking to have a fix up as soon as possible.

[gwbrown]

@stevejgordon I've found the cause of this, working on a solution now. At least for the Alias API error. That this index is showing up in the _cat APIs is intentional - system indices show up in the _cat APIs, as these APIs are mainly used for troubleshooting.

I think the main difference with this index that's causing the mismatch with the assertions is that most of the other system indices aren't created until something prompts them, whereas this geoip index is created on startup 100% of the time.

I'm not sure what the best path forwards here is, because we're limited in how we can adjust the visibility of system indices generally in the _cat APIs due to BWC commitments. It would be possible (though annoying on a technical level) to prevent net-new system indices from showing up in these APIs, but that clashes with the existing behavior, and adds another new layer of complexity for users who need to troubleshoot system indices, which I already worry is too complicated.

[stevejgordon]

Thanks @gwbrown. @martijnvg, do those YAML tests run and fail for the Elasticsearch CI process? Presumably the same assertion issue will exist if the geoip downloader is enabled in the test cluster? Should we look to disable geoip in the clients team CI process? If so, can we pass the config to the Docker command we run to start the cluster?

[martijnvg]

@stevejgordon As far as I can tell the yaml tests do not fail because of the issue reported here. This is because the error doesn't reproduce when invoking the get alias api with an authenticated user (which is always the case when running the yaml tests in ES). Also by default most integration tests run with the geoip downloader disabled (only tests that test geoip processor have the geoip downloader enabled), in order to keep build time / performance at bay. I will discuss with the core/features team whether we should reconsider this (maybe just for yaml test suite, the downloader should be enabled).