ES Indices /GET api return closed indices if they have aliases assigned.

[SthPhoenix]

Elasticsearch version: 7.1.0, 7.2.0

Plugins installed: [analysis-icu, analysis-smartc, analysis-phonetic]

JVM version : bundled

OS version: Ubuntu 18.04.2

Description of the problem including expected versus actual behavior:

Actual: Elasticsearch Indices /GET API return closed indices if they has assigned aliases.
Expected: ES should return only opened indices.

Aliases are unique to closed indices, and there are no opened indices sharing same aliases.

Steps to reproduce:

1. Create new index
2. Put alias to this index
3. Close index
4. Execute GET http://elasticsearch.host:9200/* to get list of indices including closed just now.
5. Delete alias from closed index.
6. Execute GET http://elasticsearch.host:9200/* to get list of indices without closed one.

[elasticmachine]

Pinging @elastic/es-distributed

[dliappis]

Hey @SthPhoenix ,

Thanks for this bug report.

I am not sure I see the problem here. The most typical API to get a list of indices is GET _cat/indices (or GET _aliases). Using either API the state of the index after closing it seems correct to me whether there is an alias or not.

Testing closing an index without an alias shows:

DELETE testindex

PUT testindex

GET /_cat/indices
yellow open testindex            ygpMPnV1R1SCc-DwP6-IlQ 1 1 0 0   230b   230b

GET /_aliases
{
  "testindex" : {
    "aliases" : { }
  }
}

POST /testindex/_close

GET /_cat/indices
yellow close testindex            ygpMPnV1R1SCc-DwP6-IlQ 1 1                  

GET /_aliases
{
  "testindex" : {
    "aliases" : { }
  }
}

Testing closing an index with an alias shows:

DELETE testindex

PUT testindex

POST /_aliases
{
  "actions": [
    {
      "add": {
        "index": "testindex", "alias": "testalias"
      }
    }]
}

GET /_cat/indices
yellow open testindex            fAq8AxBASpasUPJdAE2OmA 1 1 0 0   230b   230b

GET /_aliases
{
"testindex" : {
    "aliases" : {
      "testalias" : { }
    }
  }
}

POST /testindex/_close

GET /_cat/indices
yellow close testindex            fAq8AxBASpasUPJdAE2OmA 1 1                  

GET /_aliases
{
  "testindex" : {
    "aliases" : {
      "testalias" : { }
    }
  }
}

[SthPhoenix]

Yes, _cat API works as intended. Problem is that Get Index API have inconsistent behavior.

Actually I noticed it while using official python client, where es.indicies.get('*') in previous versions of ES returned only opened indices, and after 7.1 it start returning closed ones.

[dliappis]

@SthPhoenix I can't reproduce this (on 7.2.0 at least):

I tried:

1. Created index (PUT testindex)
2. Assign alias

   POST /_aliases
   {
     "actions": [
       {
         "add": {
           "index": "testindex", "alias": "testalias"
         }
       }]
   }
   

3. Check GET _all (or GET *) and shows the open index as expected.
4. Close index (POST /testindex/_close)
5. Check GET _all (or GET *) and the closed index testindex isn't listed.

Are these the same reproduction steps you followed?

[SthPhoenix]

I am using python client (7.0.0) for this, but yes, steps are the same. For checking indices list I also used curl.
Tested today on ES 7.3, got the same bug.
If it might be the case, this is not clean install, ES was updated step by step from ES 6.5.1

[dliappis]

For checking indices list I also used curl.

To clarify: if you follow my earlier steps using curl, does step 5. i.e. curl -H 'Content-Type:application/json' <es_url:es_port>/_all?pretty list testindex?

[SthPhoenix]

Yes, using curl gives same strange behavior.
Also tested this with clean docker containers 7.1 to 7.3 - and it works just as expected, that's weird. I'll test it more thoroughly and let you know if I could figure out the reason.

[tlrx]

@SthPhoenix Did you manage to reproduce or to figure out the reason?

Looking at the code, the GET Indices API should only return opened indices on 6.x, 7.x and master as it uses the "strict expand open" option.

It is possible that the Elasticsearch Python Client sets a different indices options that allow closed indices to be included in the response, but looking at the client code it does not seem to be the case.

Also, I'm curious to know if any of the clusters you tested have the Security enabled?

[SthPhoenix]

Hi @tlrx ! I won't have access to the cluster with this error till the end of next week.

It is possible that the Elasticsearch Python Client sets a different indices options that allow closed indices to be included in the response, but looking at the client code it does not seem to be the case.

Since this behavior is also reproduced with curl requests, I think ES python client is out of question.

Also, I'm curious to know if any of the clusters you tested have the Security enabled?

Yes, I have security enabled on the cluster.
I have updated to ES 7.1, enabled security and later found this issue, but disabling security didn't help.

[tlrx]

@SthPhoenix Thanks for the extra information.

Since this behavior is also reproduced with curl requests, I think ES python client is out of question.

Agreed, I just wanted to double check.

Yes, I have security enabled on the cluster.

That's what I suspected, and indeed this bug can be reproduced on 7.1/7.2 (maybe also later versions) but it only appears when security is enabled.