Translog reader didn't release file handler, nearly millions translog still opened, with new translog keep flushing

[wangkhc]

Elasticsearch version (bin/elasticsearch --version):
6.7.0

JVM version (java -version):
openjdk version "11.0.3-AJDK" 2019-03-09

OS version (uname -a if on a Unix-like system):
Linux es-cn-mp91cp7gj0001pi7e-def6a525-0002 3.10.0-327.ali2010.rc6.alios7.x86_64 #1 SMP Tue May 23 13:42:10 CST 2017 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

1. Millions of translog file opened containing files created serveral days ago, which should be trimmed.
2. this node has thousands of shards (2c8g vm)
3. indexing already stopped by user, but still no help.

Steps to reproduce:
this is from aliyun es customer, we didn't figure out how did it happened.

snapshot : lsof translog-3.tlog

[martijnvg]

Is the flush api used frequently? This can cause the creation of many small translog files.
Can you share node stats for this particular node (GET /_nodes/{node-id}/stats/_all)?

this node has thousands of shards

That is a large number of shards for a single node. I would suggest to either reduce the number of shards or add more data nodes. In case of the many translog files, having this many shards, can cause the number of translog files to grow to a large number on this node.

Also lsof reports open file handles many times over. Combined with having many shards on this node, the number of translog open file handles that lsof reports can be a lot.

Recently an improvement (#47414) was made that restricts the number of translog files in cases where the translog files are very small. This improvement is part of the 6.8.4 release, so perhaps it makes sense to upgrade to this version.

[wangkhc]

Is the flush api used frequently?

No, this user already stopped any operations to this cluster.

Can you share node stats for this particular node (GET /_nodes/{node-id}/stats/_all)?

Unfortunally, this cluster was already released by user for this unstable performace, I'm afraid no more scene can be acquired from this cluster :(

Also lsof reports open file handles many times over. Combined with having many shards on this node, the number of translog open file handles that lsof reports can be a lot.

The point is, taking translog-3.ckp as an example in the screenshot, this file is created several days ago, why did this file still not be released, which finally exhausted os total file handler?

Recently an improvement (#47414) was made that restricts the number of translog files in cases where the translog files are very small. This improvement is part of the 6.8.4 release, so perhaps it makes sense to upgrade to this version.

Tks, This pr seems to be a feasible workaround, but I still wonder that besides manually invoking "flush" api, is there any corner situation that can cause inner flush keeps happening?

[DaveCTurner]

There is significant confusion here. The screenshot above shows many different files called translog-3.ckp with different paths. It is the expected behaviour that some of these remain open for days at a time. Also, lsof | grep | wc gives a number that is usually wildly inaccurate as an estimate of the number of open file handles. Given that the cluster no longer exists for further investigation I think we cannot take any action here so I am closing this.