Optimize indexing by replacing sychroinzed lock of TranslogWriter.add by Disruptor in async mode

[dengweisysu]

When indexing big data in high speed, sychronized lock of TranslogWriter.add will waste a lot of time. Here is some situations that Completing for sync lock happens:

• with many other thread writing the same index shard
• flush trigger by IndexService.AsyncTranslogFSync
• flush trigger when translog reach flush_threshold_size
• flush trigger when rolling generation

for user, translog durability strategy was set async , but there are still much performance losed (about 50% in my situation) compared with no translog writing. So I try to using RingBuffer(Disruptor) to make translog adding lock free for writing thread.

Test scenarios:

• machine: Cpu=24 core, memory=64G
• doc count: 10 million
• field count per doc: 400+
• translog size per doc: about 3k （34G tlog generate for 10 million doc）

elasticsearch node config and index config:

# large size to reduce times of full flush
translog.flush_threshold_size: 30G （default 512M）
index.translog.sync_interval: 60s （default 5s）

indices.memory.index_buffer_size: 20% (default 10%)

Indexing elapsed time for each scene list below:

• translog open with async durability model : 18 minutes
• translog close (change source code) : 12 minutes
• translog writing async using disruptor(change source code) : 12 minutes

reference: https://lmax-exchange.github.io/disruptor/

[elasticmachine]

Pinging @elastic/es-distributed

[dengweisysu]

Disruptor use sun.misc.Unsafe internal, and this is forbidden in thirdPartyAudit.... I'm thinking other solution

[dengweisysu]

use LinkedBlockingQueue instead of disruptor.

Indexing elapsed time for each scene list below:

• translog open with async durability model : 18 minutes
• translog close (change source code) : 12 minutes
• translog writing async using disruptor(change source code) : 12 minutes
• translog writing async using LinkedBlockQueue(change source code) : about 13 minutes

[s1monw]

hey @dengweisysu thanks for opening this. I would like to take a couple of steps back. I skimmed the code and I understand what you are after. Yet, the translog is a very delicate part of the system which in some cases warrants performance hits over complexity. That said, I would like to understand what you are trying to improve and where are the bottlenecks. I understand that there is a lock around writing the bytes out and there are locks on when we roll over generations. Many different parts play a role here so it's crucial to understand what the bottleneck before we drop many guarantees by making everything async, threaded or queued.

you are referring to a some kind of benchmark, can you share how you benchmark the system? I have to be able to reproduce the problem you are seeing, I can try to do this myself if you can't provide the code for this scenario.

[dengweisysu]

hey @dengweisysu thanks for opening this. I would like to take a couple of steps back. I skimmed the code and I understand what you are after. Yet, the translog is a very delicate part of the system which in some cases warrants performance hits over complexity. That said, I would like to understand what you are trying to improve and where are the bottlenecks. I understand that there is a lock around writing the bytes out and there are locks on when we roll over generations. Many different parts play a role here so it's crucial to understand what the bottleneck before we drop many guarantees by making everything async, threaded or queued.

you are referring to a some kind of benchmark, can you share how you benchmark the system? I have to be able to reproduce the problem you are seeing, I can try to do this myself if you can't provide the code for this scenario.

Test scenarios:

machine: Cpu=24 core (2.4G Hz ), memory=64G
doc count: 10 million (user info, one doc per user, so it's 10 million diffent user info)
field count per doc: 400+ ,most of them are long,integer,keyword(no text field) and docvalues is disable
doc id : user id from our system
translog size per doc: about 3k （34G tlog generate for 10 million doc）

main index settings is below:

    "refresh_interval" : "-1",
    "translog" : {
      "generation_threshold_size" : "64mb", (this is default value)
      "flush_threshold_size" : "30gb",
      "sync_interval" : "60s",
      "durability" : "async"
    },
    "number_of_replicas" : "0",
    "number_of_shards" : "1",
    "merge" : {
      "scheduler" : {
        "max_thread_count" : "1"
      },
      "policy" : {
        "segments_per_tier" : "30"
      }
    }

we use Spark to write data,. When writing, cpu is nearly about 90%+., 24 write thread is busily writing.

In pull request discuss, we found that when i make the config to avoid fsync and rollgeneration, performance will improve a lot.

[dengweisysu]

jvm heap size is 31G

[dengweisysu]

this test is done on v6.6.2
As I known, there are nearly no different in dealing with translog with head version of master branch