-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Closed as not planned
Labels
:Data Management/StatsStatistics tracking and retrieval APIsStatistics tracking and retrieval APIs>enhancementTeam:Data ManagementMeta label for data/management teamMeta label for data/management team
Description
In the high-concurrency write log scenario, we found that hot_threads is always on the translog.sync operation. After a long investigation, I found that the application client often consumes several Kafka topics at the same time to form a bulkrequest and write it to Elasticsearch. Each topic corresponds to an index, resulting in a very small bulk quest to each shard, so the translog sync operation is very frequent .
This problem can be alleviated by setting translog async. In order to quickly locate the problem, I think it is necessary to add stats for bulk per shard, including bulk_shard_sizes, bulk_shard_lantency, and bulk_shard_qps.
dakrone, zhichen and wangkhc
Metadata
Metadata
Assignees
Labels
:Data Management/StatsStatistics tracking and retrieval APIsStatistics tracking and retrieval APIs>enhancementTeam:Data ManagementMeta label for data/management teamMeta label for data/management team