Don't allow `bulk` and `indexing` thread pools size larger than number of CPU cores

[mikemccand]

I've seen a couple cases recently where users "tuned" up the number of threads in the thread pools for indexing, and then didn't realize this leads to crazy issues like Lucene writing an insane number of segments and doing an insane number of merges, etc.

I don't see any reason to allow this value to exceed the number of CPU cores?

[jpountz]

+1

[s1monw]

I guess that's a good limit to have

[makeyang]

then how to deal with high concurrency indexing situation?
I "tuned" up the number of threads just in order to handle it although I know well its side effort.

[mikemccand]

@makeyang increasing the bulk and index thread pool sizes beyond the number of cores your hardware does not help indexing throughput ... it will hurt it, as you're seeing in #15538. See https://www.elastic.co/blog/performance-indexing-2-0 for ways to optimize for indexing.

[makeyang]

@mikemccand
it's not about indexing performance.
my situation is here: I have a ES cluster and many other teams will write their data into this cluster, the data is very samll though the clients are many, so sometime when all bulk/index thread is occupied, the client will get rejecte exception. the write is not heavy at all. cpu/memory is all used little. but reject exception happened, how to deal with this situation?

[mikemccand]

@makeyang in that case you should increase your bulk queue_size, not the size.

[makeyang]

@mikemccand got it, thanks.
I have another question to ask: I have done load test(especially for indexing performance) and found out that single node can handle 40000 docs/second, over this, ES will throw reject exception to client. but at the max throughput, the node's resouce is not completely occupied,
cpu usage is 80%
iops is 170(spin disk, I have done iops test with other tools, iops can reacheve 500).
so my question is why it can't write more while resouce is not fully occupied?
what is bottleneck and how to identify it?

[makeyang]

@mikemccand please help to share any thoughts, insights, anything

[mikemccand]

@makeyang did you try the suggestions from the above blog post?

[makeyang]

yeah. try all of them.