Internal: reduce the size of the search thread pool.

Follow-up of #9135. We initially decreased the stack size because it would end
up a lot of memory when there are many threads. But we also have some thread
pools that may be oversized, in particular the search thread pool.

This commit proposes to decrease the default search thread pool size from
`3 * num_procs` to `3 * num_procs / 2 + 1`. This is large enough to be sure
that we can use all the machine resources even with a search-only work load
but not too large in order to not consume too much memory because of the stack
size and thread locals.

Author: jpountz

src/main/java/org/elasticsearch/cache/recycler/PageCacheRecycler.java

@@ -59,8 +59,8 @@ private static int maximumSearchThreadPoolSize(ThreadPool threadPool, Settings s
         assert searchThreadPool != null;
         final int maxSize = searchThreadPool.getMax();
         if (maxSize <= 0) {
-            // happens with cached thread pools, let's assume there are at most 3x ${number of processors} threads
-            return 3 * EsExecutors.boundedNumberOfProcessors(settings);
+            // happens with cached thread pools, let's assume there are at most 2x ${number of processors} threads
+            return 2 * EsExecutors.boundedNumberOfProcessors(settings);
         } else {
             return maxSize;
         }

src/main/java/org/elasticsearch/threadpool/ThreadPool.java

@@ -113,7 +113,7 @@ public ThreadPool(Settings settings, @Nullable NodeSettingsService nodeSettingsS
                 .put(Names.INDEX, settingsBuilder().put("type", "fixed").put("size", availableProcessors).put("queue_size", 200).build())
                 .put(Names.BULK, settingsBuilder().put("type", "fixed").put("size", availableProcessors).put("queue_size", 50).build())
                 .put(Names.GET, settingsBuilder().put("type", "fixed").put("size", availableProcessors).put("queue_size", 1000).build())
-                .put(Names.SEARCH, settingsBuilder().put("type", "fixed").put("size", availableProcessors * 3).put("queue_size", 1000).build())
+                .put(Names.SEARCH, settingsBuilder().put("type", "fixed").put("size", ((availableProcessors * 3) / 2) + 1).put("queue_size", 1000).build())
                 .put(Names.SUGGEST, settingsBuilder().put("type", "fixed").put("size", availableProcessors).put("queue_size", 1000).build())
                 .put(Names.PERCOLATE, settingsBuilder().put("type", "fixed").put("size", availableProcessors).put("queue_size", 1000).build())
                 .put(Names.MANAGEMENT, settingsBuilder().put("type", "scaling").put("keep_alive", "5m").put("size", 5).build())