Speed up PK lookups at index time.

[jpountz]

At index time Elasticsearch needs to look up the version associated with the
_uid of the document that is being indexed, which is often the bottleneck for
indexing.

While reviewing the output of the jfr telemetry from a Rally benchmark, I saw
that significant time was spent in ConcurrentHashMap#get and ThreadLocal#get.
The reason is that we cache lookup objects per thread and segment, and for every
indexed document, we first need to look up the cache associated with this
segment (ConcurrentHashMap#get) and then get a state that is local to the
current thread (ThreadLocal#get). So if you are indexing N documents per
second and have S segments, both these methods will be called N*S times per
second.

This commit changes version lookup to use a cache per index reader rather than
per segment. While this makes cache entries live for less long, we now only need
to do one call to ConcurrentHashMap#get and ThreadLocal#get per indexed
document.

NOTE: I had to remove IndexReader#getCombinedCoreAndDeletesKey from the
forbidden APIs in order to make it work.

Here are some screenshots of the hot methods as reported when indexing 100M empty documents with rally using 1 shard, 0 replicas and otherwise default settings.
Before:

After:

ThreadLocal#get is no longer among the hottest methods after the change, and while ConcurrentHashMap#get is still there, its main callers are now the live version map and the circuit breaker, not the state we maintain for version lookups in the index.

Here is the report from rally compare on two races that have been run without telemetry (so that it does not affect the benchmark results). The speedup looks real.

                             Metric    Baseline    Contender                Diff
-----------------------------------  ----------  -----------  ------------------
   Min Indexing Throughput [docs/s]       67415        73665  +6250.00000
Median Indexing Throughput [docs/s]       73608      80540.3  +6932.33333
   Max Indexing Throughput [docs/s]       75151        81799  +6648.00000
                Indexing time [min]     113.854      94.9646    -18.88890
                   Merge time [min]     15.0901      11.0337     -4.05640
                 Refresh time [min]     1.52247      1.57633     +0.05387
                   Flush time [min]    0.901483     0.874733     -0.02675
          Merge throttle time [min]     5.95868       2.5531     -3.40558
       Median CPU usage (index) [%]     566.462      568.981     +2.51969
             Total Young Gen GC [s]     113.929      116.784     +2.85500
               Total Old Gen GC [s]      18.633       19.979     +1.34600
                    Index size [GB]     1.88654      1.87796     -0.00858
               Totally written [GB]     20.4271      19.9763     -0.45073
                      Segment count          22           34    +12.00000

[jpountz]

This is basically a revert of #14070. I'm stalling this on making cache keys / closed listeners less trappy.

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[rjernst]

@jpountz Is this PR worth reviving?

[jpountz]

Absolutely!

[jpountz]

I rebased this PR!

[s1monw]

left a small comment LGTM otherwise

[s1monw]

can you leave a comment why we moved to this? you can also reference this PR for documentation purposes

[jpountz]

+1 I had more comments initially but they apparently got lost with the rebase