Drop in indexing performance and increase in agg latency after mappings/aggs switch to java time in #36363

[dliappis]

#36363 (merged to master in daa2ec8) has caused a performance drop manifested in both a drop in indexing throughput (see: https://elasticsearch-benchmarks.elastic.co/#tracks/nyc-taxis/nightly/30d for http_logs / nyc_taxis / nested / noaa) as well as latency increase in aggs (see: nyc_taxis, nested, http_logs) and gc (nested) on 2018-01-23.

• Before: nyc_taxis median indexing throughput: 81421 docs/s
• After: nyc_taxis median indexing throughput: 65981 docs/s

Reproduction steps:

Java time commit daa2ec8:

esrally --pipeline="from-sources-complete" --track="nyc_taxis" --car="4gheap" --runtime-jdk=8 --challenge="append-no-conflicts" --revision="daa2ec8" --include-tasks="delete-index,create-index,check-cluster-health,index,refresh-after-index"

Previous commit 7b3dd30:

esrally --pipeline="from-sources-complete" --track="nyc_taxis" --car="4gheap" --runtime-jdk=8 --challenge="append-no-conflicts" --revision="7b3dd30" --include-tasks="delete-index,create-index,check-cluster-health,index,refresh-after-index"

Note: If you are using Oracle JDK you can also get a profile by specifying --telemetry=jfr.

[elasticmachine]

Pinging @elastic/es-core-infra

[spinscale]

this is still open for the aggs case, but I have a fix ready... running benchmarks at the moment