Merge remote-tracking branch 'origin/master' into all-sql-fn-null

Author: Andras Palinkas

benchmarks/README.md

@@ -24,12 +24,24 @@ If you want to run a specific benchmark class like, say,
 `MemoryStatsBenchmark`, you can use `--args`:
 
 ```
-gradlew -p benchmarks run --args ' MemoryStatsBenchmark'
+gradlew -p benchmarks run --args 'MemoryStatsBenchmark'
+```
+
+Everything in the `'` gets sent on the command line to JMH.
+
+You can set benchmark parameters with `-p`:
+```
+gradlew -p benchmarks/ run --args 'RoundingBenchmark.round -prounder=es -prange="2000-10-01 to 2000-11-01" -pzone=America/New_York -pinterval=10d -pcount=1000000'
+```
+
+The benchmark code defines default values for the parameters so if
+you leave out any out JMH will run with each default value, one after
+the other. This will run with `interval` set to `calendar year` then
+`calendar hour` then `10d` then `5d` then `1h`:
+```
+gradlew -p benchmarks/ run --args 'RoundingBenchmark.round -prounder=es -prange="2000-10-01 to 2000-11-01" -pzone=America/New_York -pcount=1000000'
 ```
 
-Everything in the `'` gets sent on the command line to JMH. The leading ` `
-inside the `'`s is important. Without it parameters are sometimes sent to
-gradle.
 
 ## Adding Microbenchmarks
 
@@ -55,7 +67,8 @@ To get realistic results, you should exercise care when running benchmarks. Here
 * Use the integrated profilers in JMH to dig deeper if benchmark results to not match your hypotheses:
     * Add `-prof gc` to the options to check whether the garbage collector runs during a microbenchmarks and skews
    your results. If so, try to force a GC between runs (`-gc true`) but watch out for the caveats.
-    * Add `-prof perf` or `-prof perfasm` (both only available on Linux) to see hotspots.
+    * Add `-prof perf` or `-prof perfasm` (both only available on Linux, see Disassembling below) to see hotspots.
+    * Add `-prof async` to see hotspots.
 * Have your benchmarks peer-reviewed.
 
 ### Don't
@@ -66,6 +79,8 @@ To get realistic results, you should exercise care when running benchmarks. Here
 
 ## Disassembling
 
+NOTE: Linux only. Sorry Mac and Windows.
+
 Disassembling is fun! Maybe not always useful, but always fun! Generally, you'll want to install `perf` and FCML's `hsdis`.
 `perf` is generally available via `apg-get install perf` or `pacman -S perf`. FCML is a little more involved. This worked
 on 2020-08-01:
@@ -88,3 +103,30 @@ gradlew -p benchmarks run --args ' MemoryStatsBenchmark -jvmArgs "-XX:+UnlockDia
 ```
 
 If you want `perf` to find the hot methods for you then do add `-prof:perfasm`.
+
+## Async Profiler
+
+Note: Linux and Mac only. Sorry Windows.
+
+The async profiler is neat because it does not suffer from the safepoint
+bias problem. And because it makes pretty flame graphs!
+
+Let user processes read performance stuff:
+```
+sudo bash
+echo 0 > /proc/sys/kernel/kptr_restrict
+echo 1 > /proc/sys/kernel/perf_event_paranoid
+exit
+```
+
+Grab the async profiler from https://github.com/jvm-profiling-tools/async-profiler
+and run `prof async` like so:
+```
+gradlew -p benchmarks/ run --args 'LongKeyedBucketOrdsBenchmark.multiBucket -prof "async:libPath=/home/nik9000/Downloads/tmp/async-profiler-1.8.3-linux-x64/build/libasyncProfiler.so;dir=/tmp/prof;output=flamegraph"'
+```
+
+If you are on mac this'll warn you that you downloaded the shared library from
+the internet. You'll need to go to settings and allow it to run.
+
+The profiler tells you it'll be more accurate if you install debug symbols
+with the JVM. I didn't and the results looked pretty good to me. (2021-02-01)

benchmarks/build.gradle

@@ -29,7 +29,7 @@ archivesBaseName = 'elasticsearch-benchmarks'
 tasks.named("test").configure { enabled = false }
 
 dependencies {
-  api( project(":server")) {
+  api(project(":server")) {
     // JMH ships with the conflicting version 4.6. This prevents us from using jopt-simple in benchmarks (which should be ok) but allows
     // us to invoke the JMH uberjar as usual.
     exclude group: 'net.sf.jopt-simple', module: 'jopt-simple'
@@ -61,12 +61,6 @@ tasks.named("dependenciesGraph").configure {  enabled = false }
 tasks.named("thirdPartyAudit").configure {
   ignoreViolations(
           // these classes intentionally use JDK internal API (and this is ok since the project is maintained by Oracle employees)
-          'org.openjdk.jmh.profile.AbstractHotspotProfiler',
-          'org.openjdk.jmh.profile.HotspotThreadProfiler',
-          'org.openjdk.jmh.profile.HotspotClassloadingProfiler',
-          'org.openjdk.jmh.profile.HotspotCompilationProfiler',
-          'org.openjdk.jmh.profile.HotspotMemoryProfiler',
-          'org.openjdk.jmh.profile.HotspotRuntimeProfiler',
           'org.openjdk.jmh.util.Utils'
   )
 }

buildSrc/version.properties

@@ -42,7 +42,7 @@ securemock        = 1.2
 mocksocket        = 1.2
 
 # benchmark dependencies
-jmh               = 1.19
+jmh               = 1.26
 
 # test dependencies
 # when updating this version, also update :qa:evil-tests

client/benchmark/src/main/java/org/elasticsearch/client/benchmark/AbstractBenchmark.java

@@ -72,10 +72,10 @@ private void runBulkIndexBenchmark(String[] args) throws Exception {
         String indexFilePath = args[2];
         String indexName = args[3];
         String typeName = args[4];
-        int totalDocs = Integer.valueOf(args[5]);
+        float totalDocs = Float.valueOf(args[5]);
         int bulkSize = Integer.valueOf(args[6]);
 
-        int totalIterationCount = (int) Math.floor(totalDocs / bulkSize);
+        int totalIterationCount = (int) Math.ceil(totalDocs / bulkSize);
         // consider 40% of all iterations as warmup iterations
         int warmupIterations = (int) (0.4d * totalIterationCount);
         int iterations = totalIterationCount - warmupIterations;

client/benchmark/src/main/java/org/elasticsearch/client/benchmark/ops/bulk/BulkBenchmarkTask.java

@@ -171,7 +171,7 @@ public void run() {
                     logger.warn("Error while executing bulk request", ex);
                 }
                 long stop = System.nanoTime();
-                if (iteration < warmupIterations) {
+                if (iteration >= warmupIterations) {
                     sampleRecorder.addSample(new Sample("bulk", start, start, stop, success));
                 }
             }

docs/reference/ccr/getting-started.asciidoc

@@ -64,6 +64,10 @@ that are periodically created in a remote cluster
 You can manually create follower indices to replicate specific indices on a
 remote cluster, or configure auto-follow patterns to replicate rolling time series indices.
 
+TIP: If you want to replicate data across clusters in the cloud, you can
+link:{cloud}/ec-enable-ccs.html[configure remote clusters on {ess}]. Then, you
+can <<modules-cross-cluster-search,search across clusters>> and set up {ccr}.
+
 video::https://static-www.elastic.co/v3/assets/bltefdd0b53724fa2ce/blt994089f5e841ad69/5f6265de6f40ab4648b5cf9b/ccr-setup-video-edited.mp4[width=700, height=500, options="autoplay,loop"]
 
 [[ccr-getting-started-prerequisites]]

docs/reference/search/point-in-time-api.asciidoc

@@ -49,7 +49,7 @@ POST /_search <1>
 and {ref}/search-request-body.html#request-body-search-preference[`preference`]
 as these parameters are copied from the point in time.
 <2> The `id` parameter tells Elasticsearch to execute the request using contexts
-from this point int time.
+from this point in time.
 <3> The `keep_alive` parameter tells Elasticsearch how long it should extend
 the time to live of the point in time.
 

docs/reference/search/search-your-data/search-across-clusters.asciidoc

@@ -28,6 +28,10 @@ The following APIs support {ccs}:
 
 To perform a {ccs}, you must have at least one remote cluster configured.
 
+TIP: If you want to search across clusters in the cloud, you can
+link:{cloud}/ec-enable-ccs.html[configure remote clusters on {ess}]. Then, you
+can search across clusters and <<ccr-getting-started,set up {ccr}>>.
+
 The following <<cluster-update-settings,cluster update settings>> API request
 adds three remote clusters:`cluster_one`, `cluster_two`, and `cluster_three`.
 

docs/reference/snapshot-restore/apis/clean-up-repo-api.asciidoc

@@ -31,6 +31,12 @@ POST /_snapshot/my_repository/_cleanup
 
 `POST /_snapshot/<repository>/_cleanup`
 
+[[clean-up-snapshot-repo-api-prereqs]]
+==== {api-prereq-title}
+
+* If the {es} {security-features} are enabled, you must have the `manage`
+<<privileges-list-cluster,cluster privilege>> to use this API.
+
 [[clean-up-snapshot-repo-api-desc]]
 ==== {api-description-title}
 

docs/reference/snapshot-restore/apis/clone-snapshot-api.asciidoc

@@ -20,6 +20,12 @@ PUT /_snapshot/my_repository/source_snapshot/_clone/target_snapshot
 
 `PUT /_snapshot/<repository>/<source_snapshot>/_clone/<target_snapshot>`
 
+[[clone-snapshot-api-prereqs]]
+==== {api-prereq-title}
+
+* If the {es} {security-features} are enabled, you must have the `manage`
+<<privileges-list-cluster,cluster privilege>> to use this API.
+
 [[clone-snapshot-api-desc]]
 ==== {api-description-title}