Merge branch 'master' into rollover-max-single-shard-size

Author: joegallo

.ci/teamcity.init.gradle

@@ -1,20 +1,9 @@
 /*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
  */
 
 gradle.settingsEvaluated { settings ->

CONTRIBUTING.md

@@ -442,35 +442,26 @@ We require license headers on all Java files. With the exception of the
 top-level `x-pack` directory, all contributed code should have the following
 license header unless instructed otherwise:
 
-    /*
-     * Licensed to Elasticsearch under one or more contributor
-     * license agreements. See the NOTICE file distributed with
-     * this work for additional information regarding copyright
-     * ownership. Elasticsearch licenses this file to you under
-     * the Apache License, Version 2.0 (the "License"); you may
-     * not use this file except in compliance with the License.
-     * You may obtain a copy of the License at
-     *
-     *    http://www.apache.org/licenses/LICENSE-2.0
-     *
-     * Unless required by applicable law or agreed to in writing,
-     * software distributed under the License is distributed on an
-     * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-     * KIND, either express or implied.  See the License for the
-     * specific language governing permissions and limitations
-     * under the License.
-     */
+        /*
+         * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+         * or more contributor license agreements. Licensed under the Elastic License
+         * 2.0 and the Server Side Public License, v 1; you may not use this file except
+         * in compliance with, at your election, the Elastic License 2.0 or the Server
+         * Side Public License, v 1.
+         */
 
 The top-level `x-pack` directory contains code covered by the [Elastic
-license](licenses/ELASTIC-LICENSE.txt). Community contributions to this code are
+license](licenses/ELASTIC-LICENSE-2.0.txt). Community contributions to this code are
 welcome, and should have the following license header unless instructed
 otherwise:
 
-    /*
-     * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
-     * or more contributor license agreements. Licensed under the Elastic License;
-     * you may not use this file except in compliance with the Elastic License.
-     */
+        /*
+         * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+         * or more contributor license agreements. Licensed under the Elastic License
+         * 2.0; you may not use this file except in compliance with the Elastic License
+         * 2.0.
+         */
+
 
 It is important that the only code covered by the Elastic licence is contained
 within the top-level `x-pack` directory. The build will fail its pre-commit

LICENSE.txt

@@ -1,10 +1,7 @@
-Source code in this repository is covered by one of three licenses: (i) the
-Apache License 2.0 (ii) an Apache License 2.0 compatible license (iii) the
-Elastic License. The default license throughout the repository is Apache License
-2.0 unless the header specifies another license. Elastic Licensed code is found
-only in the x-pack directory.
-
-The build produces two sets of binaries - one set that falls under the Elastic
-License and another set that falls under Apache License 2.0. The binaries that
-contain `-oss` in the artifact name are licensed under Apache License 2.0 and
-these binaries do not package any code from the x-pack directory.
+Source code in this repository is covered by (i) a dual license under the Server
+Side Public License, v 1 and the Elastic License 2.0 or (ii) an Apache License
+2.0 compatible license or (iii) solely under the Elastic License 2.0, in each
+case, as noted in the applicable header. The default throughout the repository
+is a dual license under the Server Side Public License, v 1 and the Elastic
+License 2.0, unless the header specifies another license. Code that is licensed
+solely under the Elastic License 2.0 is found only in the x-pack folder.

NOTICE.txt

@@ -1,5 +1,5 @@
 Elasticsearch
-Copyright 2009-2018 Elasticsearch
+Copyright 2009-2021 Elasticsearch
 
 This product includes software developed by The Apache Software
 Foundation (http://www.apache.org/).

Vagrantfile

@@ -4,22 +4,11 @@
 # This Vagrantfile exists to test packaging. Read more about its use in the
 # vagrant section in TESTING.asciidoc.
 
-# Licensed to Elasticsearch under one or more contributor
-# license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright
-# ownership. Elasticsearch licenses this file to you under
-# the Apache License, Version 2.0 (the "License"); you may
-# not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.
+# Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+# or more contributor license agreements. Licensed under the Elastic License
+# 2.0 and the Server Side Public License, v 1; you may not use this file except
+# in compliance with, at your election, the Elastic License 2.0 or the Server
+# Side Public License, v 1.
 
 define_opts = {
   autostart: false

benchmarks/README.md

@@ -24,12 +24,24 @@ If you want to run a specific benchmark class like, say,
 `MemoryStatsBenchmark`, you can use `--args`:
 
 ```
-gradlew -p benchmarks run --args ' MemoryStatsBenchmark'
+gradlew -p benchmarks run --args 'MemoryStatsBenchmark'
+```
+
+Everything in the `'` gets sent on the command line to JMH.
+
+You can set benchmark parameters with `-p`:
+```
+gradlew -p benchmarks/ run --args 'RoundingBenchmark.round -prounder=es -prange="2000-10-01 to 2000-11-01" -pzone=America/New_York -pinterval=10d -pcount=1000000'
+```
+
+The benchmark code defines default values for the parameters so if
+you leave out any out JMH will run with each default value, one after
+the other. This will run with `interval` set to `calendar year` then
+`calendar hour` then `10d` then `5d` then `1h`:
+```
+gradlew -p benchmarks/ run --args 'RoundingBenchmark.round -prounder=es -prange="2000-10-01 to 2000-11-01" -pzone=America/New_York -pcount=1000000'
 ```
 
-Everything in the `'` gets sent on the command line to JMH. The leading ` `
-inside the `'`s is important. Without it parameters are sometimes sent to
-gradle.
 
 ## Adding Microbenchmarks
 
@@ -55,7 +67,8 @@ To get realistic results, you should exercise care when running benchmarks. Here
 * Use the integrated profilers in JMH to dig deeper if benchmark results to not match your hypotheses:
     * Add `-prof gc` to the options to check whether the garbage collector runs during a microbenchmarks and skews
    your results. If so, try to force a GC between runs (`-gc true`) but watch out for the caveats.
-    * Add `-prof perf` or `-prof perfasm` (both only available on Linux) to see hotspots.
+    * Add `-prof perf` or `-prof perfasm` (both only available on Linux, see Disassembling below) to see hotspots.
+    * Add `-prof async` to see hotspots.
 * Have your benchmarks peer-reviewed.
 
 ### Don't
@@ -66,6 +79,8 @@ To get realistic results, you should exercise care when running benchmarks. Here
 
 ## Disassembling
 
+NOTE: Linux only. Sorry Mac and Windows.
+
 Disassembling is fun! Maybe not always useful, but always fun! Generally, you'll want to install `perf` and FCML's `hsdis`.
 `perf` is generally available via `apg-get install perf` or `pacman -S perf`. FCML is a little more involved. This worked
 on 2020-08-01:
@@ -88,3 +103,30 @@ gradlew -p benchmarks run --args ' MemoryStatsBenchmark -jvmArgs "-XX:+UnlockDia
 ```
 
 If you want `perf` to find the hot methods for you then do add `-prof:perfasm`.
+
+## Async Profiler
+
+Note: Linux and Mac only. Sorry Windows.
+
+The async profiler is neat because it does not suffer from the safepoint
+bias problem. And because it makes pretty flame graphs!
+
+Let user processes read performance stuff:
+```
+sudo bash
+echo 0 > /proc/sys/kernel/kptr_restrict
+echo 1 > /proc/sys/kernel/perf_event_paranoid
+exit
+```
+
+Grab the async profiler from https://github.com/jvm-profiling-tools/async-profiler
+and run `prof async` like so:
+```
+gradlew -p benchmarks/ run --args 'LongKeyedBucketOrdsBenchmark.multiBucket -prof "async:libPath=/home/nik9000/Downloads/tmp/async-profiler-1.8.3-linux-x64/build/libasyncProfiler.so;dir=/tmp/prof;output=flamegraph"'
+```
+
+If you are on mac this'll warn you that you downloaded the shared library from
+the internet. You'll need to go to settings and allow it to run.
+
+The profiler tells you it'll be more accurate if you install debug symbols
+with the JVM. I didn't and the results looked pretty good to me. (2021-02-01)

benchmarks/build.gradle

@@ -1,22 +1,11 @@
 import org.elasticsearch.gradle.info.BuildParams
 
 /*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
  */
 
 apply plugin: 'elasticsearch.build'
@@ -29,7 +18,7 @@ archivesBaseName = 'elasticsearch-benchmarks'
 tasks.named("test").configure { enabled = false }
 
 dependencies {
-  api( project(":server")) {
+  api(project(":server")) {
     // JMH ships with the conflicting version 4.6. This prevents us from using jopt-simple in benchmarks (which should be ok) but allows
     // us to invoke the JMH uberjar as usual.
     exclude group: 'net.sf.jopt-simple', module: 'jopt-simple'
@@ -61,12 +50,6 @@ tasks.named("dependenciesGraph").configure {  enabled = false }
 tasks.named("thirdPartyAudit").configure {
   ignoreViolations(
           // these classes intentionally use JDK internal API (and this is ok since the project is maintained by Oracle employees)
-          'org.openjdk.jmh.profile.AbstractHotspotProfiler',
-          'org.openjdk.jmh.profile.HotspotThreadProfiler',
-          'org.openjdk.jmh.profile.HotspotClassloadingProfiler',
-          'org.openjdk.jmh.profile.HotspotCompilationProfiler',
-          'org.openjdk.jmh.profile.HotspotMemoryProfiler',
-          'org.openjdk.jmh.profile.HotspotRuntimeProfiler',
           'org.openjdk.jmh.util.Utils'
   )
 }

benchmarks/src/main/java/org/elasticsearch/benchmark/fs/AvailableIndexFoldersBenchmark.java

@@ -1,20 +1,9 @@
 /*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
  */
 package org.elasticsearch.benchmark.fs;
 

benchmarks/src/main/java/org/elasticsearch/benchmark/indices/breaker/MemoryStatsBenchmark.java

@@ -1,20 +1,9 @@
 /*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
  */
 package org.elasticsearch.benchmark.indices.breaker;
 

benchmarks/src/main/java/org/elasticsearch/benchmark/routing/allocation/AllocationBenchmark.java

@@ -1,20 +1,9 @@
 /*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
  */
 package org.elasticsearch.benchmark.routing.allocation;