Merge master & resolve conflicts

.gitignore

@@ -5,6 +5,13 @@
 *.ipr
 *.iws
 build-idea/
+out/
+
+# include shared intellij config
+!.idea/scopes/x_pack.xml
+!.idea/inspectionProfiles/Project_Default.xml
+!.idea/runConfigurations/Debug_Elasticsearch.xml
+
 # These files are generated in the main tree by IntelliJ
 benchmarks/src/main/generated/*
 

CONTRIBUTING.md

@@ -103,17 +103,14 @@ be used to test against other JDKs as well, this is not only limited to JDK 11.
 > Note: It is also required to have `JAVA8_HOME`, `JAVA9_HOME`, `JAVA10_HOME`
 and `JAVA11_HOME`, and `JAVA12_HOME` available so that the tests can pass.
 
-> Warning: do not use `sdkman` for Java installations which do not have proper
-`jrunscript` for jdk distributions.
-
 Elasticsearch uses the Gradle wrapper for its build. You can execute Gradle
 using the wrapper via the `gradlew` script on Unix systems or `gradlew.bat`
 script on Windows in the root of the repository. The examples below show the
 usage on Unix.
 
-We support development in the Eclipse and IntelliJ IDEs.
-For Eclipse, the minimum version that we support is [4.13][eclipse].
-For IntelliJ, the minimum version that we support is [IntelliJ 2017.2][intellij].
+We support development in IntelliJ versions IntelliJ 2019.2 and
+onwards. We would like to support Eclipse, but few of us use it and has fallen
+into [disrepair][eclipse].
 
 [Docker](https://docs.docker.com/install/) is required for building some Elasticsearch artifacts and executing certain test suites. You can run Elasticsearch without building all the artifacts with:
 
@@ -123,48 +120,21 @@ You can access Elasticsearch with:
 
     curl -u elastic:password localhost:9200
 
-### Configuring IDEs And Running Tests
-
-Eclipse users can automatically configure their IDE: `./gradlew eclipse`
-then `File: Import: Gradle : Existing Gradle Project`.
-Additionally you will want to ensure that Eclipse is using 2048m of heap by modifying
-`eclipse.ini` accordingly to avoid GC overhead and OOM errors.
-
-IntelliJ users can automatically configure their IDE: `./gradlew idea`
-then `File->New Project From Existing Sources`. Point to the root of
-the source directory, select
-`Import project from external model->Gradle`, enable
-`Use auto-import`. In order to run tests directly from
-IDEA 2017.2 and above, it is required to disable the IDEA run launcher in order to avoid
-`idea_rt.jar` causing "jar hell". This can be achieved by adding the
-`-Didea.no.launcher=true` [JVM
-option](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties).
-Alternatively, `idea.no.launcher=true` can be set in the
-[`idea.properties`](https://www.jetbrains.com/help/idea/file-idea-properties.html)
-file which can be accessed under Help > Edit Custom Properties (this will require a
-restart of IDEA). For IDEA 2017.3 and above, in addition to the JVM option, you will need to go to
-`Run->Edit Configurations->...->Defaults->JUnit` and verify that the `Shorten command line` setting is set to
-`user-local default: none`. You may also need to [remove `ant-javafx.jar` from your
-classpath](https://github.com/elastic/elasticsearch/issues/14348) if that is
-reported as a source of jar hell.
-
-To run an instance of elasticsearch from the source code run `./gradlew run`
-
-The Elasticsearch codebase makes heavy use of Java `assert`s and the
-test runner requires that assertions be enabled within the JVM. This
-can be accomplished by passing the flag `-ea` to the JVM on startup.
-
-For IntelliJ, go to
-`Run->Edit Configurations...->Defaults->JUnit->VM options` and input
-`-ea`.
-
-For Eclipse, go to `Preferences->Java->Installed JREs` and add `-ea` to
-`VM Arguments`.
-
-Some tests related to locale testing also require the flag
-`-Djava.locale.providers` to be set. Set the VM options/VM arguments for
-IntelliJ or Eclipse like describe above to use
-`-Djava.locale.providers=SPI,COMPAT`.
+### Importing the project into IntelliJ IDEA
+
+Elasticsearch builds using Java 13. When importing into IntelliJ you will need
+to define an appropriate SDK. The convention is that **this SDK should be named
+"13"** so that the project import will detect it automatically. For more details
+on defining an SDK in IntelliJ please refer to [their documentation](https://www.jetbrains.com/help/idea/sdk.html#define-sdk).
+SDK definitions are global, so you can add the JDK from any project, or after
+project import. Importing with a missing JDK will still work, IntelliJ will
+simply report a problem and will refuse to build until resolved.
+
+You can import the Elasticsearch project into IntelliJ IDEA via:
+
+ - Select **File > Open**
+ - In the subsequent dialog navigate to the root `build.gradle` file
+ - In the subsequent dialog select **Open as Project**
 
 ### REST Endpoint Conventions
 
@@ -212,14 +182,7 @@ Please follow these formatting guidelines:
   part of a file. Please format such sections sympathetically with the rest
   of the code, while keeping lines to maximum length of 76 characters.
 * Wildcard imports (`import foo.bar.baz.*`) are forbidden and will cause
-  the build to fail. This can be done automatically by your IDE:
-   * Eclipse: `Preferences->Java->Code Style->Organize Imports`. There are
-     two boxes labeled "`Number of (static )? imports needed for .*`". Set
-     their values to 99999 or some other absurdly high value.
-   * IntelliJ: `Preferences/Settings->Editor->Code Style->Java->Imports`.
-     There are two configuration options: `Class count to use import with
-     '*'` and `Names count to use static import with '*'`. Set their values
-     to 99999 or some other absurdly high value.
+  the build to fail.
 * If *absolutely* necessary, you can disable formatting for regions of code
   with the `// tag::NAME` and `// end::NAME` directives, but note that
   these are intended for use in documentation, so please make it clear what
@@ -234,9 +197,6 @@ Please follow these formatting guidelines:
 
 #### Editor / IDE Support
 
-Eclipse IDEs can import the file [.eclipseformat.xml]
-directly.
-
 IntelliJ IDEs can
 [import](https://blog.jetbrains.com/idea/2014/01/intellij-idea-13-importing-code-formatter-settings-from-eclipse/)
 the same settings file, and / or use the [Eclipse Code
@@ -393,26 +353,9 @@ It is important that the only code covered by the Elastic licence is contained
 within the top-level `x-pack` directory. The build will fail its pre-commit
 checks if contributed code does not have the appropriate license headers.
 
-You may find it helpful to configure your IDE to automatically insert the
-appropriate license header depending on the part of the project to which you are
-contributing.
-
-#### IntelliJ: Copyright & Scope Profiles
-
-To have IntelliJ insert the correct license, it is necessary to create to copyright profiles.
-These may potentially be called `apache2` and `commercial`. These can be created in
-`Preferences/Settings->Editor->Copyright->Copyright Profiles`. To associate these profiles to
-their respective directories, two "Scopes" will need to be created. These can be created in
-`Preferences/Settings->Appearances & Behavior->Scopes`. When creating scopes, be sure to choose
-the `shared` scope type. Create a scope, `apache2`, with
-the associated pattern of `!file[group:x-pack]:*/`. This pattern will exclude all the files contained in
-the `x-pack` directory. The other scope, `commercial`, will have the inverse pattern of `file[group:x-pack]:*/`.
-The two scopes, together, should account for all the files in the project. To associate the scopes
-with their copyright-profiles, go into `Preferences/Settings->Editor>Copyright` and use the `+` to add
-the associations `apache2/apache2` and `commercial/commercial`.
-
-Configuring these options in IntelliJ can be quite buggy, so do not be alarmed if you have to open/close
-the settings window and/or restart IntelliJ to see your changes take effect.
+> **NOTE:** If you have imported the project into IntelliJ IDEA the project will
+> be automatically configured to add the correct license header to new source
+> files based on the source location.
 
 ### Creating A Distribution
 
@@ -425,7 +368,7 @@ cd elasticsearch/
 To build a darwin-tar distribution, run this command:
 
 ```sh
-./gradlew -p distribution/archives/darwin-tar assemble --parallel
+./gradlew -p distribution/archives/darwin-tar assemble
 ```
 
 You will find the distribution under:
@@ -435,9 +378,12 @@ To create all build artifacts (e.g., plugins and Javadocs) as well as
 distributions in all formats, run this command:
 
 ```sh
-./gradlew assemble --parallel
+./gradlew assemble
 ```
 
+> **NOTE:** Running the task above will fail if you don't have a available
+> Docker installation.
+
 The package distributions (Debian and RPM) can be found under:
 `./distribution/packages/(deb|rpm|oss-deb|oss-rpm)/build/distributions/`
 
@@ -570,10 +516,6 @@ known as "transitive" dependencies".</dd>
 should not be shipped with the project because it is "provided" by the runtime
 somehow. Elasticsearch plugins use this configuration to include dependencies
 that are bundled with Elasticsearch's server.</dd>
-<dt>`bundle`</dt><dd>Only available in projects with the shadow plugin,
-dependencies with this configuration are bundled into the jar produced by the
-build. Since IDEs do not understand this configuration we rig them to treat
-dependencies in this configuration as `compile` dependencies.</dd>
 <dt>`testCompile`</dt><dd>Code that is on the classpath for compiling tests
 that are part of this project but not production code. The canonical example
 of this is `junit`.</dd>
@@ -608,6 +550,5 @@ Finally, we require that you run `./gradlew check` before submitting a
 non-documentation contribution. This is mentioned above, but it is worth
 repeating in this section because it has come up in this context.
 
-[eclipse]: https://download.eclipse.org/eclipse/downloads/drops4/R-4.13-201909161045/
 [intellij]: https://blog.jetbrains.com/idea/2017/07/intellij-idea-2017-2-is-here-smart-sleek-and-snappy/
-[shadow-plugin]: https://github.com/johnrengelman/shadow
+[eclipse]: https://github.com/elastic/elasticsearch/issues/53664

TESTING.asciidoc

@@ -20,9 +20,9 @@ To create a platform-specific build including the x-pack modules, use the
 following depending on your operating system:
 
 -----------------------------
-./gradlew :distribution:archives:linux-tar:assemble --parallel
-./gradlew :distribution:archives:darwin-tar:assemble --parallel
-./gradlew :distribution:archives:windows-zip:assemble --parallel
+./gradlew :distribution:archives:linux-tar:assemble
+./gradlew :distribution:archives:darwin-tar:assemble
+./gradlew :distribution:archives:windows-zip:assemble
 -----------------------------
 
 === Running Elasticsearch from a checkout
@@ -51,6 +51,10 @@ recommended to configure the IDE to initiate multiple listening attempts. In cas
 is called "Auto restart" and needs to be checked. In case of Eclipse, "Connection limit" setting
 needs to be configured with a greater value (ie 10 or more).
 
+NOTE: If you have imported the project into IntelliJ according to the instructions in
+link:/CONTRIBUTING.md#importing-the-project-into-intellij-idea[CONTRIBUTING.md] then a debug run configuration
+named "Debug Elasticsearch" will be created for you and configured appropriately.
+
 ==== Distribution
 
 By default a node is started with the zip distribution.

build.gradle

@@ -17,20 +17,21 @@
  * under the License.
  */
 
-
 import com.avast.gradle.dockercompose.tasks.ComposePull
 import com.github.jengelman.gradle.plugins.shadow.ShadowPlugin
+import de.thetaphi.forbiddenapis.gradle.ForbiddenApisPlugin
 import org.apache.tools.ant.taskdefs.condition.Os
 import org.elasticsearch.gradle.BuildPlugin
 import org.elasticsearch.gradle.BwcVersions
 import org.elasticsearch.gradle.Version
 import org.elasticsearch.gradle.VersionProperties
 import org.elasticsearch.gradle.plugin.PluginBuildPlugin
+import org.gradle.plugins.ide.eclipse.model.AccessRule
 import org.gradle.plugins.ide.eclipse.model.SourceFolder
 import org.gradle.util.DistributionLocator
 import org.gradle.util.GradleVersion
 
-import static org.elasticsearch.gradle.tool.Boilerplate.maybeConfigure
+import static org.elasticsearch.gradle.util.GradleUtils.maybeConfigure
 
 plugins {
   id 'lifecycle-base'
@@ -43,6 +44,7 @@ apply plugin: 'nebula.info-scm'
 apply from: 'gradle/build-scan.gradle'
 apply from: 'gradle/build-complete.gradle'
 apply from: 'gradle/runtime-jdk-provision.gradle'
+apply from: 'gradle/ide.gradle'
 
 // common maven publishing configuration
 allprojects {
@@ -180,7 +182,6 @@ allprojects {
       System.getProperty("eclipse.application") != null ||    // Detects gradle launched from the Eclipse compiler server
       gradle.startParameter.taskNames.contains('eclipse') ||  // Detects gradle launched from the command line to do eclipse stuff
       gradle.startParameter.taskNames.contains('cleanEclipse')
-    isIdea = System.getProperty("idea.active") != null || gradle.startParameter.taskNames.contains('idea') || gradle.startParameter.taskNames.contains('cleanIdea')
 
     // for BWC testing
     bwcVersions = versions
@@ -356,36 +357,6 @@ gradle.projectsEvaluated {
   }
 }
 
-// intellij configuration
-allprojects {
-  apply plugin: 'idea'
-
-  if (isIdea) {
-    project.buildDir = file('build-idea')
-  }
-  idea {
-    module {
-      inheritOutputDirs = false
-      outputDir = file('build-idea/classes/main')
-      testOutputDir = file('build-idea/classes/test')
-
-      // also ignore other possible build dirs
-      excludeDirs += file('build')
-      excludeDirs += file('build-eclipse')
-    }
-  }
-
-  tasks.named('cleanIdea') {
-    delete 'build-idea'
-  }
-}
-
-idea {
-  project {
-    vcs = 'Git'
-  }
-}
-
 // eclipse configuration
 allprojects {
   apply plugin: 'eclipse'
@@ -408,6 +379,21 @@ allprojects {
       }
     }
   }
+  /*
+   * Allow accessing com/sun/net/httpserver in projects that have
+   * configured forbidden apis to allow it.
+   */
+  plugins.withType(ForbiddenApisPlugin) {
+    eclipse.classpath.file.whenMerged { classpath ->
+      if (false == forbiddenApisTest.bundledSignatures.contains('jdk-non-portable')) {
+        classpath.entries
+          .findAll { it.kind == "con" && it.toString().contains("org.eclipse.jdt.launching.JRE_CONTAINER") }
+          .each {
+            it.accessRules.add(new AccessRule("accessible", "com/sun/net/httpserver/*"))
+          }
+      }
+    }
+  }
 
   File licenseHeaderFile
   String prefix = ':x-pack'
@@ -559,3 +545,15 @@ allprojects {
     }
   }
 }
+
+// TODO: remove this once 7.7 is released and the 7.x branch is 7.8
+subprojects {
+  pluginManager.withPlugin('elasticsearch.testclusters') {
+    testClusters.all {
+      if (org.elasticsearch.gradle.info.BuildParams.isSnapshotBuild() == false) {
+        systemProperty 'es.itv2_feature_flag_registered', 'true'
+        systemProperty 'es.datastreams_feature_flag_registered', 'true'
+      }
+    }
+  }
+}

buildSrc/build.gradle

@@ -187,8 +187,10 @@ if (project != rootProject) {
     distribution project(':distribution:archives:oss-windows-zip')
     distribution project(':distribution:archives:darwin-tar')
     distribution project(':distribution:archives:oss-darwin-tar')
+    distribution project(':distribution:archives:linux-aarch64-tar')
     distribution project(':distribution:archives:linux-tar')
     distribution project(':distribution:archives:oss-linux-tar')
+    distribution project(':distribution:archives:oss-linux-aarch64-tar')
   }
 
   // for external projects we want to remove the marker file indicating we are running the Elasticsearch project