Merge branch 'main' into rollup-use-smile-to-replace-map

* main: (175 commits)
  Fix integration test on Windows (elastic#89894)
  Avoiding the use of dynamic map keys in the cluster_formation results of the stable master health indicator (elastic#89842)
  Mute org.elasticsearch.tracing.apm.ApmIT.testCapturesTracesForHttpTraffic (elastic#89891)
  Fix typos in audit event types (elastic#89886)
  Synthetic _source: support histogram field (elastic#89833)
  [TSDB] Rename rollup public API to downsample  (elastic#89809)
  Format script values access (elastic#89780)
  [DOCS] Simplifies composite aggregation recommendation (elastic#89878)
  [DOCS] Update CCS compatibility matrix for 8.3 (elastic#88906)
  Fix memory leak when double invoking RestChannel.sendResponse (elastic#89873)
  [ML] Add processor autoscaling decider (elastic#89645)
  Update disk-usage.asciidoc (elastic#89709) (elastic#89874)
  Add allocation deciders in createComponents (elastic#89836)
  Mute flaky H3LatLonGeometryTest.testIndexPoints (elastic#89870)
  Fix typo in get-snapshot-status-api doc (elastic#89865)
  Picking master eligible node at random in the master stability health indicator (elastic#89841)
  Do not reuse the client after a disruption elastic#89815 (elastic#89866)
  [ML] Distribute trained model allocations across availability zones (elastic#89822)
  Increment clientCalledCount before onResponse (elastic#89858)
  AwaitsFix for elastic#89867
  ...

# Conflicts:
#	x-pack/plugin/rollup/src/main/java/org/elasticsearch/xpack/downsample/RollupShardIndexer.java

Author: weizijun

.ci/bwcVersions

@@ -55,6 +55,7 @@ BWC_VERSION:
   - "7.17.4"
   - "7.17.5"
   - "7.17.6"
+  - "7.17.7"
   - "8.0.0"
   - "8.0.1"
   - "8.1.0"
@@ -69,6 +70,7 @@ BWC_VERSION:
   - "8.3.1"
   - "8.3.2"
   - "8.3.3"
-  - "8.3.4"
   - "8.4.0"
+  - "8.4.1"
+  - "8.4.2"
   - "8.5.0"

.ci/jobs.t/elastic+elasticsearch+periodic+ear.yml

@@ -1,7 +1,6 @@
 ---
 - job:
     name: elastic+elasticsearch+%BRANCH%+periodic+ear
-    workspace: /dev/shm/elastic+elasticsearch+%BRANCH%+periodic+ear
     display-name: "elastic / elasticsearch # %BRANCH% - encryption at rest"
     description: "The Elasticsearch %BRANCH% branch encryption at rest compatibility tests.\n\n"
     node: packaging-large

.ci/snapshotBwcVersions

@@ -1,5 +1,4 @@
 BWC_VERSION:
-  - "7.17.6"
-  - "8.3.4"
-  - "8.4.0"
+  - "7.17.7"
+  - "8.4.2"
   - "8.5.0"

.github/workflows/docs-preview-links.yml

@@ -0,0 +1,26 @@
+name: Docs Preview Links
+
+on:
+  pull_request_target:
+    types: [opened]
+    paths:
+      - '**.asciidoc'
+
+jobs:
+  doc-preview:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/github-script@v6
+        name: Add doc preview links
+        with:
+          script: |
+            const pr = context.payload.pull_request;
+            const comment = `Documentation preview:
+              - ✨ [Changed pages](https://${context.repo.repo}_${pr.number}.docs-preview.app.elstc.co/diff)`;
+            
+            github.rest.issues.createComment({
+              issue_number: context.issue.number,
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              body: comment,
+            });

.github/workflows/gradle-wrapper-validation.yml

@@ -1,6 +1,9 @@
 name: "Validate Gradle Wrapper"
 on: [push]
 
+permissions:
+  contents: read
+
 jobs:
   validation:
     name: "Validation"

README.asciidoc

@@ -26,9 +26,156 @@ If you prefer to install and manage Elasticsearch yourself, you can download
 the latest version from 
 https://www.elastic.co/downloads/elasticsearch[elastic.co/downloads/elasticsearch].
 
-For more installation options, see the
-https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html[Elasticsearch installation
-documentation].
+=== Run Elasticsearch locally
+
+//// 
+IMPORTANT: This content is replicated in the Elasticsearch guide. 
+If you make changes, you must also update setup/set-up-local-dev-deployment.asciidoc.
+////
+
+To try out Elasticsearch on your own machine, we recommend using Docker
+and running both Elasticsearch and Kibana.
+Docker images are available from the https://www.docker.elastic.co[Elastic Docker registry].
+
+NOTE: Starting in Elasticsearch 8.0, security is enabled by default. 
+The first time you start Elasticsearch, TLS encryption is configured automatically, 
+a password is generated for the `elastic` user, 
+and a Kibana enrollment token is created so you can connect Kibana to your secured cluster.
+
+For other installation options, see the
+https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html[Elasticsearch installation documentation].
+
+**Start Elasticsearch**
+
+. Install and start https://www.docker.com/products/docker-desktop[Docker
+Desktop]. Go to **Preferences > Resources > Advanced** and set Memory to at least 4GB.
+
+. Start an Elasticsearch container:
++
+----
+docker network create elastic
+docker pull docker.elastic.co/elasticsearch/elasticsearch:{version} <1>
+docker run --name elasticsearch --net elastic -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -t docker.elastic.co/elasticsearch/elasticsearch:{version}
+----
+<1> Replace {version} with the version of Elasticsearch you want to run.
++
+When you start Elasticsearch for the first time, the generated `elastic` user password and
+Kibana enrollment token are output to the terminal.
++
+NOTE: You might need to scroll back a bit in the terminal to view the password 
+and enrollment token.
+
+. Copy the generated password and enrollment token and save them in a secure 
+location. These values are shown only when you start Elasticsearch for the first time.
+You'll use these to enroll Kibana with your Elasticsearch cluster and log in.
+
+**Start Kibana**
+
+Kibana enables you to easily send requests to Elasticsearch and analyze, visualize, and manage data interactively.
+
+. In a new terminal session, start Kibana and connect it to your Elasticsearch container:
++
+----
+docker pull docker.elastic.co/kibana/kibana:{version} <1>
+docker run --name kibana --net elastic -p 5601:5601 docker.elastic.co/kibana/kibana:{version}
+----
+<1> Replace {version} with the version of Kibana you want to run.
++
+When you start Kibana, a unique URL is output to your terminal.
+
+. To access Kibana, open the generated URL in your browser.
+
+  .. Paste the enrollment token that you copied when starting
+  Elasticsearch and click the button to connect your Kibana instance with Elasticsearch.
+
+  .. Log in to Kibana as the `elastic` user with the password that was generated
+  when you started Elasticsearch.
+
+**Send requests to Elasticsearch**
+
+You send data and other requests to Elasticsearch through REST APIs. 
+You can interact with Elasticsearch using any client that sends HTTP requests, 
+such as the https://www.elastic.co/guide/en/elasticsearch/client/index.html[Elasticsearch
+language clients] and https://curl.se[curl]. 
+Kibana's developer console provides an easy way to experiment and test requests. 
+To access the console, go to **Management > Dev Tools**.
+
+**Add data**
+
+You index data into Elasticsearch by sending JSON objects (documents) through the REST APIs.  
+Whether you have structured or unstructured text, numerical data, or geospatial data, 
+Elasticsearch efficiently stores and indexes it in a way that supports fast searches. 
+
+For timestamped data such as logs and metrics, you typically add documents to a
+data stream made up of multiple auto-generated backing indices.
+
+To add a single document to an index, submit an HTTP post request that targets the index. 
+
+----
+POST /customer/_doc/1
+{
+  "firstname": "Jennifer",
+  "lastname": "Walters"
+}
+----
+
+This request automatically creates the `customer` index if it doesn't exist, 
+adds a new document that has an ID of 1, and 
+stores and indexes the `firstname` and `lastname` fields.
+
+The new document is available immediately from any node in the cluster. 
+You can retrieve it with a GET request that specifies its document ID:
+
+----
+GET /customer/_doc/1
+----
+
+To add multiple documents in one request, use the `_bulk` API.
+Bulk data must be newline-delimited JSON (NDJSON). 
+Each line must end in a newline character (`\n`), including the last line.
+
+----
+PUT customer/_bulk
+{ "create": { } }
+{ "firstname": "Monica","lastname":"Rambeau"}
+{ "create": { } }
+{ "firstname": "Carol","lastname":"Danvers"}
+{ "create": { } }
+{ "firstname": "Wanda","lastname":"Maximoff"}
+{ "create": { } }
+{ "firstname": "Jennifer","lastname":"Takeda"}
+----
+
+**Search**
+
+Indexed documents are available for search in near real-time. 
+The following search matches all customers with a first name of _Jennifer_ 
+in the `customer` index.
+
+----
+GET customer/_search
+{
+  "query" : {
+    "match" : { "firstname": "Jennifer" }  
+  }
+}
+----
+
+**Explore**
+
+You can use Discover in Kibana to interactively search and filter your data.
+From there, you can start creating visualizations and building and sharing dashboards.
+
+To get started, create a _data view_ that connects to one or more Elasticsearch indices,
+data streams, or index aliases.
+
+. Go to **Management > Stack Management > Kibana > Data Views**.
+. Select **Create data view**.
+. Enter a name for the data view and a pattern that matches one or more indices, 
+such as _customer_. 
+. Select **Save data view to Kibana**.  
+
+To start exploring, go to **Analytics > Discover**.
 
 [[upgrade]]
 == Upgrade

TESTING.asciidoc

@@ -113,7 +113,7 @@ password: `elastic-password`.
 You may need to customize the cluster configuration for the ./gradlew run task.
 You can simply find the task in the source code and configure it there.
 (The task is currently defined in build-tools-internal/src/main/groovy/elasticsearch.run.gradle)
-However, this requires modifying a source controlled file and subject to accidental commits.
+However, this requires modifying a source controlled file and is subject to accidental commits.
 Alternatively, you can use a Gradle init script to inject custom build logic with the -I flag to configure this task locally.
 
 For example:
@@ -142,6 +142,40 @@ Now the http.p12 file will be placed in the config directory of the running clus
 Assuming you have the http.ssl.keystore setup correctly, you can now use HTTPS with ./gradlew run without the risk
 of accidentally committing your local configurations.
 
+==== Multiple nodes in the test cluster for ./gradlew run
+
+Another desired customization for ./gradlew run might be to run multiple
+nodes with different setting for each node. For example, you may want to debug a coordinating only node that fans out
+to one or more data nodes. To do this, increase the numberOfNodes and add specific configuration for each
+of the nodes.  For example, the following will instruct the first node (:9200) to be a coordinating only node,
+and all other nodes to be master, data_hot, data_content nodes.
+-------------------------------------
+testClusters.register("runTask") {
+     ...
+    numberOfNodes = 2
+    def cluster = testClusters.named("runTask").get()
+    cluster.getNodes().each { node ->
+      node.setting('cluster.initial_master_nodes', cluster.getLastNode().getName())
+      node.setting('node.roles', '[master,data_hot,data_content]')
+    }
+    cluster.getFirstNode().setting('node.roles', '[]')
+   ...
+}
+-------------------------------------
+
+You can also place this config in custom init script (see above) to avoid accidental commits.
+If you are passing in the --debug-jvm flag with multiple nodes, you will need multiple remote debuggers running. One
+for each node listening at port 5007, 5008, 5009, and so on. Ensure that each remote debugger has auto restart enabled.
+
+==== Manually testing cross cluster search
+
+Use ./gradlew run-ccs to launch 2 clusters wired together for the purposes of cross cluster search.
+For example send a search request "my_remote_cluster:*/_search" to the querying cluster (:9200) to query data
+in the fulfilling cluster.
+
+If you are passing in the --debug-jvm flag, you will need two remote debuggers running. One at port 5007 and another
+one at port 5008. Ensure that each remote debugger has auto restart enabled.
+
 === Test case filtering.
 
 You can run a single test, provided that you specify the Gradle project. See the documentation on

benchmarks/src/main/java/org/elasticsearch/benchmark/script/ScriptScoreBenchmark.java

@@ -40,6 +40,7 @@
 import org.elasticsearch.script.ScoreScript;
 import org.elasticsearch.script.ScriptModule;
 import org.elasticsearch.search.lookup.SearchLookup;
+import org.elasticsearch.search.lookup.SourceLookup;
 import org.openjdk.jmh.annotations.Benchmark;
 import org.openjdk.jmh.annotations.BenchmarkMode;
 import org.openjdk.jmh.annotations.Fork;
@@ -88,7 +89,8 @@ public class ScriptScoreBenchmark {
     private final CircuitBreakerService breakerService = new NoneCircuitBreakerService();
     private final SearchLookup lookup = new SearchLookup(
         fieldTypes::get,
-        (mft, lookup, fdo) -> mft.fielddataBuilder(FieldDataContext.noRuntimeFields("benchmark")).build(fieldDataCache, breakerService)
+        (mft, lookup, fdo) -> mft.fielddataBuilder(FieldDataContext.noRuntimeFields("benchmark")).build(fieldDataCache, breakerService),
+        new SourceLookup.ReaderSourceProvider()
     );
 
     @Param({ "expression", "metal", "painless_cast", "painless_def" })

benchmarks/src/main/java/org/elasticsearch/benchmark/search/fetch/subphase/FetchSourcePhaseBenchmark.java

@@ -79,8 +79,7 @@ private BytesReference buildBigExample(String extraText) throws IOException {
 
     @Benchmark
     public BytesReference filterObjects() throws IOException {
-        SourceLookup lookup = new SourceLookup();
-        lookup.setSource(sourceBytes);
+        SourceLookup lookup = new SourceLookup(new SourceLookup.BytesSourceProvider(sourceBytes));
         Object value = lookup.filter(fetchContext);
         return FetchSourcePhase.objectToBytes(value, XContentType.JSON, Math.min(1024, lookup.internalSourceRef().length()));
     }

build-conventions/src/main/java/org/elasticsearch/gradle/internal/conventions/info/GitInfo.java

@@ -75,18 +75,25 @@ public static GitInfo gitInfo(File rootDir) {
                 head = dotGit.resolve("HEAD");
                 gitDir = dotGit;
             } else {
-                // this is a git worktree, follow the pointer to the repository
-                final Path workTree = Paths.get(readFirstLine(dotGit).substring("gitdir:".length()).trim());
-                if (Files.exists(workTree) == false) {
-                    return new GitInfo("unknown", "unknown");
-                }
-                head = workTree.resolve("HEAD");
-                final Path commonDir = Paths.get(readFirstLine(workTree.resolve("commondir")));
-                if (commonDir.isAbsolute()) {
-                    gitDir = commonDir;
+                // this is a git worktree or submodule, follow the pointer to the repository
+                final Path reference = Paths.get(readFirstLine(dotGit).substring("gitdir:".length()).trim());
+                if (reference.getParent().endsWith("modules")) {
+                    // this is a git submodule so follow the reference to the git repo
+                    gitDir = rootDir.toPath().resolve(reference);
+                    head = gitDir.resolve("HEAD");
                 } else {
-                    // this is the common case
-                    gitDir = workTree.resolve(commonDir);
+                    // this is a worktree so resolve the root repo directory
+                    if (Files.exists(reference) == false) {
+                        return new GitInfo("unknown", "unknown");
+                    }
+                    head = reference.resolve("HEAD");
+                    final Path commonDir = Paths.get(readFirstLine(reference.resolve("commondir")));
+                    if (commonDir.isAbsolute()) {
+                        gitDir = commonDir;
+                    } else {
+                        // this is the common case
+                        gitDir = reference.resolve(commonDir);
+                    }
                 }
             }
             final String ref = readFirstLine(head);