Configure heap dump path out of the box

The JVM defaults to dumping the heap to the working directory of
Elasticsearch. For the RPM and Debian packages, this location is
/usr/share/elasticsearch. This directory is not writable by the
elasticsearch user, so by default heap dumps in this situation are
lost. This commit modifies the packaging for the RPM and Debian packages
to set the heap dump path to /var/lib/elasticsearch as the default
location for dumping the heap. This location is writable by the
elasticsearch user by default. We add documentation of this important
setting if /var/lib/elasticsearch is not suitable for receiving heap
dumps.

Relates #26755

Author: jasontedor

distribution/build.gradle

@@ -537,6 +537,12 @@ Map<String, String> expansionsForDistribution(distributionType) {
     'heap.min': defaultHeapSize,
     'heap.max': defaultHeapSize,
 
+    'heap.dump.path': [
+      'deb': "-XX:HeapDumpPath=/var/lib/elasticsearch",
+      'rpm': "-XX:HeapDumpPath=/var/lib/elasticsearch",
+      'def': "#-XX:HeapDumpPath=/heap/dump/path"
+    ],
+
     'stopping.timeout': [
       'rpm': 86400,
     ],

distribution/src/main/resources/config/jvm.options

@@ -80,7 +80,7 @@
 
 # specify an alternative path for heap dumps
 # ensure the directory exists and has sufficient space
-#-XX:HeapDumpPath=${heap.dump.path}
+${heap.dump.path}
 
 ## GC logging
 

docs/reference/setup/important-settings.asciidoc

@@ -12,6 +12,7 @@ configured before going into production.
 * <<network.host,`network.host`>>
 * <<unicast.hosts,`discovery.zen.ping.unicast.hosts`>>
 * <<minimum_master_nodes,`discovery.zen.minimum_master_nodes`>>
+* <<heap-dump-path,JVM heap dump path>>
 
 [float]
 [[path-settings]]
@@ -180,3 +181,23 @@ nodes should be set to `(3 / 2) + 1` or `2`:
 discovery.zen.minimum_master_nodes: 2
 --------------------------------------------------
 
+[float]
+[[heap-dump-path]]
+=== JVM heap dump path
+
+The <<rpm,RPM>> and <<deb,Debian>> package distributions default to configuring
+the JVM to dump the heap on out of memory exceptions to
+`/var/lib/elasticsearch`. If this path is not suitable for storing heap dumps,
+you should modify the entry `-XX:HeapDumpPath=/var/lib/elasticsearch` in
+<<jvm-options,`jvm.options`>> to an alternate path.  If you specify a filename
+instead of a directory, the JVM will repeatedly use the same file; this is one
+mechanism for preventing heap dumps from accumulating in the heap dump path.
+Alternatively, you can configure a scheduled task via your OS to remove heap
+dumps that are older than a configured age.
+
+Note that the archive distributions do not configure the heap dump path by
+default. Instead, the JVM will default to dumping to the working directory for
+the Elasticsearch process. If you wish to configure a heap dump path, you should
+modify the entry `#-XX:HeapDumpPath=/heap/dump/path` in
+<<jvm-options,`jvm.options`>> to remove the comment marker `#` and to specify an
+actual path.