Configure heap dump path out of the box

The JVM defaults to dumping the heap to the working directory of
Elasticsearch. For the RPM and Debian packages, this location is
/usr/share/elasticsearch. This directory is not writable by the
elasticsearch user, so by default heap dumps in this situation are
lost. This commit modifies the packaging for the RPM and Debian packages
to set the heap dump path to /var/lib/elasticsearch as the default
location for dumping the heap. This location is writable by the
elasticsearch user by default. We add documentation of this important
setting if /var/lib/elasticsearch is not suitable for receiving heap
dumps.

Relates #26755

Author: jasontedor

distribution/build.gradle

@@ -537,6 +537,12 @@ Map<String, String> expansionsForDistribution(distributionType) {
     'heap.min': defaultHeapSize,
     'heap.max': defaultHeapSize,
 
+    'heap.dump.path': [
+      'deb': "-XX:HeapDumpPath=/var/lib/elasticsearch",
+      'rpm': "-XX:HeapDumpPath=/var/lib/elasticsearch",
+      'def': "#-XX:HeapDumpPath=/heap/dump/path"
+    ],
+
     'stopping.timeout': [
       'rpm': 86400,
     ],

distribution/src/main/resources/config/jvm.options

@@ -80,7 +80,7 @@
 
 # specify an alternative path for heap dumps
 # ensure the directory exists and has sufficient space
-#-XX:HeapDumpPath=${heap.dump.path}
+${heap.dump.path}
 
 ## GC logging
 

docs/reference/migration/migrate_6_0/packaging.asciidoc

@@ -76,3 +76,17 @@ environment used to start Elasticsearch (and various supporting scripts). This
 legacy feature could be useful when there were several environment variables
 useful for configuring JVM options; this functionality had previously been
 replaced by <<jvm-options>>. Therefore, `ES_INCLUDE` has been removed.
+
+==== Heap dump path default
+
+Previous versions of Elasticsearch did not configure the JVM heap dump
+path. As such, the JVM would default to the working directory for the
+JVM. For the RPM and Debian package distributions this was a problem
+because the default would be `/usr/share/elasticsearch`, a path not
+writable by the `elasticsearch` user meaning that out of the box heap
+dumps would not be produced. It is a tragedy to not have this diagnostic
+information by default. Starting in Elasticsearch 6.0.0, the RPM and
+Debian packages <<heap-dump-path,configure the heap dump path>> to be
+`/var/lib/elasticsearch`. If this location is not suitable for receiving
+heap dumps (e.g., due to disk space concerns), you should adjust this
+path to a more suitable location.

docs/reference/setup/important-settings.asciidoc

@@ -12,6 +12,7 @@ configured before going into production.
 * <<network.host,`network.host`>>
 * <<unicast.hosts,`discovery.zen.ping.unicast.hosts`>>
 * <<minimum_master_nodes,`discovery.zen.minimum_master_nodes`>>
+* <<heap-dump-path,JVM heap dump path>>
 
 [float]
 [[path-settings]]
@@ -180,3 +181,23 @@ nodes should be set to `(3 / 2) + 1` or `2`:
 discovery.zen.minimum_master_nodes: 2
 --------------------------------------------------
 
+[float]
+[[heap-dump-path]]
+=== JVM heap dump path
+
+The <<rpm,RPM>> and <<deb,Debian>> package distributions default to configuring
+the JVM to dump the heap on out of memory exceptions to
+`/var/lib/elasticsearch`. If this path is not suitable for storing heap dumps,
+you should modify the entry `-XX:HeapDumpPath=/var/lib/elasticsearch` in
+<<jvm-options,`jvm.options`>> to an alternate path.  If you specify a filename
+instead of a directory, the JVM will repeatedly use the same file; this is one
+mechanism for preventing heap dumps from accumulating in the heap dump path.
+Alternatively, you can configure a scheduled task via your OS to remove heap
+dumps that are older than a configured age.
+
+Note that the archive distributions do not configure the heap dump path by
+default. Instead, the JVM will default to dumping to the working directory for
+the Elasticsearch process. If you wish to configure a heap dump path, you should
+modify the entry `#-XX:HeapDumpPath=/heap/dump/path` in
+<<jvm-options,`jvm.options`>> to remove the comment marker `#` and to specify an
+actual path.