Bootstrap test for ability to write OOM heap dump

[robin13]

Currently it is possible for elasticsearch users to have -XX:+HeapDumpOnOutOfMemoryError enabled (it is enabled by default), but in many installations (rpm/deb) the default location ($ES_HOME which is usually /var/share/elasticsearch) is not writable.
Alternatively the location can be explicitly defined with -XX:HeapDumpPath= but this location may not be writable... so if an OOM happens, no heapdump can be written, which in turn greatly complicates debugging the root cause.
It would be great to as a first step have this as part of bootstrap checks, initially creating a warning and in a later version actually blocking elasticsearch starting.
This could be difficult considering the Java security manager...

[jasontedor]

As I mentioned in another channel, this is not going to work as a bootstrap check. The location could be a location that we do not grant security manager file permissions for, but the Elasticsearch user could have filesystem permissions for and therefore the JVM will have no trouble writing the heap dump from this perspective should it exit on out of memory exception. Trying to check if we have filesystem permissions will die with a security manager access control exception. While we could grant ourselves the requisite permissions, this strikes me as a bad idea as this location can be an arbitrary location on disk.

[geekpete]

How about if it's an rpm/deb install that we put this check in the service manager script?
This way before the elasticsearch process is kicked off, the script can test this location is writable.
Probably a bit hacky as the script needs to read configs and check if the setting is writable by the configured elastic user, but it's also specific to those users who might have the common packaging issue causing it.

There's also the approach of ensuring any default packaging/installs on common (or all supported?) OSes works properly for writing heap dumps out of the box and fix any packaging bug if it exists. I didn't see any existing packaging bugs related to heap dump location default writability that I could find.

[jasontedor]

Sorry, parsing the JVM options file from the startup scripts is not something that I would be willing to add as it's too messy.