Elastic Server automatically shutdown

[dobariya]

I have three elastic servers all masters having total 32 GB ram each,16 GB allocated to elastic servers.
From last few days any one of three servers randomly automatically goes down and the java.exe is killed automatically.Even no error is logged in elastic search logs.

Am using virtual environment with network file system to store data and logs
Elastic search version is 1.7.0
Indices count:around 300
Total documents in whole elastic search : around 60 million

Below is my config

cluster.name: Cluster1

cluster.routing.allocation.disk.threshold_enabled: false
script.disable_dynamic: false
node.name: "Master1"

node.master: true
node.data: true

index.query.bool.max_clause_count: 50100
indices.fielddata.cache.size: 25%
indices.fielddata.cache.expire: 5m
action.disable_delete_all_indices: true
indices.cluster.send_refresh_mapping: false
index.cache.field.type: soft

path.data: \nas5\Elasticsearch\Data

path.logs: \nas5\Elasticsearch\Logs\Master1

bootstrap.mlockall: true

http.max_content_length: 999mb

indices.recovery.max_bytes_per_sec: 100mb

indices.recovery.concurrent_streams: 5

above config goes same for all three servers Master1,Master2,Master3.

[dbblackdiamond]

Hi,

I am having the same problem. I have a 4 nodes cluster and node seems to randomly crash. The nodes are running on Ubuntu 15.04, kernel version 3.19.0.28. This is a small installation, so each node is configured with 512MB of RAM. I have tried increasing it and it didn't make any difference.
There are no messages in the /var/log/elasticsearch/clustername.log file.
Below is the content of the /var/log/apport.log file:

ERROR: apport (pid 13340) Thu Sep 17 02:25:53 2015: called for pid 28872, signal 6, core limit 0
ERROR: apport (pid 13340) Thu Sep 17 02:25:53 2015: executable: /usr/lib/jvm/java-7-oracle/jre/bin/java (command line "/usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.7.1.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch")
ERROR: apport (pid 13340) Thu Sep 17 02:25:53 2015: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 13340) Thu Sep 17 02:25:57 2015: core dump exceeded 49 MiB, dropped from /var/crash/_usr_lib_jvm_java-7-oracle_jre_bin_java.112.crash to avoid memory overflow
ERROR: apport (pid 13340) Thu Sep 17 02:25:57 2015: wrote report /var/crash/_usr_lib_jvm_java-7-oracle_jre_bin_java.112.crash

Below is the content of the /var/crash/_usr_lib_jvm_java-7-oracle_jre_bin_java.112.crash:

ProblemType: Crash
Architecture: amd64
CrashCounter: 1
Date: Thu Sep 17 02:25:53 2015
DistroRelease: Ubuntu 15.04
ExecutablePath: /usr/lib/jvm/java-7-oracle/jre/bin/java
ExecutableTimestamp: 1442115528
ProcCmdline: /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOut
OfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.7.1.jar:/usr/sh
are/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elast
icsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
ProcCwd: /
ProcEnviron:
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/false
ProcMaps:
 00400000-00401000 r-xp 00000000 08:01 258460                             /usr/lib/jvm/java-7-oracle/jre/bin/java
 00600000-00601000 rw-p 00000000 08:01 258460                             /usr/lib/jvm/java-7-oracle/jre/bin/java
 02232000-0309f000 rw-p 00000000 00:00 0                                  [heap]
 bae00000-cae00000 rw-p 00000000 00:00 0 
 cae00000-fae00000 rw-p 00000000 00:00 0 
 fae00000-ff0ec000 rw-p 00000000 00:00 0 
 ff0ec000-100000000 rw-p 00000000 00:00 0 
 7f3616b70000-7f3617867000 r--s 00000000 08:01 492968                     /var/opt/elasticsearch/data/graylog2/nodes/0/indices/graylog_0/3/index/_clw_Lucene41_0.tim
 7f3618258000-7f3618c48000 r--s 00000000 08:01 492605                     /var/opt/elasticsearch/data/graylog2/nodes/0/indices/.marvel-2015.09.16/0/index/_i8x_Lucene41_0.tim
 7f3618c48000-7f3619a4a000 r--s 00000000 08:01 492645                     /var/opt/elasticsearch/data/graylog2/nodes/0/indices/graylog_0/0/index/_ipe_Lucene41_0.tim
 7f361a818000-7f361b331000 r--s 00000000 08:01 492865                     /var/opt/elasticsearch/data/graylog2/nodes/0/indices/.marvel-2015.09.16/0/index/_6c5_Lucene41_0.tim
 7f361c707000-7f361c70a000 ---p 00000000 00:00 0 
 7f361c70a000-7f361c808000 rw-p 00000000 00:00 0 
 7f361c8d5000-7f361c8d8000 ---p 00000000 00:00 0 
 7f361c8d8000-7f361c9d6000 rw-p 00000000 00:00 0                          [stack:11126]
 7f361c9d6000-7f361c9d9000 ---p 00000000 00:00 0 
 7f361c9d9000-7f361cad7000 rw-p 00000000 00:00 0                          [stack:28934]
 7f361cad7000-7f361cada000 ---p 00000000 00:00 0 
 7f361cada000-7f361cbd8000 rw-p 00000000 00:00 0                          [stack:28933]
 7f361dd8c000-7f361dd8f000 ---p 00000000 00:00 0 
 7f361dd8f000-7f361de8d000 rw-p 00000000 00:00 0                          [stack:28935]
 7f361de8d000-7f361de90000 ---p 00000000 00:00 0 
 7f361de90000-7f361df8e000 rw-p 00000000 00:00 0                          [stack:28931]
 7f361df8e000-7f361df91000 ---p 00000000 00:00 0 
 7f361df91000-7f361e08f000 rw-p 00000000 00:00 0                          [stack:28930]
 7f361e08f000-7f361e092000 ---p 00000000 00:00 0 
 7f361e092000-7f361e190000 rw-p 00000000 00:00 0                          [stack:10651]
 7f361e760000-7f361eaef000 r--s 00000000 08:01 492585                     /var/opt/elasticsearch/data/graylog2/nodes/0/indices/.marvel-2015.09.17/0/index/_1j0_Lucene41_0.tim
 7f361ee7c000-7f361ee7f000 ---p 00000000 00:00 0 
 7f361ee7f000-7f361ef7d000 rw-p 00000000 00:00 0                          [stack:28928]
 7f361ef7d000-7f361ef80000 ---p 00000000 00:00 0 
 7f361ef80000-7f361f07e000 rw-p 00000000 00:00 0                          [stack:13312]
 7f361f07e000-7f361f081000 ---p 00000000 00:00 0 
 7f361f081000-7f361f17f000 rw-p 00000000 00:00 0                          [stack:28926]
 7f361f17f000-7f361f182000 ---p 00000000 00:00 0 
 7f361f182000-7f361f280000 rw-p 00000000 00:00 0                          [stack:28925]
 7f361f458000-7f361f6a1000 r--s 00000000 08:01 492617                     /var/opt/elasticsearch/data/graylog2/nodes/0/indices/graylog_0/3/index/_pkb_Lucene41_0.tim
 7f361f6a7000-7f361f6aa000 ---p 00000000 00:00 0 
 7f361f6aa000-7f361f7a8000 rw-p 00000000 00:00 0 
 7f361f7a8000-7f361fffd000 r--s 00000000 08:01 492672                     /var/opt/elasticsearch/data/graylog2/nodes/0/indices/.marvel-2015.09.16/0/index/_ayb_Lucene41_0.tim
 7f3620000000-7f362015b000 rw-p 00000000 00:00 0 
 7f362015b000-7f3624000000 ---p 00000000 00:00 0 
 7f3624000000-7f3625802000 rw-p 00000000 00:00 0 
 7f3625802000-7f3628000000 ---p 00000000 00:00 0 
 7f3628000000-7f3628036000 rw-p 00000000 00:00 0 
 7f3628036000-7f362c000000 ---p 00000000 00:00 0 
 7f362c000000-7f362c128000 rw-p 00000000 00:00 0 
 7f362c128000-7f3630000000 ---p 00000000 00:00 0 
 7f3630000000-7f363004d000 rw-p 00000000 00:00 0 
 7f363004d000-7f3634000000 ---p 00000000 00:00 0 
 7f3634000000-7f363426c000 rw-p 00000000 00:00 0 
 7f363426c000-7f3638000000 ---p 00000000 00:00 0 
 7f3638008000-7f3638009000 r--s 00000000 08:01 492793                     /var/opt/elasticsearch/data/graylog2/nodes/0/indices/.marvel-2015.09.16/0/index/_6c5_Lucene410_0.dvd
 7f36380bf000-7f36380c3000 r--s 000f7000 08:01 258384                     /usr/lib/jvm/java-7-oracle/jre/lib/ext/localedata.jar
 7f3638157000-7f363815a000 ---p 00000000 00:00 0 
 7f363815a000-7f3638258000 rw-p 00000000 00:00 0                          [stack:28924]
 7f3638278000-7f3638313000 r--s 00000000 08:01 492969                     /var/opt/elasticsearch/data/graylog2/nodes/0/indices/graylog_0/3/index/_clw_Lucene410_0.dvd
 7f3638314000-7f3638317000 ---p 00000000 00:00 0 
 7f3638317000-7f3638415000 rw-p 00000000 00:00 0 
 7f3638415000-7f3638418000 ---p 00000000 00:00 0 
 7f3638418000-7f3638516000 rw-p 00000000 00:00 0 
 7f3638516000-7f3638519000 ---p 00000000 00:00 0 
 7f3638519000-7f3638617000 rw-p 00000000 00:00 0 
 7f3638617000-7f363861a000 ---p 00000000 00:00 0 
 7f363861a000-7f3638718000 rw-p 00000000 00:00 0 
 7f3638730000-7f3638731000 r--s 00000000 08:01 492629                     /var/opt/elasticsearch/data/graylog2/nodes/0/indices/.marvel-2015.09.17/0/index/_1j0_Lucene410_0.dvd
 7f3638738000-7f363874e000 r--s 00000000 08:01 492619                     /var/opt/elasticsearch/data/graylog2/nodes/0/indices/graylog_0/3/index/_pkb_Lucene410_0.dvd
 7f3638750000-7f36387be000 r--s 00000000 08:01 257053                     /var/opt/elasticsearch/data/graylog2/nodes/0/indices/.marvel-2015.09.13/0/index/_by_Lucene41_0.tim
 7f36387c4000-7f36387c7000 ---p 00000000 00:00 0 
 7f36387c7000-7f36388c5000 rw-p 00000000 00:00 0 
 7f36388c5000-7f36388c8000 ---p 00000000 00:00 0 
 7f36388c8000-7f36389c6000 rw-p 00000000 00:00 0 
 7f36389c6000-7f36389c9000 ---p 00000000 00:00 0 
 7f36389c9000-7f3638ac7000 rw-p 00000000 00:00 0 
 7f3638ac7000-7f3638aca000 ---p 00000000 00:00 0 
 7f3638aca000-7f3638bc8000 rw-p 00000000 00:00 0 
 7f3638bc8000-7f3638ef7000 r--s 00000000 08:01 492600                     /var/opt/elasticsearch/data/graylog2/nodes/0/indices/.marvel-2015.09.16/0/index/_foo_Lucene41_0.tim
 7f3638ef8000-7f3638fa0000 r--s 00000000 08:01 492657                     /var/opt/elasticsearch/data/graylog2/nodes/0/indices/graylog_0/0/index/_ipe_Lucene410_0.dvd
 7f3638fa8000-7f3638fa9000 r--s 00000000 08:01 257050                     /var/opt/elasticsearch/data/graylog2/nodes/0/indices/.marvel-2015.09.13/0/index/_by_Lucene410_0.dvd
 7f3638fb0000-7f3638fb1000 r--s 00000000 08:01 492675                     /var/opt/elasticsearch/data/graylog2/nodes/0/indices/.marvel-2015.09.16/0/index/_ayb_Lucene410_0.dvd
 7f3638fb8000-7f3638fb9000 r--s 00000000 08:01 492608                     /var/opt/elasticsearch/data/graylog2/nodes/0/indices/.marvel-2015.09.16/0/index/_i8x_Lucene410_0.dvd
 7f3638fc0000-7f3638fc1000 r--s 00000000 08:01 492602                     /var/opt/elasticsearch/data/graylog2/nodes/0/indices/.marvel-2015.09.16/0/index/_foo_Lucene410_0.dvd
 7f3638fc6000-7f3638fc9000 ---p 00000000 00:00 0 
 7f3638fc9000-7f36390c7000 rw-p 00000000 00:00 0                          [stack:28923]
 7f36390c7000-7f36390ca000 ---p 00000000 00:00 0 
 7f36390ca000-7f36391c8000 rw-p 00000000 00:00 0                          [stack:28939]
 7f36391c8000-7f36391cb000 ---p 00000000 00:00 0 
 7f36391cb000-7f36392c9000 rw-p 00000000 00:00 0 
 7f36392c9000-7f36392cc000 ---p 00000000 00:00 0 
 7f36392cc000-7f36393ca000 rw-p 00000000 00:00 0 
 7f36393ca000-7f36393cd000 ---p 00000000 00:00 0 
 7f36393cd000-7f36394cb000 rw-p 00000000 00:00 0                          [stack:28940]
 7f36394cb000-7f36394ce000 ---p 00000000 00:00 0 
 7f36394ce000-7f36395cc000 rw-p 00000000 00:00 0                          [stack:28915]
 7f36395cc000-7f36395cf000 ---p 00000000 00:00 0 
 7f36395cf000-7f36396cd000 rw-p 00000000 00:00 0                          [stack:28914]
 7f36396cd000-7f36396d0000 ---p 00000000 00:00 0 
 7f36396d0000-7f36397ce000 rw-p 00000000 00:00 0                          [stack:28913]
 7f36397ce000-7f36397d1000 ---p 00000000 00:00 0 
 7f36397d1000-7f36398cf000 rw-p 00000000 00:00 0                          [stack:28912]
 7f36398cf000-7f36398d2000 ---p 00000000 00:00 0 
 7f36398d2000-7f36399d0000 rw-p 00000000 00:00 0                          [stack:28911]
 7f36399d0000-7f36399d3000 ---p 00000000 00:00 0 
 7f36399d3000-7f3639ad1000 rw-p 00000000 00:00 0 
 7f3639ad1000-7f3639ad4000 ---p 00000000 00:00 0 
 7f3639ad4000-7f3639bd2000 rw-p 00000000 00:00 0                          [stack:28907]
 7f3639bd2000-7f3639bd5000 ---p 00000000 00:00 0 
 7f3639bd5000-7f3639cd3000 rw-p 00000000 00:00 0 
 7f3639cd3000-7f3639cd6000 ---p 00000000 00:00 0 
 7f3639cd6000-7f3639dd4000 rw-p 00000000 00:00 0                          [stack:29136]
 7f3639dd4000-7f3639dd7000 ---p 00000000 00:00 0 
 7f3639dd7000-7f3639ed5000 rw-p 00000000 00:00 0                          [stack:28904]
 7f3639ed5000-7f3639ed8000 ---p 00000000 00:00 0 
 7f3639ed8000-7f3639fd6000 rw-p 00000000 00:00 0                          [stack:28903]
 7f3639fd6000-7f3639fd9000 ---p 00000000 00:00 0 
 7f3639fd9000-7f363a0d7000 rw-p 00000000 00:00 0                          [stack:28902]
 7f363a0d7000-7f363a0da000 ---p 00000000 00:00 0 
 7f363a0da000-7f363a1d8000 rw-p 00000000 00:00 0                          [stack:28901]
 7f363a1d8000-7f363a1db000 ---p 00000000 00:00 0 
 7f363a1db000-7f363a2d9000 rw-p 00000000 00:00 0                          [stack:28900]
 7f363a2d9000-7f363a2dc000 ---p 00000000 00:00 0 
 7f363a2dc000-7f363a3da000 rw-p 00000000 00:00 0                          [stack:28899]
 7f363a3da000-7f363a3dd000 ---p 00000000 00:00 0 
 7f363a3dd000-7f363a4db000 rw-p 00000000 00:00 0                          [stack:28898]
 7f363a4db000-7f363a4de000 ---p 00000000 00:00 0 
 7f363a4de000-7f363a5dc000 rw-p 00000000 00:00 0                          [stack:28897]
 7f363a5dc000-7f363a5df000 ---p 00000000 00:00 0 
 7f363a5df000-7f363a6dd000 rw-p 00000000 00:00 0                          [stack:28896]
 7f363a6dd000-7f363a6e0000 ---p 00000000 00:00 0 
 7f363a6e0000-7f363a7de000 rw-p 00000000 00:00 0                          [stack:28895]
 7f363a7de000-7f363a7e1000 ---p 00000000 00:00 0 
 7f363a7e1000-7f363a8df000 rw-p 00000000 00:00 0                          [stack:28894]
 7f363a8df000-7f363a8e2000 ---p 00000000 00:00 0 
 7f363a8e2000-7f363a9e0000 rw-p 00000000 00:00 0                          [stack:28893]
 7f363a9e0000-7f363aa0f000 r-xp 00000000 08:01 255559                     /usr/share/elasticsearch/lib/sigar/libsigar-amd64-linux.so
 7f363aa0f000-7f363ab0e000 ---p 0002f000 08:01 255559                     /usr/share/elasticsearch/lib/sigar/libsigar-amd64-linux.so
 7f363ab0e000-7f363ab14000 rw-p 0002e000 08:01 255559                     /usr/share/elasticsearch/lib/sigar/libsigar-amd64-linux.so
 7f363ab14000-7f363ab17000 rw-p 00000000 00:00 0 
 7f363ab17000-7f363ab1a000 ---p 00000000 00:00 0 
 7f363ab1a000-7f363ac18000 rw-p 00000000 00:00 0                          [stack:28891]
 7f363ac18000-7f363ac1a000 r--s 00010000 08:01 255931                     /usr/share/elasticsearch/plugins/marvel/marvel-1.3.1.jar
 7f363ac20000-7f363ac30000 r-xp 00000000 08:01 257747                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/libnio.so
 7f363ac30000-7f363ae30000 ---p 00010000 08:01 257747                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/libnio.so
 7f363ae30000-7f363ae31000 rw-p 00010000 08:01 257747                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/libnio.so
 7f363ae33000-7f363ae37000 r--s 0008b000 08:01 258373                     /usr/lib/jvm/java-7-oracle/jre/lib/jsse.jar
 7f363ae3a000-7f363ae41000 r--s 00062000 08:01 255562                     /usr/share/elasticsearch/lib/sigar/sigar-1.6.4.jar
 7f363ae47000-7f363ae4d000 r--s 0002f000 08:01 255579                     /usr/share/elasticsearch/lib/lucene-queries-4.10.4.jar
 7f363ae52000-7f363ae63000 r--s 000b2000 08:01 255569                     /usr/share/elasticsearch/lib/jts-1.13.jar
 7f363ae68000-7f363ae69000 r--s 00008000 08:01 255577                     /usr/share/elasticsearch/lib/lucene-memory-4.10.4.jar
 7f363ae6f000-7f363ae78000 r--s 00057000 08:01 255580                     /usr/share/elasticsearch/lib/lucene-queryparser-4.10.4.jar
 7f363ae7e000-7f363ae81000 r--s 00026000 08:01 255563                     /usr/share/elasticsearch/lib/antlr-runtime-3.5.jar
 7f363ae88000-7f363ae8a000 r--s 00008000 08:01 255566                     /usr/share/elasticsearch/lib/asm-commons-4.1.jar
 7f363ae90000-7f363aeba000 r--s 00248000 08:01 255572                     /usr/share/elasticsearch/lib/lucene-core-4.10.4.jar
 7f363aebe000-7f363af2d000 r--s 0063e000 08:01 255567                     /usr/share/elasticsearch/lib/groovy-all-2.4.4.jar
 7f363af30000-7f363b230000 rw-p 00000000 00:00 0 
 7f363b230000-7f363b247000 r-xp 00000000 08:01 257781                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/libnet.so
 7f363b247000-7f363b446000 ---p 00017000 08:01 257781                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/libnet.so
 7f363b446000-7f363b447000 rw-p 00016000 08:01 257781                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/libnet.so
 7f363b448000-7f363b450000 r-xp 00000000 08:01 257755                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/libmanagement.so
 7f363b450000-7f363b64f000 ---p 00008000 08:01 257755                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/libmanagement.so
 7f363b64f000-7f363b650000 rw-p 00007000 08:01 257755                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/libmanagement.so
 7f363b654000-7f363b766000 r--s 00c54000 08:01 255557                     /usr/share/elasticsearch/lib/elasticsearch-1.7.1.jar
 7f363b76a000-7f363b76b000 ---p 00000000 00:00 0 
 7f363b76b000-7f363b86b000 rw-p 00000000 00:00 0                          [stack:28890]
 7f363b86b000-7f363b86e000 ---p 00000000 00:00 0 
 7f363b86e000-7f363b96c000 rw-p 00000000 00:00 0                          [stack:28889]
 7f363b96c000-7f363b96f000 ---p 00000000 00:00 0 
 7f363b96f000-7f363ba6d000 rw-p 00000000 00:00 0                          [stack:28888]
 7f363ba6d000-7f363ba70000 ---p 00000000 00:00 0 
 7f363ba70000-7f363bb6e000 rw-p 00000000 00:00 0                          [stack:28887]
 7f363bb6e000-7f363bb71000 ---p 00000000 00:00 0 
 7f363bb71000-7f363bc6f000 rw-p 00000000 00:00 0                          [stack:28886]
 7f363bc6f000-7f363bc72000 ---p 00000000 00:00 0 
 7f363bc72000-7f363bd70000 rw-p 00000000 00:00 0                          [stack:28885]
 7f363bd70000-7f363bef9000 r--p 00000000 08:01 54985                      /usr/lib/locale/locale-archive
 7f363beff000-7f363bf02000 ---p 00000000 00:00 0 
 7f363bf02000-7f363c000000 rw-p 00000000 00:00 0                          [stack:28884]
 7f363c000000-7f363c29e000 rw-p 00000000 00:00 0 
 7f363c29e000-7f3640000000 ---p 00000000 00:00 0 
 7f3640003000-7f3640006000 r--s 0001b000 08:01 255581                     /usr/share/elasticsearch/lib/lucene-sandbox-4.10.4.jar
 7f364000c000-7f3640010000 r--s 000dc000 08:01 255568                     /usr/share/elasticsearch/lib/jna-4.1.0.jar
 7f3640017000-7f364001a000 r--s 0001f000 08:01 255575                     /usr/share/elasticsearch/lib/lucene-highlighter-4.10.4.jar
 7f364001c000-7f364001f000 r--s 0001c000 08:01 255582                     /usr/share/elasticsearch/lib/lucene-spatial-4.10.4.jar
 7f3640026000-7f364004e000 rw-p 00000000 00:00 0 
 7f364004e000-7f3640051000 ---p 00000000 00:00 0 
 7f3640051000-7f364014f000 rw-p 00000000 00:00 0                          [stack:28883]
 7f364014f000-7f3640150000 ---p 00000000 00:00 0 
 7f3640150000-7f364029b000 rw-p 00000000 00:00 0                          [stack:28882]
 7f364029b000-7f364045b000 r--s 039fb000 08:01 258429                     /usr/lib/jvm/java-7-oracle/jre/lib/rt.jar
 7f3640462000-7f3643dae000 rw-p 00000000 00:00 0 
 7f3643dae000-7f3643f2e000 rw-p 00000000 00:00 0 
 7f3643f2e000-7f3643fae000 rw-p 00000000 00:00 0 
 7f3643fae000-7f364412e000 rw-p 00000000 00:00 0 
 7f364412e000-7f3644150000 rw-p 00000000 00:00 0 
 7f3644150000-7f3644157000 rw-p 00000000 00:00 0 
 7f3644157000-7f3644158000 rw-p 00000000 00:00 0 
 7f3644158000-7f36449c8000 rwxp 00000000 00:00 0 
 7f36449c8000-7f3647158000 rw-p 00000000 00:00 0 
 7f3647158000-7f364716f000 r-xp 00000000 08:01 257776                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/libzip.so
 7f364716f000-7f364736f000 ---p 00017000 08:01 257776                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/libzip.so
 7f364736f000-7f3647370000 rw-p 00017000 08:01 257776                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/libzip.so
 7f3647370000-7f364737c000 r-xp 00000000 08:01 2212                       /lib/x86_64-linux-gnu/libnss_files-2.21.so
 7f364737c000-7f364757b000 ---p 0000c000 08:01 2212                       /lib/x86_64-linux-gnu/libnss_files-2.21.so
 7f364757b000-7f364757c000 r--p 0000b000 08:01 2212                       /lib/x86_64-linux-gnu/libnss_files-2.21.so
 7f364757c000-7f364757d000 rw-p 0000c000 08:01 2212                       /lib/x86_64-linux-gnu/libnss_files-2.21.so
 7f3647580000-7f364758b000 r-xp 00000000 08:01 2200                       /lib/x86_64-linux-gnu/libnss_nis-2.21.so
 7f364758b000-7f364778a000 ---p 0000b000 08:01 2200                       /lib/x86_64-linux-gnu/libnss_nis-2.21.so
 7f364778a000-7f364778b000 r--p 0000a000 08:01 2200                       /lib/x86_64-linux-gnu/libnss_nis-2.21.so
 7f364778b000-7f364778c000 rw-p 0000b000 08:01 2200                       /lib/x86_64-linux-gnu/libnss_nis-2.21.so
 7f3647790000-7f36477a7000 r-xp 00000000 08:01 2218                       /lib/x86_64-linux-gnu/libnsl-2.21.so
 7f36477a7000-7f36479a6000 ---p 00017000 08:01 2218                       /lib/x86_64-linux-gnu/libnsl-2.21.so
 7f36479a6000-7f36479a7000 r--p 00016000 08:01 2218                       /lib/x86_64-linux-gnu/libnsl-2.21.so
 7f36479a7000-7f36479a8000 rw-p 00017000 08:01 2218                       /lib/x86_64-linux-gnu/libnsl-2.21.so
 7f36479a8000-7f36479aa000 rw-p 00000000 00:00 0 
 7f36479b0000-7f36479b8000 r-xp 00000000 08:01 2214                       /lib/x86_64-linux-gnu/libnss_compat-2.21.so
 7f36479b8000-7f3647bb7000 ---p 00008000 08:01 2214                       /lib/x86_64-linux-gnu/libnss_compat-2.21.so
 7f3647bb7000-7f3647bb8000 r--p 00007000 08:01 2214                       /lib/x86_64-linux-gnu/libnss_compat-2.21.so
 7f3647bb8000-7f3647bb9000 rw-p 00008000 08:01 2214                       /lib/x86_64-linux-gnu/libnss_compat-2.21.so
 7f3647bc0000-7f3647be9000 r-xp 00000000 08:01 257742                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/libjava.so
 7f3647be9000-7f3647de9000 ---p 00029000 08:01 257742                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/libjava.so
 7f3647de9000-7f3647deb000 rw-p 00029000 08:01 257742                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/libjava.so
 7f3647df0000-7f3647dfd000 r-xp 00000000 08:01 257769                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/libverify.so
 7f3647dfd000-7f3647ffc000 ---p 0000d000 08:01 257769                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/libverify.so
 7f3647ffc000-7f3647ffe000 rw-p 0000c000 08:01 257769                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/libverify.so
 7f3648000000-7f3648cdc000 rw-p 00000000 00:00 0 
 7f3648cdc000-7f364c000000 ---p 00000000 00:00 0 
 7f364c007000-7f364c009000 r--s 00017000 08:01 255584                     /usr/share/elasticsearch/lib/spatial4j-0.4.1.jar
 7f364c009000-7f364c00b000 r--s 00011000 08:01 255573                     /usr/share/elasticsearch/lib/lucene-expressions-4.10.4.jar
 7f364c00b000-7f364c00c000 r--s 0000b000 08:01 255565                     /usr/share/elasticsearch/lib/asm-4.1.jar
 7f364c00e000-7f364c016000 r--s 00066000 08:01 255564                     /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar
 7f364c016000-7f364c018000 r--s 00016000 08:01 255578                     /usr/share/elasticsearch/lib/lucene-misc-4.10.4.jar
 7f364c019000-7f364c01c000 r--s 00029000 08:01 255583                     /usr/share/elasticsearch/lib/lucene-suggest-4.10.4.jar
 7f364c01e000-7f364c020000 r--s 0000e000 08:01 255576                     /usr/share/elasticsearch/lib/lucene-join-4.10.4.jar
 7f364c020000-7f364c023000 r--s 00018000 08:01 255574                     /usr/share/elasticsearch/lib/lucene-grouping-4.10.4.jar
 7f364c025000-7f364c026000 ---p 00000000 00:00 0 
 7f364c026000-7f364c188000 rw-p 00000000 00:00 0                          [stack:28881]
 7f364c188000-7f364c190000 rw-p 00000000 00:00 0 
 7f364c190000-7f364c197000 r-xp 00000000 08:01 2219                       /lib/x86_64-linux-gnu/librt-2.21.so
 7f364c197000-7f364c396000 ---p 00007000 08:01 2219                       /lib/x86_64-linux-gnu/librt-2.21.so
 7f364c396000-7f364c397000 r--p 00006000 08:01 2219                       /lib/x86_64-linux-gnu/librt-2.21.so
 7f364c397000-7f364c398000 rw-p 00007000 08:01 2219                       /lib/x86_64-linux-gnu/librt-2.21.so
 7f364c398000-7f364c49f000 r-xp 00000000 08:01 2217                       /lib/x86_64-linux-gnu/libm-2.21.so
 7f364c49f000-7f364c69e000 ---p 00107000 08:01 2217                       /lib/x86_64-linux-gnu/libm-2.21.so
 7f364c69e000-7f364c69f000 r--p 00106000 08:01 2217                       /lib/x86_64-linux-gnu/libm-2.21.so
 7f364c69f000-7f364c6a0000 rw-p 00107000 08:01 2217                       /lib/x86_64-linux-gnu/libm-2.21.so
 7f364c6a0000-7f364d216000 r-xp 00000000 08:01 257796                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so
 7f364d216000-7f364d415000 ---p 00b76000 08:01 257796                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so
 7f364d415000-7f364d4d9000 rw-p 00b75000 08:01 257796                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so
 7f364d4d9000-7f364d51a000 rw-p 00000000 00:00 0 
 7f364d520000-7f364d6e0000 r-xp 00000000 08:01 2222                       /lib/x86_64-linux-gnu/libc-2.21.so
 7f364d6e0000-7f364d8e0000 ---p 001c0000 08:01 2222                       /lib/x86_64-linux-gnu/libc-2.21.so
 7f364d8e0000-7f364d8e4000 r--p 001c0000 08:01 2222                       /lib/x86_64-linux-gnu/libc-2.21.so
 7f364d8e4000-7f364d8e6000 rw-p 001c4000 08:01 2222                       /lib/x86_64-linux-gnu/libc-2.21.so
 7f364d8e6000-7f364d8ea000 rw-p 00000000 00:00 0 
 7f364d8f0000-7f364d8f3000 r-xp 00000000 08:01 2205                       /lib/x86_64-linux-gnu/libdl-2.21.so
 7f364d8f3000-7f364daf2000 ---p 00003000 08:01 2205                       /lib/x86_64-linux-gnu/libdl-2.21.so
 7f364daf2000-7f364daf3000 r--p 00002000 08:01 2205                       /lib/x86_64-linux-gnu/libdl-2.21.so
 7f364daf3000-7f364daf4000 rw-p 00003000 08:01 2205                       /lib/x86_64-linux-gnu/libdl-2.21.so
 7f364daf8000-7f364db0d000 r-xp 00000000 08:01 257788                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/jli/libjli.so
 7f364db0d000-7f364dd0c000 ---p 00015000 08:01 257788                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/jli/libjli.so
 7f364dd0c000-7f364dd0d000 rw-p 00014000 08:01 257788                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/jli/libjli.so
 7f364dd10000-7f364dd28000 r-xp 00000000 08:01 2209                       /lib/x86_64-linux-gnu/libpthread-2.21.so
 7f364dd28000-7f364df28000 ---p 00018000 08:01 2209                       /lib/x86_64-linux-gnu/libpthread-2.21.so
 7f364df28000-7f364df29000 r--p 00018000 08:01 2209                       /lib/x86_64-linux-gnu/libpthread-2.21.so
 7f364df29000-7f364df2a000 rw-p 00019000 08:01 2209                       /lib/x86_64-linux-gnu/libpthread-2.21.so
 7f364df2a000-7f364df2e000 rw-p 00000000 00:00 0 
 7f364df30000-7f364df54000 r-xp 00000000 08:01 2208                       /lib/x86_64-linux-gnu/ld-2.21.so
 7f364df57000-7f364df60000 r--s 0006f000 08:01 255570                     /usr/share/elasticsearch/lib/log4j-1.2.17.jar
 7f364df67000-7f364df78000 r--s 0018f000 08:01 255571                     /usr/share/elasticsearch/lib/lucene-analyzers-common-4.10.4.jar
 7f364df78000-7f364df9a000 rw-p 00000000 00:00 0 
 7f364df9a000-7f364e038000 rw-p 00000000 00:00 0 
 7f364e038000-7f364e040000 rw-s 00000000 08:01 255598                     /tmp/hsperfdata_elasticsearch/28872 (deleted)
 7f364e046000-7f364e049000 ---p 00000000 00:00 0 
 7f364e049000-7f364e148000 rw-p 00000000 00:00 0                          [stack:28880]
 7f364e14c000-7f364e14e000 rw-p 00000000 00:00 0 
 7f364e14e000-7f364e14f000 r--p 00000000 00:00 0 
 7f364e14f000-7f364e153000 rw-p 00000000 00:00 0 
 7f364e153000-7f364e154000 r--p 00023000 08:01 2208                       /lib/x86_64-linux-gnu/ld-2.21.so
 7f364e154000-7f364e155000 rw-p 00024000 08:01 2208                       /lib/x86_64-linux-gnu/ld-2.21.so
 7f364e155000-7f364e157000 rw-p 00000000 00:00 0 
 7fff042c1000-7fff042e2000 rw-p 00000000 00:00 0                          [stack]
 7fff04358000-7fff0435a000 r--p 00000000 00:00 0                          [vvar]
 7fff0435a000-7fff0435c000 r-xp 00000000 00:00 0                          [vdso]
 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
ProcStatus:
 Name:  java
 State: D (disk sleep)
 Tgid:  28872
 Ngid:  0
 Pid:   28872
 PPid:  1
 TracerPid: 0
 Uid:   112 112 112 112
 Gid:   120 120 120 120
 FDSize:    512
 Groups:    120 
 VmPeak:     2038676 kB
 VmSize:     1977020 kB
 VmLck:        0 kB
 VmPin:        0 kB
 VmHWM:   351900 kB
 VmRSS:   322512 kB
 VmData:     1849952 kB
 VmStk:      136 kB
 VmExe:        4 kB
 VmLib:    15844 kB
 VmPTE:     1112 kB
 VmSwap:       42280 kB
 Threads:   47
 SigQ:  0/1924
 SigPnd:    0000000000000000
 ShdPnd:    0000000000000000
 SigBlk:    0000000000000000
 SigIgn:    0000000000000000
 SigCgt:    2000000181005ccf
 CapInh:    0000000000000000
 CapPrm:    0000000000000000
 CapEff:    0000000000000000
 CapBnd:    0000003fffffffff
 Seccomp:   0
 Cpus_allowed:  1
 Cpus_allowed_list: 0
 Mems_allowed:  00000000,00000001
 Mems_allowed_list: 0
 voluntary_ctxt_switches:   31
 nonvoluntary_ctxt_switches:    1
Signal: 6
Uname: Linux 3.19.0-28-generic x86_64
UserGroups: 

Below is the content of my /etc/elasticsearch/elasticsearch.yml:

##################### Elasticsearch Configuration Example #####################

# This file contains an overview of various configuration settings,
# targeted at operations staff. Application developers should
# consult the guide at <http://elasticsearch.org/guide>.
#
# The installation procedure is covered at
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html>.
#
# Elasticsearch comes with reasonable defaults for most settings,
# so you can try it out without bothering with configuration.
#
# Most of the time, these defaults are just fine for running a production
# cluster. If you're fine-tuning your cluster, or wondering about the
# effect of certain configuration option, please _do ask_ on the
# mailing list or IRC channel [http://elasticsearch.org/community].

# Any element in the configuration can be replaced with environment variables
# by placing them in ${...} notation. For example:
#
#node.rack: ${RACK_ENV_VAR}

# For information on supported formats and syntax for the config file, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html>


################################### Cluster ###################################

# Cluster name identifies your cluster for auto-discovery. If you're running
# multiple clusters on the same network, make sure you're using unique names.
#
cluster.name: "graylog2"


#################################### Node #####################################

# Node names are generated dynamically on startup, so you're relieved
# from configuring them manually. You can tie this node to a specific name:
#
#node.name: "Franz Kafka"

# Every node can be configured to allow or deny being eligible as the master,
# and to allow or deny to store the data.
#
# Allow this node to be eligible as a master node (enabled by default):
#
node.master: true
#
# Allow this node to store data (enabled by default):
#
node.data: true

# You can exploit these settings to design advanced cluster topologies.
#
# 1. You want this node to never become a master node, only to hold data.
#    This will be the "workhorse" of your cluster.
#
#node.master: false
#node.data: true
#
# 2. You want this node to only serve as a master: to not store any data and
#    to have free resources. This will be the "coordinator" of your cluster.
#
#node.master: true
#node.data: false
#
# 3. You want this node to be neither master nor data node, but
#    to act as a "search load balancer" (fetching data from nodes,
#    aggregating results, etc.)
#
#node.master: false
#node.data: false

# Use the Cluster Health API [http://localhost:9200/_cluster/health], the
# Node Info API [http://localhost:9200/_nodes] or GUI tools
# such as <http://www.elasticsearch.org/overview/marvel/>,
# <http://github.com/karmi/elasticsearch-paramedic>,
# <http://github.com/lukas-vlcek/bigdesk> and
# <http://mobz.github.com/elasticsearch-head> to inspect the cluster state.

# A node can have generic attributes associated with it, which can later be used
# for customized shard allocation filtering, or allocation awareness. An attribute
# is a simple key value pair, similar to node.key: value, here is an example:
#
#node.rack: rack314

# By default, multiple nodes are allowed to start from the same installation location
# to disable it, set the following:
#node.max_local_storage_nodes: 1


#################################### Index ####################################

# You can set a number of options (such as shard/replica options, mapping
# or analyzer definitions, translog settings, ...) for indices globally,
# in this file.
#
# Note, that it makes more sense to configure index settings specifically for
# a certain index, either when creating it or by using the index templates API.
#
# See <http://elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules.html> and
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/indices-create-index.html>
# for more information.

# Set the number of shards (splits) of an index (5 by default):
#
#index.number_of_shards: 5

# Set the number of replicas (additional copies) of an index (1 by default):
#
#index.number_of_replicas: 1

# Note, that for development on a local machine, with small indices, it usually
# makes sense to "disable" the distributed features:
#
#index.number_of_shards: 1
#index.number_of_replicas: 0

# These settings directly affect the performance of index and search operations
# in your cluster. Assuming you have enough machines to hold shards and
# replicas, the rule of thumb is:
#
# 1. Having more *shards* enhances the _indexing_ performance and allows to
#    _distribute_ a big index across machines.
# 2. Having more *replicas* enhances the _search_ performance and improves the
#    cluster _availability_.
#
# The "number_of_shards" is a one-time setting for an index.
#
# The "number_of_replicas" can be increased or decreased anytime,
# by using the Index Update Settings API.
#
# Elasticsearch takes care about load balancing, relocating, gathering the
# results from nodes, etc. Experiment with different settings to fine-tune
# your setup.

# Use the Index Status API (<http://localhost:9200/A/_status>) to inspect
# the index status.


#################################### Paths ####################################

# Path to directory containing configuration (this file and logging.yml):
#
#path.conf: /path/to/conf

# Path to directory where to store index data allocated for this node.
#
path.data: /var/opt/elasticsearch/data
#
# Can optionally include more than one location, causing data to be striped across
# the locations (a la RAID 0) on a file level, favouring locations with most free
# space on creation. For example:
#
#path.data: /path/to/data1,/path/to/data2

# Path to temporary files:
#
path.work: /var/opt/elasticsearch/temp

# Path to log files:
#
#path.logs: /path/to/logs

# Path to where plugins are installed:
#
#path.plugins: /path/to/plugins


#################################### Plugin ###################################

# If a plugin listed here is not installed for current node, the node will not start.
#
#plugin.mandatory: mapper-attachments,lang-groovy


################################### Memory ####################################

# Elasticsearch performs poorly when JVM starts swapping: you should ensure that
# it _never_ swaps.
#
# Set this property to true to lock the memory:
#
#bootstrap.mlockall: true

# Make sure that the ES_MIN_MEM and ES_MAX_MEM environment variables are set
# to the same value, and that the machine has enough memory to allocate
# for Elasticsearch, leaving enough memory for the operating system itself.
#
# You should also make sure that the Elasticsearch process is allowed to lock
# the memory, eg. by using `ulimit -l unlimited`.


############################## Network And HTTP ###############################

# Elasticsearch, by default, binds itself to the 0.0.0.0 address, and listens
# on port [9200-9300] for HTTP traffic and on port [9300-9400] for node-to-node
# communication. (the range means that if the port is busy, it will automatically
# try the next port).

# Set the bind address specifically (IPv4 or IPv6):
#
#network.bind_host: 192.168.0.1

# Set the address other nodes will use to communicate with this node. If not
# set, it is automatically derived. It must point to an actual IP address.
#
#network.publish_host: 192.168.0.1

# Set both 'bind_host' and 'publish_host':
#
network.host: 192.168.1.133

# Set a custom port for the node to node communication (9300 by default):
#
#transport.tcp.port: 9300

# Enable compression for all communication between nodes (disabled by default):
#
#transport.tcp.compress: true

# Set a custom port to listen for HTTP traffic:
#
#http.port: 9200

# Set a custom allowed content length:
#
#http.max_content_length: 100mb

# Disable HTTP completely:
#
#http.enabled: false


################################### Gateway ###################################

# The gateway allows for persisting the cluster state between full cluster
# restarts. Every change to the state (such as adding an index) will be stored
# in the gateway, and when the cluster starts up for the first time,
# it will read its state from the gateway.

# There are several types of gateway implementations. For more information, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html>.

# The default gateway type is the "local" gateway (recommended):
#
#gateway.type: local

# Settings below control how and when to start the initial recovery process on
# a full cluster restart (to reuse as much local data as possible when using shared
# gateway).

# Allow recovery process after N nodes in a cluster are up:
#
#gateway.recover_after_nodes: 1

# Set the timeout to initiate the recovery process, once the N nodes
# from previous setting are up (accepts time value):
#
#gateway.recover_after_time: 5m

# Set how many nodes are expected in this cluster. Once these N nodes
# are up (and recover_after_nodes is met), begin recovery process immediately
# (without waiting for recover_after_time to expire):
#
#gateway.expected_nodes: 2


############################# Recovery Throttling #############################

# These settings allow to control the process of shards allocation between
# nodes during initial recovery, replica allocation, rebalancing,
# or when adding and removing nodes.

# Set the number of concurrent recoveries happening on a node:
#
# 1. During the initial recovery
#
#cluster.routing.allocation.node_initial_primaries_recoveries: 4
#
# 2. During adding/removing nodes, rebalancing, etc
#
#cluster.routing.allocation.node_concurrent_recoveries: 2

# Set to throttle throughput when recovering (eg. 100mb, by default 20mb):
#
#indices.recovery.max_bytes_per_sec: 20mb

# Set to limit the number of open concurrent streams when
# recovering a shard from a peer:
#
#indices.recovery.concurrent_streams: 5


################################## Discovery ##################################

# Discovery infrastructure ensures nodes can be found within a cluster
# and master node is elected. Multicast discovery is the default.

# Set to ensure a node sees N other master eligible nodes to be considered
# operational within the cluster. This should be set to a quorum/majority of 
# the master-eligible nodes in the cluster.
#
#discovery.zen.minimum_master_nodes: 1

# Set the time to wait for ping responses from other nodes when discovering.
# Set this option to a higher value on a slow or congested network
# to minimize discovery failures:
#
#discovery.zen.ping.timeout: 3s

# For more information, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html>

# Unicast discovery allows to explicitly control which nodes will be used
# to discover the cluster. It can be used when multicast is not present,
# or to restrict the cluster communication-wise.
#
# 1. Disable multicast discovery (enabled by default):
#
#discovery.zen.ping.multicast.enabled: false
#
# 2. Configure an initial list of master nodes in the cluster
#    to perform discovery when new nodes (master or data) are started:
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2:port"]

# EC2 discovery allows to use AWS EC2 API in order to perform discovery.
#
# You have to install the cloud-aws plugin for enabling the EC2 discovery.
#
# For more information, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html>
#
# See <http://elasticsearch.org/tutorials/elasticsearch-on-ec2/>
# for a step-by-step tutorial.

# GCE discovery allows to use Google Compute Engine API in order to perform discovery.
#
# You have to install the cloud-gce plugin for enabling the GCE discovery.
#
# For more information, see <https://github.com/elasticsearch/elasticsearch-cloud-gce>.

# Azure discovery allows to use Azure API in order to perform discovery.
#
# You have to install the cloud-azure plugin for enabling the Azure discovery.
#
# For more information, see <https://github.com/elasticsearch/elasticsearch-cloud-azure>.

################################## Slow Log ##################################

# Shard level query and fetch threshold logging.

#index.search.slowlog.threshold.query.warn: 10s
#index.search.slowlog.threshold.query.info: 5s
#index.search.slowlog.threshold.query.debug: 2s
#index.search.slowlog.threshold.query.trace: 500ms

#index.search.slowlog.threshold.fetch.warn: 1s
#index.search.slowlog.threshold.fetch.info: 800ms
#index.search.slowlog.threshold.fetch.debug: 500ms
#index.search.slowlog.threshold.fetch.trace: 200ms

#index.indexing.slowlog.threshold.index.warn: 10s
#index.indexing.slowlog.threshold.index.info: 5s
#index.indexing.slowlog.threshold.index.debug: 2s
#index.indexing.slowlog.threshold.index.trace: 500ms

################################## GC Logging ################################

#monitor.jvm.gc.young.warn: 1000ms
#monitor.jvm.gc.young.info: 700ms
#monitor.jvm.gc.young.debug: 400ms

#monitor.jvm.gc.old.warn: 10s
#monitor.jvm.gc.old.info: 5s
#monitor.jvm.gc.old.debug: 2s

################################## Security ################################

# Uncomment if you want to enable JSONP as a valid return transport on the
# http server. With this enabled, it may pose a security risk, so disabling
# it unless you need it is recommended (it is disabled by default).
#
#http.jsonp.enable: true

Below is the output from "curl -XGET http://192.168.1.131:9200/_cluster/health?pretty

{
  "cluster_name" : "graylog2",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 9,
  "active_shards" : 18,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0
}

Any idea about what is going on? I am running Elasticsearch 1.7.1, installed from the debian .deb file, pulled directly from the elasticsearch.org website.

Thanks a lot,
Bertrand.

[imotov]

If it is always the same server that is crashing and all servers have the same version of JVM and os, it might be faulty memory on this server. Could you try running a memory diagnostics test to rule faulty memory out?

[dbblackdiamond]

Unfortunately, it isn't always the same server. Also, all servers are running as a Virtual Machine on the same physical server, so if it was faulty memory, wouldn't that affect all the virtual machines?

[dobariya]

yes correct the same situation is with me as well. Just for clarification my nodes are running on windows server 2008 R2 virtual machines

[clintongormley]

There is a good chance it is libsigar causing this. try deleting the lib/sigar directory

2.0 doesn't rely on sigar at all

[dbblackdiamond]

OK... I have removed the lib/sigar directory on 2 of my 4 nodes. Let see if it makes any difference.

[dbblackdiamond]

Hi, so far, my cluster has been pretty stable. Unfortunately, the host, that runs it, crashed and I had to restart everything, but for the last 48 hours, it has been stable. I'll keep monitoring it and will report back.

[dbblackdiamond]

I have looked at my cluster this morning and again, it looks like the java executable crashed again.
Below is the content of the apport.log file:

ERROR: apport (pid 30379) Thu Sep 24 06:57:53 2015: called for pid 2038, signal 6, core limit 0
ERROR: apport (pid 30379) Thu Sep 24 06:57:53 2015: executable: /usr/lib/jvm/java-7-oracle/jre/bin/java (command lin
e "/usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiating
OccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfil
e.encoding=UTF-8 -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elastic
search/lib/elasticsearch-1.7.1.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.pidfile=
/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var
/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.config=/etc/elasticsearch/elasticsearc
h.yml -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch")
ERROR: apport (pid 30379) Thu Sep 24 06:57:53 2015: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 30379) Thu Sep 24 06:57:58 2015: core dump exceeded 56 MiB, dropped from /var/crash/_usr_lib_jvm_
java-7-oracle_jre_bin_java.112.crash to avoid memory overflow
ERROR: apport (pid 30379) Thu Sep 24 06:57:58 2015: wrote report /var/crash/_usr_lib_jvm_java-7-oracle_jre_bin_java.
112.crash

Below is the content of the .crash file:

ProblemType: Crash
Architecture: amd64
CrashCounter: 1
Date: Thu Sep 24 06:57:53 2015
DistroRelease: Ubuntu 15.04
ExecutablePath: /usr/lib/jvm/java-7-oracle/jre/bin/java
ExecutableTimestamp: 1442115528
ProcCmdline: /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMS
InitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplic
itGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch -cp :/usr/sha
re/elasticsearch/lib/elasticsearch-1.7.1.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -De
s.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path
.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.config=/etc/elasticsearch/el
asticsearch.yml -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
ProcCwd: /
ProcEnviron:
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/false
ProcMaps:
 00400000-00401000 r-xp 00000000 08:01 258460                             /usr/lib/jvm/java-7-oracle/jre/bin/java
 00600000-00601000 rw-p 00000000 08:01 258460                             /usr/lib/jvm/java-7-oracle/jre/bin/java
 01818000-02758000 rw-p 00000000 00:00 0                                  [heap]
 bae00000-cae00000 rw-p 00000000 00:00 0 
 cae00000-fae00000 rw-p 00000000 00:00 0 
 fae00000-ff063000 rw-p 00000000 00:00 0 
 ff063000-100000000 rw-p 00000000 00:00 0 
 7ff4cf168000-7ff4cf5be000 r--s 00000000 08:01 493109                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.24/0/index/_2o4_Lucene41_0.tim
 7ff4cfe70000-7ff4d0039000 r--s 00000000 08:01 493077                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.23/0/index/_a5d_Lucene41_0.tim
 7ff4d03d0000-7ff4d09cd000 r--s 00000000 08:01 492675                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.23/0/index/_8z4_Lucene41_0.tim
 7ff4d09d0000-7ff4d1469000 r--s 00000000 08:01 257032                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/2/index/_3rdp_Lucene41_0.tim
 7ff4d1470000-7ff4d1a56000 r--s 00000000 08:01 493016                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.23/0/index/_596_Lucene41_0.tim
 7ff4d1a57000-7ff4d1a5a000 ---p 00000000 00:00 0 
 7ff4d1a5a000-7ff4d1b58000 rw-p 00000000 00:00 0 
 7ff4d1b58000-7ff4d1ef2000 r--s 00000000 08:01 492653                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.22/0/index/_3t8_Lucene41_0.tim
 7ff4d1ef7000-7ff4d1efa000 ---p 00000000 00:00 0 
 7ff4d1efa000-7ff4d1ff8000 rw-p 00000000 00:00 0 
 7ff4d2037000-7ff4d203a000 ---p 00000000 00:00 0 
 7ff4d203a000-7ff4d2138000 rw-p 00000000 00:00 0 
 7ff4d2138000-7ff4d21b7000 r--s 00000000 08:01 257131                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/2/index/_3rdp_Lucene410_0.dvd
 7ff4d21b8000-7ff4d225d000 r--s 00000000 08:01 493070                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/0/index/_37ie_Lucene410_0.dvd
 7ff4d2260000-7ff4d2e42000 r--s 00000000 08:01 493054                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/0/index/_37ie_Lucene41_0.tim
 7ff4d2e48000-7ff4d2e8e000 r--s 00000000 08:01 257176                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/2/index/_39f1_Lucene410_0.dvd
 7ff4d2e90000-7ff4d3487000 r--s 00000000 08:01 257174                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/2/index/_39f1_Lucene41_0.tim
 7ff4d34a8000-7ff4d34ea000 r--s 00000000 08:01 493155                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/1/index/_3acs_Lucene410_0.dvd
 7ff4d34f0000-7ff4d3a92000 r--s 00000000 08:01 493047                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/1/index/_3acs_Lucene41_0.tim
 7ff4d3a98000-7ff4d3ad5000 r--s 00000000 08:01 493035                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/0/index/_37gf_Lucene410_0.dvd
 7ff4d3ad8000-7ff4d401a000 r--s 00000000 08:01 493029                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/0/index/_37gf_Lucene41_0.tim
 7ff4d4030000-7ff4d4031000 r--s 00000000 08:01 493068                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.24/0/index/_2o4_Lucene410_0.dvd
 7ff4d4040000-7ff4d4041000 r--s 00000000 08:01 492663                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.23/0/index/_8z4_Lucene410_0.dvd
 7ff4d4048000-7ff4d4049000 r--s 00000000 08:01 493081                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.23/0/index/_a5d_Lucene410_0.dvd
 7ff4d4050000-7ff4d4051000 r--s 00000000 08:01 492655                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.22/0/index/_3t8_Lucene410_0.dvd
 7ff4d4058000-7ff4d4059000 r--s 00000000 08:01 493027                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.23/0/index/_596_Lucene410_0.dvd
 7ff4d405b000-7ff4d405e000 ---p 00000000 00:00 0 
 7ff4d405e000-7ff4d415c000 rw-p 00000000 00:00 0                          [stack:2099]
 7ff4d415c000-7ff4d415f000 ---p 00000000 00:00 0 
 7ff4d415f000-7ff4d425d000 rw-p 00000000 00:00 0                          [stack:2098]
 7ff4d425d000-7ff4d4260000 ---p 00000000 00:00 0 
 7ff4d4260000-7ff4d435e000 rw-p 00000000 00:00 0                          [stack:2097]
 7ff4d435e000-7ff4d4361000 ---p 00000000 00:00 0 
 7ff4d4361000-7ff4d445f000 rw-p 00000000 00:00 0                          [stack:2096]
 7ff4d445f000-7ff4d4462000 ---p 00000000 00:00 0 
 7ff4d4462000-7ff4d4560000 rw-p 00000000 00:00 0                          [stack:2095]
 7ff4d4560000-7ff4d4561000 r--s 00000000 08:01 492944                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.17/0/index/_d6y_Lucene410_0.dvd
 7ff4d4568000-7ff4d48db000 r--s 00000000 08:01 492955                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.17/0/index/_d6y_Lucene41_0.tim
 7ff4d48e0000-7ff4d48e1000 r--s 00000000 08:01 492936                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.17/0/index/_brt_Lucene410_0.dvd
 7ff4d48e8000-7ff4d549c000 r--s 00000000 08:01 492919                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.17/0/index/_brt_Lucene41_0.tim
 7ff4d54a0000-7ff4d54a2000 r--s 00000000 08:01 492995                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.17/0/index/_6lc_Lucene410_0.dvd
 7ff4d54a8000-7ff4d63fe000 r--s 00000000 08:01 493006                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.17/0/index/_6lc_Lucene41_0.tim
 7ff4d6400000-7ff4d6401000 r--s 00000000 08:01 492863                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.20/0/index/_54b_Lucene410_0.dvd
 7ff4d6408000-7ff4d65e2000 r--s 00000000 08:01 492857                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.20/0/index/_54b_Lucene41_0.tim
 7ff4d65e8000-7ff4d65ea000 r--s 00000000 08:01 492888                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.20/0/index/_4gn_Lucene410_0.dvd
 7ff4d65f0000-7ff4d701b000 r--s 00000000 08:01 492876                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.20/0/index/_4gn_Lucene41_0.tim
 7ff4d7020000-7ff4d7098000 r--s 00000000 08:01 257042                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/2/index/_2qip_Lucene410_0.dvd
 7ff4d7098000-7ff4d7acb000 r--s 00000000 08:01 257060                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/2/index/_2qip_Lucene41_0.tim
 7ff4d7ad0000-7ff4d7b29000 r--s 00000000 08:01 257057                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/2/index/_8fk_Lucene410_0.dvd
 7ff4d7b30000-7ff4d8301000 r--s 00000000 08:01 257159                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/2/index/_8fk_Lucene41_0.tim
 7ff4d8308000-7ff4d83ae000 r--s 00000000 08:01 257051                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/2/index/_1eq3_Lucene410_0.dvd
 7ff4d83b0000-7ff4d90de000 r--s 00000000 08:01 257155                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/2/index/_1eq3_Lucene41_0.tim
 7ff4d90e0000-7ff4d90e1000 r--s 00000000 08:01 492800                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.19/0/index/_9sy_Lucene410_0.dvd
 7ff4d90e8000-7ff4d94ae000 r--s 00000000 08:01 492795                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.19/0/index/_9sy_Lucene41_0.tim
 7ff4d94b0000-7ff4d94b1000 r--s 00000000 08:01 492809                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.19/0/index/_cf6_Lucene410_0.dvd
 7ff4d94b8000-7ff4d9c08000 r--s 00000000 08:01 492841                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.19/0/index/_cf6_Lucene41_0.tim
 7ff4d9c08000-7ff4d9c09000 r--s 00000000 08:01 492778                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.19/0/index/_825_Lucene410_0.dvd
 7ff4d9c10000-7ff4da13f000 r--s 00000000 08:01 492827                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.19/0/index/_825_Lucene41_0.tim
 7ff4da140000-7ff4da141000 r--s 00000000 08:01 492759                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.19/0/index/_4o5_Lucene410_0.dvd
 7ff4da148000-7ff4daaa8000 r--s 00000000 08:01 492822                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.19/0/index/_4o5_Lucene41_0.tim
 7ff4daaa8000-7ff4daaa9000 r--s 00000000 08:01 492706                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.18/0/index/_bht_Lucene410_0.dvd
 7ff4daab0000-7ff4dad85000 r--s 00000000 08:01 492700                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.18/0/index/_bht_Lucene41_0.tim
 7ff4dad88000-7ff4db746000 r--s 00000000 08:01 492821                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.18/0/index/_dqy_Lucene41_0.tim
 7ff4db748000-7ff4dbf20000 r--s 00000000 08:01 492685                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.18/0/index/_7ud_Lucene41_0.tim
 7ff4dbf20000-7ff4dc9df000 r--s 00000000 08:01 492731                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.18/0/index/_4jg_Lucene41_0.tim
 7ff4dc9e4000-7ff4dc9e7000 ---p 00000000 00:00 0 
 7ff4dc9e7000-7ff4dcae5000 rw-p 00000000 00:00 0 
 7ff4dcae5000-7ff4dcae8000 ---p 00000000 00:00 0 
 7ff4dcae8000-7ff4dcbe6000 rw-p 00000000 00:00 0                          [stack:2086]
 7ff4dcbe6000-7ff4dcbe9000 ---p 00000000 00:00 0 
 7ff4dcbe9000-7ff4dcce7000 rw-p 00000000 00:00 0                          [stack:29076]
 7ff4dcce7000-7ff4dccea000 ---p 00000000 00:00 0 
 7ff4dccea000-7ff4dcde8000 rw-p 00000000 00:00 0                          [stack:2084]
 7ff4dcde8000-7ff4dce62000 r--s 00000000 08:01 493270                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/1/index/_2tgr_Lucene410_0.dvd
 7ff4dce68000-7ff4dd8b6000 r--s 00000000 08:01 493061                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/1/index/_2tgr_Lucene41_0.tim
 7ff4dd8b8000-7ff4ddfd1000 r--s 00000000 08:01 493368                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/1/index/_1sj0_Lucene41_0.tim
 7ff4ddfd8000-7ff4ded9a000 r--s 00000000 08:01 493259                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/1/index/_hy9_Lucene41_0.tim
 7ff4ded9f000-7ff4deda2000 ---p 00000000 00:00 0 
 7ff4deda2000-7ff4deea0000 rw-p 00000000 00:00 0                          [stack:2083]
 7ff4deea0000-7ff4df1f7000 r--s 00000000 08:01 492622                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/0/index/_2avh_Lucene41_0.tim
 7ff4df1f8000-7ff4dfffa000 r--s 00000000 08:01 492567                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/0/index/_ipe_Lucene41_0.tim
 7ff4e0000000-7ff4e0143000 rw-p 00000000 00:00 0 
 7ff4e0143000-7ff4e4000000 ---p 00000000 00:00 0 
 7ff4e4000000-7ff4e5530000 rw-p 00000000 00:00 0 
 7ff4e5530000-7ff4e8000000 ---p 00000000 00:00 0 
 7ff4e8000000-7ff4e8030000 rw-p 00000000 00:00 0 
 7ff4e8030000-7ff4ec000000 ---p 00000000 00:00 0 
 7ff4ec000000-7ff4ec136000 rw-p 00000000 00:00 0 
 7ff4ec136000-7ff4f0000000 ---p 00000000 00:00 0 
 7ff4f0000000-7ff4f0272000 rw-p 00000000 00:00 0 
 7ff4f0272000-7ff4f4000000 ---p 00000000 00:00 0 
 7ff4f4000000-7ff4f4036000 rw-p 00000000 00:00 0 
 7ff4f4036000-7ff4f8000000 ---p 00000000 00:00 0 
 7ff4f8000000-7ff4f8098000 rw-p 00000000 00:00 0 
 7ff4f8098000-7ff4fc000000 ---p 00000000 00:00 0 
 7ff4fc000000-7ff4fc001000 r--s 00000000 08:01 492732                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.18/0/index/_dqy_Lucene410_0.dvd
 7ff4fc008000-7ff4fc055000 r--s 00000000 08:01 492918                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/1/index/_1sj0_Lucene410_0.dvd
 7ff4fc058000-7ff4fc0fe000 r--s 00000000 08:01 493291                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/1/index/_hy9_Lucene410_0.dvd
 7ff4fc100000-7ff4fc123000 r--s 00000000 08:01 492610                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/0/index/_2avh_Lucene410_0.dvd
 7ff4fc128000-7ff4fc1a0000 r--s 00000000 08:01 492565                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/0/index/_1wf6_Lucene410_0.dvd
 7ff4fc1a0000-7ff4fcbff000 r--s 00000000 08:01 492559                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/0/index/_1wf6_Lucene41_0.tim
 7ff4fcc00000-7ff4fcca8000 r--s 00000000 08:01 492589                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/graylog_0/0/index/_ipe_Lucene410_0.dvd
 7ff4fccae000-7ff4fccb1000 ---p 00000000 00:00 0 
 7ff4fccb1000-7ff4fcdaf000 rw-p 00000000 00:00 0                          [stack:2248]
 7ff4fcdaf000-7ff4fcdb2000 ---p 00000000 00:00 0 
 7ff4fcdb2000-7ff4fceb0000 rw-p 00000000 00:00 0                          [stack:2106]
 7ff4fceb0000-7ff4fceb3000 ---p 00000000 00:00 0 
 7ff4fceb3000-7ff4fcfb1000 rw-p 00000000 00:00 0                          [stack:6942]
 7ff4fcfb1000-7ff4fcfb4000 ---p 00000000 00:00 0 
 7ff4fcfb4000-7ff4fd0b2000 rw-p 00000000 00:00 0                          [stack:2105]
 7ff4fd0b2000-7ff4fd0b5000 ---p 00000000 00:00 0 
 7ff4fd0b5000-7ff4fd1b3000 rw-p 00000000 00:00 0                          [stack:2078]
 7ff4fd1b3000-7ff4fd1b6000 ---p 00000000 00:00 0 
 7ff4fd1b6000-7ff4fd2b4000 rw-p 00000000 00:00 0                          [stack:2077]
 7ff4fd2b4000-7ff4fd2b7000 ---p 00000000 00:00 0 
 7ff4fd2b7000-7ff4fd3b5000 rw-p 00000000 00:00 0                          [stack:2076]
 7ff4fd3b5000-7ff4fd3b8000 ---p 00000000 00:00 0 
 7ff4fd3b8000-7ff4fd4b6000 rw-p 00000000 00:00 0                          [stack:2075]
 7ff4fd4b6000-7ff4fd4b9000 ---p 00000000 00:00 0 
 7ff4fd4b9000-7ff4fd5b7000 rw-p 00000000 00:00 0                          [stack:2074]
 7ff4fd5b7000-7ff4fd5ba000 ---p 00000000 00:00 0 
 7ff4fd5ba000-7ff4fd6b8000 rw-p 00000000 00:00 0                          [stack:2169]
 7ff4fd6b8000-7ff4fd6bb000 ---p 00000000 00:00 0 
 7ff4fd6bb000-7ff4fd7b9000 rw-p 00000000 00:00 0                          [stack:2072]
 7ff4fd7b9000-7ff4fd7bc000 ---p 00000000 00:00 0 
 7ff4fd7bc000-7ff4fd8ba000 rw-p 00000000 00:00 0                          [stack:2101]
 7ff4fd8ba000-7ff4fd8bd000 ---p 00000000 00:00 0 
 7ff4fd8bd000-7ff4fd9bb000 rw-p 00000000 00:00 0                          [stack:29981]
 7ff4fd9bb000-7ff4fd9be000 ---p 00000000 00:00 0 
 7ff4fd9be000-7ff4fdabc000 rw-p 00000000 00:00 0                          [stack:2069]
 7ff4fdabc000-7ff4fdabf000 ---p 00000000 00:00 0 
 7ff4fdabf000-7ff4fdbbd000 rw-p 00000000 00:00 0                          [stack:2068]
 7ff4fdbbd000-7ff4fdbc0000 ---p 00000000 00:00 0 
 7ff4fdbc0000-7ff4fdcbe000 rw-p 00000000 00:00 0                          [stack:2067]
 7ff4fdcbe000-7ff4fdcc1000 ---p 00000000 00:00 0 
 7ff4fdcc1000-7ff4fddbf000 rw-p 00000000 00:00 0                          [stack:2066]
 7ff4fddbf000-7ff4fddc2000 ---p 00000000 00:00 0 
 7ff4fddc2000-7ff4fdec0000 rw-p 00000000 00:00 0                          [stack:2065]
 7ff4fdec0000-7ff4fdec3000 ---p 00000000 00:00 0 
 7ff4fdec3000-7ff4fdfc1000 rw-p 00000000 00:00 0                          [stack:2064]
 7ff4fdfc1000-7ff4fdfc4000 ---p 00000000 00:00 0 
 7ff4fdfc4000-7ff4fe0c2000 rw-p 00000000 00:00 0                          [stack:2063]
 7ff4fe0c2000-7ff4fe0c5000 ---p 00000000 00:00 0 
 7ff4fe0c5000-7ff4fe1c3000 rw-p 00000000 00:00 0                          [stack:2062]
 7ff4fe1c3000-7ff4fe1c6000 ---p 00000000 00:00 0 
 7ff4fe1c6000-7ff4fe2c4000 rw-p 00000000 00:00 0                          [stack:2061]
 7ff4fe2c4000-7ff4fe2c7000 ---p 00000000 00:00 0 
 7ff4fe2c7000-7ff4fe3c5000 rw-p 00000000 00:00 0                          [stack:2060]
 7ff4fe3c5000-7ff4fe3c8000 ---p 00000000 00:00 0 
 7ff4fe3c8000-7ff4fe4c6000 rw-p 00000000 00:00 0                          [stack:2059]
 7ff4fe4c6000-7ff4fe4c9000 ---p 00000000 00:00 0 
 7ff4fe4c9000-7ff4fe5c7000 rw-p 00000000 00:00 0                          [stack:2058]
 7ff4fe5c7000-7ff4fe5ca000 ---p 00000000 00:00 0 
 7ff4fe5ca000-7ff4fe6c8000 rw-p 00000000 00:00 0                          [stack:2057]
 7ff4fe6c8000-7ff4fe6d8000 r-xp 00000000 08:01 257747                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/l
ibnio.so
 7ff4fe6d8000-7ff4fe8d8000 ---p 00010000 08:01 257747                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/l
ibnio.so
 7ff4fe8d8000-7ff4fe8d9000 rw-p 00010000 08:01 257747                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/l
ibnio.so
 7ff4fe8e0000-7ff4febe0000 rw-p 00000000 00:00 0 
 7ff4febe0000-7ff4febf7000 r-xp 00000000 08:01 257781                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/l
ibnet.so
 7ff4febf7000-7ff4fedf6000 ---p 00017000 08:01 257781                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/l
ibnet.so
 7ff4fedf6000-7ff4fedf7000 rw-p 00016000 08:01 257781                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/l
ibnet.so
 7ff4fedf8000-7ff4fee00000 r-xp 00000000 08:01 257755                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/l
ibmanagement.so
 7ff4fee00000-7ff4fefff000 ---p 00008000 08:01 257755                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/l
ibmanagement.so
 7ff4fefff000-7ff4ff000000 rw-p 00007000 08:01 257755                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/l
ibmanagement.so
 7ff4ff000000-7ff501000000 rw-p 00000000 00:00 0 
 7ff501000000-7ff501900000 rwxp 00000000 00:00 0 
 7ff501900000-7ff504cd6000 rw-p 00000000 00:00 0 
 7ff504cd6000-7ff508000000 ---p 00000000 00:00 0 
 7ff508000000-7ff508001000 r--s 00000000 08:01 492684                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.18/0/index/_7ud_Lucene410_0.dvd
 7ff508008000-7ff508009000 r--s 00000000 08:01 492715                     /var/opt/elasticsearch/data/graylog2/nodes
/0/indices/.marvel-2015.09.18/0/index/_4jg_Lucene410_0.dvd
 7ff50800f000-7ff508013000 r--s 000f7000 08:01 258384                     /usr/lib/jvm/java-7-oracle/jre/lib/ext/loc
aledata.jar
 7ff508018000-7ff50801a000 r--s 00010000 08:01 255931                     /usr/share/elasticsearch/plugins/marvel/ma
rvel-1.3.1.jar
 7ff50801b000-7ff50801f000 r--s 0008b000 08:01 258373                     /usr/lib/jvm/java-7-oracle/jre/lib/jsse.ja
r
 7ff50801f000-7ff508025000 r--s 0002f000 08:01 255579                     /usr/share/elasticsearch/lib/lucene-querie
s-4.10.4.jar
 7ff50802a000-7ff50803b000 r--s 000b2000 08:01 255569                     /usr/share/elasticsearch/lib/jts-1.13.jar
 7ff508040000-7ff508041000 r--s 00008000 08:01 255577                     /usr/share/elasticsearch/lib/lucene-memory
-4.10.4.jar
 7ff508047000-7ff508050000 r--s 00057000 08:01 255580                     /usr/share/elasticsearch/lib/lucene-queryp
arser-4.10.4.jar
 7ff508056000-7ff508059000 r--s 00026000 08:01 255563                     /usr/share/elasticsearch/lib/antlr-runtime
-3.5.jar
 7ff508060000-7ff508062000 r--s 00008000 08:01 255566                     /usr/share/elasticsearch/lib/asm-commons-4
.1.jar
 7ff508063000-7ff508066000 r--s 0001b000 08:01 255581                     /usr/share/elasticsearch/lib/lucene-sandbo
x-4.10.4.jar
 7ff50806c000-7ff508070000 r--s 000dc000 08:01 255568                     /usr/share/elasticsearch/lib/jna-4.1.0.jar
 7ff508077000-7ff50807a000 r--s 0001f000 08:01 255575                     /usr/share/elasticsearch/lib/lucene-highli
ghter-4.10.4.jar
 7ff50807c000-7ff50807f000 r--s 0001c000 08:01 255582                     /usr/share/elasticsearch/lib/lucene-spatia
l-4.10.4.jar
 7ff50807f000-7ff508081000 r--s 00017000 08:01 255584                     /usr/share/elasticsearch/lib/spatial4j-0.4
.1.jar
 7ff508081000-7ff508083000 r--s 00011000 08:01 255573                     /usr/share/elasticsearch/lib/lucene-expres
sions-4.10.4.jar
 7ff508088000-7ff5080b2000 r--s 00248000 08:01 255572                     /usr/share/elasticsearch/lib/lucene-core-4
.10.4.jar
 7ff5080b3000-7ff5080b4000 r--s 0000b000 08:01 255565                     /usr/share/elasticsearch/lib/asm-4.1.jar
 7ff5080b6000-7ff5080be000 r--s 00066000 08:01 255564                     /usr/share/elasticsearch/lib/apache-log4j-
extras-1.2.17.jar
 7ff5080be000-7ff5080c0000 r--s 00016000 08:01 255578                     /usr/share/elasticsearch/lib/lucene-misc-4
.10.4.jar
 7ff5080c1000-7ff5080c4000 r--s 00029000 08:01 255583                     /usr/share/elasticsearch/lib/lucene-sugges
t-4.10.4.jar
 7ff5080c6000-7ff508135000 r--s 0063e000 08:01 255567                     /usr/share/elasticsearch/lib/groovy-all-2.
4.4.jar
 7ff508136000-7ff508138000 r--s 0000e000 08:01 255576                     /usr/share/elasticsearch/lib/lucene-join-4
.10.4.jar
 7ff508138000-7ff50813b000 r--s 00018000 08:01 255574                     /usr/share/elasticsearch/lib/lucene-groupi
ng-4.10.4.jar
 7ff50813f000-7ff508148000 r--s 0006f000 08:01 255570                     /usr/share/elasticsearch/lib/log4j-1.2.17.
jar
 7ff50814f000-7ff508160000 r--s 0018f000 08:01 255571                     /usr/share/elasticsearch/lib/lucene-analyz
ers-common-4.10.4.jar
 7ff508164000-7ff50818c000 rw-p 00000000 00:00 0 
 7ff50818c000-7ff50829e000 r--s 00c54000 08:01 255557                     /usr/share/elasticsearch/lib/elasticsearch
-1.7.1.jar
 7ff5082a2000-7ff5082a3000 ---p 00000000 00:00 0 
 7ff5082a3000-7ff5083a3000 rw-p 00000000 00:00 0                          [stack:2056]
 7ff5083a3000-7ff5083a6000 ---p 00000000 00:00 0 
 7ff5083a6000-7ff5084a4000 rw-p 00000000 00:00 0                          [stack:2055]
 7ff5084a4000-7ff5084a7000 ---p 00000000 00:00 0 
 7ff5084a7000-7ff5085a5000 rw-p 00000000 00:00 0                          [stack:2054]
 7ff5085a5000-7ff5085a8000 ---p 00000000 00:00 0 
 7ff5085a8000-7ff5086a6000 rw-p 00000000 00:00 0                          [stack:2053]
 7ff5086a6000-7ff5086a9000 ---p 00000000 00:00 0 
 7ff5086a9000-7ff5087a7000 rw-p 00000000 00:00 0                          [stack:2052]
 7ff5087a7000-7ff5087aa000 ---p 00000000 00:00 0 
 7ff5087aa000-7ff5088a8000 rw-p 00000000 00:00 0                          [stack:2051]
 7ff5088a8000-7ff508a31000 r--p 00000000 08:01 54985                      /usr/lib/locale/locale-archive
 7ff508a35000-7ff508a38000 ---p 00000000 00:00 0 
 7ff508a38000-7ff508b36000 rw-p 00000000 00:00 0                          [stack:2050]
 7ff508b36000-7ff508b39000 ---p 00000000 00:00 0 
 7ff508b39000-7ff508c37000 rw-p 00000000 00:00 0                          [stack:2049]
 7ff508c37000-7ff508c38000 ---p 00000000 00:00 0 
 7ff508c38000-7ff508d83000 rw-p 00000000 00:00 0                          [stack:2048]
 7ff508d83000-7ff508f43000 r--s 039fb000 08:01 258429                     /usr/lib/jvm/java-7-oracle/jre/lib/rt.jar
 7ff508f49000-7ff509042000 rw-p 00000000 00:00 0 
 7ff509042000-7ff509043000 ---p 00000000 00:00 0 
 7ff509043000-7ff50a9d6000 rw-p 00000000 00:00 0                          [stack:2047]
 7ff50a9d6000-7ff50ab56000 rw-p 00000000 00:00 0 
 7ff50ab56000-7ff50abd6000 rw-p 00000000 00:00 0 
 7ff50abd6000-7ff50ad56000 rw-p 00000000 00:00 0 
 7ff50ad56000-7ff50ad78000 rw-p 00000000 00:00 0 
 7ff50ad78000-7ff50ad7f000 rw-p 00000000 00:00 0 
 7ff50ad7f000-7ff50ad80000 rw-p 00000000 00:00 0 
 7ff50ad80000-7ff50ad97000 r-xp 00000000 08:01 257776                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/l
ibzip.so
 7ff50ad97000-7ff50af97000 ---p 00017000 08:01 257776                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/l
ibzip.so
 7ff50af97000-7ff50af98000 rw-p 00017000 08:01 257776                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/l
ibzip.so
 7ff50af98000-7ff50afa4000 r-xp 00000000 08:01 2212                       /lib/x86_64-linux-gnu/libnss_files-2.21.so
 7ff50afa4000-7ff50b1a3000 ---p 0000c000 08:01 2212                       /lib/x86_64-linux-gnu/libnss_files-2.21.so
 7ff50b1a3000-7ff50b1a4000 r--p 0000b000 08:01 2212                       /lib/x86_64-linux-gnu/libnss_files-2.21.so
 7ff50b1a4000-7ff50b1a5000 rw-p 0000c000 08:01 2212                       /lib/x86_64-linux-gnu/libnss_files-2.21.so
 7ff50b1a8000-7ff50b1b3000 r-xp 00000000 08:01 2200                       /lib/x86_64-linux-gnu/libnss_nis-2.21.so
 7ff50b1b3000-7ff50b3b2000 ---p 0000b000 08:01 2200                       /lib/x86_64-linux-gnu/libnss_nis-2.21.so
 7ff50b3b2000-7ff50b3b3000 r--p 0000a000 08:01 2200                       /lib/x86_64-linux-gnu/libnss_nis-2.21.so
 7ff50b3b3000-7ff50b3b4000 rw-p 0000b000 08:01 2200                       /lib/x86_64-linux-gnu/libnss_nis-2.21.so
 7ff50b3b8000-7ff50b3cf000 r-xp 00000000 08:01 2218                       /lib/x86_64-linux-gnu/libnsl-2.21.so
 7ff50b3cf000-7ff50b5ce000 ---p 00017000 08:01 2218                       /lib/x86_64-linux-gnu/libnsl-2.21.so
 7ff50b5ce000-7ff50b5cf000 r--p 00016000 08:01 2218                       /lib/x86_64-linux-gnu/libnsl-2.21.so
 7ff50b5cf000-7ff50b5d0000 rw-p 00017000 08:01 2218                       /lib/x86_64-linux-gnu/libnsl-2.21.so
 7ff50b5d0000-7ff50b5d2000 rw-p 00000000 00:00 0 
 7ff50b5d8000-7ff50b5e0000 r-xp 00000000 08:01 2214                       /lib/x86_64-linux-gnu/libnss_compat-2.21.s
o
 7ff50b5e0000-7ff50b7df000 ---p 00008000 08:01 2214                       /lib/x86_64-linux-gnu/libnss_compat-2.21.s
o
 7ff50b7df000-7ff50b7e0000 r--p 00007000 08:01 2214                       /lib/x86_64-linux-gnu/libnss_compat-2.21.s
o
 7ff50b7e0000-7ff50b7e1000 rw-p 00008000 08:01 2214                       /lib/x86_64-linux-gnu/libnss_compat-2.21.s
o
 7ff50b7e8000-7ff50b811000 r-xp 00000000 08:01 257742                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/l
ibjava.so
 7ff50b811000-7ff50ba11000 ---p 00029000 08:01 257742                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/l
ibjava.so
 7ff50ba11000-7ff50ba13000 rw-p 00029000 08:01 257742                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/l
ibjava.so
 7ff50ba18000-7ff50ba25000 r-xp 00000000 08:01 257769                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/l
ibverify.so
 7ff50ba25000-7ff50bc24000 ---p 0000d000 08:01 257769                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/l
ibverify.so
 7ff50bc24000-7ff50bc26000 rw-p 0000c000 08:01 257769                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/l
ibverify.so
 7ff50bc28000-7ff50bc2f000 r-xp 00000000 08:01 2219                       /lib/x86_64-linux-gnu/librt-2.21.so
 7ff50bc2f000-7ff50be2e000 ---p 00007000 08:01 2219                       /lib/x86_64-linux-gnu/librt-2.21.so
 7ff50be2e000-7ff50be2f000 r--p 00006000 08:01 2219                       /lib/x86_64-linux-gnu/librt-2.21.so
 7ff50be2f000-7ff50be30000 rw-p 00007000 08:01 2219                       /lib/x86_64-linux-gnu/librt-2.21.so
 7ff50be30000-7ff50bf37000 r-xp 00000000 08:01 2217                       /lib/x86_64-linux-gnu/libm-2.21.so
 7ff50bf37000-7ff50c136000 ---p 00107000 08:01 2217                       /lib/x86_64-linux-gnu/libm-2.21.so
 7ff50c136000-7ff50c137000 r--p 00106000 08:01 2217                       /lib/x86_64-linux-gnu/libm-2.21.so
 7ff50c137000-7ff50c138000 rw-p 00107000 08:01 2217                       /lib/x86_64-linux-gnu/libm-2.21.so
 7ff50c138000-7ff50ccae000 r-xp 00000000 08:01 257796                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/s
erver/libjvm.so
 7ff50ccae000-7ff50cead000 ---p 00b76000 08:01 257796                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/s
erver/libjvm.so
 7ff50cead000-7ff50cf71000 rw-p 00b75000 08:01 257796                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/s
erver/libjvm.so
 7ff50cf71000-7ff50cfb2000 rw-p 00000000 00:00 0 
 7ff50cfb8000-7ff50d178000 r-xp 00000000 08:01 2222                       /lib/x86_64-linux-gnu/libc-2.21.so
 7ff50d178000-7ff50d378000 ---p 001c0000 08:01 2222                       /lib/x86_64-linux-gnu/libc-2.21.so
 7ff50d378000-7ff50d37c000 r--p 001c0000 08:01 2222                       /lib/x86_64-linux-gnu/libc-2.21.so
 7ff50d37c000-7ff50d37e000 rw-p 001c4000 08:01 2222                       /lib/x86_64-linux-gnu/libc-2.21.so
 7ff50d37e000-7ff50d382000 rw-p 00000000 00:00 0 
 7ff50d388000-7ff50d38b000 r-xp 00000000 08:01 2205                       /lib/x86_64-linux-gnu/libdl-2.21.so
 7ff50d38b000-7ff50d58a000 ---p 00003000 08:01 2205                       /lib/x86_64-linux-gnu/libdl-2.21.so
 7ff50d58a000-7ff50d58b000 r--p 00002000 08:01 2205                       /lib/x86_64-linux-gnu/libdl-2.21.so
 7ff50d58b000-7ff50d58c000 rw-p 00003000 08:01 2205                       /lib/x86_64-linux-gnu/libdl-2.21.so
 7ff50d590000-7ff50d5a5000 r-xp 00000000 08:01 257788                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/j
li/libjli.so
 7ff50d5a5000-7ff50d7a4000 ---p 00015000 08:01 257788                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/j
li/libjli.so
 7ff50d7a4000-7ff50d7a5000 rw-p 00014000 08:01 257788                     /usr/lib/jvm/java-7-oracle/jre/lib/amd64/j
li/libjli.so
 7ff50d7a8000-7ff50d7c0000 r-xp 00000000 08:01 2209                       /lib/x86_64-linux-gnu/libpthread-2.21.so
 7ff50d7c0000-7ff50d9c0000 ---p 00018000 08:01 2209                       /lib/x86_64-linux-gnu/libpthread-2.21.so
 7ff50d9c0000-7ff50d9c1000 r--p 00018000 08:01 2209                       /lib/x86_64-linux-gnu/libpthread-2.21.so
 7ff50d9c1000-7ff50d9c2000 rw-p 00019000 08:01 2209                       /lib/x86_64-linux-gnu/libpthread-2.21.so
 7ff50d9c2000-7ff50d9c6000 rw-p 00000000 00:00 0 
 7ff50d9c8000-7ff50d9ec000 r-xp 00000000 08:01 2208                       /lib/x86_64-linux-gnu/ld-2.21.so
 7ff50d9ee000-7ff50da10000 rw-p 00000000 00:00 0 
 7ff50da10000-7ff50da18000 rw-p 00000000 00:00 0 
 7ff50da18000-7ff50da3c000 rw-p 00000000 00:00 0 
 7ff50da3c000-7ff50dad8000 rw-p 00000000 00:00 0 
 7ff50dad8000-7ff50dae0000 rw-s 00000000 08:01 257125                     /tmp/hsperfdata_elasticsearch/2038 (delete
d)
 7ff50dae3000-7ff50dae5000 rw-p 00000000 00:00 0 
 7ff50dae5000-7ff50dae6000 r--p 00000000 00:00 0 
 7ff50dae6000-7ff50dae9000 ---p 00000000 00:00 0 
 7ff50dae9000-7ff50dbeb000 rw-p 00000000 00:00 0                          [stack:2046]
 7ff50dbeb000-7ff50dbec000 r--p 00023000 08:01 2208                       /lib/x86_64-linux-gnu/ld-2.21.so
 7ff50dbec000-7ff50dbed000 rw-p 00024000 08:01 2208                       /lib/x86_64-linux-gnu/ld-2.21.so
 7ff50dbed000-7ff50dbf0000 rw-p 00000000 00:00 0 
 7fffd07ba000-7fffd07db000 rw-p 00000000 00:00 0                          [stack]
 7fffd07e0000-7fffd07e2000 r--p 00000000 00:00 0                          [vvar]
 7fffd07e2000-7fffd07e4000 r-xp 00000000 00:00 0                          [vdso]
 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
ProcStatus:
 Name:  java
 State: D (disk sleep)
 Tgid:  2038
 Ngid:  0
 Pid:   2038
 PPid:  1
 TracerPid: 0
 Uid:   112 112 112 112
 Gid:   120 120 120 120
 FDSize:    512
 Groups:    120 
 VmPeak:     2164628 kB
 VmSize:     2160164 kB
 VmLck:        0 kB
 VmPin:        0 kB
 VmHWM:   383944 kB
 VmRSS:   312008 kB
 VmData:     1840504 kB
 VmStk:      136 kB
 VmExe:        4 kB
 VmLib:    15656 kB
 VmPTE:     1308 kB
 VmSwap:       64412 kB
 Threads:   47
 SigQ:  0/1924
 SigPnd:    0000000000000000
 ShdPnd:    0000000000000000
 SigBlk:    0000000000000000
 SigIgn:    0000000000000000
 SigCgt:    2000000181005ccf
 CapInh:    0000000000000000
 CapPrm:    0000000000000000
 CapEff:    0000000000000000
 CapBnd:    0000003fffffffff
 Seccomp:   0
 Cpus_allowed:  1
 Cpus_allowed_list: 0
 Mems_allowed:  00000000,00000001
 Mems_allowed_list: 0
 voluntary_ctxt_switches:   29
 nonvoluntary_ctxt_switches:    3
Signal: 6
Uname: Linux 3.19.0-28-generic x86_64
UserGroups: 

I have been searching for further information and, unfortunately, can't find anything as to why the java process crashed.
Any idea?

Thanks a lot,
Bertrand.

[clintongormley]

I'd suggest upgrading your JVM. Also, giving Elasticsearch 512MB is pretty damn limited, especially with 300 indices. That is probably unrelated to this JVM crash though.

[dbblackdiamond]

I have upgraded the RAM on the VMs to 1GB, although I don't see any out of memory error messages.

[clintongormley]

@dbblackdiamond Have you upgraded the JVM, which was my primary advice?

[dbblackdiamond]

@clintongormley: yes I have upgraded my virtual machines from 512MB of RAM to 1GB of RAM. I haven't touched the Java settings though.

[dobariya]

@clintongormley: I used debug diagnostic tool by Microsoft which gave me exception report occurred in java.exe.
I tried uploading the file but it does not allow saying something really went wrong.Please let me know how can i transfer the report which might help you to trance the issue

[dbblackdiamond]

@clintongormley: I have increased the memory even more on one of my servers. I increased it to 4GB of RAM for the server and changed the JVM Heap Settings to use 2GB of RAM. So far, things have been stable for all servers. I'll keep monitoring it.