java.lang.ArrayIndexOutOfBoundsException: -65531 indexing doc with geometry

[sbocconi]

Elasticsearch version:
"version": {
"number": "5.2.0",
"build_hash": "24e05b9",
"build_date": "2017-01-24T19:52:35.800Z",
"build_snapshot": false,
"lucene_version": "6.4.0"
}
also
"number" : "5.5.0",
"build_hash" : "260387d",
"build_date" : "2017-06-30T23:16:05.735Z",
"build_snapshot" : false,
"lucene_version" : "6.6.0"

Plugins installed: []

JVM version:
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

OS version:
Ubuntu 16.04.2 LTS

Description of the problem including expected versus actual behavior:
While trying to index a ndjson with MultiPolygon geometries, a failed engine [lucene commit failed] java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot commit
error is thrown with Caused by: java.lang.ArrayIndexOutOfBoundsException: -65531

Steps to reproduce:
With a script:
Script and data to reproduce the problem:

data is here
Script is here

Run the script in same dir as data, wait ~4.5 hours on a single node ES 32GB quad core.

Manually:

1. create an index with the following mapping:

{
  "cshapes": {
    "mappings": {
      "hg:Country": {
        "_all": {
          "enabled": false
        },
        "properties": {
          "dataset": {
            "type": "keyword"
          },
          "geometry": {
            "type": "geo_shape",
            "tree": "quadtree",
            "precision": "10.0m"
          },
          "id": {
            "type": "keyword"
          },
          "name": {
            "type": "text",
            "index": false,
            "fields": {
              "analyzed": {
                "type": "text"
              },
              "exact": {
                "type": "text",
                "analyzer": "lowercase"
              }
            }
          },
          "type": {
            "type": "keyword"
          },
          "uri": {
            "type": "keyword"
          },
          "validSince": {
            "type": "date",
            "format": "date_optional_time"
          },
          "validSinceTimestamp": {
            "type": "long"
          },
          "validUntil": {
            "type": "date",
            "format": "date_optional_time"
          },
          "validUntilTimestamp": {
            "type": "long"
          }
        }
      },
      "_default_": {
        "_all": {
          "enabled": false
        },
        "properties": {
          "dataset": {
            "type": "keyword"
          },
          "geometry": {
            "type": "geo_shape",
            "tree": "quadtree",
            "precision": "10.0m"
          },
          "id": {
            "type": "keyword"
          },
          "name": {
            "type": "text",
            "index": false,
            "fields": {
              "analyzed": {
                "type": "text"
              },
              "exact": {
                "type": "text",
                "analyzer": "lowercase"
              }
            }
          },
          "type": {
            "type": "keyword"
          },
          "uri": {
            "type": "keyword"
          },
          "validSince": {
            "type": "date",
            "format": "date_optional_time"
          },
          "validUntil": {
            "type": "date",
            "format": "date_optional_time"
          }
        }
      }
    }
  }
}

2. Bulk index the file at https://github.com/histograph/data/blob/master/cshapes/cshapes.pits.ndjson

Provide logs (if relevant):

[2017-03-21T02:05:47,759][WARN ][o.e.i.e.Engine           ] [ip-10-1-0-29-histograph] [cshapes][0] failed engine [lucene commit failed]
java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot commit
	at org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4635) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3073) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3232) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3195) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.elasticsearch.index.engine.InternalEngine.commitIndexWriter(InternalEngine.java:1296) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.engine.InternalEngine.flush(InternalEngine.java:803) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.shard.IndexShard.flush(IndexShard.java:777) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.indices.flush.SyncedFlushService.performPreSyncedFlush(SyncedFlushService.java:414) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.indices.flush.SyncedFlushService.access$1000(SyncedFlushService.java:70) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.indices.flush.SyncedFlushService$PreSyncedFlushTransportHandler.messageReceived(SyncedFlushService.java:696) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.indices.flush.SyncedFlushService$PreSyncedFlushTransportHandler.messageReceived(SyncedFlushService.java:692) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:610) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.2.0.jar:5.2.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: java.lang.ArrayIndexOutOfBoundsException: -65531
	at org.apache.lucene.util.ByteBlockPool.setBytesRef(ByteBlockPool.java:308) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.util.BytesRefHash.equals(BytesRefHash.java:183) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.util.BytesRefHash.findHash(BytesRefHash.java:337) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:255) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:149) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:766) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:417) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:373) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:231) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1575) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1320) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:558) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.engine.InternalEngine.innerIndex(InternalEngine.java:520) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:409) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:556) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:546) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:213) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:348) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.index(TransportShardBulkAction.java:155) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.handleItem(TransportShardBulkAction.java:134) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:120) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:73) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:76) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:49) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:914) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:884) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:327) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:262) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:864) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:861) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:142) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1652) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:873) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.access$400(TransportReplicationAction.java:92) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:279) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:258) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:250) ~[elasticsearch-5.2.0.jar:5.2.0]
	... 7 more
[2017-03-21T02:05:47,771][WARN ][o.e.i.c.IndicesClusterStateService] [ip-10-1-0-29-histograph] [[cshapes][0]] marking and sending shard failed due to [shard failure, reason [lucene commit failed]]
java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot commit
	at org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4635) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3073) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3232) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3195) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.elasticsearch.index.engine.InternalEngine.commitIndexWriter(InternalEngine.java:1296) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.engine.InternalEngine.flush(InternalEngine.java:803) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.shard.IndexShard.flush(IndexShard.java:777) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.indices.flush.SyncedFlushService.performPreSyncedFlush(SyncedFlushService.java:414) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.indices.flush.SyncedFlushService.access$1000(SyncedFlushService.java:70) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.indices.flush.SyncedFlushService$PreSyncedFlushTransportHandler.messageReceived(SyncedFlushService.java:696) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.indices.flush.SyncedFlushService$PreSyncedFlushTransportHandler.messageReceived(SyncedFlushService.java:692) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:610) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.2.0.jar:5.2.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: java.lang.ArrayIndexOutOfBoundsException: -65531
	at org.apache.lucene.util.ByteBlockPool.setBytesRef(ByteBlockPool.java:308) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.util.BytesRefHash.equals(BytesRefHash.java:183) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.util.BytesRefHash.findHash(BytesRefHash.java:337) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:255) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:149) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:766) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:417) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:373) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:231) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1575) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1320) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:558) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.engine.InternalEngine.innerIndex(InternalEngine.java:520) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:409) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:556) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:546) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:213) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:348) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.index(TransportShardBulkAction.java:155) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.handleItem(TransportShardBulkAction.java:134) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:120) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:73) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:76) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:49) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:914) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:884) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:327) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:262) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:864) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:861) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:142) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1652) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:873) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.access$400(TransportReplicationAction.java:92) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:279) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:258) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:250) ~[elasticsearch-5.2.0.jar:5.2.0]
	... 7 more
[2017-03-21T02:05:47,762][DEBUG][o.e.a.b.TransportShardBulkAction] [ip-10-1-0-29-histograph] 763],[-73.776673,64.294709],[-73.823336,64.324709],[-73.833618,64.331663],[-73.778746,64.406518],[-73.765564,64.409427],[-73.7468,64.408045],[-73.730836,64.386382]]],[[[-78.019164,62.591936],[-78.008346,62.593609],[-77.867218,62.589154],[-77.850555,62.582773],[-77.841673,62.568054],[-77.839309,62.553463],[-77.848755,62.543191],[-77.862209,62.539436],[-77.873046,62.537773],[-77.885009,62.5375],[-77.913055,62.539436],[-78.103336,62.559154],[-78.113746,62.566382],[-78.107773,62.581863],[-78.0475,62.591936],[-78.030836,62.593318],[-78.019164,62.591936]]],[[[-96.610546,71.290545],[-96.581118,71.293591],[-96.563327,71.292209],[-96.546955,71.289154],[-96.535555,71.282763],[-96.471391,71.229145],[-96.484173,71.206236],[-96.561109,71.208327],[-96.578064,71.210536],[-96.6286,71.220263],[-96.641455,71.231573],[-96.650836,71.290118],[-96.610546,71.290545]]],[[[-123.338336,49.5061],[-123.329864,49.498609],[-123.3239,49.4886],[-123.319455,49.474709],[-123.317782,49.464154],[-123.320846,49.449991],[-123.329864,49.442627],[-123.360282,49.433054],[-123.375,49.433327],[-123.445118,49.440127],[-123.459591,49.468873],[-123.453055,49.495545],[-123.445264,49.515273],[-123.436664,49.522218],[-123.380827,49.536109],[-123.365409,49.535554],[-123.355,49.531936],[-123.338336,49.5061]]],[[[-126.912509,50.613609],[-127.020973,50.639164],[-127.042773,50.6375],[-127.058327,50.632491],[-127.101936,50.627491],[-127.127355,50.627773],[-127.142991,50.635345],[-127.129436,50.654154],[-127.109164,50.665264],[-127.090146,50.669436],[-126.890291,50.667218],[-126.876509,50.664709],[-126.834236,50.628464],[-126.9025,50.613882],[-126.912509,50.613609]]],[[[-108.359446,67.899991],[-108.412218,67.885818],[-108.646955,67.869427],[-108.664991,67.870673],[-108.646664,67.887209],[-108.585009,67.915545],[-108.5661,67.922763],[-108.544727,67.928318],[-108.531109,67.928863],[-108.375546,67.9211],[-108.364436,67.915545],[-108.361109,67.905254],[-108.359446,67.899991]]],[[[-109.576946,68.232482],[-109.581946,68.225818],[-109.594164,68.214427],[-109.673891,68.173309],[-109.762509,68.143327],[-109.772509,68.139982],[-109.783891,68.1375],[-109.811664,68.136109],[-109.828055,68.136936],[-109.845682,68.141518],[-109.855418,68.150818],[-109.845555,68.158036],[-109.771946,68.188309],[-109.676936,68.224154],[-109.644173,68.232209],[-109.5889,68.245254],[-109.571673,68.244218],[-109.576946,68.232482]]],[[[-64.897509,64.4086],[-64.820009,64.379427],[-64.772146,64.344918],[-64.849727,64.307482],[-64.861391,64.307482],[-64.877209,64.313309],[-64.888064,64.321109],[-64.937773,64.361645],[-64.944991,64.370527],[-64.955564,64.383882],[-64.959173,64.401518],[-64.949164,64.412491],[-64.928891,64.418036],[-64.906246,64.414709],[-64.897509,64.4086]]],[[[-64.054836,63.736591],[-64.043336,63.734163],[-64.030291,63.729718],[-64.027364,63.695891],[-64.161118,63.674436],[-64.181109,63.675827],[-64.200291,63.685545],[-64.208346,63.697491],[-64.212091,63.7093],[-64.208891,63.721927],[-64.180282,63.742218],[-64.170682,63.746518],[-64.077436,63.757773],[-64.054836,63.736591]]],[[[-90.72,63.543054],[-90.705282,63.538054],[-90.679164,63.520827],[-90.676655,63.511109],[-90.688891,63.506382],[-90.793609,63.494154],[-90.804446,63.493327],[-90.816955,63.495827],[-90.877209,63.514718],[-90.933318,63.534163],[-90.967091,63.548045],[-90.957509,63.551382],[-90.771118,63.551936],[-90.748046,63.550273],[-90.72,63.543054]]],[[[-125.867491,49.233327],[-125.829182,49.226382],[-125.818073,49.220545],[-125.799155,49.208327],[-125.788891,49.172764],[-125.797091,49.149154],[-125.817291,49.1261],[-125.860282,49.134436],[-125.908336,49.163182],[-125.925827,49.190827],[-125.930827,49.218109],[-125.878882,49.235827],[-125.867491,49.233327]]],[[[-71.044155,71.142763],[-71.060273,71.118318],[-71.076264,71.104845],[-71.1125,71.089427],[-71.143336,71.082491],[-71.239436,71.062618],[-71.252918,71.070273],[-71.255282,71.080273],[-71.247364,71.094154],[-71.229718,71.110545],[-71.218336,71.118045],[-71.200564,71.125809],[-71.1489,71.142209],[-71.118055,71.149427],[-71.082227,71.155254],[-71.063609,71.156936],[-71.036527,71.152691],[-71.044155,71.142763]]],[[[-89.915009,68.913318],[-89.9525,68.926373],[-89.978336,68.933873],[-90,68.937573],[-90.031955,68.943036],[-90.075564,68.948027],[-90.070555,68.981936],[-89.946382,69.010273],[-89.933609,69.011654],[-89.913891,69.005827],[-89.9075,68.920118],[-89.915009,68.913318]]],[[[-74.707782,72.207218],[-74.744155,72.184418],[-74.763064,72.175263],[-74.778336,72.171373],[-74.796664,72.168045],[-74.841382,72.166654],[-74.933609,72.170818],[-74.947082,72.175118],[-74.932218,72.187763],[-74.916109,72.197209],[-74.906109,72.202209],[-74.882218,72.210818],[-74.850827,72.217754],[-74.813327,72.223036],[-74.775009,72.2272],[-74.703891,72.230545],[-74.6743,72.224636],[-74.707782,72.207218]]],[[[-98.573627,74.334991],[-98.535282,74.328873],[-98.521664,74.324709],[-98.513346,74.316236],[-98.525009,74.310527],[-98.568618,74.3047],[-98.657227,74.299427],[-98.746946,74.298036],[-98.810273,74.298327],[-98.831682,74.299145],[-98.859791,74.307682],[-98.7525,74.334154],[-98.718336,74.336654],[-98.630827,74.342482],[-98.616655,74.341936],[-98.585827,74.338591],[-98.573627,74.334991]]],[[[-128.308046,52.128873],[-128.313627,52.129718],[-128.364991,52.162491],[-128.373046,52.185264],[-128.378464,52.216936],[-128.373736,52.226236],[-128.356655,52.235827],[-128.345273,52.2386],[-128.297246,52.237209],[-128.229355,52.215373],[-128.299991,52.133609],[-128.308046,52.128873]]],[[[-80.170546,59.673882],[-80.205836,59.665263],[-80.222782,59.660273],[-80.231391,59.653736],[-80.240346,59.635754],[-80.260836,59.6236],[-80.2775,59.6186],[-80.319173,59.612209],[-80.329727,59.6125],[-80.341464,59.618182],[-80.295273,59.678327],[-80.232773,59.725263],[-80.222227,59.7236],[-80.171391,59.715273],[-80.149727,59.707773],[-80.155691,59.680409],[-80.170546,59.673882]]],[[[-62.231382,66.269436],[-62.183882,66.2372],[-62.199164,66.216936],[-62.405,66.2186],[-62.418473,66.220682],[-62.428336,66.2311],[-62.419164,66.237763],[-62.319455,66.269436],[-62.301936,66.274991],[-62.282782,66.278882],[-62.261673,66.280273],[-62.245973,66.277763],[-62.231382,66.269436]]],[[[-113.778609,77.104154],[-113.806955,77.104154],[-113.853336,77.105545],[-113.880546,77.108036],[-113.908891,77.113036],[-113.927909,77.121509],[-113.9293,77.132482],[-113.920836,77.1411],[-113.901109,77.146382],[-113.880827,77.149718],[-113.859446,77.151382],[-113.798609,77.152482],[-113.774173,77.151654],[-113.7225,77.148036],[-113.699155,77.144718],[-113.675555,77.140273],[-113.662636,77.127482],[-113.679173,77.116927],[-113.703055,77.111645],[-113.727782,77.108036],[-113.753064,77.105545],[-113.778609,77.104154]]],[[[-107.791109,66.988309],[-107.790136,66.982482],[-107.826946,66.898318],[-107.8993,66.858873],[-107.923046,66.850536],[-107.935,66.849154],[-107.945827,66.854009],[-107.834036,67.006236],[-107.816964,67.009154],[-107.806664,67.005827],[-107.795,66.997209],[-107.791109,66.988309]]],[[[-92.158336,62.390545],[-92.223618,62.355554],[-92.306109,62.351663],[-92.339718,62.354718],[-92.349727,62.356654],[-92.372218,62.389436],[-92.351109,62.412627],[-92.319455,62.415263],[-92.308336,62.414436],[-92.162218,62.402218],[-92.140418,62.397218],[-92.158336,62.390545]]],[[[-95.314709,67.238591],[-95.339718,67.209854],[-95.361664,67.197754],[-95.373609,67.196363],[-95.400555,67.197209],[-95.415282,67.199709],[-95.430827,67.202773],[-95.5275,67.223036],[-95.550073,67.231791],[-95.542218,67.238591],[-95.391109,67.263045],[-95.377782,67.262773],[-95.317227,67.255554],[-95.307973,67.246163],[-95.314709,67.238591]]],[[[-79.866391,53.169445],[-79.858891,53.166936],[-79.846118,53.160545],[-79.795836,53.116391],[-79.788327,53.104436],[-79.7932,53.094718],[-79.839718,53.083045],[-79.909164,53.081936],[-79.922909,53.083464],[-79.930973,53.091936],[-79.9386,53.122491],[-79.9386,53.134991],[-79.932782,53.150127],[-79.902082,53.1736],[-79.887509,53.174436],[-79.877491,53.173045],[-79.866391,53.169445]]],[[[-62.961109,67.0547],[-62.941109,67.043591],[-62.9225,67.0311],[-62.914446,67.023036],[-62.915273,67.012218],[-62.938046,67.005827],[-62.977773,67.006654],[-63.009727,67.009991],[-63.037782,67.015827],[-63.068336,67.025545],[-63.092773,67.035809],[-63.123891,67.049145],[-63.137018,67.058518],[-63.133055,67.071936],[-63.113891,67.079573],[-63.100082,67.079709],[-63.002227,67.069445],[-62.978336,67.062763],[-62.961109,67.0547]]],[[[-110.816964,67.940263],[-110.865827,67.892491],[-110.880282,67.8875],[-111.059436,67.849154],[-111.071673,67.847491],[-111.087909,67.854354],[-111.070282,67.8672],[-110.848891,67.954709],[-110.839173,67.958036],[-110.826682,67.959718],[-110.81,67.952963],[-110.816964,67.940263]]],[[[-124.025555,49.767773],[-124.046109,49.756382],[-124.060546,49.744991],[-124.074718,49.733327],[-124.090846,49.715545],[-124.1011,49.700273],[-124.105564,49.689427],[-124.107227,49.677218],[-124.112209,49.662354],[-124.126373,49.651382],[-124.139182,49.650545],[-124.153609,49.655545],[-124.186946,49.668882],[-124.196664,49.676936],[-124.200691,49.704018],[-124.141182,49.750273],[-124.091664,49.767773],[-124.032782,49.777773],[-124.018618,49.774154],[-124.025555,49.767773]]],[[[-116.260273,70.549991],[-116.243327,70.550263],[-116.189436,70.545536],[-116.140555,70.538591],[-116.1293,70.534145],[-116.291109,70.515545],[-116.323336,70.513609],[-116.446109,70.508882],[-116.464718,70.509154],[-116.477782,70.511936],[-116.495418,70.521382],[-116.47,70.538036],[-116.303046,70.5522],[-116.287782,70.553318],[-116.260273,70.549991]]],[[[-61.693327,57.368045],[-61.6775,57.357218],[-61.632218,57.337773],[-61.621664,57.335545],[-61.608336,57.335273],[-61.591455,57.327773],[-61.608336,57.308327],[-61.655136,57.290409],[-61.737364,57.291109],[-61.753336,57.302491],[-61.766664,57.317909],[-61.7675,57.328327],[-61.747782,57.366027],[-61.726946,57.374436],[-61.7025,57.372763],[-61.693327,57.368045]]]]},"dataset":"cshapes","validSinceTimestamp":-678585600,"validUntilTimestamp":1341014400}]}
java.lang.ArrayIndexOutOfBoundsException: -65531
	at org.apache.lucene.util.ByteBlockPool.setBytesRef(ByteBlockPool.java:308) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.util.BytesRefHash.equals(BytesRefHash.java:183) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.util.BytesRefHash.findHash(BytesRefHash.java:337) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:255) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:149) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:766) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:417) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:373) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:231) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1575) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1320) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:558) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.engine.InternalEngine.innerIndex(InternalEngine.java:520) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:409) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:556) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:546) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:213) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:348) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.index(TransportShardBulkAction.java:155) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.handleItem(TransportShardBulkAction.java:134) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:120) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:73) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:76) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:49) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:914) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:884) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:327) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:262) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:864) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:861) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:142) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1652) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:873) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.access$400(TransportReplicationAction.java:92) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:279) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:258) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:250) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:610) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596) [elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.2.0.jar:5.2.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
[2017-03-21T02:05:47,779][WARN ][o.e.c.a.s.ShardStateAction] [ip-10-1-0-29-histograph] [cshapes][0] received shard failed for shard id [[cshapes][0]], allocation id [7TttPv9gRV-OtyVVRsrEQA], primary term [0], message [shard failure, reason [lucene commit failed]], failure [IllegalStateException[this writer hit an unrecoverable error; cannot commit]; nested: ArrayIndexOutOfBoundsException[-65531]; ]
java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot commit
	at org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4635) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3073) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3232) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3195) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.elasticsearch.index.engine.InternalEngine.commitIndexWriter(InternalEngine.java:1296) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.engine.InternalEngine.flush(InternalEngine.java:803) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.shard.IndexShard.flush(IndexShard.java:777) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.indices.flush.SyncedFlushService.performPreSyncedFlush(SyncedFlushService.java:414) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.indices.flush.SyncedFlushService.access$1000(SyncedFlushService.java:70) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.indices.flush.SyncedFlushService$PreSyncedFlushTransportHandler.messageReceived(SyncedFlushService.java:696) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.indices.flush.SyncedFlushService$PreSyncedFlushTransportHandler.messageReceived(SyncedFlushService.java:692) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:610) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.2.0.jar:5.2.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: java.lang.ArrayIndexOutOfBoundsException: -65531
	at org.apache.lucene.util.ByteBlockPool.setBytesRef(ByteBlockPool.java:308) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.util.BytesRefHash.equals(BytesRefHash.java:183) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.util.BytesRefHash.findHash(BytesRefHash.java:337) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:255) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:149) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:766) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:417) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:373) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:231) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1575) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1320) ~[lucene-core-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:57:29]
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:558) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.engine.InternalEngine.innerIndex(InternalEngine.java:520) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:409) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:556) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:546) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:213) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:348) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.index(TransportShardBulkAction.java:155) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.handleItem(TransportShardBulkAction.java:134) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:120) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:73) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:76) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:49) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:914) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:884) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:327) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:262) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:864) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:861) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:142) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1652) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:873) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.access$400(TransportReplicationAction.java:92) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:279) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:258) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:250) ~[elasticsearch-5.2.0.jar:5.2.0]
	... 7 more

[sbocconi]

Possibly related to #22786

[colings86]

@nknize are you able to take a look at this?

[sbocconi]

Any progress on this issue?
Thanks

[aj7]

we are getting quite a bit of 'array-out-of-bounds' error importing some geo-shape objects. Could it be that if the size is too big, then this becomes a problem? Any ideas please?

[2017-07-26T12:02:36,395][DEBUG][o.e.a.b.TransportShardBulkAction] [my_data_node] [my_index][4] failed to execute bulk item (index) BulkShardRequest [[my_index][4]] containing [index {[my_index][FeatureCollection][AV1-j4-ITghY5gkRJfoj], source[n/a, actual length: [4mb], max length: 2kb]}]
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [geometry]
	at org.elasticsearch.index.mapper.GeoShapeFieldMapper.parse(GeoShapeFieldMapper.java:473) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:450) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:467) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:383) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:373) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:93) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:66) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:277) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:536) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.shard.IndexShard.prepareIndexOnPrimary(IndexShard.java:513) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.prepareIndexOperationOnPrimary(TransportShardBulkAction.java:450) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:458) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:143) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:113) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:69) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:939) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:908) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:322) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:264) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:888) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:885) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:147) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1656) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:897) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.access$400(TransportReplicationAction.java:93) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:281) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:260) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:252) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:627) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.4.1.jar:5.4.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
	at org.elasticsearch.common.geo.builders.PolygonBuilder.assign(PolygonBuilder.java:483) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.common.geo.builders.PolygonBuilder.compose(PolygonBuilder.java:455) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.common.geo.builders.PolygonBuilder.coordinates(PolygonBuilder.java:221) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.common.geo.builders.MultiPolygonBuilder.build(MultiPolygonBuilder.java:129) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.GeoShapeFieldMapper.parse(GeoShapeFieldMapper.java:455) ~[elasticsearch-5.4.1.jar:5.4.1]
	... 36 more

[sbocconi]

Script and data to reproduce the problem:

data is here
Script is here

Run the script in same dir as data, wait ~4.5 hours on a single node ES 32GB quad core.

[imotov]

I was able to reproduce the issue on 6.4 and it is definitely unrelated to #25933 and #22786. The issues #25933 and #22786 were caused by a bug in the polygon parsing logic that couldn't handle malformed polygons with holes that were protruding out of the outer shell in a certain way and. These issues were fixed by #27685.

In this case the issue is caused by the shape of post-1948 Canada that at 10m precision produces a huge number of tokens that cannot be correctly handled by Lucene causing an ArrayOutOfBounds exception in ByteBlockPool most likely due to numeric overflow somewhere in BytesRefHash. The exception is not handling gracefully and results in the engine failure that in turn causes the shard failure.

I was able to shrink the reproduction script in size and time required to reproduce the issue, but it still needs about 1 hour to generate enough tokens to cause the overflow on my machine.

[imotov]

So, just to see the impact of #36751 I indexed the same shape that was crashing the shard after 1h+ of the indexing time and it took 0.016 sec. I then modified the script in the original issue that was failing halfway after 4.5 hours of indexing and it took about 1 sec. I also ran the server with 1G instead of 16G that were required before.

[imotov]

Opened LUCENE-8614

[imotov]

I don't think there is anything we can/should do to address this on the elasticsearch side. The short term solution would be to change the exception thrown by lucene so it doesn't cause unrecoverable engine failure and this will hopefully addressed by LUCENE-8614. The long term solution is #36751 and deprecation of quadtrees. So, I am going to close the issues.