Geo_shape indexing issue - java.lang.ArrayIndexOutOfBoundsException

[aj7]

Elasticsearch version: 5.4.1

Plugins installed: []

JVM version (java -version): java version "1.8.0_131"

OS version (uname -a if on a Unix-like system): MacOS 10.11.6

Description of the problem including expected versus actual behavior:
Have been trying to index a lot of geometry, and getting consistently the same problems with those geometry/polygons that seem to contain a lot of vertices. Have tried importing the shapefile, and geojson through ogr2ogr. also tried generating a json file to use with bulk update. The error I am getting is an ArrayOutofBoundsException.

Also to note, if I upload the JSON file without specifying a mapping i.e. a geo_shape, then the data gets sucked in, but I would expect it's then not indexed.

As I have consistently been able to reproduce this with different upload routes, I am assuming it has to be a bug - is there a way around it?

Error I am getting is below:
https://gist.github.com/aj7/12fa99044901d7db8b42a5d5d5753bf8

I am happy to attach the json file if someone want's to try and see for themselves?

[danielmitterdorfer]

@aj7 It would be very helpful if you can create a minimal reproduction scenario (data + API calls / curl commands).

[aj7]

@danielmitterdorfer Sure, I have put the info in the gist

Hope this is fine?

[danielmitterdorfer]

Based on the data from the gist I could create a minimal reproduction scenario for Elasticsearch 5.4.1 consisting of only one document (attached compressed as geometry.json.zip).

curl -XDELETE "http://localhost:9200/speedlimit"

curl -XPUT "http://localhost:9200/speedlimit" -d '
{
  "mappings": {
    "speedlimit": {
      "properties": {
        "geometry": {
          "type": "geo_shape",
          "tree": "quadtree",
          "precision": "100m"
        }
      }
    }
  }
}'
# ensure that you've unzipped geometry.json.zip from the attachment above
curl -XPOST 'http://localhost:9200/_bulk?pretty' --data-binary @geometry.json

Will lead to:

{
  "took" : 33813,
  "errors" : true,
  "items" : [
    {
      "index" : {
        "_index" : "speedlimit",
        "_type" : "speedlimit",
        "_id" : "AV2EWf3T4RDVowrFsHtQ",
        "status" : 400,
        "error" : {
          "type" : "mapper_parsing_exception",
          "reason" : "failed to parse [geometry]",
          "caused_by" : {
            "type" : "array_index_out_of_bounds_exception",
            "reason" : "-1"
          }
        }
      }
    }
  ]
}

and in the logs:

org.elasticsearch.index.mapper.MapperParsingException: failed to parse [geometry]
	at org.elasticsearch.index.mapper.GeoShapeFieldMapper.parse(GeoShapeFieldMapper.java:473) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:450) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:467) ~[elasticsearch-5.4.1.jar:5.4.1]
[...]
Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
	at org.elasticsearch.common.geo.builders.PolygonBuilder.assign(PolygonBuilder.java:483) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.common.geo.builders.PolygonBuilder.compose(PolygonBuilder.java:455) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.common.geo.builders.PolygonBuilder.coordinates(PolygonBuilder.java:221) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.common.geo.builders.MultiPolygonBuilder.build(MultiPolygonBuilder.java:129) ~[elasticsearch-5.4.1.jar:5.4.1]
	at org.elasticsearch.index.mapper.GeoShapeFieldMapper.parse(GeoShapeFieldMapper.java:455) ~[elasticsearch-5.4.1.jar:5.4.1]
	... 36 more

[apeters]

I have nothing to add to this ticket other than to +1 @aj7 on this issue. We too have seen this error with either high vertex count polygons or polygons with very high precision. Sometime the workaround is to simplify the geometry with something like PostGIS but often that can compromise the data.
And, for what it's worth, it's not too hard to generate polygons like this (imagine a highly precise polygon representing country borders).
We would love for this to be fixed, as we've run into this on more then one occasion.

[rkuchvarskyy]

We also faced with this issue
Simple example cause
"java.lang.ArrayIndexOutOfBoundsException: -1\r\n\tat org.elasticsearch.common.geo.builders.PolygonBuilder.assign(PolygonBuilder.java:483)\r\n\tat org.elasticsearch.common.geo.builders.PolygonBuilder.compose(PolygonBuilder.java:455)\r\n\tat org.elasticsearch.common.geo.builders.PolygonBuilder.coordinates(PolygonBuilder.java:221)\r\n\tat org.elasticsearch.common.geo.builders.PolygonBuilder.buildGeometry(PolygonBuilder.java:251)\r\n\tat org.elasticsearch.common.geo.builders.PolygonBuilder.build(PolygonBuilder.java:226)\r\n\tat

{"type":"Polygon","coordinates":[[[-10,10.0000000000002],[-5,-5],[10,5],[-10,10.0000000000002]],[[-10,10],[0,5],[0,0],[-10,10]]]}

[DaveCTurner]

This seems to be because of an error in the geometry you're trying to index, although it would certainly be helpful if there was a more descriptive error message given. The attached zip file, for instance, contains self-intersections so I think it's right to reject it.

Here is a shorter reproduction:

$ curl -XDELETE "http://localhost:9200/speedlimit"
{"acknowledged":true}
$ curl -XPUT "http://localhost:9200/speedlimit" -H'Content-type: application/json' -d '{"mappings":{"speedlimit":{"properties":{"geometry":{"type":"geo_shape","tree":"quadtree","precision":"100m"}}}}}'
{"acknowledged":true,"shards_acknowledged":true,"index":"speedlimit"}
$ curl -XPOST "http://localhost:9200/speedlimit/speedlimit" -H'Content-type: application/json' -d '{"geometry":{"coordinates":[[[[4,3],[3,2],[3,3],[4,3]],[[4,2],[3,1],[4,1],[4,2]]]],"type":"MultiPolygon"}}'
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse [geometry]"}],"type":"mapper_parsing_exception","reason":"failed to parse [geometry]","caused_by":{"type":"array_index_out_of_bounds_exception","reason":"-1"}},"status":400}

In this case the given geometry contains two triangles, the first of which is clockwise and the second is anticlockwise, indicating that one is an enclave (hole) within the other. However the triangles do not touch.

[DaveCTurner]

@rkuchvarskyy the example you give also is not a valid geometry because the "hole" is not completely within the outer polygon. It's too small to see the error, but if I exaggerate the problem a bit your shape looks like this:

[rkuchvarskyy]

@DaveCTurner I understood this but I expect some validation exception but not java.lang.ArrayIndexOutOfBoundsException