Ingest processor fails if `host.ip` contains an array of IPs

[rwaight]

Elasticsearch version: 7.3.1 running on Elastic Cloud

Description of the problem including expected versus actual behavior:
When attempting to enrich events with geoIP information in beats, with netinfo.enabled: true defined, the Elasticsearch Ingest processor fails if host.ip contains an array of IPs. The expected behavior is that an array of IPs can be ingested into Elasticsearch. The current workaround is to disable netinfo (i.e., netinfo.enabled: false)

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

1. Configure adding the geoip-info ingest pipeline in order to enrich events with geoIP information

2. Configure the Elasticsearch output to use the geoip-info pipeline (i.e. output.elasticsearch.pipeline: geoip-info)

3. Configure a top-level processor in *beats (this impacts auditbeat, filebeat, heartbeat, metricbeat, packetbeat, and winlogbeat) to add geo information, with netinfo.enabled: true

configuration from `packetbeat.yml`

processors:
  - add_host_metadata:
      netinfo.enabled: true
  - add_cloud_metadata: ~
  - add_fields:
      when.network.source.ip: 10.100.10.0/24
      fields:
        source.geo.location:
          lat: 42
          lon: -93
        source.geo.continent_name: North America
        source.geo.region_iso_code: US-IA
        source.geo.country_iso_code: US
        source.geo.region_name: Iowa
      target: ''
  - add_fields:
      when.network.destination.ip: 10.100.10.0/24
      fields:
        destination.geo.location:
          lat: 42
          lon: -93
        destination.geo.continent_name: North America
        destination.geo.region_iso_code: US-IA
        destination.geo.country_iso_code: US
        destination.geo.region_name: Iowa
      target: ''

4. Start packetbeat, it is unable to publish events to Elasticsearch, when not in DEBUG mode, the only response generated is:

ERROR    pipeline/output.go:121    Failed to publish events: temporary bulk send failure

Provide logs (if relevant):

elasticsearch/client.go:533    Bulk item insert failed (i=0, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}

more of the same `Bulk item insert failed` logs

2019-08-30T10:04:26.038-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:339	PublishEvents: 35 events have been published to elasticsearch in 130.0856ms.
2019-08-30T10:04:26.038-0500	DEBUG	[tcp]	tcp/tcp.go:177	tcp flow id: 0xc000e29900
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=0, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[decoder]	decoder/decoder.go:216	flow id flags: 3089
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=1, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[flows]	flows/flows.go:94	get flow
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=2, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[flows]	flows/flows.go:96	lookup flow: {3089 0 255 255 255 12 255 255 255 255 255 20 24 1 0 1} => [124 173 116 146 217 104 248 177 86 165 135 119 104 198 38 169 10 100 10 20 187 1 193 226 3 0 0 0 0 0 0 0]
2019-08-30T10:04:26.039-0500	DEBUG	[flows]	flows/flows.go:89	unlock flows
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=3, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[sniffer]	sniffer/sniffer.go:209	Packet number: 49991
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=4, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[decoder]	decoder/decoder.go:173	decode packet data
2019-08-30T10:04:26.039-0500	DEBUG	[flows]	flows/flows.go:84	lock flows
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=5, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[flows]	flows/flowid.go:157	flowid: add eth
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=6, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[decoder]	decoder/decoder.go:246	IPv4 packet
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=7, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=8, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[flows]	flows/flowid.go:163	flowid: add ipv4
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=9, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[decoder]	decoder/decoder.go:287	TCP packet
2019-08-30T10:04:26.039-0500	DEBUG	[flows]	flows/flowid.go:230	flowid: add tcp
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=10, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[decoder]	decoder/decoder.go:216	flow id flags: 1041
2019-08-30T10:04:26.039-0500	DEBUG	[flows]	flows/flows.go:94	get flow
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=11, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[flows]	flows/flows.go:96	lookup flow: {1041 0 255 255 255 12 255 255 255 255 255 20 255 1 0 1} => [0 80 182 150 119 253 248 177 86 165 135 119 10 100 10 15 10 100 10 20 21 194 61 13]
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=12, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=13, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[flows]	flows/flows.go:89	unlock flows
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=14, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=15, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[sniffer]	sniffer/sniffer.go:209	Packet number: 49992
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=16, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[decoder]	decoder/decoder.go:173	decode packet data
2019-08-30T10:04:26.039-0500	DEBUG	[flows]	flows/flows.go:84	lock flows
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=17, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=18, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=19, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[flows]	flows/flowid.go:157	flowid: add eth
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=20, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[decoder]	decoder/decoder.go:246	IPv4 packet
2019-08-30T10:04:26.039-0500	DEBUG	[flows]	flows/flowid.go:163	flowid: add ipv4
2019-08-30T10:04:26.039-0500	DEBUG	[decoder]	decoder/decoder.go:287	TCP packet
2019-08-30T10:04:26.039-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=21, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.039-0500	DEBUG	[flows]	flows/flowid.go:230	flowid: add tcp
2019-08-30T10:04:26.040-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=22, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.040-0500	DEBUG	[decoder]	decoder/decoder.go:216	flow id flags: 1041
2019-08-30T10:04:26.040-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=23, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.040-0500	DEBUG	[flows]	flows/flows.go:94	get flow
2019-08-30T10:04:26.040-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=24, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.040-0500	DEBUG	[flows]	flows/flows.go:96	lookup flow: {1041 0 255 255 255 12 255 255 255 255 255 20 255 1 0 1} => [0 80 182 150 119 253 248 177 86 165 135 119 10 100 10 15 10 100 10 20 21 194 61 13]
2019-08-30T10:04:26.040-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=25, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.040-0500	DEBUG	[flows]	flows/flows.go:89	unlock flows
2019-08-30T10:04:26.040-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=26, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.040-0500	DEBUG	[sniffer]	sniffer/sniffer.go:209	Packet number: 49993
2019-08-30T10:04:26.040-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=27, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.040-0500	DEBUG	[decoder]	decoder/decoder.go:173	decode packet data
2019-08-30T10:04:26.040-0500	DEBUG	[flows]	flows/flows.go:84	lock flows
2019-08-30T10:04:26.040-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=28, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.040-0500	DEBUG	[flows]	flows/flowid.go:157	flowid: add eth
2019-08-30T10:04:26.040-0500	DEBUG	[decoder]	decoder/decoder.go:246	IPv4 packet
2019-08-30T10:04:26.040-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=29, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.040-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=30, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.040-0500	DEBUG	[flows]	flows/flowid.go:163	flowid: add ipv4
2019-08-30T10:04:26.040-0500	DEBUG	[decoder]	decoder/decoder.go:287	TCP packet
2019-08-30T10:04:26.040-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=31, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.040-0500	DEBUG	[flows]	flows/flowid.go:230	flowid: add tcp
2019-08-30T10:04:26.040-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=32, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.040-0500	DEBUG	[flows]	flows/flowid.go:236	flowid: add tcp connection id
2019-08-30T10:04:26.040-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=33, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.040-0500	DEBUG	[tcp]	tcp/tcp.go:177	tcp flow id: 0xc000e29900
2019-08-30T10:04:26.040-0500	DEBUG	[elasticsearch]	elasticsearch/client.go:533	Bulk item insert failed (i=34, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]","caused_by":{"type":"illegal_argument_exception","reason":"field [host.ip] of type [java.util.ArrayList] cannot be cast to [java.lang.String]"}},"header":{"processor_type":"geoip"}}
2019-08-30T10:04:26.040-0500	DEBUG	[decoder]	decoder/decoder.go:216	flow id flags: 3089
2019-08-30T10:04:26.040-0500	DEBUG	[flows]	flows/flows.go:94	get flow
2019-08-30T10:04:26.040-0500	INFO	[publisher]	pipeline/retry.go:155	Drop batch

When not in DEBUG mode, the only response generated is:

ERROR    pipeline/output.go:121    Failed to publish events: temporary bulk send failure

[elasticmachine]

Pinging @elastic/es-core-features

[martijnvg]

So the ip addresses are then provided as json array in the source field (the field setting in the pipeline config)? If that is the case then is the expected behaviour is that the geo information is appended in the configured target_field (in the order matching to the order of the ip addresses in the configured source field)?

[webmat]

I think an acceptable implementation would be to loop over addresses, and simply decorate the event based on the first IP that has a match in the DB.

The most likely scenario this will happen is in situations specifically like host.ip, where a host has a bunch of IP addresses for real and virtual network interfaces. So most of them would be private, and potentially one of them would be a public address. If there's more than one public address assigned to the host, only the first one would be geolocated. The other ones would likely have yielded a similar result anyway.

[sspreitzer]

I am opting for a geoip feature where the ingest would iterates over an array of IPs and enrich the document's geoip field with an array of geoip information.

(Furthermore opening the possibility to point to point paths in visualizations. As well opening up the possibility for computed searches shortest paths, triangulations, etc..)

[rwaight]

Hi @probakowski, thank you for opening the PR! Would it be possible to backport to 7.4?

[jbaiera]

@rwaight Now that 7.5 is released I doubt there will be any further releases of the 7.4 branch

[rwaight]

Thanks for clarifying this with me, @jbaiera! I'll keep an eye on this for release in 7.6.