DateIndexNameProcessor does not support unix epoch format

[iksnalybok]

Elasticsearch+Filebeat 6.0.0-rc1.

DateIndexNameProcessor expects the date to be a String. At line 64: String date = ingestDocument.getFieldValue(field, String.class);.

The log4j2 json logger outputs timestamp as unix_ms epoch (e.g. {"timeMillis":1507099254201,…}).
I want the log to be indexed in a daily index, on the day it was generated (vs. ingested), so I tell filebeat to use an elasticsearch date_index_name pipeline. It fails with the following exception:

Caused by: java.lang.IllegalArgumentException: field [json.timeMillis] of type [java.lang.Long] cannot be cast to [java.lang.String]
at o.e.ingest.IngestDocument.cast(IngestDocument.java:542)
at o.e.ingest.IngestDocument.getFieldValue(IngestDocument.java:107)
at o.e.ingest.common.DateIndexNameProcessor.execute(DateIndexNameProcessor.java:64)
at o.e.ingest.CompoundProcessor.execute(CompoundProcessor.java:100)

Reproducer:

1. Start elasticsearch (unzip and start) and create the pipeline:

PUT /_ingest/pipeline/bugTimestampPipeline
  {
    "description": "bugTimestampPipeline",
    "processors" : [
      {
        "date_index_name" : {
          "field"             : "json.timeMillis",
          "date_formats"      : [ "UNIX_MS" ],
          "index_name_prefix" : "myDailyIndex-",
          "date_rounding"     : "d",
          "index_name_format" : "yyyy.MM.dd"
        }
      }
    ]
  }

2. Create the filebeat configuration, and run filebeat --path.config confBugTimestamp -c filebeat-bugTimestamp.yml:

• confBugTimestamp/fields.yml: a copy of <filebeatDir>/fields.yml

• confBugTimestamp/filebeat-bugTimestamp.yml

setup.kibana:
  host: "localhost:5601"

output.elasticsearch:
  hosts: ["localhost:9200"]
  pipeline: bugTimestampPipeline

filebeat.prospectors:
- type: log
  enabled: true
  paths:
  - logsBugTimestamp/*.json.log
  json.keys_under_root: false
  json.add_error_key: true
  json.message_key: message
  close_inactive: 24h
  close_renamed: true  # because Windows (https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#close-renamed)
  close_removed: true  # because Windows (https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#close-removed)

3. copy bugTimestamp.json.log into logsBugTimestamp/

• bugTimestamp.json.log

{"timeMillis":1507099254201,"level":"INFO","message":"foobar"}

[martijnvg]

@iksnalybok Thanks for sharing this bug! I'll fix this soon.