[Ingest Pipeline] Date processor ignores timezone offset XXX

[P1llus]

Currently tested in Dev Console on 7.9.2 running on ESS.

Steps to reproduce:

It seems that timezone formats that work in pure Java is being ignored when using the date processor in ingest pipelines. Example:

POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "_description",
    "processors": [
      {
        "date": {
          "field": "message",
          "target_field": "@timestamp", 
          "formats": ["EEE MMM  d HH:mm:ss YYYY XXX"],
          "timezone": "UTC"
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "message": """Wed Oct  21 11:19:17 2020 -04:00"""
      }
    }
  ]
}

This will output:

{
  "docs" : [
    {
      "doc" : {
        "_index" : "index",
        "_type" : "_doc",
        "_id" : "id",
        "_source" : {
          "@timestamp" : "2020-10-21T11:19:17.000Z",
          "message" : "Wed Oct  21 11:19:17 2020 -04:00"
        },
        "_ingest" : {
          "timestamp" : "2020-10-08T09:21:11.48095Z"
        }
      }
    }
  ]
}

As you can see even though the offset is -04:00, and date processors defaults to UTC; it does not calculate any offset?

The same behavior is seen with other offset examples like Z, z, VV etc.

The same format in Java will work (sample code provided by @marc-gr ):

import java.util.*;
import java.util.Date;
import java.text.SimpleDateFormat;
public class Main {
    public static void main(String[] args) throws Exception {
        SimpleDateFormat format = new SimpleDateFormat("E MMM  d HH:mm:ss YYYY XXX");
        String dateString = format.format( new Date()   );
        Date   date       = format.parse ( "Wed Oct  21 11:19:17 2020 -04:00" );    
        System.out.println(date);
    }
}

Anyone know if this is intended behavior?

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Ingest)

[gaobinlong]

@P1llus , I think there are two problems:

1. If timezone is set to UTC explicitly, the time zone specified in the format will be ignored, we can find that from the source code: DataFormat.java. So timezone canot be set to UTC if we want to parse the timezone from the field message .

2. When we remove the timezone parameter, nothing changed! That's because the year format YYYY is week based year, there's a bug I think: DataFormat.java. The workaround is setting the year format to yyyy.

Finally, we can use date processor as follows:

{
  "pipeline": {
    "description": "_description",
    "processors": [
      {
        "date": {
          "field": "message",
          "target_field": "@timestamp", 
          "formats": ["EEE MMM  d HH:mm:ss yyyy XXX"]
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "message": "Wed Oct  21 11:19:17 2020 -04:00"
      }
    }
  ]
}

[P1llus]

Using yyyy resolved the issue indeed @gaobinlong , thanks for the pointer there! Will take a look at the bug you mentioned and see if we should keep this issue open.

[pgomulka]

I think the bug is valid. I thought I fixed this in #51215 but I missed this use case.
I will work on it