Ingest node date processor uses stale year information on calendar year change

[pprkut]

Elasticsearch version:
5.1.1

JVM version:
1.8.0.111

OS version:
CentOS 7.3

Description of the problem including expected versus actual behavior:

I'm feeding /var/log/secure to elasticsearch, using filebeat and ingest node. This worked fine before the new year, but then I noticed entries appearing with a one year old timestamp.
Turns out, the date processor was parsing the date string "Jan 2 23:59:48" as "2016-01-02T23:59:48.000Z". Restarting elasticsearch caused new entries to get the correct year information.

More info available here.
Possibly related to logstash-plugins/logstash-filter-date#3

[clintongormley]

@talevy Did you have a solution for this in Logstash that can be ported to ingest?

[talevy]

Oops, I wasn't aware of this pre-existing issue that we had in logstash-filter-date. I see the solution we came up with in Logstash, I'll see how we can do something similar in Elasticsearch