Support JSON logging formatting

[markwalkom]

There was discussion on the Logstash list about some grok patterns for Elasticsearch logs to allow easier ingestion and someone suggested that it'd be better if Elasticsearch supported JSON output using https://github.com/logstash/log4j-jsonevent-layout.

Is it possible to do this and have it as an optional setting in the logging config?

[markwalkom]

Had a chat with a customer today and they rolled their own for this using log4j-jsonevent and a few other libraries with some config changes.

It'd be great if we could offer this out of the box.

[samcday]

Here's what we're doing:

• Drop log4j-jsonevent-layout jar-with-deps into ES lib/ dir.
• Setting appender.file.layout.type to net.logstash.log4j.JSONEventLayoutV1 in logging.yml

After that, the ES logfile will be spitting out JSON, and order is restored to the universe :)

[samcday]

If you think a PR that adds log4j-jsonevent-layout as a compile dep in ES, with a json alias for the layout.type (similar to the pattern one) would be accepted, I'm happy to raise one.

[TinLe]

+1

[markwalkom]

Any way we could get this into an early 2.X release?

[faxm0dem]

👍

[miah]

We've hit 2.0 release. ES is still on log4j 1.2, which means we still have to follow this hack workaround.

[miah]

Also should point out that the logstash jsonevent jar doesn't work with ES 2.0 in my experience. Logs are produced (in our non-json aka console logs) until the node has finished starting, then all logging ceases.

[jeremydonahue]

+1 for this feature.

@miah I tested the logstash jsonevent jar with ES 2.0 today and was unable to get it working at all. I tried building from source and a jar downloaded from maven.org. If I understand correctly, log4j version 2 supports json as a layout option. How much work is involved with upgrading log4j?

[kimchy]

I think that "just doing json layout" is not enough. My suggestion is to enhance our custom logging format to allow for parameters, something like this event happened at {timestamp:date} and took {timeInMillis:duration} (just an example). Then, we can produce a proper structured logging that makes any structured format more usable, specifically if we create a common schema around it (for duration, ...). Building a JSON layout on top of it using our logging abstraction will then be simpler.

Note, I am not a fan of the concept of MDC and such, I think this will be a much better solution.

[faxm0dem]

@kimchy can you elaborate on why you don't like MDC? I'm a bit surprised this is coming from someone who developed a tool that is used a lot for indexing structured logs often containing MDC ;-)