Audit logging structured entries

[albertzaharovits]

To facilitate machine readability, log entries should be structured in a dictionary-like format. Structured logging is also known as semantic or typed logging. Examples of such structured formats are: key value pairs, JSON and XML. In general, flat text entries might be more concise and trade ambiguity for human readability. Machine readable entries are more verbose and explicit, shunning ambiguity.

The status quo for the audit logging is a mixture of the templated and the structured entry type. In general, the templated entry type is a good tradeoff between user readability and the dictionary-like format. It is not ambiguous and is also more concise compared to the structured type. Here is an example entry from the current implementation:

[2018-05-30T07:11:39,605] [transport] [access_granted]  origin_type=[local_node], origin_address=[127.0.0.1], principal=[_xpack_security], roles=[superuser], action=[indices:data/read/search[free_context/scroll]]

Unfortunately, in the current implementation, entries are ambiguous because the set of allowed characters in user and role names includes '=', ' ', '[' and ']'. Fixing this would allow for unambiguous templated log entries.

Unambiguous means machine readable, i.e. you can write a regex to parse a specific field, so why bother with the verbose structured format? The answer is that the next step after parsing is indexing, and indexing requires that event fields have a name and a type. The parser of a templated entry type infers the field name and type from the token's position. But there may be several entry types with different positional parsing required, so one has to create several patterns. But what if patterns overlap (several can match the same line), or more commonly, what if parsers assign the same field name to fields with different data types. This is very possible because the code which generates entries and the code which parses and assigns type names are in different projects. For that matter, it is also relatively easy for the generating and the parsing pieces to go out of sync with each other, requiring integration tests...

Lecture over, here is the proposal:
The format of the entries should be JSON objects, UTF-8 encoded. This format is dictionary-like, unambiguous, where each field has a name and special characters in values are escaped. The new format will coexist alongside the present one, in an separate file trail.
Field names should align to the elastic common schema (ECS). ECS encourages dot notation for field names, i.e. related fields are similarly prefixed. This proposal will follow this suggestion, but note that entries will not be nested, values are not dictionaries.
The stretched goal is to have the format configurable, so even SYSLOG message format from RFC5424 might be supported. One possible avenue for this are messages in log4j 2.

This proposal is a narrowing of #8786 to the scope of the audit log.
Relates to #29881 .

[elasticmachine]

Pinging @elastic/es-security

[clintongormley]

When we have discussed structured logging in the past, we had agreed to go down the path of structured logging instead of JSON because it is human readable yet machine parseable, and is more succinct.

I think it'd be a shame to have to use JSON for the audit logs. I understand the issues with escaping, and I don't think we want several regexes to parse different log files. Instead I'm thinking of something like:

CPU high [cpu:90%] on [node:abcdefg][ip:1.2.3.4]

So we can extract the message plus each individual field using the same process for every log line.

@rjernst You use structured logging in a previous job - you must have faced similar problems. Any suggestions for this?

[albertzaharovits]

@clintongormley in the context of my pretentious lingo about petty matters, JSON is a structured logging example.
What I think you mean is that in the past templated logging has been preferred over structured because of its conciseness and human readability.

Basically the difference between templated and structured logging is if values are accompanied by the field name.
The example you propose is a novel breed of templated where values have field names. I haven't seen this before.
What I've seen, is that entries have a "message" field that is free text to soothe the human soul.
So, using your example, this should look like this:

{ "message": "CPU is high at 90% on node abcdefg with ip 1.2.3.4",
  "cpu": "90%",
  "node": "abcdefg",
  "ip": "1.2.3.4"
}

The JSON will not be nested, so a one line entry (unformatted with spaces) should still be readable by the typical IT admin eye.

I am leaning towards the flat JSON with the text "message" field because it is the canonical way (parsers don't require Regexp), although it loses on the conciseness field. WDYT?

[pcsanwald]

My personal experience here is that at my last job I used winston using the default JSON formatting feeding into papertrail. I found the lack of conciseness to be extremely annoying, and as a business we never derived any particular value from the JSON structure. that is obviously anecdotal. I set it up that way because it seemed like a good idea at the time, but I would think twice before doing it again.

Also, since it's not trivial for things like postgres to log in JSON format, anyone using an alternate or additional log aggregation service is going to likely be dealing with templated logs regardless. And in the case of audit logging, there will very often be a 3rd party aggregation service involved somehow (certainly this was the case when I worked at a large investment bank).

I'm a bit curious why we can't support both formats via config?

[clintongormley]

It's still not fantastically human readable, and we do want to optimise for readability so that sysadmins don't hate us at 3am. The most important thing for me is that we use the same style of logging across all logs generated by Elasticsearch. Think about what a stack trace would look like in JSON? Awful!

/cc @elastic/es-core-infra for opinions.

I'm a bit curious why we can't support both formats via config?

Of course this is a possibility, but it'd be much nicer to be able to satisfy both needs out of the box, so that centralised logging doesn't require config changes that then make local access unreadable.

[rjernst]

@rjernst You use structured logging in a previous job - you must have faced similar problems. Any suggestions for this?

I'm torn on which is more readable, json or text. In my previous use of structured logging, it was for search request timings (among other things). It was a multi line format with an end of record marker and became hard to read because it was packed together so tightly with no extra spacing. The format would roughly look something like:

DATE
path=...
ip=...
timers=timer1:57,timer2:230
...
EOR

So there was a general set of "timers" (which could change in code based on the request/parts of the code executed), and the times were always in milliseconds. I believe there were some other generalized keys with subkeys, but the timers I think is the most illustrative for my point: having types of the values based on structure (eg all subkeys of "timers") makes the server side a lot more flexible.

Obviously this could be in json as well, it would just be more verbose. In some cases (large entries), the json is probably easier to read (if pretty printing is used). I would just make sure in this case that the superfluous formatting (ie spaces/newlines) are removed before transferring the records.

[rjernst]

I should also mention my use was a dedicated log for search requests, while I think part of our dilemma is we have multiple logs for different purposes that we are trying to have a unified format for. IMO logs with general messages should look different than request logs, etc.

[jasontedor]

I am concerned by the fact that this discussion is being driven on a targeted issue (structured logging for audit logging) rather than more broadly within Elasticsearch. Is there a reason that we are approaching this from this angle?

[albertzaharovits]

Is there a reason that we are approaching this from this angle?

I have tried to scope my work to achieve the removal of audit indexing #29881. To that end, only logfile audit has to be machine readable so that it's easy to parse it (without utilizing multiple regexp). The topic did touched matters that are not audit logfile specific, such as stacktraces.

IMO log formats can differ: request, slowlog, audit can have different formats, although I agree it would be desirable that all logs are templated or all logs are JSON-like. In the light of this, given that in the current implementation, we are mostly on the templated bandwagon with all the logs (and no JSON-like), I am now swayed to the templated proposal that @clintongormley proposed, which has names for fields, so that a single regexp is required to parse all entries, and there is no possibility for multiple ones to overlap. It is also readable (entries are phrases instead of tabular) being alike to the general log.