Investigate `immediateFlush` performance impact for the audit log

[albertzaharovits]

This was suggested by @bizybot here https://github.com/elastic/elasticsearch/pull/33894/files#r220517146 .

As long as we're not using async appenders, and currently we do not, we can enable
the log4j appender property named immediateFlush . This makes sure every log event
is written before the log() call returns. This increases the reliability of the audit log,
an important property, but it's performance impact has to be gauged first.

It is worth pointing out, that we might as well recommend enabling it in the docs, but not
actually enabling it on the default distribution, the reason being that we might not be able
to evaluate the performance impact accurately on each file system / OS combination and
parameters.

@danielmitterdorfer do you think rally can help with this performance evaluation?
Do you think this is actually something we can quantify given our current support matrix?

[danielmitterdorfer]

@danielmitterdorfer do you think rally can help with this performance evaluation?

In general it can, as it is just a load generator. But I guess evaluating this might get tricky because you need to benchmark a scenario which produces enough audit log entries (or even better: a varying degree of audit log entries across different scenario so you can better estimate the overhead). I think it would make sense to benchmark on one platform first and then expand as needed.

[jaymode]

@albertzaharovits @bizybot the default for the RollingFile appender has immediateFlush set to true according to https://logging.apache.org/log4j/2.x/manual/appenders.html. So do we really need to do anything here?

[bizybot]

Thanks @jaymode Sorry I missed it When set to true - the default, I guess we need not do anything here.