From 0b1bedf41d95eae62b742c8a819bdcf30d6a3ba6 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Thu, 18 Oct 2018 14:13:18 +0300 Subject: [PATCH 01/10] Finally .... --- .../settings/audit-settings.asciidoc | 46 +++++++++++++------ .../security/auditing/output-logfile.asciidoc | 38 +++++++++++++-- .../en/security/auditing/overview.asciidoc | 15 ++++-- 3 files changed, 77 insertions(+), 22 deletions(-) diff --git a/docs/reference/settings/audit-settings.asciidoc b/docs/reference/settings/audit-settings.asciidoc index 524198df58c47..a62302e638989 100644 --- a/docs/reference/settings/audit-settings.asciidoc +++ b/docs/reference/settings/audit-settings.asciidoc @@ -18,10 +18,17 @@ Set to `true` to enable auditing on the node. The default value is `false`. `xpack.security.audit.outputs`:: Specifies where audit logs are output. For example: `[ index, logfile ]`. The default value is `logfile`, which puts the auditing events in a dedicated -`_access.log` file on the node. You can also specify `index`, which -puts the auditing events in an {es} index that is prefixed with -`.security_audit_log`. The index can reside on the same cluster or a separate -cluster. +file named `_audit.log` on each node. +You can also specify `index`, which puts the auditing events in an {es} index +that is prefixed with `.security_audit_log`. The index can reside on the same +cluster or a separate cluster. + +For backwards compatibility reasons, the logfile contents are duplicated to +another `_access.log` file with an improved format. +It is recommended that you disable this backwards compatible output file. To do +that, change its logger level to `off` in the `log4j2.properties` file from +`ES_PATH_CONF`. + + -- TIP: If the index is unavailable, it is possible for auditing events to @@ -57,17 +64,26 @@ audited in plain text when including the request body in audit events. [[node-audit-settings]] ==== Local Node Info Settings -`xpack.security.audit.logfile.prefix.emit_node_name`:: -Specifies whether to include the node's name in the local node info. The -default value is `true`. - -`xpack.security.audit.logfile.prefix.emit_node_host_address`:: -Specifies whether to include the node's IP address in the local node info. The -default value is `false`. - -`xpack.security.audit.logfile.prefix.emit_node_host_name`:: -Specifies whether to include the node's host name in the local node info. The -default value is `false`. +`xpack.security.audit.logfile.emit_node_name`:: +Specifies whether to include the <> as a field in each +audit event. +The default value is `true`. + +`xpack.security.audit.logfile.emit_node_host_address`:: +Specifies whether to include the node's IP address as a field in each audit event. +The default value is `false`. + +`xpack.security.audit.logfile.emit_node_host_name`:: +Specifies whether to include the node's host name as a field in each audit event. +The default value is `false`. + +`xpack.security.audit.logfile.emit_node_id`:: +Specifies whether to include the node id as a field in each audit event. +This is available for the new format only. +Unlike <>, whose value might change if the administrator +changes the setting in the config file, the node id will persist across cluster +restarts and the administrator has no handle to change it. +The default value is `true`. [[index-audit-settings]] ==== Audit Log Indexing Configuration Settings diff --git a/x-pack/docs/en/security/auditing/output-logfile.asciidoc b/x-pack/docs/en/security/auditing/output-logfile.asciidoc index ee33f618f9665..6f55142fe3a12 100644 --- a/x-pack/docs/en/security/auditing/output-logfile.asciidoc +++ b/x-pack/docs/en/security/auditing/output-logfile.asciidoc @@ -3,13 +3,39 @@ === Logfile audit output The `logfile` audit output is the default output for auditing. It writes data to -the `_access.log` file in the logs directory. +the `_audit.log` file in the logs directory. To maintain +compatibility with releases prior to 6.5.0, another file `_access.log` +is also generated next to it. They differ in the output format but the contents +are similar. For systems that had not automatically ingesting the audit file +for search or analytics it is strongly recommended to only keep the newer format. +Turning off the deprecated output format can be achieved by disabling the logger +in the `log4j2.properties` file in `ES_PATH_CONF` (hint: there is a config comment +about it). [float] [[audit-log-entry-format]] === Log entry format -The format of a log entry is: +This is describing the format for log entries as they are appended to the +new `_audit.log` file. + +- Each log entry is a one line JSON document and each one is printed on a separate line. +- The fields of a log entry are ordered. However, if a field does not have a value it + will not be printed. The precise line pattern, together with the complete field + order, are specified in the `log4j2.properties` config file. +- The log entry does not contain nested inner JSON objects, i.e. the doc is flat. +- The field names follow a dotted notation to flatten inner objects. +- A field's value can be a string, a number or an array of strings. +- A field's value, a request body as well, will be escaped as per the JSON RFC 4627. + +There is a list of <> specifying the +set of fields for each sog entry type. + +[float] +[[deprecated-audit-log-entry-format]] +=== Deprecated log entry format + +This is the format for a log entries as appended to `_access.log`: [source,txt] ---------------------------------------------------------------------------- @@ -48,8 +74,14 @@ audited in plain text when including the request body in audit events. [[logging-file]] You can also configure how the logfile is written in the `log4j2.properties` file located in `ES_PATH_CONF`. By default, audit information is appended to the -`_access.log` file located in the standard Elasticsearch `logs` directory +`_audit.log` file located in the standard Elasticsearch `logs` directory (typically located at `$ES_HOME/logs`). The file rolls over on a daily basis. +The deprecated logfile audit format (`_access.log`) can be disabled +from the same `log4j2.properties` file (hint: look for the comment +instructing to set the log level to `off`). The deprecated format is a duplication +of information that is in place to assure backwards compatibility. If you are +not strict about the audit format it is strongly recommended to only use the +`_audit.log` log appender. [float] [[audit-log-ignore-policy]] diff --git a/x-pack/docs/en/security/auditing/overview.asciidoc b/x-pack/docs/en/security/auditing/overview.asciidoc index b874af3d1c43c..e97cdcc32c6c2 100644 --- a/x-pack/docs/en/security/auditing/overview.asciidoc +++ b/x-pack/docs/en/security/auditing/overview.asciidoc @@ -16,11 +16,18 @@ must set `xpack.security.audit.enabled` to `true` in `elasticsearch.yml`. {Security} provides two ways to persist audit logs: * The <> output, which persists events to - a dedicated `_access.log` file on the host's file system. -* The <> output, which persists events to an Elasticsearch index. -The audit index can reside on the same cluster, or a separate cluster. + a dedicated `_audit.log` file on the host's file system. + For backwards compatibility reasons, a file named `_access.log` + is also generated. This has a format compatible to the one in releases prior + to 6.5.0. Between the two, the contents are identical, therefore it is recommended + to disable the `_access.log` output. For this, turn off the + logger in the `log4j2.properties` config file from `ES_PATH_CONF`. (hint: look + out for a comment in the config file). +* The <> output, which persists events to an Elasticsearch + index. The audit index can reside on the same cluster, or a separate cluster. -By default, only the `logfile` output is used when enabling auditing. +By default, only the `logfile` output is used when enabling auditing, +implicitly outputing to both `_audit.log` and `_access.log`. To facilitate browsing and analyzing the events, you can also enable indexing by setting `xpack.security.audit.outputs` in `elasticsearch.yml`: From a8d8c71ddac4a5a575e4914ba60a1099fe713796 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Thu, 18 Oct 2018 19:11:40 +0300 Subject: [PATCH 02/10] Addressed Jay's feedback --- docs/reference/settings/audit-settings.asciidoc | 8 ++++---- x-pack/docs/en/security/auditing/output-logfile.asciidoc | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/reference/settings/audit-settings.asciidoc b/docs/reference/settings/audit-settings.asciidoc index a62302e638989..8da3aa6ba1df2 100644 --- a/docs/reference/settings/audit-settings.asciidoc +++ b/docs/reference/settings/audit-settings.asciidoc @@ -24,10 +24,10 @@ that is prefixed with `.security_audit_log`. The index can reside on the same cluster or a separate cluster. For backwards compatibility reasons, the logfile contents are duplicated to -another `_access.log` file with an improved format. -It is recommended that you disable this backwards compatible output file. To do -that, change its logger level to `off` in the `log4j2.properties` file from -`ES_PATH_CONF`. +the `_access.log` file with the format for versions prior to 6.5.0. +If the backwards compatible format is not required, it should be disabled. +To do that, change its logger level to `off` in the `log4j2.properties` file +from `ES_PATH_CONF`. + -- diff --git a/x-pack/docs/en/security/auditing/output-logfile.asciidoc b/x-pack/docs/en/security/auditing/output-logfile.asciidoc index 6f55142fe3a12..e37247141135a 100644 --- a/x-pack/docs/en/security/auditing/output-logfile.asciidoc +++ b/x-pack/docs/en/security/auditing/output-logfile.asciidoc @@ -6,8 +6,8 @@ The `logfile` audit output is the default output for auditing. It writes data to the `_audit.log` file in the logs directory. To maintain compatibility with releases prior to 6.5.0, another file `_access.log` is also generated next to it. They differ in the output format but the contents -are similar. For systems that had not automatically ingesting the audit file -for search or analytics it is strongly recommended to only keep the newer format. +are similar. For systems that are not ingesting the audit file for search or +analytics it is strongly recommended to only keep the newer format. Turning off the deprecated output format can be achieved by disabling the logger in the `log4j2.properties` file in `ES_PATH_CONF` (hint: there is a config comment about it). From 5a6d6c0c534a98a11f908bfe389f793051a46ff5 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Tue, 23 Oct 2018 22:12:46 +0300 Subject: [PATCH 03/10] Lisa's feedback --- docs/reference/settings/audit-settings.asciidoc | 14 ++++++++------ .../en/security/auditing/output-logfile.asciidoc | 4 +++- x-pack/docs/en/security/auditing/overview.asciidoc | 6 +----- 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/reference/settings/audit-settings.asciidoc b/docs/reference/settings/audit-settings.asciidoc index 8da3aa6ba1df2..55a898acce184 100644 --- a/docs/reference/settings/audit-settings.asciidoc +++ b/docs/reference/settings/audit-settings.asciidoc @@ -23,11 +23,12 @@ You can also specify `index`, which puts the auditing events in an {es} index that is prefixed with `.security_audit_log`. The index can reside on the same cluster or a separate cluster. -For backwards compatibility reasons, the logfile contents are duplicated to -the `_access.log` file with the format for versions prior to 6.5.0. +For backwards compatibility reasons, if you use the logfile output type, a +`_access.log` file is also created. It contains the same +information, but it uses the older (pre-6.5.0) formatting style. If the backwards compatible format is not required, it should be disabled. -To do that, change its logger level to `off` in the `log4j2.properties` file -from `ES_PATH_CONF`. +To do that, change its logger level to `off` in the `log4j2.properties` file. +For more information, see <>. + -- @@ -79,10 +80,11 @@ The default value is `false`. `xpack.security.audit.logfile.emit_node_id`:: Specifies whether to include the node id as a field in each audit event. -This is available for the new format only. +This is available for the new format only. That is to say, this information +does not exist in the `_access.log` file. Unlike <>, whose value might change if the administrator changes the setting in the config file, the node id will persist across cluster -restarts and the administrator has no handle to change it. +restarts and the administrator cannot change it. The default value is `true`. [[index-audit-settings]] diff --git a/x-pack/docs/en/security/auditing/output-logfile.asciidoc b/x-pack/docs/en/security/auditing/output-logfile.asciidoc index e37247141135a..00f14fbec0f86 100644 --- a/x-pack/docs/en/security/auditing/output-logfile.asciidoc +++ b/x-pack/docs/en/security/auditing/output-logfile.asciidoc @@ -9,8 +9,10 @@ is also generated next to it. They differ in the output format but the contents are similar. For systems that are not ingesting the audit file for search or analytics it is strongly recommended to only keep the newer format. Turning off the deprecated output format can be achieved by disabling the logger -in the `log4j2.properties` file in `ES_PATH_CONF` (hint: there is a config comment +in the `log4j2.properties` file (hint: there is a config comment about it). +For more information, see <>. + [float] [[audit-log-entry-format]] diff --git a/x-pack/docs/en/security/auditing/overview.asciidoc b/x-pack/docs/en/security/auditing/overview.asciidoc index e97cdcc32c6c2..6f04e17d83138 100644 --- a/x-pack/docs/en/security/auditing/overview.asciidoc +++ b/x-pack/docs/en/security/auditing/overview.asciidoc @@ -18,11 +18,7 @@ must set `xpack.security.audit.enabled` to `true` in `elasticsearch.yml`. * The <> output, which persists events to a dedicated `_audit.log` file on the host's file system. For backwards compatibility reasons, a file named `_access.log` - is also generated. This has a format compatible to the one in releases prior - to 6.5.0. Between the two, the contents are identical, therefore it is recommended - to disable the `_access.log` output. For this, turn off the - logger in the `log4j2.properties` config file from `ES_PATH_CONF`. (hint: look - out for a comment in the config file). + is also generated. * The <> output, which persists events to an Elasticsearch index. The audit index can reside on the same cluster, or a separate cluster. From de3d362df72e41825f5f84c17d50c8eb89743bce Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Tue, 23 Oct 2018 22:18:09 +0300 Subject: [PATCH 04/10] fix links --- docs/reference/settings/audit-settings.asciidoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/reference/settings/audit-settings.asciidoc b/docs/reference/settings/audit-settings.asciidoc index 55a898acce184..c63813ce3b13f 100644 --- a/docs/reference/settings/audit-settings.asciidoc +++ b/docs/reference/settings/audit-settings.asciidoc @@ -66,8 +66,8 @@ audited in plain text when including the request body in audit events. ==== Local Node Info Settings `xpack.security.audit.logfile.emit_node_name`:: -Specifies whether to include the <> as a field in each -audit event. +Specifies whether to include the {ref}/node.name.html[node name] as a field in +each audit event. The default value is `true`. `xpack.security.audit.logfile.emit_node_host_address`:: @@ -82,7 +82,7 @@ The default value is `false`. Specifies whether to include the node id as a field in each audit event. This is available for the new format only. That is to say, this information does not exist in the `_access.log` file. -Unlike <>, whose value might change if the administrator +Unlike {ref}/node.name.html[node name], whose value might change if the administrator changes the setting in the config file, the node id will persist across cluster restarts and the administrator cannot change it. The default value is `true`. From 445edd2ec4d65f79a344df95e0cb9cc480fbfc9e Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Tue, 23 Oct 2018 22:45:33 +0300 Subject: [PATCH 05/10] More feedback --- x-pack/docs/en/security/auditing/output-logfile.asciidoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/x-pack/docs/en/security/auditing/output-logfile.asciidoc b/x-pack/docs/en/security/auditing/output-logfile.asciidoc index 00f14fbec0f86..4cefc592562fb 100644 --- a/x-pack/docs/en/security/auditing/output-logfile.asciidoc +++ b/x-pack/docs/en/security/auditing/output-logfile.asciidoc @@ -4,14 +4,14 @@ The `logfile` audit output is the default output for auditing. It writes data to the `_audit.log` file in the logs directory. To maintain -compatibility with releases prior to 6.5.0, another file `_access.log` -is also generated next to it. They differ in the output format but the contents +compatibility with releases prior to 6.5.0, a `_access.log` file +is also generated. They differ in the output format but the contents are similar. For systems that are not ingesting the audit file for search or analytics it is strongly recommended to only keep the newer format. Turning off the deprecated output format can be achieved by disabling the logger in the `log4j2.properties` file (hint: there is a config comment about it). -For more information, see <>. +For more information, see {ref}/logging.html#configuring-logging-levels[configuring-logging]. [float] From 4f5b88b0e161ea52b31910514afd7f7a16f7e069 Mon Sep 17 00:00:00 2001 From: Lisa Cawley Date: Tue, 23 Oct 2018 23:23:27 +0300 Subject: [PATCH 06/10] Update docs/reference/settings/audit-settings.asciidoc Co-Authored-By: albertzaharovits --- docs/reference/settings/audit-settings.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/settings/audit-settings.asciidoc b/docs/reference/settings/audit-settings.asciidoc index c63813ce3b13f..35f5b859f0ae0 100644 --- a/docs/reference/settings/audit-settings.asciidoc +++ b/docs/reference/settings/audit-settings.asciidoc @@ -66,7 +66,7 @@ audited in plain text when including the request body in audit events. ==== Local Node Info Settings `xpack.security.audit.logfile.emit_node_name`:: -Specifies whether to include the {ref}/node.name.html[node name] as a field in +Specifies whether to include the <> as a field in each audit event. The default value is `true`. From 15b35a0ee1c652250483be2da7fee7f5e215cf24 Mon Sep 17 00:00:00 2001 From: Lisa Cawley Date: Tue, 23 Oct 2018 23:23:34 +0300 Subject: [PATCH 07/10] Update docs/reference/settings/audit-settings.asciidoc Co-Authored-By: albertzaharovits --- docs/reference/settings/audit-settings.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/settings/audit-settings.asciidoc b/docs/reference/settings/audit-settings.asciidoc index 35f5b859f0ae0..ec661a1f30c9f 100644 --- a/docs/reference/settings/audit-settings.asciidoc +++ b/docs/reference/settings/audit-settings.asciidoc @@ -82,7 +82,7 @@ The default value is `false`. Specifies whether to include the node id as a field in each audit event. This is available for the new format only. That is to say, this information does not exist in the `_access.log` file. -Unlike {ref}/node.name.html[node name], whose value might change if the administrator +Unlike <>, whose value might change if the administrator changes the setting in the config file, the node id will persist across cluster restarts and the administrator cannot change it. The default value is `true`. From 5d1055cb4217444c55a4a0db1d877e1635143895 Mon Sep 17 00:00:00 2001 From: Lisa Cawley Date: Tue, 23 Oct 2018 23:23:44 +0300 Subject: [PATCH 08/10] Update x-pack/docs/en/security/auditing/output-logfile.asciidoc Co-Authored-By: albertzaharovits --- x-pack/docs/en/security/auditing/output-logfile.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/auditing/output-logfile.asciidoc b/x-pack/docs/en/security/auditing/output-logfile.asciidoc index 4cefc592562fb..b3e29e2e7608c 100644 --- a/x-pack/docs/en/security/auditing/output-logfile.asciidoc +++ b/x-pack/docs/en/security/auditing/output-logfile.asciidoc @@ -18,7 +18,7 @@ For more information, see {ref}/logging.html#configuring-logging-levels[configur [[audit-log-entry-format]] === Log entry format -This is describing the format for log entries as they are appended to the +The log entries in the `_audit.log` file new `_audit.log` file. - Each log entry is a one line JSON document and each one is printed on a separate line. From 4dc2b1837072ded952c3cb45a89afc16339a2ccd Mon Sep 17 00:00:00 2001 From: Lisa Cawley Date: Tue, 23 Oct 2018 23:24:13 +0300 Subject: [PATCH 09/10] Update x-pack/docs/en/security/auditing/output-logfile.asciidoc Co-Authored-By: albertzaharovits --- x-pack/docs/en/security/auditing/output-logfile.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/auditing/output-logfile.asciidoc b/x-pack/docs/en/security/auditing/output-logfile.asciidoc index b3e29e2e7608c..2519964affb6e 100644 --- a/x-pack/docs/en/security/auditing/output-logfile.asciidoc +++ b/x-pack/docs/en/security/auditing/output-logfile.asciidoc @@ -19,7 +19,7 @@ For more information, see {ref}/logging.html#configuring-logging-levels[configur === Log entry format The log entries in the `_audit.log` file -new `_audit.log` file. +have the following format: - Each log entry is a one line JSON document and each one is printed on a separate line. - The fields of a log entry are ordered. However, if a field does not have a value it From 000778e8e2e80d7931166ebf5a62d827bf910d22 Mon Sep 17 00:00:00 2001 From: Lisa Cawley Date: Tue, 23 Oct 2018 23:24:34 +0300 Subject: [PATCH 10/10] Update x-pack/docs/en/security/auditing/output-logfile.asciidoc Co-Authored-By: albertzaharovits --- x-pack/docs/en/security/auditing/output-logfile.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/auditing/output-logfile.asciidoc b/x-pack/docs/en/security/auditing/output-logfile.asciidoc index 2519964affb6e..d9e7eb81c3f3a 100644 --- a/x-pack/docs/en/security/auditing/output-logfile.asciidoc +++ b/x-pack/docs/en/security/auditing/output-logfile.asciidoc @@ -37,7 +37,7 @@ set of fields for each sog entry type. [[deprecated-audit-log-entry-format]] === Deprecated log entry format -This is the format for a log entries as appended to `_access.log`: +The log entries in the `_access.log` file have the following format: [source,txt] ----------------------------------------------------------------------------