Skip to content

[7.x] [DOCS] EQL: Document search API's tiebreaker_field param (#57935) #58540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jun 26, 2020
Merged

[7.x] [DOCS] EQL: Document search API's tiebreaker_field param (#57935) #58540

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
aee3451
[DOCS] EQL: Document search API's `tiebreaker_field` param (#57935)
jrodewig Jun 25, 2020
6bb221f
[DOCS] Fix EQL search snippet for tiebreaker example
jrodewig Jun 25, 2020
e7e5eb2
Add doc type
jrodewig Jun 25, 2020
43df3fd
Merge branch '7.x' into backport__57935-7x
elasticmachine Jun 25, 2020
605f450
Merge branch '7.x' into backport__57935-7x
elasticmachine Jun 26, 2020
0240540
add missing type
jrodewig Jun 26, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 47 additions & 18 deletions docs/reference/eql/eql-search-api.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,15 @@ In {es}, EQL assumes each document in an index corresponds to an event.
----
PUT /my_index/_bulk?refresh
{"index":{"_index" : "my_index", "_id" : "1"}}
{ "@timestamp": "2020-12-06T11:04:05.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process" }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{ "@timestamp": "2020-12-06T11:04:05.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "edwCRnyD","sequence": 1 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "my_index", "_id" : "2"}}
{ "@timestamp": "2020-12-06T11:04:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file" }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{ "@timestamp": "2020-12-06T11:04:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file", "id": "dGCHwoeS", "sequence": 2 }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "my_index", "_id" : "3"}}
{ "@timestamp": "2020-12-07T11:06:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process" }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{ "@timestamp": "2020-12-07T11:06:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "cMyt5SZ2", "sequence": 3 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "my_index", "_id" : "4"}}
{ "@timestamp": "2020-12-07T11:07:08.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file" }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{ "@timestamp": "2020-12-07T11:07:08.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file", "id": "bYA7gPay", "sequence": 4 }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "my_index", "_id" : "5"}}
{ "@timestamp": "2020-12-07T11:07:09.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process" }, "process": { "name": "regsvr32.exe", "path": "C:\\Windows\\System32\\regsvr32.exe" } }
{ "@timestamp": "2020-12-07T11:07:09.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "aR3NWVOs", "sequence": 5 }, "process": { "name": "regsvr32.exe", "path": "C:\\Windows\\System32\\regsvr32.exe" } }
----
// TESTSETUP
////
Expand Down Expand Up @@ -231,7 +231,20 @@ return.
For <<eql-sequences,sequence queries>>, the maximum number of matching sequences
to return.
+
Defaults to `50`. Values must be greater than `0`.
Defaults to `50`. This value must be greater than `0`.

[[eql-search-api-tiebreaker-field]]
`tiebreaker_field`::
(Optional, string)
Field used to sort events with the same
<<eql-search-api-timestamp-field,timestamp field>> value. Defaults to
`event.sequence`, as defined in the {ecs-ref}/ecs-event.html[Elastic Common
Schema (ECS)].
+
By default, matching events in the search response are sorted by timestamp,
converted to milliseconds since the https://en.wikipedia.org/wiki/Unix_time[Unix
epoch], in ascending order. If two or more events share the same timestamp, this
field is used to sort the events in ascending, lexicographic order.

[[eql-search-api-timestamp-field]]
`timestamp_field`::
Expand Down Expand Up @@ -422,11 +435,11 @@ Original JSON body passed for the event at index time.

`sort`::
(array)
Integer used as the sort value for the event.
Array of field values used to sort the event.
+
By default, this is the event's <<eql-search-api-timestamp-field,timestamp
value>>, converted to milliseconds since the
https://en.wikipedia.org/wiki/Unix_time[Unix epoch].
By default, the first item in the array is the event's
<<eql-search-api-timestamp-field,timestamp value>>, converted to milliseconds
since the https://en.wikipedia.org/wiki/Unix_time[Unix epoch].
======
=====

Expand Down Expand Up @@ -460,11 +473,11 @@ Original JSON body passed for the event at index time.

`sort`::
(array)
Integer used as the sort value for the event.
Array of field values used to sort the event.
+
By default, this is the event's <<eql-search-api-timestamp-field,timestamp
value>>, converted to milliseconds since the
https://en.wikipedia.org/wiki/Unix_time[Unix epoch].
By default, the first item in the array is the event's
<<eql-search-api-timestamp-field,timestamp value>>, converted to milliseconds
since the https://en.wikipedia.org/wiki/Unix_time[Unix epoch].
=====
====

Expand Down Expand Up @@ -495,6 +508,10 @@ property are sorted by <<eql-search-api-timestamp-field,timestamp>>, converted
to milliseconds since the https://en.wikipedia.org/wiki/Unix_time[Unix epoch],
in ascending order.

If two or more events share the same timestamp, the
<<eql-search-api-tiebreaker-field,`tiebreaker_field`>> field is used to sort
the events in ascending, lexicographic order.

[source,console-result]
----
{
Expand All @@ -519,7 +536,9 @@ in ascending order.
"id": "8a4f500d"
},
"event": {
"category": "file"
"category": "file",
"id": "dGCHwoeS",
"sequence": 2,
},
"file": {
"accessed": "2020-12-07T11:07:08.000Z",
Expand Down Expand Up @@ -548,7 +567,9 @@ in ascending order.
"id": "8a4f500d"
},
"event": {
"category": "file"
"category": "file",
"id": "bYA7gPay",
"sequence": 4
},
"file": {
"accessed": "2020-12-07T11:07:08.000Z",
Expand Down Expand Up @@ -612,6 +633,10 @@ the `hits.sequences.events` property are sorted by
<<eql-search-api-timestamp-field,timestamp>>, converted to milliseconds since
the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order.

If two or more events share the same timestamp, the
<<eql-search-api-tiebreaker-field,`tiebreaker_field`>> field is used to sort
the events in ascending, lexicographic order.

[source,console-result]
----
{
Expand Down Expand Up @@ -641,7 +666,9 @@ the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order.
"id": "8a4f500d"
},
"event": {
"category": "file"
"category": "file",
"id": "bYA7gPay",
"sequence": 4
},
"file": {
"accessed": "2020-12-07T11:07:08.000Z",
Expand Down Expand Up @@ -675,7 +702,9 @@ the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order.
"id": "8a4f500d"
},
"event": {
"category": "process"
"category": "process",
"id": "aR3NWVOs",
"sequence": 5
},
"process": {
"name": "regsvr32.exe",
Expand Down
146 changes: 135 additions & 11 deletions docs/reference/eql/search.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,15 @@ The following <<docs-bulk,bulk API>> request adds some example log data to the
----
PUT /sec_logs/_bulk?refresh
{"index":{"_index" : "sec_logs", "_id" : "1"}}
{ "@timestamp": "2020-12-06T11:04:05.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process" }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{ "@timestamp": "2020-12-06T11:04:05.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "edwCRnyD","sequence": 1 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "2"}}
{ "@timestamp": "2020-12-06T11:04:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file" }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{ "@timestamp": "2020-12-06T11:04:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file", "id": "dGCHwoeS", "sequence": 2 }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "3"}}
{ "@timestamp": "2020-12-07T11:06:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process" }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{ "@timestamp": "2020-12-07T11:06:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "cMyt5SZ2", "sequence": 3 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "4"}}
{ "@timestamp": "2020-12-07T11:07:08.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file" }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{ "@timestamp": "2020-12-07T11:07:08.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file", "id": "bYA7gPay", "sequence": 4 }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "5"}}
{ "@timestamp": "2020-12-07T11:07:09.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process" }, "process": { "name": "regsvr32.exe", "path": "C:\\Windows\\System32\\regsvr32.exe" } }
{ "@timestamp": "2020-12-07T11:07:09.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "aR3NWVOs", "sequence": 5 }, "process": { "name": "regsvr32.exe", "path": "C:\\Windows\\System32\\regsvr32.exe" } }
----
// TESTSETUP

Expand Down Expand Up @@ -91,7 +91,9 @@ https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order.
"id": "8a4f500d"
},
"event": {
"category": "process"
"category": "process",
"id": "edwCRnyD",
"sequence": 1
},
"process": {
"name": "cmd.exe",
Expand All @@ -113,7 +115,9 @@ https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order.
"id": "8a4f500d"
},
"event": {
"category": "process"
"category": "process",
"id": "cMyt5SZ2",
"sequence": 3
},
"process": {
"name": "cmd.exe",
Expand Down Expand Up @@ -199,7 +203,9 @@ the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order.
"id": "8a4f500d"
},
"event": {
"category": "file"
"category": "file",
"id": "bYA7gPay",
"sequence": 4
},
"file": {
"accessed": "2020-12-07T11:07:08.000Z",
Expand Down Expand Up @@ -233,7 +239,9 @@ the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order.
"id": "8a4f500d"
},
"event": {
"category": "process"
"category": "process",
"id": "aR3NWVOs",
"sequence": 5
},
"process": {
"name": "regsvr32.exe",
Expand Down Expand Up @@ -322,7 +330,9 @@ contains the shared `agent.id` value for each matching event.
"id": "8a4f500d"
},
"event": {
"category": "file"
"category": "file",
"id": "bYA7gPay",
"sequence": 4
},
"file": {
"accessed": "2020-12-07T11:07:08.000Z",
Expand Down Expand Up @@ -356,7 +366,9 @@ contains the shared `agent.id` value for each matching event.
"id": "8a4f500d"
},
"event": {
"category": "process"
"category": "process",
"id": "aR3NWVOs",
"sequence": 5
},
"process": {
"name": "regsvr32.exe",
Expand Down Expand Up @@ -433,6 +445,118 @@ GET /sec_logs/_eql/search
----
====

[discrete]
[[eql-search-specify-a-sort-tiebreaker]]
=== Specify a sort tiebreaker

By default, the EQL search API sorts matching events in the search response by
timestamp. However, if two or more events share the same timestamp, a tiebreaker
field is used to sort the events in ascending, lexicographic order.

The EQL search API uses `event.sequence` as the default tiebreaker field. You
can use the `tiebreaker_field` parameter to specify another field.

.*Example*
[%collapsible]
====
The following request specifies `event.start` as the tiebreaker field.

[source,console]
----
GET /sec_logs/_eql/search
{
"tiebreaker_field": "event.id",
"query": """
process where process.name == "cmd.exe" and stringContains(process.path, "System32")
"""
}
----

The API returns the following response. Note the `sort` property of each
matching event contains an array of two items:

* The first item is the event's <<eql-search-api-timestamp-field,timestamp>>,
converted to milliseconds since the https://en.wikipedia.org/wiki/Unix_time[Unix
epoch].

* The second item is the event's `event.id` value. This value is used as a sort
tiebreaker for events with the same timestamp.

[source,console-result]
----
{
"is_partial": false,
"is_running": false,
"took": 34,
"timed_out": false,
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"events": [
{
"_index": "sec_logs",
"_type": "_doc",
"_id": "1",
"_score": null,
"_source": {
"@timestamp": "2020-12-06T11:04:05.000Z",
"agent": {
"id": "8a4f500d"
},
"event": {
"category": "process",
"id": "edwCRnyD",
"sequence": 1
},
"process": {
"name": "cmd.exe",
"path": "C:\\Windows\\System32\\cmd.exe"
}
},
"sort": [
1607252645000, <1>
"edwCRnyD" <2>
]
},
{
"_index": "sec_logs",
"_type": "_doc",
"_id": "3",
"_score": null,
"_source": {
"@timestamp": "2020-12-07T11:06:07.000Z",
"agent": {
"id": "8a4f500d"
},
"event": {
"category": "process",
"id": "cMyt5SZ2",
"sequence": 3
},
"process": {
"name": "cmd.exe",
"path": "C:\\Windows\\System32\\cmd.exe"
}
},
"sort": [
1607339167000, <1>
"cMyt5SZ2" <2>
]
}
]
}
}
----
// TESTRESPONSE[s/"took": 34/"took": $body.took/]
<1> The event's <<eql-search-api-timestamp-field,timestamp>>, converted to
milliseconds since the https://en.wikipedia.org/wiki/Unix_time[Unix
epoch]
<2> The event's `event.id` value.
====


[discrete]
[[eql-search-filter-query-dsl]]
=== Filter using query DSL
Expand Down