From 05a2540e1c55b7f7983b9faa3184e07d4dfc3ff7 Mon Sep 17 00:00:00 2001 From: Dimitris Athanasiou Date: Thu, 5 Jul 2018 11:41:14 +0100 Subject: [PATCH 01/10] [ML][DOCS] Add documentation for detector rules and filters --- x-pack/docs/build.gradle | 10 + x-pack/docs/en/ml/api-quickref.asciidoc | 9 + x-pack/docs/en/ml/configuring.asciidoc | 4 + .../docs/en/ml/detector-custom-rules.asciidoc | 232 ++++++++++++++++++ x-pack/docs/en/rest-api/defs.asciidoc | 3 + x-pack/docs/en/rest-api/ml-api.asciidoc | 12 + .../en/rest-api/ml/delete-filter.asciidoc | 52 ++++ .../en/rest-api/ml/filterresource.asciidoc | 14 ++ .../docs/en/rest-api/ml/get-filter.asciidoc | 83 +++++++ .../docs/en/rest-api/ml/jobresource.asciidoc | 76 +++++- .../docs/en/rest-api/ml/put-filter.asciidoc | 66 +++++ .../en/rest-api/ml/update-filter.asciidoc | 69 ++++++ .../docs/en/rest-api/ml/update-job.asciidoc | 23 +- 13 files changed, 643 insertions(+), 10 deletions(-) create mode 100644 x-pack/docs/en/ml/detector-custom-rules.asciidoc create mode 100644 x-pack/docs/en/rest-api/ml/delete-filter.asciidoc create mode 100644 x-pack/docs/en/rest-api/ml/filterresource.asciidoc create mode 100644 x-pack/docs/en/rest-api/ml/get-filter.asciidoc create mode 100644 x-pack/docs/en/rest-api/ml/put-filter.asciidoc create mode 100644 x-pack/docs/en/rest-api/ml/update-filter.asciidoc diff --git a/x-pack/docs/build.gradle b/x-pack/docs/build.gradle index 2ae26044be5ab..70f6061985a6a 100644 --- a/x-pack/docs/build.gradle +++ b/x-pack/docs/build.gradle @@ -309,6 +309,16 @@ setups['farequote_datafeed'] = setups['farequote_job'] + ''' "job_id":"farequote", "indexes":"farequote" } +''' +setups['ml_filter_safe_domains'] = ''' + - do: + xpack.ml.put_filter: + filter_id: "safe_domains" + body: > + { + "description": "A list of safe domains", + "items": ["*.google.com", "wikipedia.org"] + } ''' setups['server_metrics_index'] = ''' - do: diff --git a/x-pack/docs/en/ml/api-quickref.asciidoc b/x-pack/docs/en/ml/api-quickref.asciidoc index dc87a6ba209c2..be74167862e15 100644 --- a/x-pack/docs/en/ml/api-quickref.asciidoc +++ b/x-pack/docs/en/ml/api-quickref.asciidoc @@ -47,6 +47,15 @@ The main {ml} resources can be accessed with a variety of endpoints: * {ref}/ml-delete-calendar-job.html[DELETE /calendars/+++/jobs/+++]: Disassociate a job from a calendar * {ref}/ml-delete-calendar.html[DELETE /calendars/+++]: Delete a calendar +[float] +[[ml-api-filters]] +=== /filters/ + +* {ref}/ml-put-filter.html[PUT /filters/+++]: Create a filter +* {ref}/ml-update-filter.html[POST /filters/+++/_update]: Update a filter +* {ref}/ml-get-filter.html[GET /filters/+++]: List filters +* {ref}/ml-delete-filter.html[DELETE /filter/+++]: Delete a filter + [float] [[ml-api-datafeeds]] === /datafeeds/ diff --git a/x-pack/docs/en/ml/configuring.asciidoc b/x-pack/docs/en/ml/configuring.asciidoc index c2c6e69a71128..e35f046a82bd9 100644 --- a/x-pack/docs/en/ml/configuring.asciidoc +++ b/x-pack/docs/en/ml/configuring.asciidoc @@ -34,6 +34,7 @@ The scenarios in this section describe some best practices for generating useful * <> * <> * <> +* <> :edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/ml/customurl.asciidoc include::customurl.asciidoc[] @@ -49,3 +50,6 @@ include::populations.asciidoc[] :edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/ml/transforms.asciidoc include::transforms.asciidoc[] + +:edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/ml/detector-custom-rules.asciidoc +include::detector-custom-rules.asciidoc[] \ No newline at end of file diff --git a/x-pack/docs/en/ml/detector-custom-rules.asciidoc b/x-pack/docs/en/ml/detector-custom-rules.asciidoc new file mode 100644 index 0000000000000..cd751ce6f83e2 --- /dev/null +++ b/x-pack/docs/en/ml/detector-custom-rules.asciidoc @@ -0,0 +1,232 @@ +[role="xpack"] +[[ml-configuring-detector-custom-rules]] +=== Configuring Detector Custom Rules + +Anomaly detectors operate unsupervised and produce results that are +statistically anomalous. However, the {ml} models have no awareness +of the domain of the data being analyzed. For certain use cases, +even though a result may be statistically anomalous, there may be +domain knowledge that renders that result uninteresting. Detector +{ref}/ml-job-resource.html#ml-detector-custom-rule[custom_rules] +provide a way of instructing detectors to change their behaviour +based on such domain specific knowledge. + +Rules describe _when_ a detector should take a certain _action_ instead +of following its default behaviour. To specify the _when_ a rule uses +a `scope` and `conditions`. You can think of `scope` as the categorical +specificiation of a rule, while `conditions` are the numerical part. +A rule can have a scope, one or more conditions, or a combination of +scope and conditions. + +Let us see how those can be configured by examples. + +==== Specifying rule scope + +Let us assume we are configuring a job in order to DNS data exfiltration. +Our data contain fields "subdomain" and "highest_registered_domain". +We can use a detector that looks like `high_info_content(subdomain) over highest_registered_domain`. +If we run such a job it is possible that we discover a lot of anomalies on frequently +used domains that we have reasons to trust. As security analysts, we are not interested in +such anomalies. Ideally, we could instruct the detector to skip results for domains that we consider +safe. Using a rule with a scope allows us to achieve this. + +First, we need to create a list with our safe domains. Those lists are called `filters` in {ml}. +Filters can be shared across jobs. + +We create our filter using the {ref}/ml-put-filter.html[put filter API]: + +[source,js] +---------------------------------- +PUT _xpack/ml/filters/safe_domains +{ + "description": "Our list of safe domains", + "items": ["safe.com", "trusted.com"] +} +---------------------------------- +// CONSOLE + +Now, we can create our job specifying a scope that uses the filter for the `highest_registered_domain` field: + +[source,js] +---------------------------------- +PUT _xpack/ml/anomaly_detectors/dns_exfiltration_with_rule +{ + "analysis_config" : { + "bucket_span":"5m", + "detectors" :[{ + "function":"high_info_content", + "field_name": "subdomain", + "over_field_name": "highest_registered_domain", + "custom_rules": [{ + "actions": ["skip_result"], + "scope": { + "highest_registered_domain": { + "filter_id": "safe_domains", + "filter_type": "include" + } + } + }] + }] + }, + "data_description" : { + "time_field":"timestamp" + } +} +---------------------------------- +// CONSOLE + +As time advances and we see more data and more results, we might encounter new domains that +we want to add in the filter. We can do that by using the {ref}/ml-update-filter.html[update filter API]: + +[source,js] +---------------------------------- +POST _xpack/ml/filters/safe_domains/_update +{ + "add_items": ["another-safe.com"] +} +---------------------------------- +// CONSOLE +// TEST[setup:ml_filter_safe_domains] + +Note that we can provide scope for any of the partition/over/by fields. +In the following example we scope multiple fields: + +[source,js] +---------------------------------- +PUT _xpack/ml/anomaly_detectors/scoping_multiple_fields +{ + "analysis_config" : { + "bucket_span":"5m", + "detectors" :[{ + "function":"count", + "partition_field_name": "my_partition", + "over_field_name": "my_over", + "by_field_name": "my_by", + "custom_rules": [{ + "actions": ["skip_result"], + "scope": { + "my_partition": { + "filter_id": "filter_1" + }, + "my_over": { + "filter_id": "filter_2" + }, + "my_by": { + "filter_id": "filter_3" + } + } + }] + }] + }, + "data_description" : { + "time_field":"timestamp" + } +} +---------------------------------- +// CONSOLE + +Such a detector will skip results when the values of all 3 scoped fields +are included in the referenced filters. + +==== Specifying rule conditions + +Imagine a detector that looks for anomalies in cpu utilization. +Given a machine that is idle for long enough, small movement in cpu could +result in anomalous results where the `actual` value is quite small, e.g. 0.02. +Given our knowledge about how cpu utilization behaves we might determine that +anomalies with such small actual values are not interesting for investigation. + +Let us now configure a job with a rule that will skip results where cpu +utilization is less than 0.20. + +[source,js] +---------------------------------- +PUT _xpack/ml/anomaly_detectors/cpu_with_rule +{ + "analysis_config" : { + "bucket_span":"5m", + "detectors" :[{ + "function":"high_mean", + "field_name": "cpu_utilization", + "custom_rules": [{ + "actions": ["skip_result"], + "conditions": [ + { + "applies_to": "actual", + "operator": "lt", + "value": 0.20 + } + ] + }] + }] + }, + "data_description" : { + "time_field":"timestamp" + } +} +---------------------------------- +// CONSOLE + +When there are multiple conditions they are combined with a logical `and`. +This is useful when we want the rule to apply to a range. We simply create +a rule with two conditions, one for each end of the desired range. + +Here is an example where a cound detector will skip results when the count +is greater than 30 and less than 50: + +[source,js] +---------------------------------- +PUT _xpack/ml/anomaly_detectors/rule_with_range +{ + "analysis_config" : { + "bucket_span":"5m", + "detectors" :[{ + "function":"count", + "custom_rules": [{ + "actions": ["skip_result"], + "conditions": [ + { + "applies_to": "actual", + "operator": "gt", + "value": 30 + }, + { + "applies_to": "actual", + "operator": "lt", + "value": 50 + } + ] + }] + }] + }, + "data_description" : { + "time_field":"timestamp" + } +} +---------------------------------- +// CONSOLE + +==== Rules in the life-cycle of a job + +Rules only apply for results created after the rules were created. +Let us imagine that we have configured a job and it has been running +for some time. After observing its results we decide that we can employ +rules in order to get rid of some uninteresting results. We can use +the update-job API to do so. However, the rule we added will only be in effect +for any results created from the moment we added the rule onwards. Past results +will remain unaffected. + +==== Using rules VS filtering data + +It might appear like using rules is just another way of filtering the data +that feeds into a job. For example, a rule that skips results when the +partition field value is in a filter sounds equivalent to having a query +that filters out such documents. But it is not. There is a fundamental +difference. When the data is filtered before reaching a job it is as if they +do never existed for the job. With rules, the data still reaches the job and +depending the rule actions they are affecting the job. + +For example, a rule with the `skip_result` action means all data will still +be modeled. On the other hand, a rule with the `skip_model_update` action means +results will still be created even though the model will not be updated by +data matched by a rule. diff --git a/x-pack/docs/en/rest-api/defs.asciidoc b/x-pack/docs/en/rest-api/defs.asciidoc index 99600472a0930..349ce343c7ae9 100644 --- a/x-pack/docs/en/rest-api/defs.asciidoc +++ b/x-pack/docs/en/rest-api/defs.asciidoc @@ -8,6 +8,7 @@ job configuration options. * <> * <> * <> +* <> * <> * <> * <> @@ -19,6 +20,8 @@ include::ml/calendarresource.asciidoc[] [role="xpack"] include::ml/datafeedresource.asciidoc[] [role="xpack"] +include::ml/filterresource.asciidoc[] +[role="xpack"] include::ml/jobresource.asciidoc[] [role="xpack"] include::ml/jobcounts.asciidoc[] diff --git a/x-pack/docs/en/rest-api/ml-api.asciidoc b/x-pack/docs/en/rest-api/ml-api.asciidoc index e9a987cc4a709..b48e9f934042d 100644 --- a/x-pack/docs/en/rest-api/ml-api.asciidoc +++ b/x-pack/docs/en/rest-api/ml-api.asciidoc @@ -15,6 +15,14 @@ machine learning APIs and in advanced job configuration options in Kibana. * <>, <> * <>, <> +[float] +[[ml-api-filter-endpoint]] +=== Filters + +* <>, <> +* <> +* <> + [float] [[ml-api-datafeed-endpoint]] === {dfeeds-cap} @@ -69,11 +77,13 @@ include::ml/close-job.asciidoc[] //CREATE include::ml/put-calendar.asciidoc[] include::ml/put-datafeed.asciidoc[] +include::ml/put-filter.asciidoc[] include::ml/put-job.asciidoc[] //DELETE include::ml/delete-calendar.asciidoc[] include::ml/delete-datafeed.asciidoc[] include::ml/delete-calendar-event.asciidoc[] +include::ml/delete-filter.asciidoc[] include::ml/delete-job.asciidoc[] include::ml/delete-calendar-job.asciidoc[] include::ml/delete-snapshot.asciidoc[] @@ -93,6 +103,7 @@ include::ml/get-job.asciidoc[] include::ml/get-job-stats.asciidoc[] include::ml/get-snapshot.asciidoc[] include::ml/get-calendar-event.asciidoc[] +include::ml/get-filter.asciidoc[] include::ml/get-record.asciidoc[] //OPEN include::ml/open-job.asciidoc[] @@ -107,6 +118,7 @@ include::ml/start-datafeed.asciidoc[] include::ml/stop-datafeed.asciidoc[] //UPDATE include::ml/update-datafeed.asciidoc[] +include::ml/update-filter.asciidoc[] include::ml/update-job.asciidoc[] include::ml/update-snapshot.asciidoc[] //VALIDATE diff --git a/x-pack/docs/en/rest-api/ml/delete-filter.asciidoc b/x-pack/docs/en/rest-api/ml/delete-filter.asciidoc new file mode 100644 index 0000000000000..5ff827d61a19b --- /dev/null +++ b/x-pack/docs/en/rest-api/ml/delete-filter.asciidoc @@ -0,0 +1,52 @@ +[role="xpack"] +[[ml-delete-filter]] +=== Delete Filter API +++++ +Delete Filter +++++ + +Deletes a filter. + + +==== Request + +`DELETE _xpack/ml/filters/` + + +==== Description + +This API deletes a filter. A filter is only allowed to be +deleted if it is not used by any job. + + +==== Path Parameters + +`filter_id` (required):: + (string) Identifier for the calendar. + + +==== Authorization + +You must have `manage_ml`, or `manage` cluster privileges to use this API. +For more information, see {xpack-ref}/security-privileges.html[Security Privileges]. + + +==== Examples + +The following example deletes the `safe_domains` filter: + +[source,js] +-------------------------------------------------- +DELETE _xpack/ml/filters/safe_domains +-------------------------------------------------- +// CONSOLE +// TEST[setup:ml_filter_safe_domains] + +When the filter is deleted, you receive the following results: +[source,js] +---- +{ + "acknowledged": true +} +---- +//TESTRESPONSE diff --git a/x-pack/docs/en/rest-api/ml/filterresource.asciidoc b/x-pack/docs/en/rest-api/ml/filterresource.asciidoc new file mode 100644 index 0000000000000..1733c2245f50e --- /dev/null +++ b/x-pack/docs/en/rest-api/ml/filterresource.asciidoc @@ -0,0 +1,14 @@ +[role="xpack"] +[[ml-filter-resource]] +=== Filter Resources + +A filter resource has the following properties: + +`filter_id`:: + (string) A string that uniquely identifies the calendar. + +`description`:: + (array) A description of the filter. + +`items`:: + (array of strings) An array of strings which is the filter item list. diff --git a/x-pack/docs/en/rest-api/ml/get-filter.asciidoc b/x-pack/docs/en/rest-api/ml/get-filter.asciidoc new file mode 100644 index 0000000000000..a5b59dd936940 --- /dev/null +++ b/x-pack/docs/en/rest-api/ml/get-filter.asciidoc @@ -0,0 +1,83 @@ +[role="xpack"] +[[ml-get-filter]] +=== Get Filters API +++++ +Get Filters +++++ + +Retrieves filters. + + +==== Request + +`GET _xpack/ml/filters/` + + +`GET _xpack/ml/filters/` + + +===== Description + +You can get a single filter or all filters. + + +==== Path Parameters + +`filter_id`:: + (string) Identifier for the filter. + + +==== Request Body + +`from`::: + (integer) Skips the specified number of filters. + +`size`::: + (integer) Specifies the maximum number of filters to obtain. + + +==== Results + +The API returns the following information: + +`filters`:: + (array) An array of filter resources. + For more information, see <>. + + +==== Authorization + +You must have `monitor_ml`, `monitor`, `manage_ml`, or `manage` cluster +privileges to use this API. For more information, see +{xpack-ref}/security-privileges.html[Security Privileges]. + + +==== Examples + +The following example gets configuration information for the `safe_domains` +filter: + +[source,js] +-------------------------------------------------- +GET _xpack/ml/filters/safe_domains +-------------------------------------------------- +// CONSOLE +// TEST[setup:ml_filter_safe_domains] + +The API returns the following results: +[source,js] +---- +{ + "count": 1, + "filters": [ + { + "filter_id": "safe_domains", + "description": "A list of safe domains", + "items": [ + "*.google.com", + "wikipedia.org" + ] + } + ] +} +---- +//TESTRESPONSE diff --git a/x-pack/docs/en/rest-api/ml/jobresource.asciidoc b/x-pack/docs/en/rest-api/ml/jobresource.asciidoc index bb959fd728cb6..a16fb9ada20c0 100644 --- a/x-pack/docs/en/rest-api/ml/jobresource.asciidoc +++ b/x-pack/docs/en/rest-api/ml/jobresource.asciidoc @@ -262,7 +262,12 @@ NOTE: The `field_name` cannot contain double quotes or backslashes. `use_null`:: (boolean) Defines whether a new series is used as the null series - when there is no value for the by or partition fields. The default value is `false`. + + when there is no value for the by or partition fields. The default value is `false`. + +`custom_rules`:: + (array) An array of rule objects, which enable customizing how the detector works. + For example, a rule may dictate to the detector conditions under which results should be skipped. + For more information see <>. + + -- IMPORTANT: Field names are case sensitive, for example a field named 'Bytes' @@ -270,9 +275,9 @@ is different from one named 'bytes'. -- -After you create a job, the only property you can change in the detector -configuration object is the `detector_description`; all other properties are -informational. +After you create a job, the only properties you can change in the detector +configuration object are the `detector_description` and the `custom_rules`; +all other properties are informational. [float] [[ml-datadescription]] @@ -408,6 +413,69 @@ the categorization field value came from. For more information, see {xpack-ref}/ml-configuring-categories.html[Categorizing Log Messages]. +[float] +[[ml-detector-custom-rule]] +==== Detector Custom Rule + +Custom rules enable you to customize the way detectors operate. +By configuring rules you can provide domain specific knowledge to the model +so that the quality of the results better matches a specific use case. + +Detectors produce statistically valid anomalies but they have no awareness +of the domain that is being modeled. For example, a `median(cpu)` detector +might pick anomalies where the cpu utilization is very small but statistically +anomalous given the history of a machine. A rule can be employed to instruct +the detector to skip results where the cpu value is small enough. + +Another example would be a `high_count by ip` detector that may result in +anomalies for IP addresses that are known to be safe. Similarly, a rule can +instruct the detector to skip results when the IP is in a safe list. Such safe +lists can be created using `filters` and they can be shared among rules for different jobs. + +WARNING: Careless use of rules may result in degrading the quality of the results. + +A rule has the following properties: + +`actions`:: + (array) The set of actions to be triggered when the rule applies. + The available actions include: + + `skip_result`::: the result will not be created. Note this also means this result + will not impact the scoring of results. The model be normally updated with the + corresponding series value. This is the default action and it serves to skip + results that undesired. + `skip_model_update`::: the value for that series will not be used to update the model. + If there is an anomalous record for that series it will be normally created. This + is suitable when certain values are expected to be consistently anomalous and they + affect the model in a way that negatively impacts the rest of the results. + +`scope`:: + (object) An optional scope of series where the rule applies. By default the scope + includes all series. Scoping is allowed for any of the partition/by/over fields. + To add a scope for a field add the field name as a key in the scope object and + set its value to an object with properties: + `filter_id`:: + (string) The id of the filter to be used + `filter_type`:: + (string) Either `include` (the rule applies for values in the filter) + or `exclude` (the rule applies for values not in the filter). Defaults + to `include`. + +`conditions`:: + (array) An optional array of numeric conditions when the rule applies. + Multiple conditions are combined together with a logical `AND`. + A condition has the following properties: + + `applies_to`::: + (string) Specifies the result property to which the condition applies. + The available options are `actual`, `typical`, `diff_from_typical`, `time`. + `operator`::: + (string) Specifies the condition operator. The available options are + `gt` (greater than), `gte` (greater than or equals), `lt` (less than) and `lte` (less than or equals). + `value`::: + (double) The value that is compared against the `applied_to` field using the `operator`. + +A rule is required to either have a non-empty scope or at least one condition. +For more examples see {xpack-ref}/ml-configuring-detector-custom-rules.html[Configuring Detector Custom Rules]. + [float] [[ml-apilimits]] ==== Analysis Limits diff --git a/x-pack/docs/en/rest-api/ml/put-filter.asciidoc b/x-pack/docs/en/rest-api/ml/put-filter.asciidoc new file mode 100644 index 0000000000000..4c172d7334fd5 --- /dev/null +++ b/x-pack/docs/en/rest-api/ml/put-filter.asciidoc @@ -0,0 +1,66 @@ +[role="xpack"] +[[ml-put-filter]] +=== Create Filter API +++++ +Create Filter +++++ + +Instantiates a filter. + +==== Request + +`PUT _xpack/ml/filters/` + +===== Description + +A filter contains a list of strings and can +be referenced by ML anomaly detectors' `custom_rules`. + +==== Path Parameters + +`filter_id` (required):: + (string) Identifier for the filter. + + +==== Request Body + +`description`:: + (string) A description of the filter. +`items`:: + (array of strings) The items of the filter. + A wildcard `*` can be used at the beginning + or the end of an item. Up to 10000 items + are allowed in each filter. + + +==== Authorization + +You must have `manage_ml`, or `manage` cluster privileges to use this API. +For more information, see +{xpack-ref}/security-privileges.html[Security Privileges]. + + +==== Examples + +The following example creates the `safe_domains` filter: + +[source,js] +-------------------------------------------------- +PUT _xpack/ml/filters/safe_domains +{ + "description": "A list of safe domains", + "items": ["*.google.com", "wikipedia.org"] +} +-------------------------------------------------- +// CONSOLE + +When the filter is created, you receive the following results: +[source,js] +---- +{ + "filter_id": "safe_domains", + "description": "A list of safe domains", + "items": ["*.google.com", "wikipedia.org"] +} +---- +//TESTRESPONSE diff --git a/x-pack/docs/en/rest-api/ml/update-filter.asciidoc b/x-pack/docs/en/rest-api/ml/update-filter.asciidoc new file mode 100644 index 0000000000000..8a1ce131c4981 --- /dev/null +++ b/x-pack/docs/en/rest-api/ml/update-filter.asciidoc @@ -0,0 +1,69 @@ +[role="xpack"] +[[ml-update-filter]] +=== Update Filter API +++++ +Update Filter +++++ + +Posts scheduled events in a calendar. + +==== Request + +`POST _xpack/ml/filters//_update` + + +==== Description + +This API allows updating a filter by changing its +description, or adding/removing items. + +==== Path Parameters + +`filter_id` (required):: + (string) Identifier for the filter. + + +==== Request Body + +`description`:: + (string) A description for the filter. See <>. +`add_items`:: + (array of strings) The items to add to the filter. +`remove_items`:: + (array of strings) The items to remove from the filter. + + +==== Authorization + +You must have `manage_ml`, or `manage` cluster privileges to use this API. +For more information, see +{xpack-ref}/security-privileges.html[Security Privileges]. + + +==== Examples + +You can change the description, add and remove items to the `safe_domains` filter as follows: + +[source,js] +-------------------------------------------------- +POST _xpack/ml/filters/safe_domains/_update +{ + "description": "Updated list of domains", + "add_items": ["*.myorg.com"], + "remove_items": ["wikipedia.org"] +} +-------------------------------------------------- +// CONSOLE +// TEST[setup:ml_filter_safe_domains] + +The API returns the following results: + +[source,js] +---- +{ + "filter_id": "safe_domains", + "description": "Updated list of domains", + "items": ["*.google.com", "*.myorg.com"] +} +---- +//TESTRESPONSE diff --git a/x-pack/docs/en/rest-api/ml/update-job.asciidoc b/x-pack/docs/en/rest-api/ml/update-job.asciidoc index 23046febc2e45..852745e9dd907 100644 --- a/x-pack/docs/en/rest-api/ml/update-job.asciidoc +++ b/x-pack/docs/en/rest-api/ml/update-job.asciidoc @@ -35,6 +35,8 @@ each periodic persistence of the model. See <>. | Yes |`description` |A description of the job. See <>. | No +|`detectors` |An array of <>. | No + |`groups` |A list of job groups. See <>. | No |`model_plot_config`: `enabled` |If true, enables calculation and storage of the @@ -58,12 +60,6 @@ if the job is open when you make the update, you must stop the data feed, close the job, then restart the data feed and open the job for the changes to take effect. -//|`analysis_config`: `detectors`: `detector_index` | A unique identifier of the -//detector. Matches the order of detectors returned by -//<>, starting from 0. | No -//|`analysis_config`: `detectors`: `detector_description` |A description of the -//detector. See <>. | No - [NOTE] -- * You can update the `analysis_limits` only while the job is closed. @@ -73,6 +69,21 @@ of `hard_limit`, this means that it was unable to process some data. You might want to re-run this job with an increased `model_memory_limit`. -- +[[ml-detector-update]] +==== Detector Update Objects + +A detector update object has the following properties: + +`detector_index`:: + (integer) The identifier of the detector to update. + +`description`:: + (string) The new description for the detector. + +`custom_rules`:: + (array) The new list of <> for the detector. + +No other detector property can be updated. ==== Authorization From 9f72c1285b561e9832c988ab0cd42a8589c40a57 Mon Sep 17 00:00:00 2001 From: lcawl Date: Thu, 19 Jul 2018 12:22:19 -0700 Subject: [PATCH 02/10] [DOCS] Fixes small typos --- x-pack/docs/en/ml/detector-custom-rules.asciidoc | 10 +++++----- x-pack/docs/en/rest-api/ml/delete-filter.asciidoc | 7 ++++--- x-pack/docs/en/rest-api/ml/filterresource.asciidoc | 2 +- x-pack/docs/en/rest-api/ml/put-filter.asciidoc | 6 ++++-- x-pack/docs/en/rest-api/ml/update-filter.asciidoc | 12 +++++------- 5 files changed, 19 insertions(+), 18 deletions(-) diff --git a/x-pack/docs/en/ml/detector-custom-rules.asciidoc b/x-pack/docs/en/ml/detector-custom-rules.asciidoc index cd751ce6f83e2..e72f68a6710b8 100644 --- a/x-pack/docs/en/ml/detector-custom-rules.asciidoc +++ b/x-pack/docs/en/ml/detector-custom-rules.asciidoc @@ -1,6 +1,6 @@ [role="xpack"] [[ml-configuring-detector-custom-rules]] -=== Configuring Detector Custom Rules +=== Customizing detectors with rules and filters Anomaly detectors operate unsupervised and produce results that are statistically anomalous. However, the {ml} models have no awareness @@ -171,7 +171,7 @@ When there are multiple conditions they are combined with a logical `and`. This is useful when we want the rule to apply to a range. We simply create a rule with two conditions, one for each end of the desired range. -Here is an example where a cound detector will skip results when the count +Here is an example where a count detector will skip results when the count is greater than 30 and less than 50: [source,js] @@ -208,7 +208,7 @@ PUT _xpack/ml/anomaly_detectors/rule_with_range ==== Rules in the life-cycle of a job -Rules only apply for results created after the rules were created. +Rules only affect results created after the rules were applied. Let us imagine that we have configured a job and it has been running for some time. After observing its results we decide that we can employ rules in order to get rid of some uninteresting results. We can use @@ -223,8 +223,8 @@ that feeds into a job. For example, a rule that skips results when the partition field value is in a filter sounds equivalent to having a query that filters out such documents. But it is not. There is a fundamental difference. When the data is filtered before reaching a job it is as if they -do never existed for the job. With rules, the data still reaches the job and -depending the rule actions they are affecting the job. +never existed for the job. With rules, the data still reaches the job and +affects its behavior (depending on the rule actions). For example, a rule with the `skip_result` action means all data will still be modeled. On the other hand, a rule with the `skip_model_update` action means diff --git a/x-pack/docs/en/rest-api/ml/delete-filter.asciidoc b/x-pack/docs/en/rest-api/ml/delete-filter.asciidoc index 5ff827d61a19b..be318d6c38bda 100644 --- a/x-pack/docs/en/rest-api/ml/delete-filter.asciidoc +++ b/x-pack/docs/en/rest-api/ml/delete-filter.asciidoc @@ -15,14 +15,15 @@ Deletes a filter. ==== Description -This API deletes a filter. A filter is only allowed to be -deleted if it is not used by any job. +This API deletes a filter. +If a {ml} job references the filter, you cannot delete the filter. You must +update or delete the job before you can delete the filter. ==== Path Parameters `filter_id` (required):: - (string) Identifier for the calendar. + (string) Identifier for the filter. ==== Authorization diff --git a/x-pack/docs/en/rest-api/ml/filterresource.asciidoc b/x-pack/docs/en/rest-api/ml/filterresource.asciidoc index 1733c2245f50e..26aca1b7c8f47 100644 --- a/x-pack/docs/en/rest-api/ml/filterresource.asciidoc +++ b/x-pack/docs/en/rest-api/ml/filterresource.asciidoc @@ -5,7 +5,7 @@ A filter resource has the following properties: `filter_id`:: - (string) A string that uniquely identifies the calendar. + (string) A string that uniquely identifies the filter. `description`:: (array) A description of the filter. diff --git a/x-pack/docs/en/rest-api/ml/put-filter.asciidoc b/x-pack/docs/en/rest-api/ml/put-filter.asciidoc index 4c172d7334fd5..2cdaae413fedd 100644 --- a/x-pack/docs/en/rest-api/ml/put-filter.asciidoc +++ b/x-pack/docs/en/rest-api/ml/put-filter.asciidoc @@ -13,8 +13,9 @@ Instantiates a filter. ===== Description -A filter contains a list of strings and can -be referenced by ML anomaly detectors' `custom_rules`. +A filter contains a list of strings. +It can be used by one or more jobs. Specifically, filters are referenced in +the `custom_rules` property of <>. ==== Path Parameters @@ -26,6 +27,7 @@ be referenced by ML anomaly detectors' `custom_rules`. `description`:: (string) A description of the filter. + `items`:: (array of strings) The items of the filter. A wildcard `*` can be used at the beginning diff --git a/x-pack/docs/en/rest-api/ml/update-filter.asciidoc b/x-pack/docs/en/rest-api/ml/update-filter.asciidoc index 8a1ce131c4981..1b6760dfed654 100644 --- a/x-pack/docs/en/rest-api/ml/update-filter.asciidoc +++ b/x-pack/docs/en/rest-api/ml/update-filter.asciidoc @@ -5,17 +5,13 @@ Update Filter ++++ -Posts scheduled events in a calendar. +Updates the description of a filter, adds items, or removes items. ==== Request `POST _xpack/ml/filters//_update` - -==== Description - -This API allows updating a filter by changing its -description, or adding/removing items. +//==== Description ==== Path Parameters @@ -26,9 +22,11 @@ description, or adding/removing items. ==== Request Body `description`:: - (string) A description for the filter. See <>. + (string) A description for the filter. See <>. + `add_items`:: (array of strings) The items to add to the filter. + `remove_items`:: (array of strings) The items to remove from the filter. From 882a60291a6da96c56ff7a7b037112c3ea8c89e4 Mon Sep 17 00:00:00 2001 From: lcawl Date: Thu, 19 Jul 2018 16:35:30 -0700 Subject: [PATCH 03/10] [DOCS] Addresses review feedback --- .../docs/en/rest-api/ml/jobresource.asciidoc | 20 +++++++++---------- .../docs/en/rest-api/ml/put-filter.asciidoc | 2 +- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/x-pack/docs/en/rest-api/ml/jobresource.asciidoc b/x-pack/docs/en/rest-api/ml/jobresource.asciidoc index a16fb9ada20c0..8552e83ba2845 100644 --- a/x-pack/docs/en/rest-api/ml/jobresource.asciidoc +++ b/x-pack/docs/en/rest-api/ml/jobresource.asciidoc @@ -439,22 +439,22 @@ A rule has the following properties: `actions`:: (array) The set of actions to be triggered when the rule applies. The available actions include: + - `skip_result`::: the result will not be created. Note this also means this result - will not impact the scoring of results. The model be normally updated with the - corresponding series value. This is the default action and it serves to skip - results that undesired. - `skip_model_update`::: the value for that series will not be used to update the model. - If there is an anomalous record for that series it will be normally created. This - is suitable when certain values are expected to be consistently anomalous and they - affect the model in a way that negatively impacts the rest of the results. - + `skip_result`::: The result will not be created. Note this also means this + result will not impact the scoring of results. This is the default value. + Unless you also specify `skip_model_update`, the model will be updated as + usual with the corresponding series value. + `skip_model_update`::: The value for that series will not be used to update + the model. Unless you also specify `skip_result`, the results will be created + as usual. This action is suitable when certain values are expected to be + consistently anomalous and they affect the model in a way that negatively + impacts the rest of the results. `scope`:: (object) An optional scope of series where the rule applies. By default the scope includes all series. Scoping is allowed for any of the partition/by/over fields. To add a scope for a field add the field name as a key in the scope object and set its value to an object with properties: `filter_id`:: - (string) The id of the filter to be used + (string) The id of the <> to be used. `filter_type`:: (string) Either `include` (the rule applies for values in the filter) or `exclude` (the rule applies for values not in the filter). Defaults diff --git a/x-pack/docs/en/rest-api/ml/put-filter.asciidoc b/x-pack/docs/en/rest-api/ml/put-filter.asciidoc index 2cdaae413fedd..4ce1b66ba06c2 100644 --- a/x-pack/docs/en/rest-api/ml/put-filter.asciidoc +++ b/x-pack/docs/en/rest-api/ml/put-filter.asciidoc @@ -56,7 +56,7 @@ PUT _xpack/ml/filters/safe_domains -------------------------------------------------- // CONSOLE -When the filter is created, you receive the following results: +When the filter is created, you receive the following response: [source,js] ---- { From 635aea5c13366c4f0bbf906a7f9d321fafd44f83 Mon Sep 17 00:00:00 2001 From: lcawl Date: Thu, 19 Jul 2018 17:14:31 -0700 Subject: [PATCH 04/10] [DOCS] Moves overview info to Stack Overview --- .../docs/en/ml/detector-custom-rules.asciidoc | 49 +++++++++---------- .../docs/en/rest-api/ml/jobresource.asciidoc | 19 ++----- 2 files changed, 27 insertions(+), 41 deletions(-) diff --git a/x-pack/docs/en/ml/detector-custom-rules.asciidoc b/x-pack/docs/en/ml/detector-custom-rules.asciidoc index e72f68a6710b8..e9a284b825729 100644 --- a/x-pack/docs/en/ml/detector-custom-rules.asciidoc +++ b/x-pack/docs/en/ml/detector-custom-rules.asciidoc @@ -2,19 +2,14 @@ [[ml-configuring-detector-custom-rules]] === Customizing detectors with rules and filters -Anomaly detectors operate unsupervised and produce results that are -statistically anomalous. However, the {ml} models have no awareness -of the domain of the data being analyzed. For certain use cases, -even though a result may be statistically anomalous, there may be -domain knowledge that renders that result uninteresting. Detector -{ref}/ml-job-resource.html#ml-detector-custom-rule[custom_rules] -provide a way of instructing detectors to change their behaviour -based on such domain specific knowledge. +Rules and filters enable you to change the behavior of anomaly detectors based +on domain-specific knowledge. +//TO-DO: Add link to rules overview page Rules describe _when_ a detector should take a certain _action_ instead -of following its default behaviour. To specify the _when_ a rule uses +of following its default behavior. To specify the _when_ a rule uses a `scope` and `conditions`. You can think of `scope` as the categorical -specificiation of a rule, while `conditions` are the numerical part. +specification of a rule, while `conditions` are the numerical part. A rule can have a scope, one or more conditions, or a combination of scope and conditions. @@ -25,13 +20,14 @@ Let us see how those can be configured by examples. Let us assume we are configuring a job in order to DNS data exfiltration. Our data contain fields "subdomain" and "highest_registered_domain". We can use a detector that looks like `high_info_content(subdomain) over highest_registered_domain`. -If we run such a job it is possible that we discover a lot of anomalies on frequently -used domains that we have reasons to trust. As security analysts, we are not interested in -such anomalies. Ideally, we could instruct the detector to skip results for domains that we consider -safe. Using a rule with a scope allows us to achieve this. +If we run such a job it is possible that we discover a lot of anomalies on +frequently used domains that we have reasons to trust. As security analysts, we +are not interested in such anomalies. Ideally, we could instruct the detector to +skip results for domains that we consider safe. Using a rule with a scope allows +us to achieve this. -First, we need to create a list with our safe domains. Those lists are called `filters` in {ml}. -Filters can be shared across jobs. +First, we need to create a list with our safe domains. Those lists are called +`filters` in {ml}. Filters can be shared across jobs. We create our filter using the {ref}/ml-put-filter.html[put filter API]: @@ -45,7 +41,8 @@ PUT _xpack/ml/filters/safe_domains ---------------------------------- // CONSOLE -Now, we can create our job specifying a scope that uses the filter for the `highest_registered_domain` field: +Now, we can create our job specifying a scope that uses the filter for the +`highest_registered_domain` field: [source,js] ---------------------------------- @@ -75,8 +72,9 @@ PUT _xpack/ml/anomaly_detectors/dns_exfiltration_with_rule ---------------------------------- // CONSOLE -As time advances and we see more data and more results, we might encounter new domains that -we want to add in the filter. We can do that by using the {ref}/ml-update-filter.html[update filter API]: +As time advances and we see more data and more results, we might encounter new +domains that we want to add in the filter. We can do that by using the +{ref}/ml-update-filter.html[update filter API]: [source,js] ---------------------------------- @@ -130,13 +128,14 @@ are included in the referenced filters. ==== Specifying rule conditions -Imagine a detector that looks for anomalies in cpu utilization. -Given a machine that is idle for long enough, small movement in cpu could -result in anomalous results where the `actual` value is quite small, e.g. 0.02. -Given our knowledge about how cpu utilization behaves we might determine that -anomalies with such small actual values are not interesting for investigation. +Imagine a detector that looks for anomalies in CPU utilization. +Given a machine that is idle for long enough, small movement in CPU could +result in anomalous results where the `actual` value is quite small, for +example, 0.02. Given our knowledge about how CPU utilization behaves we might +determine that anomalies with such small actual values are not interesting for +investigation. -Let us now configure a job with a rule that will skip results where cpu +Let us now configure a job with a rule that will skip results where CPU utilization is less than 0.20. [source,js] diff --git a/x-pack/docs/en/rest-api/ml/jobresource.asciidoc b/x-pack/docs/en/rest-api/ml/jobresource.asciidoc index 8552e83ba2845..becf5497f27c4 100644 --- a/x-pack/docs/en/rest-api/ml/jobresource.asciidoc +++ b/x-pack/docs/en/rest-api/ml/jobresource.asciidoc @@ -418,21 +418,7 @@ For more information, see ==== Detector Custom Rule Custom rules enable you to customize the way detectors operate. -By configuring rules you can provide domain specific knowledge to the model -so that the quality of the results better matches a specific use case. - -Detectors produce statistically valid anomalies but they have no awareness -of the domain that is being modeled. For example, a `median(cpu)` detector -might pick anomalies where the cpu utilization is very small but statistically -anomalous given the history of a machine. A rule can be employed to instruct -the detector to skip results where the cpu value is small enough. - -Another example would be a `high_count by ip` detector that may result in -anomalies for IP addresses that are known to be safe. Similarly, a rule can -instruct the detector to skip results when the IP is in a safe list. Such safe -lists can be created using `filters` and they can be shared among rules for different jobs. - -WARNING: Careless use of rules may result in degrading the quality of the results. +//TO-DO: Add link to rules overview A rule has the following properties: @@ -474,7 +460,8 @@ A rule has the following properties: (double) The value that is compared against the `applied_to` field using the `operator`. A rule is required to either have a non-empty scope or at least one condition. -For more examples see {xpack-ref}/ml-configuring-detector-custom-rules.html[Configuring Detector Custom Rules]. +For more examples see +{stack-ov}/ml-configuring-detector-custom-rules.html[Configuring Detector Custom Rules]. [float] [[ml-apilimits]] From 5b255f2dcd3d89b71a6419486a55f168e27a1f48 Mon Sep 17 00:00:00 2001 From: lcawl Date: Fri, 20 Jul 2018 09:15:24 -0700 Subject: [PATCH 05/10] [DOCS] Added rule limitations to function reference --- x-pack/docs/en/ml/functions/geo.asciidoc | 2 ++ x-pack/docs/en/ml/functions/metric.asciidoc | 3 +++ x-pack/docs/en/ml/functions/rare.asciidoc | 2 ++ x-pack/docs/en/rest-api/ml/jobresource.asciidoc | 2 +- 4 files changed, 8 insertions(+), 1 deletion(-) diff --git a/x-pack/docs/en/ml/functions/geo.asciidoc b/x-pack/docs/en/ml/functions/geo.asciidoc index e9685b46e1677..5bcf6c3394558 100644 --- a/x-pack/docs/en/ml/functions/geo.asciidoc +++ b/x-pack/docs/en/ml/functions/geo.asciidoc @@ -8,6 +8,8 @@ input data. The {xpackml} features include the following geographic function: `lat_long`. NOTE: You cannot create forecasts for jobs that contain geographic functions. +You also cannot add rules with conditions to detectors that use geographic +functions. [float] [[ml-lat-long]] diff --git a/x-pack/docs/en/ml/functions/metric.asciidoc b/x-pack/docs/en/ml/functions/metric.asciidoc index 3ee5179702720..eba7c439f53f7 100644 --- a/x-pack/docs/en/ml/functions/metric.asciidoc +++ b/x-pack/docs/en/ml/functions/metric.asciidoc @@ -15,6 +15,9 @@ The {xpackml} features include the following metric functions: * <> * xref:ml-metric-varp[`varp`, `high_varp`, `low_varp`] +NOTE: You cannot add rules with conditions to detectors that use metric +functions. + [float] [[ml-metric-min]] ==== Min diff --git a/x-pack/docs/en/ml/functions/rare.asciidoc b/x-pack/docs/en/ml/functions/rare.asciidoc index fc30918b508f1..1531285a7add2 100644 --- a/x-pack/docs/en/ml/functions/rare.asciidoc +++ b/x-pack/docs/en/ml/functions/rare.asciidoc @@ -15,6 +15,8 @@ number of times (frequency) rare values occur. `exclude_frequent`. * You cannot create forecasts for jobs that contain `rare` or `freq_rare` functions. +* You cannot add rules with conditions to detectors that use `rare` or +`freq_rare` functions. * Shorter bucket spans (less than 1 hour, for example) are recommended when looking for rare events. The functions model whether something happens in a bucket at least once. With longer bucket spans, it is more likely that diff --git a/x-pack/docs/en/rest-api/ml/jobresource.asciidoc b/x-pack/docs/en/rest-api/ml/jobresource.asciidoc index becf5497f27c4..4861b3ba67a63 100644 --- a/x-pack/docs/en/rest-api/ml/jobresource.asciidoc +++ b/x-pack/docs/en/rest-api/ml/jobresource.asciidoc @@ -245,7 +245,7 @@ NOTE: The `field_name` cannot contain double quotes or backslashes. -- `function`:: - (string) The analysis function that is used. + (string) The analysis function that is used. For example, `count`, `rare`, `mean`, `min`, `max`, and `sum`. For more information, see {xpack-ref}/ml-functions.html[Function Reference]. From b2d74f8e8aba19276d874bcf7296ced3ab0c3200 Mon Sep 17 00:00:00 2001 From: lcawl Date: Fri, 20 Jul 2018 09:41:32 -0700 Subject: [PATCH 06/10] [DOCS] Clarifies rule restriction for metric function --- x-pack/docs/en/ml/functions/metric.asciidoc | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/x-pack/docs/en/ml/functions/metric.asciidoc b/x-pack/docs/en/ml/functions/metric.asciidoc index eba7c439f53f7..9d6f3515a029c 100644 --- a/x-pack/docs/en/ml/functions/metric.asciidoc +++ b/x-pack/docs/en/ml/functions/metric.asciidoc @@ -15,8 +15,8 @@ The {xpackml} features include the following metric functions: * <> * xref:ml-metric-varp[`varp`, `high_varp`, `low_varp`] -NOTE: You cannot add rules with conditions to detectors that use metric -functions. +NOTE: You cannot add rules with conditions to detectors that use the `metric` +function. [float] [[ml-metric-min]] @@ -224,7 +224,6 @@ mean `responsetime` for each application over time. It detects when the mean The `metric` function combines `min`, `max`, and `mean` functions. You can use it as a shorthand for a combined analysis. If you do not specify a function in a detector, this is the default function. -//TBD: Is that default behavior still true? High- and low-sided functions are not applicable. You cannot use this function when a `summary_count_field_name` is specified. From 2bb56233eaac251b7dc7ac5af6385654377ebe11 Mon Sep 17 00:00:00 2001 From: lcawl Date: Fri, 20 Jul 2018 09:49:52 -0700 Subject: [PATCH 07/10] [DOCS] Adds function limitations to rule conditions --- x-pack/docs/en/rest-api/ml/jobresource.asciidoc | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/x-pack/docs/en/rest-api/ml/jobresource.asciidoc b/x-pack/docs/en/rest-api/ml/jobresource.asciidoc index 4861b3ba67a63..0f60ded6503df 100644 --- a/x-pack/docs/en/rest-api/ml/jobresource.asciidoc +++ b/x-pack/docs/en/rest-api/ml/jobresource.asciidoc @@ -458,6 +458,12 @@ A rule has the following properties: `gt` (greater than), `gte` (greater than or equals), `lt` (less than) and `lte` (less than or equals). `value`::: (double) The value that is compared against the `applied_to` field using the `operator`. ++ +-- +NOTE: If your detector uses `lat_long`, `metric`, `rare`, or `freq_rare` functions, +you cannot specify `conditions` for your rule. + +-- A rule is required to either have a non-empty scope or at least one condition. For more examples see From d8747407fbfe085f0c03292d37e320c687318ad5 Mon Sep 17 00:00:00 2001 From: lcawl Date: Fri, 20 Jul 2018 09:58:48 -0700 Subject: [PATCH 08/10] [DOCS] Fixes formatting of limitations in rule conditions --- .../docs/en/rest-api/ml/jobresource.asciidoc | 20 ++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/x-pack/docs/en/rest-api/ml/jobresource.asciidoc b/x-pack/docs/en/rest-api/ml/jobresource.asciidoc index 0f60ded6503df..98602602a562f 100644 --- a/x-pack/docs/en/rest-api/ml/jobresource.asciidoc +++ b/x-pack/docs/en/rest-api/ml/jobresource.asciidoc @@ -449,20 +449,22 @@ A rule has the following properties: `conditions`:: (array) An optional array of numeric conditions when the rule applies. Multiple conditions are combined together with a logical `AND`. - A condition has the following properties: + - `applies_to`::: ++ +-- +NOTE: If your detector uses `lat_long`, `metric`, `rare`, or `freq_rare` +functions, you cannot specify `conditions` for your rule. + + +A condition has the following properties: + +`applies_to`::: (string) Specifies the result property to which the condition applies. The available options are `actual`, `typical`, `diff_from_typical`, `time`. - `operator`::: +`operator`::: (string) Specifies the condition operator. The available options are `gt` (greater than), `gte` (greater than or equals), `lt` (less than) and `lte` (less than or equals). - `value`::: +`value`::: (double) The value that is compared against the `applied_to` field using the `operator`. -+ --- -NOTE: If your detector uses `lat_long`, `metric`, `rare`, or `freq_rare` functions, -you cannot specify `conditions` for your rule. - -- A rule is required to either have a non-empty scope or at least one condition. From 0f8b4afc13e9276e146a7662a361bb5ab1e10714 Mon Sep 17 00:00:00 2001 From: Dimitris Athanasiou Date: Mon, 23 Jul 2018 12:33:56 +0100 Subject: [PATCH 09/10] [DOCS] Remove confusing comment about results scoring for skip_result --- .../docs/en/rest-api/ml/jobresource.asciidoc | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/x-pack/docs/en/rest-api/ml/jobresource.asciidoc b/x-pack/docs/en/rest-api/ml/jobresource.asciidoc index 98602602a562f..283f18bf2390e 100644 --- a/x-pack/docs/en/rest-api/ml/jobresource.asciidoc +++ b/x-pack/docs/en/rest-api/ml/jobresource.asciidoc @@ -424,16 +424,16 @@ A rule has the following properties: `actions`:: (array) The set of actions to be triggered when the rule applies. + If more than one action is specified the effects of all actions are combined. The available actions include: + - `skip_result`::: The result will not be created. Note this also means this - result will not impact the scoring of results. This is the default value. - Unless you also specify `skip_model_update`, the model will be updated as - usual with the corresponding series value. - `skip_model_update`::: The value for that series will not be used to update - the model. Unless you also specify `skip_result`, the results will be created - as usual. This action is suitable when certain values are expected to be - consistently anomalous and they affect the model in a way that negatively - impacts the rest of the results. + `skip_result`::: The result will not be created. This is the default value. + Unless you also specify `skip_model_update`, the model will be updated as + usual with the corresponding series value. + `skip_model_update`::: The value for that series will not be used to update + the model. Unless you also specify `skip_result`, the results will be created + as usual. This action is suitable when certain values are expected to be + consistently anomalous and they affect the model in a way that negatively + impacts the rest of the results. `scope`:: (object) An optional scope of series where the rule applies. By default the scope includes all series. Scoping is allowed for any of the partition/by/over fields. From fc47818d0127615257d0fc101b1fe87887083f2d Mon Sep 17 00:00:00 2001 From: lcawl Date: Tue, 24 Jul 2018 10:30:58 -0700 Subject: [PATCH 10/10] [DOCS] Adds links to rules overview --- x-pack/docs/en/ml/detector-custom-rules.asciidoc | 5 ++--- x-pack/docs/en/rest-api/ml/delete-filter.asciidoc | 2 +- x-pack/docs/en/rest-api/ml/filterresource.asciidoc | 2 ++ x-pack/docs/en/rest-api/ml/get-filter.asciidoc | 3 ++- x-pack/docs/en/rest-api/ml/jobresource.asciidoc | 4 ++-- x-pack/docs/en/rest-api/ml/put-filter.asciidoc | 2 +- 6 files changed, 10 insertions(+), 8 deletions(-) diff --git a/x-pack/docs/en/ml/detector-custom-rules.asciidoc b/x-pack/docs/en/ml/detector-custom-rules.asciidoc index e9a284b825729..18d516fae2de0 100644 --- a/x-pack/docs/en/ml/detector-custom-rules.asciidoc +++ b/x-pack/docs/en/ml/detector-custom-rules.asciidoc @@ -2,9 +2,8 @@ [[ml-configuring-detector-custom-rules]] === Customizing detectors with rules and filters -Rules and filters enable you to change the behavior of anomaly detectors based -on domain-specific knowledge. -//TO-DO: Add link to rules overview page +<> enable you to change the behavior of anomaly +detectors based on domain-specific knowledge. Rules describe _when_ a detector should take a certain _action_ instead of following its default behavior. To specify the _when_ a rule uses diff --git a/x-pack/docs/en/rest-api/ml/delete-filter.asciidoc b/x-pack/docs/en/rest-api/ml/delete-filter.asciidoc index be318d6c38bda..b58d2980b888a 100644 --- a/x-pack/docs/en/rest-api/ml/delete-filter.asciidoc +++ b/x-pack/docs/en/rest-api/ml/delete-filter.asciidoc @@ -15,7 +15,7 @@ Deletes a filter. ==== Description -This API deletes a filter. +This API deletes a {stack-ov}/ml-rules.html[filter]. If a {ml} job references the filter, you cannot delete the filter. You must update or delete the job before you can delete the filter. diff --git a/x-pack/docs/en/rest-api/ml/filterresource.asciidoc b/x-pack/docs/en/rest-api/ml/filterresource.asciidoc index 26aca1b7c8f47..64768da4911c4 100644 --- a/x-pack/docs/en/rest-api/ml/filterresource.asciidoc +++ b/x-pack/docs/en/rest-api/ml/filterresource.asciidoc @@ -12,3 +12,5 @@ A filter resource has the following properties: `items`:: (array of strings) An array of strings which is the filter item list. + +For more information, see {stack-ov}/ml-rules.html[Machine learning rules and filters]. diff --git a/x-pack/docs/en/rest-api/ml/get-filter.asciidoc b/x-pack/docs/en/rest-api/ml/get-filter.asciidoc index a5b59dd936940..89f40cf331251 100644 --- a/x-pack/docs/en/rest-api/ml/get-filter.asciidoc +++ b/x-pack/docs/en/rest-api/ml/get-filter.asciidoc @@ -17,7 +17,8 @@ Retrieves filters. ===== Description -You can get a single filter or all filters. +You can get a single filter or all filters. For more information, see +{stack-ov}/ml-rules.html[Machine learning rules and filters]. ==== Path Parameters diff --git a/x-pack/docs/en/rest-api/ml/jobresource.asciidoc b/x-pack/docs/en/rest-api/ml/jobresource.asciidoc index 283f18bf2390e..e4b6da11dc5f4 100644 --- a/x-pack/docs/en/rest-api/ml/jobresource.asciidoc +++ b/x-pack/docs/en/rest-api/ml/jobresource.asciidoc @@ -417,8 +417,8 @@ For more information, see [[ml-detector-custom-rule]] ==== Detector Custom Rule -Custom rules enable you to customize the way detectors operate. -//TO-DO: Add link to rules overview +{stack-ov}/ml-rules.html[Custom rules] enable you to customize the way detectors +operate. A rule has the following properties: diff --git a/x-pack/docs/en/rest-api/ml/put-filter.asciidoc b/x-pack/docs/en/rest-api/ml/put-filter.asciidoc index 4ce1b66ba06c2..d2982a56f612e 100644 --- a/x-pack/docs/en/rest-api/ml/put-filter.asciidoc +++ b/x-pack/docs/en/rest-api/ml/put-filter.asciidoc @@ -13,7 +13,7 @@ Instantiates a filter. ===== Description -A filter contains a list of strings. +A {stack-ov}/ml-rules.html[filter] contains a list of strings. It can be used by one or more jobs. Specifically, filters are referenced in the `custom_rules` property of <>.