diff --git a/docs/reference/rest-api/defs.asciidoc b/docs/reference/rest-api/defs.asciidoc index 65a0384a3d3bf..09306ce1961fb 100644 --- a/docs/reference/rest-api/defs.asciidoc +++ b/docs/reference/rest-api/defs.asciidoc @@ -15,6 +15,7 @@ These resource definitions are used in APIs related to {ml-features} and * <> * <> * <> +* <> * <> include::{es-repo-dir}/ml/apis/calendarresource.asciidoc[] @@ -25,5 +26,6 @@ include::{es-repo-dir}/ml/apis/jobcounts.asciidoc[] include::{es-repo-dir}/ml/apis/snapshotresource.asciidoc[] include::{es-repo-dir}/data-frames/apis/pivotresource.asciidoc[] include::{xes-repo-dir}/rest-api/security/role-mapping-resources.asciidoc[] +include::{es-repo-dir}/rollup/apis/rollup-jobs-objects.asciidoc[] include::{es-repo-dir}/ml/apis/resultsresource.asciidoc[] include::{es-repo-dir}/ml/apis/eventresource.asciidoc[] diff --git a/docs/reference/rollup/apis/delete-job.asciidoc b/docs/reference/rollup/apis/delete-job.asciidoc index 18c353ac73674..d08bbf43d9723 100644 --- a/docs/reference/rollup/apis/delete-job.asciidoc +++ b/docs/reference/rollup/apis/delete-job.asciidoc @@ -6,22 +6,45 @@ Delete job ++++ +Deletes an existing rollup job. + experimental[] -This API deletes an existing rollup job. A job must be *stopped* first before it can be deleted. Attempting to delete -a started job will result in an error. Similarly, attempting to delete a nonexistent job will throw an exception. +[float] +[[rollup-delete-job-request]] +==== {api-request-title} + +`DELETE _rollup/job/` + +[float] +[[rollup-delete-job-prereqs]] +==== {api-prereq-title} -.Deleting the job does not delete rolled up data -********************************** -When a job is deleted, that only removes the process that is actively monitoring and rolling up data. -It does not delete any previously rolled up data. This is by design; a user may wish to roll up a static dataset. Because -the dataset is static, once it has been fully rolled up there is no need to keep the indexing Rollup job around (as there -will be no new data). So the job may be deleted, leaving behind the rolled up data for analysis. +If the {es} {security-features} are enabled, you must have `manage` or +`manage_rollup` cluster privileges to use this API. For more information, see +{stack-ov}/security-privileges.html[Security privileges]. -If you wish to also remove the rollup data, and the rollup index only contains the data for a single job, you can simply -delete the whole rollup index. If the rollup index stores data from several jobs, you must issue a Delete-By-Query that -targets the Rollup job's ID in the rollup index: +[float] +[[rollup-delete-job-desc]] +==== {api-description-title} +A job must be *stopped* first before it can be deleted. If you attempt to delete +a started job, an error occurs. Similarly, if you attempt to delete a +nonexistent job, an exception occurs. + +[IMPORTANT] +=============================== +When a job is deleted, that only removes the process that is actively monitoring +and rolling up data. It does not delete any previously rolled up data. This is +by design; a user may wish to roll up a static dataset. Because the dataset is +static, once it has been fully rolled up there is no need to keep the indexing +rollup job around (as there will be no new data). So the job can be deleted, +leaving behind the rolled up data for analysis. + +If you wish to also remove the rollup data, and the rollup index only contains +the data for a single job, you can simply delete the whole rollup index. If the +rollup index stores data from several jobs, you must issue a delete-by-query +that targets the rollup job's ID in the rollup index. [source,js] -------------------------------------------------- @@ -35,34 +58,20 @@ POST my_rollup_index/_delete_by_query } -------------------------------------------------- // NOTCONSOLE +=============================== -********************************** -==== Request - -`DELETE _rollup/job/` - -//===== Description - -==== Path Parameters - -`job_id` (required):: - (string) Identifier for the job - - -==== Request Body - -There is no request body for the Delete Job API. - -==== Authorization +[float] +[[rollup-delete-job-path-params]] +==== {api-path-parms-title} -You must have `manage` or `manage_rollup` cluster privileges to use this API. -For more information, see -{xpack-ref}/security-privileges.html[Security Privileges]. +`` (Required):: + (string) Identifier for the job. +[float] +[[rollup-delete-job-example]] +==== {api-example-title} -==== Examples - -If we have a rollup job named `sensor`, it can be deleted with: +If you have a rollup job named `sensor`, you can delete it as follows: [source,js] -------------------------------------------------- @@ -71,7 +80,7 @@ DELETE _rollup/job/sensor // CONSOLE // TEST[setup:sensor_rollup_job] -Which will return the response: +The API returns the following response: [source,js] ---- @@ -80,34 +89,3 @@ Which will return the response: } ---- // TESTRESPONSE - -If however we try to delete a job which doesn't exist: - -[source,js] --------------------------------------------------- -DELETE _rollup/job/does_not_exist --------------------------------------------------- -// CONSOLE -// TEST[catch:missing] - -A 404 `resource_not_found` exception will be thrown: - -[source,js] ----- -{ - "error" : { - "root_cause" : [ - { - "type" : "resource_not_found_exception", - "reason" : "the task with id [does_not_exist] doesn't exist", - "stack_trace": ... - } - ], - "type" : "resource_not_found_exception", - "reason" : "the task with id [does_not_exist] doesn't exist", - "stack_trace": ... - }, - "status": 404 -} ----- -// TESTRESPONSE[s/"stack_trace": .../"stack_trace": $body.$_path/] diff --git a/docs/reference/rollup/apis/get-job.asciidoc b/docs/reference/rollup/apis/get-job.asciidoc index 27410b8d39b7d..d4bb9c415dbfb 100644 --- a/docs/reference/rollup/apis/get-job.asciidoc +++ b/docs/reference/rollup/apis/get-job.asciidoc @@ -6,41 +6,69 @@ Get job ++++ +Returns the configuration, stats and status of rollup jobs. + experimental[] -This API returns the configuration, stats and status of rollup jobs. The API can return the details for a single job, -or for all jobs. +[float] +[[rollup-get-job-request]] +==== {api-request-title} -Note: This API only returns active (both `STARTED` and `STOPPED`) jobs. If a job was created, ran for a while then deleted, -this API will not return any details about that job. +`GET _rollup/job/` -For details about a historical job, the <> may be more useful +[float] +[[rollup-get-job-prereqs]] +==== {api-prereq-title} -==== Request +If the {es} {security-features} are enabled, you must have `monitor`, +`monitor_rollup`, `manage` or `manage_rollup` cluster privileges to use this API. +For more information, see +{stack-ov}/security-privileges.html[Security privileges]. -`GET _rollup/job/` +[float] +[[rollup-get-job-desc]] +==== {api-description-title} -//===== Description +The API can return the details for a single job or for all jobs. -==== Path Parameters +NOTE: This API returns only active (both `STARTED` and `STOPPED`) jobs. If a job +was created, ran for a while, then deleted, this API does not return any details +about the job. For details about historical jobs, the +<> may be more useful. -`job_id`:: - (string) Identifier for the job to retrieve. If omitted (or `_all` is used) all jobs will be returned +[float] +[[rollup-get-job-path-params]] +==== {api-path-parms-title} +`` (Optional):: + (string) Identifier for the job to retrieve. If omitted (or `_all` is used), + all jobs are returned. -==== Request Body +[float] +[[rollup-get-job-response-body]] +==== {api-response-body-title} -There is no request body for the Get Jobs API. +`jobs`:: + An array of rollup job resources. + + `config`::: (object) The rollup job's configuration. See + <>. + + `status`::: (object) The current status of the rollup job's indexer. For + example, `"job_state" : "stopped"`. See <>. + + `stats`::: (object) Transient statistics about the rollup job, such as how + many documents have been processed and how many rollup summary docs have + been indexed. See <>. -==== Authorization +For more information, see <>. -You must have `monitor`, `monitor_rollup`, `manage` or `manage_rollup` cluster privileges to use this API. -For more information, see -{xpack-ref}/security-privileges.html[Security Privileges]. +[float] +[[rollup-get-job-example]] +==== {api-example-title} -==== Examples - -If we have already created a rollup job named `sensor`, the details about the job can be retrieved with: +If you have already created a rollup job named `sensor`, you can retrieve the +details about the job: [source,js] -------------------------------------------------- @@ -49,7 +77,7 @@ GET _rollup/job/sensor // CONSOLE // TEST[setup:sensor_rollup_job] -Which will yield the following response: +The API yields the following response: [source,js] ---- @@ -114,181 +142,5 @@ Which will yield the following response: ---- // TESTRESPONSE -The `jobs` array contains a single job (`id: sensor`) since we requested a single job in the endpoint's URL. The -details for this job contain three top-level parameters: `config`, `status` and `stats` - -`config` holds the rollup job's configuration, which is identical to the configuration that was supplied when creating -the job via the <>. - -The `status` object holds the current status of the rollup job's indexer. The possible values and their meanings are: - -- `stopped` means the indexer is paused and will not process data, even if it's cron interval triggers -- `started` means the indexer is running, but not actively indexing data. When the cron interval triggers, the job's -indexer will begin to process data -- `indexing` means the indexer is actively processing data and creating new rollup documents. When in this state, any -subsequent cron interval triggers will be ignored because the job is already active with the prior trigger -- `abort` a transient state, which is usually not witnessed by the user. The `abort` state is used if the task needs to -be shut down for some reason (job has been deleted, an unrecoverable error has been encountered, etc). Shortly after -the `abort` state is set, the job will remove itself from the cluster - -Finally, the `stats` object provides transient statistics about the rollup job, such as how many documents have been -processed and how many rollup summary docs have been indexed. These stats are not persisted, so if a node is restarted -these stats will be reset. - -If we add another job, we can see how multi-job responses are handled: - -[source,js] --------------------------------------------------- -PUT _rollup/job/sensor2 <1> -{ - "index_pattern": "sensor-*", - "rollup_index": "sensor_rollup", - "cron": "*/30 * * * * ?", - "page_size" :1000, - "groups" : { - "date_histogram": { - "field": "timestamp", - "fixed_interval": "1h", - "delay": "7d" - }, - "terms": { - "fields": ["node"] - } - }, - "metrics": [ - { - "field": "temperature", - "metrics": ["min", "max", "sum"] - }, - { - "field": "voltage", - "metrics": ["avg"] - } - ] -} - -GET _rollup/job/_all <2> --------------------------------------------------- -// CONSOLE -// TEST[setup:sensor_rollup_job] -<1> We create a second job with name `sensor2` -<2> Then request all jobs by using `_all` in the GetJobs API - -Which will yield the following response: - -[source,js] ----- -{ - "jobs" : [ - { - "config" : { - "id" : "sensor2", - "index_pattern" : "sensor-*", - "rollup_index" : "sensor_rollup", - "cron" : "*/30 * * * * ?", - "groups" : { - "date_histogram" : { - "fixed_interval" : "1h", - "delay": "7d", - "field": "timestamp", - "time_zone": "UTC" - }, - "terms" : { - "fields" : [ - "node" - ] - } - }, - "metrics" : [ - { - "field" : "temperature", - "metrics" : [ - "min", - "max", - "sum" - ] - }, - { - "field" : "voltage", - "metrics" : [ - "avg" - ] - } - ], - "timeout" : "20s", - "page_size" : 1000 - }, - "status" : { - "job_state" : "stopped" - }, - "stats" : { - "pages_processed" : 0, - "documents_processed" : 0, - "rollups_indexed" : 0, - "trigger_count" : 0, - "index_failures": 0, - "index_time_in_ms": 0, - "index_total": 0, - "search_failures": 0, - "search_time_in_ms": 0, - "search_total": 0 - } - }, - { - "config" : { - "id" : "sensor", - "index_pattern" : "sensor-*", - "rollup_index" : "sensor_rollup", - "cron" : "*/30 * * * * ?", - "groups" : { - "date_histogram" : { - "fixed_interval" : "1h", - "delay": "7d", - "field": "timestamp", - "time_zone": "UTC" - }, - "terms" : { - "fields" : [ - "node" - ] - } - }, - "metrics" : [ - { - "field" : "temperature", - "metrics" : [ - "min", - "max", - "sum" - ] - }, - { - "field" : "voltage", - "metrics" : [ - "avg" - ] - } - ], - "timeout" : "20s", - "page_size" : 1000 - }, - "status" : { - "job_state" : "stopped" - }, - "stats" : { - "pages_processed" : 0, - "documents_processed" : 0, - "rollups_indexed" : 0, - "trigger_count" : 0, - "index_failures": 0, - "index_time_in_ms": 0, - "index_total": 0, - "search_failures": 0, - "search_time_in_ms": 0, - "search_total": 0 - } - } - ] -} ----- -// NOTCONSOLE +The `jobs` array contains a single job (`id: sensor`) since we requested a +single job in the endpoint's URL. diff --git a/docs/reference/rollup/apis/put-job.asciidoc b/docs/reference/rollup/apis/put-job.asciidoc index eac71a48b4336..fb13598bbd40b 100644 --- a/docs/reference/rollup/apis/put-job.asciidoc +++ b/docs/reference/rollup/apis/put-job.asciidoc @@ -6,56 +6,74 @@ Create job ++++ -experimental[] +Creates a rollup job. -This API enables you to create a rollup job. The job will be created in a `STOPPED` state, and must be -started with the <>. +experimental[] -==== Request +[float] +[[sample-api-request]] +==== {api-request-title} `PUT _rollup/job/` -//===== Description +[float] +[[sample-api-prereqs]] +==== {api-prereq-title} -==== Path Parameters +If the {es} {security-features} are enabled, you must have `manage` or +`manage_rollup` cluster privileges to use this API. For more information, see +{stack-ov}/security-privileges.html[Security privileges]. -`job_id` (required):: - (string) Identifier for the job +[float] +[[sample-api-desc]] +==== {api-description-title} +Jobs are created in a `STOPPED` state. You can start them with the +<>. -==== Request Body +[float] +[[sample-api-path-params]] +==== {api-path-parms-title} -`index_pattern` (required):: - (string) The index, or index pattern, that you wish to rollup. Supports wildcard-style patterns (`logstash-*`). +`job_id` (required):: + (string) Identifier for the job. -`rollup_index` (required):: - (string) The index that you wish to store rollup results into. Can be shared with other rollup jobs. +[float] +[[sample-api-request-body]] +==== {api-request-body-title} -`cron` (required):: +`cron` (Required):: (string) A cron string which defines when the rollup job should be executed. -`page_size` (required):: - (int) The number of bucket results that should be processed on each iteration of the rollup indexer. A larger value - will tend to execute faster, but will require more memory during processing. +`groups` (Required):: + (object) Defines the grouping fields that are defined for this rollup job. See + <>. -`groups` (required):: - (object) Defines the grouping fields that are defined for this rollup job. See <>. +`index_pattern` (Required):: + (string) The index or index pattern to roll up. Supports wildcard-style + patterns (`logstash-*`). -`metrics`:: - (object) Defines the metrics that should be collected for each grouping tuple. See <>. +`metrics` (Optional):: + (object) Defines the metrics to collect for each grouping tuple. See + <>. -For more details about the job configuration, see <>. - -==== Authorization +`page_size` (Required):: + (integer) The number of bucket results that are processed on each iteration of + the rollup indexer. A larger value tends to execute faster, but requires more + memory during processing. -You must have `manage` or `manage_rollup` cluster privileges to use this API. -For more information, see -{xpack-ref}/security-privileges.html[Security Privileges]. +`rollup_index` (Required):: + (string) The index that contains the rollup results. The index can be shared + with other rollup jobs. +For more details about the job configuration, see <>. -==== Examples +[float] +[[sample-api-example]] +==== {api-example-title} -The following example creates a rollup job named "sensor", targeting the "sensor-*" index pattern: +The following example creates a rollup job named "sensor", targeting the +"sensor-*" index pattern: [source,js] -------------------------------------------------- diff --git a/docs/reference/rollup/apis/rollup-caps.asciidoc b/docs/reference/rollup/apis/rollup-caps.asciidoc index e50806f3c1e0e..d3c09fa1b3d7c 100644 --- a/docs/reference/rollup/apis/rollup-caps.asciidoc +++ b/docs/reference/rollup/apis/rollup-caps.asciidoc @@ -6,10 +6,76 @@ Get rollup caps ++++ +Returns the capabilities of any rollup jobs that have been configured for a +specific index or index pattern. + experimental[] +//// +[float] +[[sample-api-request]] +==== {api-request-title} +// This section show the basic endpoint, without the body or optional parameters. +// Variables should use <...> syntax +// If an API supports both PUT and POST, include both here + +`PUT //_ccr/follow` + +[float] +[[sample-api-prereqs]] +==== {api-prereq-title} +// Optional list of prerequisites. + +[float] +[[sample-api-desc]] +==== {api-description-title} +// Add a more detailed description the context. +// Link to related APIs if appropriate. + +[float] +[[sample-api-path-params]] +==== {api-path-parms-title} +// A list of all path parameters in the endpoint request + +[float] +[[sample-api-query-params]] +==== {api-query-parms-title} +// A list of optional query parameters + +[float] +[[sample-api-request-body]] +==== {api-request-body-title} +// A list of the properties you can specify in the body of the request + +[float] +[[sample-api-response-body]] +==== {api-response-body-title} +// Response body is only required for detailed responses. + +[float] +[[sample-api-example]] +==== {api-example-title} +// Optional brief example. + +//// + +[discrete] +[[get-rollup-caps-request]] +==== Request + +`GET _rollup/data/` + +[discrete] +[[get-rollup-caps-prereqs]] +==== Prerequisites -This API returns the capabilities of any rollup jobs that have been configured -for a specific index or index pattern. +If the {es} {security-features} are enabled, you must have `monitor`, +`monitor_rollup`, `manage` or `manage_rollup` cluster privileges to use this API. +For more information, see +{stack-ov}/security-privileges.html[Security privileges]. + +[discrete] +[[get-rollup-caps-desc]] +==== Description This API is useful because a rollup job is often configured to rollup only a subset of fields from the source index. Furthermore, only certain aggregations @@ -22,33 +88,31 @@ This API enables you to inspect an index and determine: 2. If yes to the first question, what fields were rolled up, what aggregations can be performed, and where does the data live? -==== Request - -`GET _rollup/data/{index}` - -//===== Description - -==== Path Parameters +[discrete] +[[get-rollup-caps-params]] +==== Path parameters `index`:: - (string) Index, indices or index-pattern to return rollup capabilities for. `_all` may be used to fetch - rollup capabilities from all jobs - - -==== Request Body + (string) Index, indices or index-pattern to return rollup capabilities for. + To fetch rollup capabilities from all jobs, use `_all`. + +[discrete] +[[get-rollup-caps-response-body]] +==== Response body -There is no request body for the Get Rollup Caps API. +The API returns the following information: -==== Authorization - -You must have `monitor`, `monitor_rollup`, `manage` or `manage_rollup` cluster privileges to use this API. -For more information, see -{xpack-ref}/security-privileges.html[Security Privileges]. +`rollup_jobs`:: + (array) An array of rollup job resources. For more information, see + <>. +[discrete] +[[get-rollup-caps-examples]] ==== Examples -Imagine we have an index named `sensor-1` full of raw data. We know that the data will grow over time, so there -will be a `sensor-2`, `sensor-3`, etc. Let's create a Rollup job that targets the index pattern `sensor-*` to accommodate +Imagine we have an index named `sensor-1` full of raw data. We know that the +data will grow over time, so there will be a `sensor-2`, `sensor-3`, etc. Let's +create a rollup job that targets the index pattern `sensor-*` to accommodate this future scaling: [source,js] @@ -84,7 +148,8 @@ PUT _rollup/job/sensor // CONSOLE // TEST[setup:sensor_index] -We can then retrieve the rollup capabilities of that index pattern (`sensor-*`) via the following command: +We can then retrieve the rollup capabilities of that index pattern (`sensor-*`) +via the following command: [source,js] -------------------------------------------------- @@ -142,17 +207,21 @@ Which will yield the following response: ---- // TESTRESPONSE -The response that is returned contains information that is similar to the original Rollup configuration, but formatted -differently. First, there are some house-keeping details: the Rollup job's ID, the index that holds the rolled data, +The response that is returned contains information that is similar to the +original rollup configuration, but formatted differently. First, there are some +house-keeping details: the rollup job's ID, the index that holds the rolled data, the index pattern that the job was targeting. -Next it shows a list of fields that contain data eligible for rollup searches. Here we see four fields: `node`, `temperature`, -`timestamp` and `voltage`. Each of these fields list the aggregations that are possible. For example, you can use a min, max -or sum aggregation on the `temperature` field, but only a `date_histogram` on `timestamp`. +Next it shows a list of fields that contain data eligible for rollup searches. +Here we see four fields: `node`, `temperature`, `timestamp` and `voltage`. Each +of these fields list the aggregations that are possible. For example, you can +use a min, max or sum aggregation on the `temperature` field, but only a +`date_histogram` on `timestamp`. -Note that the `rollup_jobs` element is an array; there can be multiple, independent jobs configured for a single index -or index pattern. Each of these jobs may have different configurations, so the API returns a list of all the various -configurations available. +Note that the `rollup_jobs` element is an array; there can be multiple, +independent jobs configured for a single index or index pattern. Each of these +jobs may have different configurations, so the API returns a list of all the +various configurations available. We could also retrieve the same information with a request to `_all`: @@ -163,7 +232,7 @@ GET _rollup/data/_all // CONSOLE // TEST[continued] -But note that if we use the concrete index name (`sensor-1`), we'll retrieve no rollup capabilities: +If we use the concrete index name (`sensor-1`): [source,js] -------------------------------------------------- @@ -172,6 +241,8 @@ GET _rollup/data/sensor-1 // CONSOLE // TEST[continued] +...we retrieve no rollup capabilities: + [source,js] ---- { @@ -180,7 +251,9 @@ GET _rollup/data/sensor-1 ---- // TESTRESPONSE -Why is this? The original rollup job was configured against a specific index pattern (`sensor-*`) not a concrete index -(`sensor-1`). So while the index belongs to the pattern, the rollup job is only valid across the entirety of the pattern -not just one of it's containing indices. So for that reason, the Rollup Capabilities API only returns information based -on the originally configured index name or pattern. +Why is this? The original rollup job was configured against a specific index +pattern (`sensor-*`) not a concrete index (`sensor-1`). So while the index +belongs to the pattern, the rollup job is only valid across the entirety of the +pattern not just one of its containing indices. So for that reason, the get +rollup capabilities API returns only information that is based on the originally +configured index name or pattern. diff --git a/docs/reference/rollup/apis/rollup-index-caps.asciidoc b/docs/reference/rollup/apis/rollup-index-caps.asciidoc index a0697ba70326e..fd299abe0178a 100644 --- a/docs/reference/rollup/apis/rollup-index-caps.asciidoc +++ b/docs/reference/rollup/apis/rollup-index-caps.asciidoc @@ -6,9 +6,59 @@ Get rollup index caps ++++ +Returns the rollup capabilities of all jobs inside of a rollup index (e.g. the +index where rollup data is stored). + experimental[] -This API returns the rollup capabilities of all jobs inside of a rollup index (e.g. the index where rollup data is stored). +//// +[float] +[[sample-api-request]] +==== {api-request-title} +// This section show the basic endpoint, without the body or optional parameters. +// Variables should use <...> syntax +// If an API supports both PUT and POST, include both here + +`PUT //_ccr/follow` + +[float] +[[sample-api-prereqs]] +==== {api-prereq-title} +// Optional list of prerequisites. + +[float] +[[sample-api-desc]] +==== {api-description-title} +// Add a more detailed description the context. +// Link to related APIs if appropriate. + +[float] +[[sample-api-path-params]] +==== {api-path-parms-title} +// A list of all path parameters in the endpoint request + +[float] +[[sample-api-query-params]] +==== {api-query-parms-title} +// A list of optional query parameters + +[float] +[[sample-api-request-body]] +==== {api-request-body-title} +// A list of the properties you can specify in the body of the request + +[float] +[[sample-api-response-body]] +==== {api-response-body-title} +// Response body is only required for detailed responses. + +[float] +[[sample-api-example]] +==== {api-example-title} +// Optional brief example. + +//// + A single rollup index may store the data for multiple rollup jobs, and may have a variety of capabilities depending on those jobs. This API will allow you to determine: diff --git a/docs/reference/rollup/apis/rollup-job-config-objects.asciidoc b/docs/reference/rollup/apis/rollup-job-config-objects.asciidoc new file mode 100644 index 0000000000000..a583bdb372b12 --- /dev/null +++ b/docs/reference/rollup/apis/rollup-job-config-objects.asciidoc @@ -0,0 +1,114 @@ +[role="xpack"] +[testenv="basic"] +[[rollup-job-config-objects]] +=== Rollup job config API objects +++++ +Rollup job config +++++ + +The `config` object holds the rollup job's configuration. It is used by the +<> and the +<>. + +`cron`:: + (string) A cron string that defines when the rollup job executes. + +`groups`:: + (object) Defines the grouping fields that are defined for this rollup job. See + <> and <>. + +`id`:: + (string) An identifier for the rollup job. This property is informational; it + is returned by the <>. + +`index_pattern`:: + (string) The index or index pattern to roll up. It supports wildcard-style + patterns such as `logstash-*`. + +`metrics`:: + (object) Defines the metrics are collected for each grouping tuple. + See <> and <>. + +`page_size`:: + (integer) The number of bucket results that are processed on each iteration of + the rollup indexer. A larger value tends to execute faster, but requires more + memory during processing. + +`rollup_index`:: + (string) The index that contains the rollup results. The index can be shared + with other rollup jobs. + +//// +`timeout`:: TBD. Exists in get jobs but not put job. +//// + +[float] +[[rollup-job-group-objects]] +==== Rollup job group API objects + +`date_histogram`:: (object) TBD + +[float] +[[rollup-job-metrics-objects]] +==== Rollup job metrics API objects + +The `metrics` resource is an array of the following properties: + +`field`:: TBD + +metrics:: (array) TBD + +//// +`config`:: + (object) TBD + +`cron`:: + (string) A cron string that defines when the rollup job runs. It defines an + interval of when to run the job's indexer. When the interval triggers, the + indexer attempts to roll up the data in the index pattern. The cron pattern is + unrelated to the time interval of the data being rolled up. For example, you + can create hourly rollups of your document (as defined in the + <>) but run the indexer only on a + daily basis at midnight, as defined by the cron. The cron pattern is defined + just like the {watcher} cron schedule. + +`fields`:: + (object) A list of fields that contain data that is eligible for rollup + searches. + +`groups`:: + (object) A list of the grouping fields. + +`index_pattern`:: + (string) The index or index pattern that is rolled up. Supports wildcard-style + patterns (for example, `logstash-*`). ++ +-- +[NOTE] +This property value cannot be a pattern that also matches the `rollup_index`. +For example, the pattern `"foo-*"` matches the rollup index `"foo-rollup"`. This +situation causes problems because the rollup job attempts to roll up its own +data at runtime. If you attempt to configure a pattern that matches the +`rollup_index`, an exception is thrown to prevent this behavior. + +-- + +`job_id`:: + (string) The unique identifier for the data that is associated with the rollup + job. It can be any alphanumeric string. + +`metrics`:: + (object) The metrics to collect for each group. + +`page_size`:: + (int) The number of bucket results that are processed on each iteration of the + rollup indexer. A larger value tends to execute faster but requires more + memory during processing. This property has no effect on how the data is + rolled up; it is merely used for tweaking the speed or memory cost of the + indexer. + +`rollup_index`:: + (string) The index that contains rollup results. This index stores all the + rollup data that is generated by the job. It can be shared by multiple rollup + jobs. +//// diff --git a/docs/reference/rollup/apis/rollup-job-stats-objects.asciidoc b/docs/reference/rollup/apis/rollup-job-stats-objects.asciidoc new file mode 100644 index 0000000000000..f8db3889df6d0 --- /dev/null +++ b/docs/reference/rollup/apis/rollup-job-stats-objects.asciidoc @@ -0,0 +1,62 @@ +[role="xpack"] +[testenv="basic"] +[[rollup-job-stats-objects]] +=== Rollup job stats and status API objects +++++ +Rollup job stats and status +++++ + +The <> retrieves the following objects, +which provide information about the status and statistics for each job. + +[float] +[[rollup-job-stats-properties]] +==== {api-definitions-title} + +`status`:: (object) The current status of the rollup job's indexer. + `job_state`::: (string) Valid values are: ++ +-- +* `stopped`: The indexer is paused and will not process data, even if its cron +interval triggers. +* `started`: The indexer is running, but not actively indexing data. When the +cron interval triggers, the job's indexer will begin to process data. +* `indexing`: The indexer is actively processing data and creating new rollup +documents. When in this state, any subsequent cron interval triggers will be +ignored because the job is already active with the prior trigger. +* `abort`: A transient state, which is usually not witnessed by the user. The +`abort` state is used if the task needs to be shut down for some reason. For +example, the job has been deleted or an unrecoverable error has been encountered. Shortly after the `abort` state is set, the job will remove itself from the +cluster. +-- + +`stats`:: + (object) Transient statistics about the rollup job. ++ +-- +NOTE: These stats are not persisted. If a node is restarted, these stats are +reset. + +-- + + `pages_processed`::: TBD. `0`. + + `documents_processed`:: The number of documents that have been processed. For + example, `0`. + + `rollups_indexed`:: The number of rollup summary docs that have been indexed. + For example, `0`. + + `trigger_count`:: TBD. `0`. + + `index_failures`:: TBD. `0`. + + `index_time_in_ms`:: TBD. `0`. + + `index_total`:: TBD. `0`. + + `search_failures`:: TBD. `0`. + + `search_time_in_ms`:: TBD. `0`. + + `search_total`:: TBD diff --git a/docs/reference/rollup/apis/rollup-jobs-objects.asciidoc b/docs/reference/rollup/apis/rollup-jobs-objects.asciidoc new file mode 100644 index 0000000000000..4cda6a59e8f03 --- /dev/null +++ b/docs/reference/rollup/apis/rollup-jobs-objects.asciidoc @@ -0,0 +1,21 @@ +[role="xpack"] +[testenv="basic"] +[[rollup-job-objects]] +=== Rollup job API objects +++++ +Rollup jobs +++++ + +The <> returns +`rollup_jobs`, which is an array of objects that provide information about the +capabilities of rollup jobs on specific indices or index patterns. + +The <> returns: + +`jobs`:: (array) TBD + `config`::: (object) TBD. See <>. + `status`::: (object) TBD. See <>. + `stats`::: (object) TBD. See <>. + +include::rollup-job-config-objects.asciidoc[] +include::rollup-job-stats-objects.asciidoc[] diff --git a/docs/reference/rollup/apis/rollup-search.asciidoc b/docs/reference/rollup/apis/rollup-search.asciidoc index ec2a554d09ff4..b5a6126804416 100644 --- a/docs/reference/rollup/apis/rollup-search.asciidoc +++ b/docs/reference/rollup/apis/rollup-search.asciidoc @@ -6,9 +6,59 @@ Rollup search ++++ +Searches rolled-up data using the standard query DSL. + experimental[] -The Rollup Search endpoint allows searching rolled-up data using the standard query DSL. The Rollup Search endpoint +//// +[float] +[[sample-api-request]] +==== {api-request-title} +// This section show the basic endpoint, without the body or optional parameters. +// Variables should use <...> syntax +// If an API supports both PUT and POST, include both here + +`PUT //_ccr/follow` + +[float] +[[sample-api-prereqs]] +==== {api-prereq-title} +// Optional list of prerequisites. + +[float] +[[sample-api-desc]] +==== {api-description-title} +// Add a more detailed description the context. +// Link to related APIs if appropriate. + +[float] +[[sample-api-path-params]] +==== {api-path-parms-title} +// A list of all path parameters in the endpoint request + +[float] +[[sample-api-query-params]] +==== {api-query-parms-title} +// A list of optional query parameters + +[float] +[[sample-api-request-body]] +==== {api-request-body-title} +// A list of the properties you can specify in the body of the request + +[float] +[[sample-api-response-body]] +==== {api-response-body-title} +// Response body is only required for detailed responses. + +[float] +[[sample-api-example]] +==== {api-example-title} +// Optional brief example. + +//// + +The Rollup Search endpoint is needed because, internally, rolled-up documents utilize a different document structure than the original data. The Rollup Search endpoint rewrites standard query DSL into a format that matches the rollup documents, then takes the response and rewrites it back to what a client would expect given the original query. diff --git a/docs/reference/rollup/apis/start-job.asciidoc b/docs/reference/rollup/apis/start-job.asciidoc index 241d070a670a0..06118faa91d53 100644 --- a/docs/reference/rollup/apis/start-job.asciidoc +++ b/docs/reference/rollup/apis/start-job.asciidoc @@ -6,9 +6,59 @@ Start job ++++ -experimental[] +Starts an existing, stopped rollup job. -This API starts an existing, stopped rollup job. If the job does not exist an exception will be thrown. +experimental[] +//// +[float] +[[sample-api-request]] +==== {api-request-title} +// This section show the basic endpoint, without the body or optional parameters. +// Variables should use <...> syntax +// If an API supports both PUT and POST, include both here + +`PUT //_ccr/follow` + +[float] +[[sample-api-prereqs]] +==== {api-prereq-title} +// Optional list of prerequisites. + +[float] +[[sample-api-desc]] +==== {api-description-title} +// Add a more detailed description the context. +// Link to related APIs if appropriate. + +[float] +[[sample-api-path-params]] +==== {api-path-parms-title} +// A list of all path parameters in the endpoint request + +[float] +[[sample-api-query-params]] +==== {api-query-parms-title} +// A list of optional query parameters + +[float] +[[sample-api-request-body]] +==== {api-request-body-title} +// A list of the properties you can specify in the body of the request + +[float] +[[sample-api-response-body]] +==== {api-response-body-title} +// Response body is only required for detailed responses. + +[float] +[[sample-api-example]] +==== {api-example-title} +// Optional brief example. + +//// + + + If the job does not exist an exception will be thrown. Starting an already started job has no action. ==== Request diff --git a/docs/reference/rollup/apis/stop-job.asciidoc b/docs/reference/rollup/apis/stop-job.asciidoc index 35162246a5fbb..59a33059f695c 100644 --- a/docs/reference/rollup/apis/stop-job.asciidoc +++ b/docs/reference/rollup/apis/stop-job.asciidoc @@ -6,9 +6,58 @@ Stop job ++++ -experimental[] +Stops an existing, started rollup job. -This API stops an existing, started rollup job. If the job does not exist an exception will be thrown. +experimental[] +//// +[float] +[[sample-api-request]] +==== {api-request-title} +// This section show the basic endpoint, without the body or optional parameters. +// Variables should use <...> syntax +// If an API supports both PUT and POST, include both here + +`PUT //_ccr/follow` + +[float] +[[sample-api-prereqs]] +==== {api-prereq-title} +// Optional list of prerequisites. + +[float] +[[sample-api-desc]] +==== {api-description-title} +// Add a more detailed description the context. +// Link to related APIs if appropriate. + +[float] +[[sample-api-path-params]] +==== {api-path-parms-title} +// A list of all path parameters in the endpoint request + +[float] +[[sample-api-query-params]] +==== {api-query-parms-title} +// A list of optional query parameters + +[float] +[[sample-api-request-body]] +==== {api-request-body-title} +// A list of the properties you can specify in the body of the request + +[float] +[[sample-api-response-body]] +==== {api-response-body-title} +// Response body is only required for detailed responses. + +[float] +[[sample-api-example]] +==== {api-example-title} +// Optional brief example. + +//// + + If the job does not exist an exception will be thrown. Stopping an already stopped job has no action. ==== Request