From ad87af30dd65ef93d20ae69d74a986a638d88a54 Mon Sep 17 00:00:00 2001 From: lcawl Date: Wed, 18 Mar 2020 18:09:29 -0700 Subject: [PATCH 1/5] [DOCS] Augments cat transforms API --- docs/reference/cat/transforms.asciidoc | 158 +++++++++++++++++- docs/reference/redirects.asciidoc | 5 - docs/reference/rest-api/common-parms.asciidoc | 62 +++++++ .../apis/get-transform-stats.asciidoc | 50 +++--- 4 files changed, 237 insertions(+), 38 deletions(-) diff --git a/docs/reference/cat/transforms.asciidoc b/docs/reference/cat/transforms.asciidoc index 78013e394ef24..70138ef7e51a3 100644 --- a/docs/reference/cat/transforms.asciidoc +++ b/docs/reference/cat/transforms.asciidoc @@ -6,23 +6,167 @@ Returns configuration and usage information about {transforms}. - [[cat-transforms-api-request]] ==== {api-request-title} +`GET /_cat/transforms/` + + +`GET /_cat/transforms/_all` + + +`GET /_cat/transforms/*` + + `GET /_cat/transforms` +[[cat-transforms-api-prereqs]] +==== {api-prereq-title} + +* If the {es} {security-features} are enabled, you must have `monitor_transform` +cluster privileges to use this API. The built-in `transform_user` role has these +privileges. For more information, see <> and +<>. //[[cat-transforms-api-desc]] //==== {api-description-title} +[[cat-transforms-api-path-params]] +==== {api-path-parms-title} + +``:: +(Optional, string) +include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-id-wildcard] + +[[cat-transforms-api-query-params]] +==== {api-query-parms-title} + +`allow_no_match`:: +(Optional, boolean) +include::{docdir}/rest-api/common-parms.asciidoc[tag=allow-no-match-transforms1] + +include::{docdir}/rest-api/common-parms.asciidoc[tag=http-format] + +`from`:: +(Optional, integer) +include::{docdir}/rest-api/common-parms.asciidoc[tag=from-transforms] + +include::{docdir}/rest-api/common-parms.asciidoc[tag=cat-h] ++ +If you do not specify which columns to include, the API returns the default +columns. If you explicitly specify one or more columns, it returns only the +specified columns. ++ +Valid columns are: + +`changes_last_detection_time`, `cldt`::: +include::{docdir}/rest-api/common-parms.asciidoc[tag=checkpointing-changes-last-detected-at] + +`checkpoint_duration_time_exp_avg`, `cdtea`, `checkpointTimeExpAvg`::: +include::{docdir}/rest-api/common-parms.asciidoc[tag=exponential-avg-checkpoint-duration-ms] + +`create_time`, `ct`, `createTime`::: +(Default) +The time the {transform} was created. + +`description`, `d`::: +(Default) +The description of the {transform}. + +`dest_index`, `di`, `destIndex`::: +(Default) +include::{docdir}/rest-api/common-parms.asciidoc[tag=dest-index] + +`document_total`, `dt`::: +The total number of documents. + +`frequency`, `f`::: +(Default) +include::{docdir}/rest-api/common-parms.asciidoc[tag=frequency] + +`id`::: +(Default) +include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-id] + +`index_failure`, `if`::: +include::{docdir}/rest-api/common-parms.asciidoc[tag=index-failures] + +`index_time`, `itime`::: +include::{docdir}/rest-api/common-parms.asciidoc[tag=index-time-ms] + +`index_total `, `it`::: +include::{docdir}/rest-api/common-parms.asciidoc[tag=index-total] + +`indexed_documents_exp_avg`, `idea`::: +include::{docdir}/rest-api/common-parms.asciidoc[tag=exponential-avg-documents-indexed] + +`invocation_total`, `itotal`::: +The total number of invocations. + +`max_page_search_size`, `mpsz`::: +(Default) +The maximum page search size. + +`page_total`, `pt`::: +include::{docdir}/rest-api/common-parms.asciidoc[tag=pages-processed] + +`pipeline`, `p`::: +(Default) +The {transform} pipeline. + +`processed_documents_exp_avg`, `pdea`::: +include::{docdir}/rest-api/common-parms.asciidoc[tag=exponential-avg-documents-processed] + +`reason`, `r`::: +The reason. + +`search_failure`, `sf`::: +include::{docdir}/rest-api/common-parms.asciidoc[tag=search-failures] + +`search_time`, `stime`::: +include::{docdir}/rest-api/common-parms.asciidoc[tag=search-time-ms] + +`search_total`, `st`::: +include::{docdir}/rest-api/common-parms.asciidoc[tag=search-total] + +`source_index`, `si`, `sourceIndex`::: +(Default) +include::{docdir}/rest-api/common-parms.asciidoc[tag=source-index-transforms] + +`state`, `s`::: +(Default) +include::{docdir}/rest-api/common-parms.asciidoc[tag=state-transform] + +`transform_type`, `tt`::: +(Default) +Indicates whether it's a batch or continuous {transform}. + +`version`, `v`::: +(Default) +The version of {es} that existed on the node when the {transform} was +created. + +include::{docdir}/rest-api/common-parms.asciidoc[tag=help] + +include::{docdir}/rest-api/common-parms.asciidoc[tag=cat-s] + +`size`:: +(Optional, integer) +include::{docdir}/rest-api/common-parms.asciidoc[tag=size-transforms] + +include::{docdir}/rest-api/common-parms.asciidoc[tag=time] -//[[cat-transforms-api-query-params]] -//==== {api-query-parms-title} +include::{docdir}/rest-api/common-parms.asciidoc[tag=cat-v] +[[cat-transforms-api-examples]] +==== {api-examples-title} -//[[cat-transforms-api-response-codes]] -//==== {api-response-codes-title} +[source,console] +-------------------------------------------------- +GET /_cat/transforms?v +-------------------------------------------------- +// TEST[skip:kibana sample data] -//[[cat-transforms-api-examples]] -//==== {api-examples-title} +[source,console-result] +---- +id create_time version source_index dest_index pipeline description transform_type frequency max_page_search_size state +ecommerce-customer-transform 2020-03-18T23:39:29.525Z 8.0.0 kibana_sample_data_ecommerce ecommerce-customers batch 1m 500 STOPPED +---- +// TESTRESPONSE[skip:kibana sample data] \ No newline at end of file diff --git a/docs/reference/redirects.asciidoc b/docs/reference/redirects.asciidoc index 655e1156f1938..065450560d0a2 100644 --- a/docs/reference/redirects.asciidoc +++ b/docs/reference/redirects.asciidoc @@ -385,8 +385,3 @@ coming::[7.x] === Component template APIs coming::[7.x] - -[role="exclude",id="cat-transform"] -=== cat transform API - -See <>. \ No newline at end of file diff --git a/docs/reference/rest-api/common-parms.asciidoc b/docs/reference/rest-api/common-parms.asciidoc index baaea29f7e7e6..42294029da992 100644 --- a/docs/reference/rest-api/common-parms.asciidoc +++ b/docs/reference/rest-api/common-parms.asciidoc @@ -99,6 +99,10 @@ tag::bytes[] (Optional, <>) Unit used to display byte values. end::bytes[] +tag::checkpointing-changes-last-detected-at[] +The timestamp when changes were last detected in the source indices. +end::checkpointing-changes-last-detected-at[] + tag::cluster-health-status[] (string) Health status of the cluster, based on the state of its primary and replica @@ -207,6 +211,20 @@ Wildcard expressions are not accepted. -- end::expand-wildcards[] +tag::exponential-avg-checkpoint-duration-ms[] +Exponential moving average of the duration of the checkpoint, in milliseconds. +end::exponential-avg-checkpoint-duration-ms[] + +tag::exponential-avg-documents-indexed[] +Exponential moving average of the number of new documents that have been +indexed. +end::exponential-avg-documents-indexed[] + +tag::exponential-avg-documents-processed[] +Exponential moving average of the number of documents that have been +processed. +end::exponential-avg-documents-processed[] + tag::field_statistics[] `field_statistics`:: (Optional, boolean) If `true`, the response includes the document count, sum of document frequencies, @@ -380,6 +398,18 @@ tag::index[] used to limit the request. end::index[] +tag::index-failures[] +The number of indexing failures. +end::index-failures[] + +tag::index-time-ms[] +The amount of time spent indexing, in milliseconds. +end::index-time-ms[] + +tag::index-total[] +The number of indices created. +end::index-total[] + tag::bulk-index[] `_index`:: (Optional, string) @@ -574,6 +604,11 @@ tag::pipeline[] (Optional, string) ID of the pipeline to use to preprocess incoming documents. end::pipeline[] +tag::pages-processed[] +The number of search or bulk index operations processed. Documents are +processed in batches instead of individually. +end::pages-processed[] + tag::path-pipeline[] ``:: (Optional, string) Comma-separated list or wildcard expression of pipeline IDs @@ -706,6 +741,14 @@ tag::scroll_size[] Defaults to 100. end::scroll_size[] +tag::search-failures[] +The number of search failures. +end::search-failures[] + +tag::search-time-ms[] +The amount of time spent searching, in milliseconds. +end::search-time-ms[] + tag::search_timeout[] `timeout`:: (Optional, <>) @@ -713,6 +756,10 @@ Explicit timeout for each search request. Defaults to no timeout. end::search_timeout[] +tag::search-total[] +The number of search operations on the source index for the {transform}. +end::search-total[] + tag::search_type[] `search_type`:: (Optional, string) The type of the search operation. Available options: @@ -802,6 +849,21 @@ A query clause that retrieves a subset of data from the source index. See <>. end::source-query-transforms[] +tag::state-transform[] +The status of the {transform}, which can be one of the following values: ++ +-- +* `aborting`: The {transform} is aborting. +* `failed`: The {transform} failed. For more information about the failure, +check the reason field. +* `indexing`: The {transform} is actively processing data and creating new +documents. +* `started`: The {transform} is running but not actively indexing data. +* `stopped`: The {transform} is stopped. +* `stopping`: The {transform} is stopping. +-- +end::state-transform[] + tag::stats[] `stats`:: (Optional, string) Specific `tag` of the request for logging and statistical diff --git a/docs/reference/transform/apis/get-transform-stats.asciidoc b/docs/reference/transform/apis/get-transform-stats.asciidoc index 1175fba036bb4..fed7eadb29948 100644 --- a/docs/reference/transform/apis/get-transform-stats.asciidoc +++ b/docs/reference/transform/apis/get-transform-stats.asciidoc @@ -77,7 +77,8 @@ informational; you cannot update their values. `checkpointing`:: (object) Contains statistics about <>. `checkpointing`.`changes_last_detected_at`::: -(date) The timestamp when changes were last detected in the source indices. +(date) +include::{docdir}/rest-api/common-parms.asciidoc[tag=checkpointing-changes-last-detected-at] `checkpointing`.`last`::: (object) Contains statistics about the last completed checkpoint. `checkpointing`.`last`.`checkpoint`:::: @@ -128,18 +129,8 @@ started. (string) The host and port where transport HTTP connections are accepted. For example, `127.0.0.1:9300`. `state`:: -(string) The status of the {transform}, which can be one of the following values: -+ --- -* `aborting`: The {transform} is aborting. -* `failed`: The {transform} failed. For more information about the failure, -check the reason field. -* `indexing`: The {transform} is actively processing data and creating new -documents. -* `started`: The {transform} is running but not actively indexing data. -* `stopped`: The {transform} is stopped. -* `stopping`: The {transform} is stopping. --- +(string) +include::{docdir}/rest-api/common-parms.asciidoc[tag=state-transform] `stats`:: (object) An object that provides statistical information about the {transform}. @@ -150,28 +141,35 @@ for the {transform}. (long) The number of documents that have been processed from the source index of the {transform}. `stats`.`exponential_avg_checkpoint_duration_ms`::: -(double) Exponential moving average of the duration of the checkpoint, in milliseconds. +(double) +include::{docdir}/rest-api/common-parms.asciidoc[tag=exponential-avg-checkpoint-duration-ms] `stats`.`exponential_avg_documents_indexed`::: -(double) Exponential moving average of the number of new documents that have been -indexed. +(double) +include::{docdir}/rest-api/common-parms.asciidoc[tag=exponential-avg-documents-indexed] `stats`.`exponential_avg_documents_processed`::: -(double) Exponential moving average of the number of documents that have been -processed. +(double) +include::{docdir}/rest-api/common-parms.asciidoc[tag=exponential-avg-documents-processed] `stats`.`index_failures`::: -(long) The number of indexing failures. +(long) +include::{docdir}/rest-api/common-parms.asciidoc[tag=index-failures] `stats`.`index_time_in_ms`::: -(long) The amount of time spent indexing, in milliseconds. +(long) +include::{docdir}/rest-api/common-parms.asciidoc[tag=index-time-ms] `stats`.`index_total`::: -(long) The number of indices created. +(long) +include::{docdir}/rest-api/common-parms.asciidoc[tag=index-total] `stats`.`pages_processed`::: -(long) The number of search or bulk index operations processed. Documents are -processed in batches instead of individually. +(long) +include::{docdir}/rest-api/common-parms.asciidoc[tag=pages-processed] `stats`.`search_failures`::: -(long) The number of search failures. +(long) +include::{docdir}/rest-api/common-parms.asciidoc[tag=search-failures] `stats`.`search_time_in_ms`::: -(long) The amount of time spent searching, in milliseconds. +(long) +include::{docdir}/rest-api/common-parms.asciidoc[tag=search-time-ms] `stats`.`search_total`::: -(long) The number of search operations on the source index for the {transform}. +(long) +include::{docdir}/rest-api/common-parms.asciidoc[tag=search-total] `stats`.`trigger_count`::: (long) The number of times the {transform} has been triggered by the scheduler. For example, the scheduler triggers the {transform} indexer to check for updates From c910a4fddd5aeb741dabdf163e5c68f508ff5d07 Mon Sep 17 00:00:00 2001 From: Lisa Cawley Date: Thu, 19 Mar 2020 17:50:54 -0700 Subject: [PATCH 2/5] Update docs/reference/cat/transforms.asciidoc Co-Authored-By: Benjamin Trent --- docs/reference/cat/transforms.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/cat/transforms.asciidoc b/docs/reference/cat/transforms.asciidoc index 70138ef7e51a3..8e4420479d723 100644 --- a/docs/reference/cat/transforms.asciidoc +++ b/docs/reference/cat/transforms.asciidoc @@ -115,7 +115,7 @@ The {transform} pipeline. include::{docdir}/rest-api/common-parms.asciidoc[tag=exponential-avg-documents-processed] `reason`, `r`::: -The reason. +The reason for the current state. `search_failure`, `sf`::: include::{docdir}/rest-api/common-parms.asciidoc[tag=search-failures] @@ -169,4 +169,4 @@ GET /_cat/transforms?v id create_time version source_index dest_index pipeline description transform_type frequency max_page_search_size state ecommerce-customer-transform 2020-03-18T23:39:29.525Z 8.0.0 kibana_sample_data_ecommerce ecommerce-customers batch 1m 500 STOPPED ---- -// TESTRESPONSE[skip:kibana sample data] \ No newline at end of file +// TESTRESPONSE[skip:kibana sample data] From fb8dd0837f5bff32223e293ec2f4e71b2bd44472 Mon Sep 17 00:00:00 2001 From: lcawl Date: Fri, 20 Mar 2020 10:25:45 -0700 Subject: [PATCH 3/5] [DOCS] Fixes typo --- docs/reference/cat/transforms.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/cat/transforms.asciidoc b/docs/reference/cat/transforms.asciidoc index 8e4420479d723..ac7bc4b019a1b 100644 --- a/docs/reference/cat/transforms.asciidoc +++ b/docs/reference/cat/transforms.asciidoc @@ -91,7 +91,7 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=index-failures] `index_time`, `itime`::: include::{docdir}/rest-api/common-parms.asciidoc[tag=index-time-ms] -`index_total `, `it`::: +`index_total`, `it`::: include::{docdir}/rest-api/common-parms.asciidoc[tag=index-total] `indexed_documents_exp_avg`, `idea`::: From d7c7a11667f59d25b175c41063df42be3fc5204b Mon Sep 17 00:00:00 2001 From: lcawl Date: Fri, 20 Mar 2020 13:56:25 -0700 Subject: [PATCH 4/5] [DOCS] Augments parameter descriptions --- docs/reference/cat/transforms.asciidoc | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/docs/reference/cat/transforms.asciidoc b/docs/reference/cat/transforms.asciidoc index ac7bc4b019a1b..b7aba455eb573 100644 --- a/docs/reference/cat/transforms.asciidoc +++ b/docs/reference/cat/transforms.asciidoc @@ -102,14 +102,14 @@ The total number of invocations. `max_page_search_size`, `mpsz`::: (Default) -The maximum page search size. +include::{docdir}/rest-api/common-parms.asciidoc[tag=pivot-max-page-search-size] `page_total`, `pt`::: include::{docdir}/rest-api/common-parms.asciidoc[tag=pages-processed] `pipeline`, `p`::: (Default) -The {transform} pipeline. +include::{docdir}/rest-api/common-parms.asciidoc[tag=dest-pipeline] `processed_documents_exp_avg`, `pdea`::: include::{docdir}/rest-api/common-parms.asciidoc[tag=exponential-avg-documents-processed] @@ -136,7 +136,7 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=state-transform] `transform_type`, `tt`::: (Default) -Indicates whether it's a batch or continuous {transform}. +Indicates the type of {transform}: `batch` or `continuous`. `version`, `v`::: (Default) @@ -160,13 +160,26 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=cat-v] [source,console] -------------------------------------------------- -GET /_cat/transforms?v +GET /_cat/transforms?v&format=json -------------------------------------------------- // TEST[skip:kibana sample data] [source,console-result] ---- -id create_time version source_index dest_index pipeline description transform_type frequency max_page_search_size state -ecommerce-customer-transform 2020-03-18T23:39:29.525Z 8.0.0 kibana_sample_data_ecommerce ecommerce-customers batch 1m 500 STOPPED +[ + { + "id" : "ecommerce_transform", + "create_time" : "2020-03-20T20:31:25.077Z", + "version" : "7.7.0", + "source_index" : "kibana_sample_data_ecommerce", + "dest_index" : "kibana_sample_data_ecommerce_transform", + "pipeline" : null, + "description" : "Maximum priced ecommerce data by customer_id in Asia", + "transform_type" : "continuous", + "frequency" : "5m", + "max_page_search_size" : "500", + "state" : "STARTED" + } +] ---- // TESTRESPONSE[skip:kibana sample data] From 3ebbe6e51ccfed84e97ff3e56ca81538756ed573 Mon Sep 17 00:00:00 2001 From: lcawl Date: Mon, 23 Mar 2020 17:23:12 -0700 Subject: [PATCH 5/5] [DOCS] Addresses feedback --- docs/reference/cat/transforms.asciidoc | 6 +++--- docs/reference/rest-api/common-parms.asciidoc | 17 +++++++++++++++++ .../transform/apis/get-transform-stats.asciidoc | 14 +++++++------- 3 files changed, 27 insertions(+), 10 deletions(-) diff --git a/docs/reference/cat/transforms.asciidoc b/docs/reference/cat/transforms.asciidoc index b7aba455eb573..81c22a31d91b1 100644 --- a/docs/reference/cat/transforms.asciidoc +++ b/docs/reference/cat/transforms.asciidoc @@ -75,7 +75,7 @@ The description of the {transform}. include::{docdir}/rest-api/common-parms.asciidoc[tag=dest-index] `document_total`, `dt`::: -The total number of documents. +include::{docdir}/rest-api/common-parms.asciidoc[tag=docs-processed] `frequency`, `f`::: (Default) @@ -98,7 +98,7 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=index-total] include::{docdir}/rest-api/common-parms.asciidoc[tag=exponential-avg-documents-indexed] `invocation_total`, `itotal`::: -The total number of invocations. +include::{docdir}/rest-api/common-parms.asciidoc[tag=trigger-count] `max_page_search_size`, `mpsz`::: (Default) @@ -115,7 +115,7 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=dest-pipeline] include::{docdir}/rest-api/common-parms.asciidoc[tag=exponential-avg-documents-processed] `reason`, `r`::: -The reason for the current state. +include::{docdir}/rest-api/common-parms.asciidoc[tag=state-transform-reason] `search_failure`, `sf`::: include::{docdir}/rest-api/common-parms.asciidoc[tag=search-failures] diff --git a/docs/reference/rest-api/common-parms.asciidoc b/docs/reference/rest-api/common-parms.asciidoc index 42294029da992..793d7044ed836 100644 --- a/docs/reference/rest-api/common-parms.asciidoc +++ b/docs/reference/rest-api/common-parms.asciidoc @@ -182,6 +182,11 @@ is based on Lucene documents. {es} reclaims the disk space of deleted Lucene documents when a segment is merged. end::docs-deleted[] +tag::docs-processed[] +The number of documents that have been processed from the source index of +the {transform}. +end::docs-processed[] + tag::enrich-policy[] Enrich policy name used to limit the request. @@ -864,6 +869,11 @@ documents. -- end::state-transform[] +tag::state-transform-reason[] +If a {transform} has a `failed` state, this property provides details about the +reason for the failure. +end::state-transform-reason[] + tag::stats[] `stats`:: (Optional, string) Specific `tag` of the request for logging and statistical @@ -959,6 +969,13 @@ expression. If you do not specify one of these options, the API returns information for all {transforms}. end::transform-id-wildcard[] +tag::trigger-count[] +The number of times the {transform} has been triggered by the scheduler. For +example, the scheduler triggers the {transform} indexer to check for updates +or ingest new data at an interval specified in the +<>. +end::trigger-count[] + tag::cat-v[] `v`:: (Optional, boolean) If `true`, the response includes column headings. diff --git a/docs/reference/transform/apis/get-transform-stats.asciidoc b/docs/reference/transform/apis/get-transform-stats.asciidoc index fed7eadb29948..57bb61ebac31d 100644 --- a/docs/reference/transform/apis/get-transform-stats.asciidoc +++ b/docs/reference/transform/apis/get-transform-stats.asciidoc @@ -128,18 +128,20 @@ started. `node`.`transport_address`::: (string) The host and port where transport HTTP connections are accepted. For example, `127.0.0.1:9300`. +`reason`:: +(string) +include::{docdir}/rest-api/common-parms.asciidoc[tag=state-transform-reason] `state`:: (string) include::{docdir}/rest-api/common-parms.asciidoc[tag=state-transform] - `stats`:: (object) An object that provides statistical information about the {transform}. `stats`.`documents_indexed`::: (long) The number of documents that have been indexed into the destination index for the {transform}. `stats`.`documents_processed`::: -(long) The number of documents that have been processed from the source index of -the {transform}. +(long) +include::{docdir}/rest-api/common-parms.asciidoc[tag=docs-processed] `stats`.`exponential_avg_checkpoint_duration_ms`::: (double) include::{docdir}/rest-api/common-parms.asciidoc[tag=exponential-avg-checkpoint-duration-ms] @@ -171,10 +173,8 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=search-time-ms] (long) include::{docdir}/rest-api/common-parms.asciidoc[tag=search-total] `stats`.`trigger_count`::: -(long) The number of times the {transform} has been triggered by the scheduler. -For example, the scheduler triggers the {transform} indexer to check for updates -or ingest new data at an interval specified in the -<>. +(long) +include::{docdir}/rest-api/common-parms.asciidoc[tag=trigger-count] [[get-transform-stats-response-codes]] ==== {api-response-codes-title}