diff --git a/docs/reference/rest-api/common-parms.asciidoc b/docs/reference/rest-api/common-parms.asciidoc index 030f49288a9ad..050a21eea00e7 100644 --- a/docs/reference/rest-api/common-parms.asciidoc +++ b/docs/reference/rest-api/common-parms.asciidoc @@ -200,8 +200,8 @@ history of operations on a shard. end::docs-deleted[] tag::docs-deleted-transform[] -The number of documents that have been deleted from the destination index -for the {transform}. +The number of documents that have been deleted from the destination index due to +the retention policy for this {transform}. end::docs-deleted-transform[] tag::docs-indexed[] @@ -990,6 +990,24 @@ The `latest` method transforms the data by finding the latest document for each unique key. end::transform-latest[] +tag::transform-retention[] +Defines a retention policy for the {transform}. Data that meets the defined +criteria is deleted from the destination index. +end::transform-retention[] + +tag::transform-retention-time[] +Specifies that the {transform} uses a time field to set the retention policy. +end::transform-retention-time[] + +tag::transform-retention-time-field[] +The date field that is used to calculate the age of the document. +end::transform-retention-time-field[] + +tag::transform-retention-time-max-age[] +Specifies the maximum age of a document in the destination index. Documents that +are older than the configured value are removed from the destination index. +end::transform-retention-time-max-age[] + tag::transform-settings[] Defines optional {transform} settings. end::transform-settings[] diff --git a/docs/reference/transform/apis/get-transform-stats.asciidoc b/docs/reference/transform/apis/get-transform-stats.asciidoc index da49ce694e22d..dbebbe2ebb4c8 100644 --- a/docs/reference/transform/apis/get-transform-stats.asciidoc +++ b/docs/reference/transform/apis/get-transform-stats.asciidoc @@ -310,6 +310,8 @@ The API returns the following results: "pages_processed" : 78, "documents_processed" : 6027, "documents_indexed" : 68, + "documents_deleted": 22, + "delete_time_in_ms": 214, "trigger_count" : 168, "index_time_in_ms" : 412, "index_total" : 20, diff --git a/docs/reference/transform/apis/preview-transform.asciidoc b/docs/reference/transform/apis/preview-transform.asciidoc index a46a48dfc07bc..1f07b5713a08f 100644 --- a/docs/reference/transform/apis/preview-transform.asciidoc +++ b/docs/reference/transform/apis/preview-transform.asciidoc @@ -120,6 +120,32 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=pivot-group-by] ==== //End pivot +//Begin retention policy +`retention_policy`:: +(Optional, object) +include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-retention] ++ +.Properties of `retention_policy` +[%collapsible%open] +==== +`time`::: +(Required, object) +include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-retention-time] ++ +.Properties of `time` +[%collapsible%open] +===== +`field`::: +(Required, string) +include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-retention-time-field] + +`max_age`::: +(Required, <>) +include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-retention-time-max-age] +===== +==== +//End retention policy + //Begin source `source`:: (Required, object) diff --git a/docs/reference/transform/apis/put-transform.asciidoc b/docs/reference/transform/apis/put-transform.asciidoc index ec9032d72335f..d97f95fa2391c 100644 --- a/docs/reference/transform/apis/put-transform.asciidoc +++ b/docs/reference/transform/apis/put-transform.asciidoc @@ -32,16 +32,17 @@ For more information, see <>, <>, and [[put-transform-desc]] == {api-description-title} -This API defines a {transform}, which copies data from source indices, +This API defines a {transform}, which copies data from source indices, transforms it, and persists it into an entity-centric destination index. If you choose to use the pivot method for your {transform}, the entities are defined by the set of `group_by` fields in the `pivot` object. If you choose to use the latest method, the entities are defined by the `unique_key` field values in the `latest` object. -You can also think of the destination index as a two-dimensional tabular data structure (known as a {dataframe}). The ID for each document in the -{dataframe} is generated from a hash of the entity, so there is a unique row -per entity. For more information, see <>. +You can also think of the destination index as a two-dimensional tabular data +structure (known as a {dataframe}). The ID for each document in the {dataframe} +is generated from a hash of the entity, so there is a unique row per entity. For +more information, see <>. When the {transform} is created, a series of validations occur to ensure its success. For example, there is a check for the existence of the @@ -151,6 +152,32 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=pivot-group-by] ==== //End pivot +//Begin retention policy +`retention_policy`:: +(Optional, object) +include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-retention] ++ +.Properties of `retention_policy` +[%collapsible%open] +==== +`time`::: +(Required, object) +include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-retention-time] ++ +.Properties of `time` +[%collapsible%open] +===== +`field`::: +(Required, string) +include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-retention-time-field] + +`max_age`::: +(Required, <>) +include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-retention-time-max-age] +===== +==== +//End retention policy + //Begin settings `settings`:: (Optional, object) @@ -279,6 +306,12 @@ PUT _transform/ecommerce_transform1 "field": "order_date", "delay": "60s" } + }, + "retention_policy": { + "time": { + "field": "order_date", + "max_age": "30d" + } } } -------------------------------------------------- diff --git a/docs/reference/transform/apis/update-transform.asciidoc b/docs/reference/transform/apis/update-transform.asciidoc index 537c8707963fc..25d0e8cadb1ea 100644 --- a/docs/reference/transform/apis/update-transform.asciidoc +++ b/docs/reference/transform/apis/update-transform.asciidoc @@ -40,8 +40,8 @@ When the {transform} is updated, a series of validations occur to ensure its success. You can use the `defer_validation` parameter to skip these checks. All updated properties except description do not take effect until after the -{transform} starts the next checkpoint. This is so there is consistency with the -pivoted data in each checkpoint. +{transform} starts the next checkpoint. This is so there is data consistency in +each checkpoint. [IMPORTANT] ==== @@ -102,6 +102,32 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=dest-pipeline] (Optional, <>) include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=frequency] +//Begin retention policy +`retention_policy`:: +(Optional, object) +include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-retention] ++ +.Properties of `retention_policy` +[%collapsible%open] +==== +`time`::: +(Required, object) +include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-retention-time] ++ +.Properties of `time` +[%collapsible%open] +===== +`field`::: +(Required, string) +include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-retention-time-field] + +`max_age`::: +(Required, <>) +include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-retention-time-max-age] +===== +==== +//End retention policy + //Begin settings `settings`:: (Optional, object)