Skip to content

[Transform] refactor cat transform to show more useful information #68232

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 2, 2021
Merged

[Transform] refactor cat transform to show more useful information #68232

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
00bc3f3
refactor cat transform to show more useful information
Jan 29, 2021
bba311b
fix defaults in cat docs
Feb 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 19 additions & 19 deletions docs/reference/cat/transforms.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,35 +57,40 @@ specified columns.
Valid columns are:

`changes_last_detection_time`, `cldt`:::
(Default)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=checkpointing-changes-last-detected-at]

`checkpoint`, `cp`:::
(Default)
The sequence number for the checkpoint.

`checkpoint_duration_time_exp_avg`, `cdtea`, `checkpointTimeExpAvg`:::
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=exponential-avg-checkpoint-duration-ms]

`create_time`, `ct`, `createTime`:::
`checkpoint_progress`, `c`, `checkpointProgress`:::
(Default)
The progress of the next checkpoint that is currently in progress.

`create_time`, `ct`, `createTime`:::
The time the {transform} was created.

`description`, `d`:::
(Default)
The description of the {transform}.

`dest_index`, `di`, `destIndex`:::
(Default)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=dest-index]

`documents_indexed`, `doci`:::
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=docs-indexed]

`docs_per_second`, `dps`:::
(Default)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-settings-docs-per-second]

`documents_processed`, `docp`:::
(Default)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=docs-processed]

`frequency`, `f`:::
(Default)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=frequency]

`id`:::
Expand All @@ -104,15 +109,17 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=index-total]
`indexed_documents_exp_avg`, `idea`:::
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=exponential-avg-documents-indexed]

`max_page_search_size`, `mpsz`:::
`last_search_time`, `lst`, `lastSearchTime`:::
(Default)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=checkpointing-last-search-time]

`max_page_search_size`, `mpsz`:::
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-settings-max-page-search-size]

`pages_processed`, `pp`:::
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=pages-processed]

`pipeline`, `p`:::
(Default)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=dest-pipeline]

`processed_documents_exp_avg`, `pdea`:::
Expand Down Expand Up @@ -142,14 +149,12 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=source-index-transform
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=state-transform]

`transform_type`, `tt`:::
(Default)
Indicates the type of {transform}: `batch` or `continuous`.

`trigger_count`, `tc`:::
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=trigger-count]

`version`, `v`:::
(Default)
The version of {es} that existed on the node when the {transform} was
created.

Expand Down Expand Up @@ -179,16 +184,11 @@ GET /_cat/transforms?v=true&format=json
[
{
"id" : "ecommerce_transform",
"create_time" : "2020-03-20T20:31:25.077Z",
"version" : "7.7.0",
"source_index" : "kibana_sample_data_ecommerce",
"dest_index" : "kibana_sample_data_ecommerce_transform",
"pipeline" : null,
"description" : "Maximum priced ecommerce data by customer_id in Asia",
"transform_type" : "continuous",
"frequency" : "5m",
"max_page_search_size" : "500",
"state" : "STARTED"
"state" : "started",
"checkpoint" : "1",
"documents_processed" : "705",
"checkpoint_progress" : "100.00",
"changes_last_detection_time" : null
}
]
----
Expand Down
5 changes: 5 additions & 0 deletions docs/reference/rest-api/common-parms.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,11 @@ tag::checkpointing-changes-last-detected-at[]
The timestamp when changes were last detected in the source indices.
end::checkpointing-changes-last-detected-at[]

tag::checkpointing-last-search-time[]
The timestamp of the last search in the source indices. This field is only
shown if the transform is running.
end::checkpointing-last-search-time[]

tag::cluster-health-status[]
(string)
Health status of the cluster, based on the state of its primary and replica
Expand Down
4 changes: 4 additions & 0 deletions docs/reference/transform/apis/get-transform-stats.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,10 @@ was created.
=====
//End checkpointing.last

`last_search_time`:::
(date)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=checkpointing-last-search-time]

//Begin checkpointing.next
`next`:::
(object) Contains statistics about the next checkpoint that is currently in
Expand Down
5 changes: 5 additions & 0 deletions .../core/src/main/java/org/elasticsearch/xpack/core/transform/transforms/TransformStats.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -289,6 +289,11 @@ public void writeTo(StreamOutput out) throws IOException {
out.writeEnum(this);
}

@Override
public String toString() {
return name().toLowerCase(Locale.ROOT);
}

public String value() {
return name().toLowerCase(Locale.ROOT);
}
Expand Down
94 changes: 83 additions & 11 deletions x-pack/plugin/src/test/resources/rest-api-spec/test/transform/transforms_cat_apis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,57 @@ setup:
}
}

- do:
index:
index: airline-data
id: 1
body: >
{
"time": "2017-02-18T00:00:00Z",
"airline": "airline1",
"responsetime": 1.0,
"event_rate": 5
}

- do:
index:
index: airline-data
id: 2
body: >
{
"time": "2017-02-18T00:30:00Z",
"airline": "airline1",
"responsetime": 1.0,
"event_rate": 6
}

- do:
index:
index: airline-data
id: 3
body: >
{
"time": "2017-02-18T01:00:00Z",
"airline": "airline2",
"responsetime": 42.0,
"event_rate": 8
}

- do:
index:
index: airline-data
id: 4
body: >
{
"time": "2017-02-18T01:01:00Z",
"airline": "airline1",
"responsetime": 42.0,
"event_rate": 7
}

- do:
indices.refresh:
index: airline-data
---
teardown:
- do:
Expand All @@ -70,9 +121,9 @@ teardown:
transform_id: "airline-transform-*"
- match:
$body: |
/^ #id \s+ create_time \s+ version \s+ source_index \s+ dest_index \s+ pipeline \s+ transform_type \s+ frequency \s+ max_page_search_size \s+ docs_per_second \s+ state \n
(airline\-transform\-latest \s+ [^\s]+ \s+ [^\s]+ \s+ airline-data \s+ airline-data-latest \s+ \s+ batch \s+ 1m \s+ 500 \s+ - \s+ STOPPED \n)+
(airline\-transform\-stats \s+ [^\s]+ \s+ [^\s]+ \s+ airline-data \s+ airline-data-by-airline \s+ \s+ batch \s+ 1m \s+ 500 \s+ - \s+ STOPPED \n)+ $/
/^ #id \s+ state \s+ checkpoint \s+ documents_processed \s+ checkpoint_progress \s+ last_search_time \s+ changes_last_detection_time \n
(airline\-transform\-latest \s+ stopped \s+ 0 \s+ 0 \s+ \s+ \s+ \n)+
(airline\-transform\-stats \s+ stopped \s+ 0 \s+ 0 \s+ \s+ \s+ \n)+ $/

---
"Test cat transform stats with column selection":
Expand All @@ -84,9 +135,8 @@ teardown:
- match:
$body: |
/^ id \s+ version \s+ source_index \s+ dest_index \s+ search_total \s+ index_total \s+ docp \s+ cdtea \s+ indexed_documents_exp_avg \n
(airline\-transform-latest \s+ [^\s]+ \s+ airline-data \s+ airline-data-latest \s+ 0 \s+ 0 \s+ 0 \s+ 0.0 \s+ 0.0 \n)+
(airline\-transform-stats \s+ [^\s]+ \s+ airline-data \s+ airline-data-by-airline \s+ 0 \s+ 0 \s+ 0 \s+ 0.0 \s+ 0.0 \n)+ $/

(airline\-transform-latest \s+ [^\s]+ \s+ airline-data \s+ airline-data-latest \s+ 0 \s+ 0 \s+ 0 \s+ 0.00 \s+ 0.00 \n)+
(airline\-transform-stats \s+ [^\s]+ \s+ airline-data \s+ airline-data-by-airline \s+ 0 \s+ 0 \s+ 0 \s+ 0.00 \s+ 0.00 \n)+ $/

---
"Test cat transform stats with batch transform":
Expand All @@ -97,7 +147,7 @@ teardown:
{
"source": {
"index": ["airline-data", "airline-data-other"],
"query": {"bool":{"filter":{"term":{"airline":"foo"}}}}
"query": {"bool":{"filter":{"term":{"airline":"airline1"}}}}
},
"dest": { "index": "airline-data-by-airline-batch" },
"pivot": {
Expand All @@ -112,8 +162,30 @@ teardown:
v: true
- match:
$body: |
/^ id \s+ create_time \s+ version \s+ source_index \s+ dest_index \s+ pipeline \s+ description \s+ transform_type \s+ frequency \s+ max_page_search_size \s+ docs_per_second \s+ state \n
(airline\-transform\-batch \s+ [^\s]+ \s+ [^\s]+ \s+ airline-data,airline-data-other \s+ airline-data-by-airline-batch \s+ \s+ description \s+ batch \s+ 1m \s+ 500 \s+ - \s+ STOPPED \n)+ $/
/^ id \s+ state \s+ checkpoint \s+ documents_processed \s+ checkpoint_progress \s+ last_search_time \s+ changes_last_detection_time \n
(airline\-transform\-batch \s+ stopped \s+ 0 \s+ 0 \s+ \s+ \s+ \n)+ $/
- do:
transform.start_transform:
transform_id: "airline-transform-batch"
- match: { acknowledged: true }

- do:
transform.stop_transform:
wait_for_checkpoint: true
transform_id: "airline-transform-batch"
wait_for_completion: true
- match: { acknowledged: true }

- do:
cat.transforms:
transform_id: "airline-transform-batch"
v: true

# see gh#62204 despite wait_for_completion is true, it might still not be stopped
- match:
$body: |
/^ id \s+ state \s+ checkpoint \s+ documents_processed \s+ checkpoint_progress \s+ last_search_time \s+ changes_last_detection_time \n
(airline\-transform\-batch \s+ stop.* \s+ 1 \s+ 3 \s+ 100.00 \s+ \s+ .* \n)+ $/
- do:
transform.delete_transform:
transform_id: "airline-transform-batch"
Expand Down Expand Up @@ -148,8 +220,8 @@ teardown:
v: true
- match:
$body: |
/^ id \s+ create_time \s+ version \s+ source_index \s+ dest_index \s+ pipeline \s+ description \s+ transform_type \s+ frequency \s+ max_page_search_size \s+ docs_per_second \s+ state \n
(airline\-transform\-continuous \s+ [^\s]+ \s+ [^\s]+ \s+ airline-data,airline-data-other \s+ airline-data-by-airline-continuous \s+ \s+ description \s+ continuous \s+ 10s \s+ 500 \s+ - \s+ STOPPED \n)+ $/
/^ id \s+ state \s+ checkpoint \s+ documents_processed \s+ checkpoint_progress \s+ last_search_time \s+ changes_last_detection_time \n
(airline\-transform\-continuous \s+ stopped \s+ 0 \s+ 0 \s+ \s+ \s+ \n)+ $/
- do:
transform.delete_transform:
transform_id: "airline-transform-continuous"
Loading