Skip to content

Painless: Add reindex docs example #34024

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

Painless: Add reindex docs example #34024

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
25b4525
Painless: Add a reindex example to the context documentation.
jdconrad Sep 25, 2018
e876161
Fix console error.
jdconrad Sep 25, 2018
368aaf6
Fix test.
jdconrad Sep 25, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/painless/painless-contexts.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,12 @@ specialized code may define new ways to use a Painless script.
| Elasticsearch Documentation
| Ingest processor | <<painless-ingest-processor-context, Painless Documentation>>
| {ref}/script-processor.html[Elasticsearch Documentation]
| Reindex | <<painless-reindex-context, Painless Documentation>>
| {ref}/docs-reindex.html[Elasticsearch Documentation]
| Update | <<painless-update-context, Painless Documentation>>
| {ref}/docs-update.html[Elasticsearch Documentation]
| Update by query | <<painless-update-by-query-context, Painless Documentation>>
| {ref}/docs-update-by-query.html[Elasticsearch Documentation]
| Reindex | <<painless-reindex-context, Painless Documentation>>
| {ref}/docs-reindex.html[Elasticsearch Documentation]
| Sort | <<painless-sort-context, Painless Documentation>>
| {ref}/search-request-sort.html[Elasticsearch Documentation]
| Similarity | <<painless-similarity-context, Painless Documentation>>
Expand Down
4 changes: 2 additions & 2 deletions docs/painless/painless-contexts/index.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
include::painless-ingest-processor-context.asciidoc[]

include::painless-reindex-context.asciidoc[]

include::painless-update-context.asciidoc[]

include::painless-update-by-query-context.asciidoc[]

include::painless-reindex-context.asciidoc[]

include::painless-sort-context.asciidoc[]

include::painless-similarity-context.asciidoc[]
Expand Down
183 changes: 182 additions & 1 deletion docs/painless/painless-contexts/painless-reindex-context.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,4 +65,185 @@ reindexed into a target index.

*API*

The standard <<painless-api-reference, Painless API>> is available.
The standard <<painless-api-reference, Painless API>> is available.

*Example*

To run this example, first follow the steps in
<<painless-context-examples, context examples>>.

Copy link
Contributor

@debadair debadair Sep 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd mention that other samples build on this one here, rather than as part of what this example does. Maybe just, "The afternoon and evening indices created by this script are used in other examples." I'd probably also move this notice below the example & its description, just before the instructions to set the mappings & call reindex.

The example reindex script accomplishes the following:

Copy link
Contributor

@debadair debadair Sep 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd just say, "This example:"

* Separates the `seats` index into two different new indexes - `afternoon` and
`evening` based on when a play begins.
* Removes the `date` and `time` fields that are extraneous since the
Copy link
Contributor

@debadair debadair Sep 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing a - after evening. (I'd be inclined to use commas.)

<<painless-ingest-processor-context, ingest example>> added a single combined
`datetime` field.
* Adds a `sold_datetime` field to each document that is filled in with the
current date and time of when a seat is sold.
* Sets the `sold` field to true along with a value for the `sold_datetime` field
for a specific subset of seats for use in future examples.

[source,Painless]
----
void setSold(Map _source) { <1>
if ((_source["number"] + _source["row"] % 4) % 3 == 0 || <2>
_source["number"] % 5 == 0) {
_source["sold"] = true; <3>
_source["sold_datetime"] = _source["datetime"] - <4>
86400000 * (1 + _source["number"] % 14);
}
}

void removeExtraneous(Map _source) { <5>
_source.remove("date"); <6>
_source.remove("time"); <7>
}

setSold(ctx["_source"]); <8>
removeExtraneous(ctx["_source"]); <9>

Instant instant = Instant.ofEpochMilli(ctx["_source"]["datetime"]); <10>
ZonedDateTime dt = instant.atZone(ZoneId.of("GMT+8")); <11>

if (dt.getHour() > 16) { <12>
ctx["_index"] = "evening"; <13>
}
----
<1> Creates a `setSold` <<painless-functions, function>> to set a seat to sold
if it meets certain criteria for use in future examples.
Note::
Copy link
Contributor

@debadair debadair Sep 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd drop the "for use in future examples" and say "...to mark a seat as sold if it meets certain criteria."

Instead of a callout for every line, I'd be inclined to break this info out into more of a narrative.

In this reindex script, the setSold <<painless-functions, function>> marks a seat as sold if it meets certain criteria. The seat number and row are extracted from the source to perform the evaluation. To mark a seat sold, the sold field is set to the boolean value true and sold_datetime is set to an arbitrary value up to 15 days prior to the document's datetime value. When setting sold_datetime, the <<precedence-operator, precedence operator>> is used to guarantee that the number of days in the equation is evaluated prior to subtracting the number of seconds from the play's date and time.

NOTE: The changes to the _source Map type value that's passed in to setSold are reflected throughout the script since it's a <<reference-types, reference type>> value.

The removeExtraneous function removes the redundant date and time fields from the document source.

When these functions are called, the ctx["_source"] reindex context variable is used to retrieve the Map reference type value that contains the document's fields.

To determine if the seat should be added to the evening index or the default afternoon index, the script:

. Converts the datetime value from a <<primitive-types, long>> type to an Instant reference type by calling the API static method ofEpochMilli.
. Converts the instant value to the DateTime reference type using the API static method atZone. The time zone is specified using the API static method of on the reference type ZoneId.
. Calls getHour to get the hour from the DateTime value and checks to see if it's past 4:00PM GMT+8 (PST).

* The changes to the `Map` type value `_source` passed in to the function
are reflected throughout the script since it's a
<<reference-types, reference type>> value.
<2> Does a simple calculation to determine if the seat is sold based on
arbitrary criteria to create a set of sold seats for use in future examples.
<3> Sets the field `sold` to the boolean value `true`.
<4> Sets the field `sold_datetime` to a value between `1` to `15` days prior to
the play's date and time.
Note::
The <<precedence-operator, precedence operator>> is used to guarantee
the number of days in the equation is evaluated prior to subtracting
the number of seconds from the play's date and time.
<5> Creates a `removeExtraneous` function to remove redundant fields from the
document.
Note::
* The changes to the `Map` type value `_source` passed in to the function
are reflected throughout the script since it's a reference type value.
<6> Removes the `date` field from the document using the API non-static method
`remove`.
<7> Removes the `time` field from the document using the API non-static method
`remove`.
<8> Uses the `setSold` function to check if a seat is sold and update the
appropriate fields if so.
Note::
* The use of the `ctx["_source"]` reindex context
<<painless-variables, variable>> to retrieve the `Map` reference type
value containing the document's fields.
<9> Uses the `removeExtraneous` function to remove redundant fields from the
document.
Note::
* The use of the `ctx["_source"]` reindex context variable to retrieve the
`Map` reference type value containing the document's fields.
<10> Creates an `Instant` reference type variable `instant` and uses the API
static method `ofEpochMilli` to convert the `datetime` field from a
<<primitive-types, `long`>> type value into an `Instant` reference type
value.
Note::
* The use of the `ctx["_source"]` reindex context variable to retrieve the
`Map` reference type value containing the document's fields.
<11> Creates a `ZonedDateTime` reference type variable `dt` and uses the API
static method `atZone` to convert the `Instant` reference type value into a
`DateTime` reference type value. The time zone is specified using the
API static method `of` on the reference type `ZoneId`.
<12> Checks to see if the hour is past `4:00PM GMT+8 (PST)`.
<13> Changes from the default `afternoon` index (specified as part of the
upcoming curl request) to the `evening` index using the ctx["_index"]
reindex context variable.

Submit the following requests:

Copy link
Contributor

@debadair debadair Sep 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of "Submit the following requests" I'd say "To use this script to reindex the seats index:"

. Create {ref}/mapping.html[mappings] for the `afternoon` index:
+
[source,js]
----
PUT /afternoon
{
"mappings": {
"seat": {
"properties": {
"theatre": { "type": "keyword" },
"play": { "type": "text" },
"actors": { "type": "text" },
"row": { "type": "integer" },
"number": { "type": "integer" },
"cost": { "type": "double" },
"sold": { "type": "boolean" },
"sold_datetime": { "type": "date" },
"datetime": { "type": "date" }
}
}
}
}
----
+
// CONSOLE

. Create {ref}/mapping.html[mappings] for the `evening` index:
+
[source,js]
----
PUT /evening
{
"mappings": {
"seat": {
"properties": {
"theatre": { "type": "keyword" },
"play": { "type": "text" },
"actors": { "type": "text" },
"row": { "type": "integer" },
"number": { "type": "integer" },
"cost": { "type": "double" },
"sold": { "type": "boolean" },
"sold_datetime": { "type": "date" },
"datetime": { "type": "date" }
}
}
}
}
----
+
// CONSOLE

. Submit the reindex request:
+
[source,js]
----
POST /_reindex
{
"source": {
"index": "seats"
},
"dest": {
"index": "afternoon"
},
"script": {
"source": "void setSold(Map _source) { if ((_source[\"number\"] + _source[\"row\"] % 4) % 3 == 0 || _source[\"number\"] % 5 == 0) { _source[\"sold\"] = true; _source[\"sold_datetime\"] = _source[\"datetime\"] - 86400000 * (1 + _source[\"number\"] % 14); } } void removeExtraneous(Map _source) { _source.remove(\"date\"); _source.remove(\"time\"); } setSold(ctx[\"_source\"]); removeExtraneous(ctx[\"_source\"]); Instant instant = Instant.ofEpochMilli(ctx[\"_source\"][\"datetime\"]); ZonedDateTime dt = instant.atZone(ZoneId.of(\"GMT+8\")); if (dt.getHour() > 16) { ctx[\"_index\"] = \"evening\"; }"
}
}
----
+
// CONSOLE
// TEST[skip: requires setup from other pages]

. The reindex request may take some time to complete even after a successful
response is received. The `afternoon` index has 18312 documents, and the
`evening` index has 17892 documents. Submit the following request to check the
number of documents in each index:
+
Copy link
Contributor

@debadair debadair Sep 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't make this a step. I'd drop the "Submit the following request" and just say, "You can use _cat/indices to check the number of documents in each index:"

[source,js]
----
GET /_cat/indices?v
----
+
// CONSOLE
Loading