5.0.0-rc Painless support for working with dates

[niemyjski]

I'm upgrading from groovy to painless and it's not so painless so far. I wanted to post this here as I think this would be a pretty common scenario and I'm having trouble with dates (gotta love dates)

PUT /test-stacks-v1
{
  "aliases": {
    "test-stacks": {}
  },
  "mappings": {
    "stacks": {
      "_all": {
        "enabled": false
      },
      "dynamic": false,
      "properties": {
        "id": {
          "type": "keyword"
        },
        "first_occurrence": {
          "type": "date"
        },
        "last_occurrence": {
          "type": "date"
        },
        "total_occurrences": {
          "type": "float"
        }
      }
    }
  }
}

PUT /test-stacks/stacks/1ecd0826e447a44e78877ab1
{
  "id": "1ecd0826e447a44e78877ab1",
  "total_occurrences": 0,
  "first_occurrence": "0001-01-01T00:00:00",
  "last_occurrence": "0001-01-01T00:00:00"
}

POST /test-stacks/stacks/1ecd0826e447a44e78877ab1/_update?retry_on_conflict=3
{
  "script": { 
    "params": {
      "minOccurrenceDateUtc": "2016-10-25T22:04:19.3272056Z",
      "maxOccurrenceDateUtc": "2016-10-25T22:04:19.3272056Z",
      "count": 1
    },
    "inline": "if (ctx._source.total_occurrences == 0 || ctx._source.first_occurrence > params.minOccurrenceDateUtc) { ctx._source.first_occurrence = params.minOccurrenceDateUtc; } if (ctx._source.last_occurrence < params.maxOccurrenceDateUtc) { ctx._source.last_occurrence = params.maxOccurrenceDateUtc; } ctx._source.total_occurrences += params.count;"
  }
}

The last script update returns

json
{
"error": {
"root_cause": [
{
"type": "remote_transport_exception",
"reason": "[8mJiEA3][local[1]][indices:data/write/update[s]]"
}
],
"type": "illegal_argument_exception",
"reason": "failed to execute script",
"caused_by": {
"type": "script_exception",
"reason": "runtime error",
"caused_by": {
"type": "class_cast_exception",
"reason": "Cannot apply [<] operation to types [java.lang.String] and [java.lang.String]."
},
"script_stack": [
"if (ctx._source.last_occurrence < params.maxOccurrenceDateUtc) { ",
" ^---- HERE"
],
"script": "if (ctx._source.total_occurrences == 0 || ctx._source.first_occurrence > params.minOccurrenceDateUtc) { ctx._source.first_occurrence = params.minOccurrenceDateUtc; } if (ctx._source.last_occurrence < params.maxOccurrenceDateUtc) { ctx._source.last_occurrence = params.maxOccurrenceDateUtc; } ctx._source.total_occurrences += params.count;",
"lang": "painless"
}
},
"status": 400
}

The original groovy script worked flawlessly in 1.7.5 (including linebreaks)

``` cs
            var request = new UpdateRequest<Stack, Stack>(GetIndexById(stackId), ElasticType.Type, stackId) {
                RetryOnConflict = 3,
                Script = @"if (ctx._source.total_occurrences == 0 || ctx._source.first_occurrence > minOccurrenceDateUtc) {
                            ctx._source.first_occurrence = minOccurrenceDateUtc;
                            }
                            if (ctx._source.last_occurrence < maxOccurrenceDateUtc) {
                            ctx._source.last_occurrence = maxOccurrenceDateUtc;
                            }
                            ctx._source.total_occurrences += count;",
                Params = new Dictionary<string, object>(3) {
                    { "minOccurrenceDateUtc", minOccurrenceDateUtc },
                    { "maxOccurrenceDateUtc", maxOccurrenceDateUtc },
                    { "count", count }
                }
            };

Is there anything special I need to do to cast a field (which is mapped as a date) and stored as a string to a date reliably?

[niemyjski]

I'll try to update (https://www.elastic.co/guide/en/elasticsearch/reference/5.0/docs-update.html) with a date example once this is figured out :)

[niemyjski]

cc @jdconrad #17992

[jasontedor]

Elastic reserves GitHub for bug reports and feature requests; please post general questions on the Elastic Discourse forum.

[niemyjski]

Considering you don't have a single painless test for dates or docs (https://github.com/elastic/elasticsearch/blob/master/modules/lang-painless/src/test/java/org/elasticsearch/painless/CastTests.java) I think this is a bug/feature report.

[niemyjski]

I've created a post on https://discuss.elastic.co/t/comparing-dates-in-painless/63943 and renamed this title

[niemyjski]

Looks like it's almost impossible to parse an ISO-8601 date with painless (http://stackoverflow.com/questions/2201925/converting-iso-8601-compliant-string-to-java-util-date) am I missing something?

[niemyjski]

To anyone wondering I was able to get this working at least locally, I'm not sure how reliable it is at parsing different date formats as I can't specify Zulu because min date time doesn't have it (pre epoch)

def sf = new SimpleDateFormat(\""yyyy-MM-dd'T'HH:mm:ss\"");
if (ctx._source.total_occurrences == 0 || sf.parse(ctx._source.first_occurrence).after(sf.parse(params.minOccurrenceDateUtc))) {
  ctx._source.first_occurrence = params.minOccurrenceDateUtc;
}
if (sf.parse(ctx._source.last_occurrence).before(sf.parse(params.maxOccurrenceDateUtc))) {
  ctx._source.last_occurrence = params.maxOccurrenceDateUtc;
}
ctx._source.total_occurrences += params.count;

[nik9000]

I'm reopening this because I think it is interesting. I mean, you have a workaround but it kind of sucks for a few reasons:

1. We end up re-parsing the params on every execution.
2. It isn't painless.
3. first_occurrence is a date and should be exposed to painless as a date.
4. We really need to document examples like this.

I don't know how many changes should stem from this issue but my instinct is "a couple".

I do want to be clear on point number 2. Painless's list of priorities are:

1. Security
2. Performance
3. Awesomeness/Painlessness
4. Compatibility with Groovy

The work around you have has trouble with 2 and 3 and 4. I don't know if we'll ever get 4, but we should be able to get 2 and 3.

[niemyjski]

Yeah, I ended up having to do even more work as stated in here (https://discuss.elastic.co/t/comparing-dates-in-painless/63943). I had to create a function to parse the date into a Instant (in a try block) and return a default value. It definitely wasn't straight forward. I appreciate you reopening this.