filtering on _timestamp field (with custom format) not working

[karol-gwaj]

run the example below:

DELETE test

POST test

PUT test/test/_mapping
{
    "test" : 
    {
        "_timestamp" : { "enabled" : true, "store" : true, "format": "yyyyMMddHHmmssSSS"}
    }
}

POST test/test/1?timestamp=20140301101010123
{
    "id" : 1
}

POST test/_search
{
    "fields": [
       "_timestamp"
    ], 
    "query": {
        "range": {
           "_timestamp": {
              "from": 20140301101010000,    
              "to": 20140301101010999
           }
        }
    }
}

i wll expect that last query will return created document (as the timestamp falls into specified range)
also after inserting, timestamp on document is not this same as the one specified in the request (it is in fact 20140301101010124 when it should be 20140301101010123)

context:

• es: 1.0.0
• jvm: 1.7.0_51
• hosted in EC2

[imotov]

This problem occurs because Elasticsearch first tries to parse timestamp as long (number of milliseconds since epoch). So, your timestamp is getting interpreted as Thu, 27 Oct 2033 12:48:30 GMT, while your ranges are getting interpreted correctly as you expect. We could flip this logic in parsing timestamp and try parsing using format first, but then it will break interface for all users who are using milliseconds (with default format just a number is interpreted as year). So, I am not really sure what would be a good solution here, but two possible workarounds here are 1) to use a format that doesn't look like long or 2) switch to milliseconds on the timestamp field.

[karol-gwaj]

sadly im stuck with timestamp in this format (as it comes from external source)
will be nice to have some configuration switch that will allow to change default parsing behavior for timestamp

[clintongormley]

Another place where we don't do the right thing with "1234" vs 1234:

DELETE /t

PUT /t
{
  "mappings": {
    "t": {
      "properties": {
        "date": {
          "type": "date",
          "format": "YYYYMMddHHmmss"
        }
      }
    }
  }
}

PUT /t/t/1
{
  "date": "20140101000000"
}

PUT /t/t/2
{
  "date": 20140101000000
}

GET /t/_search
{
  "script_fields": {
    "date": {
      "script": "doc['date'].value"
    }
  }
}

GET /t/_search
{
  "query": {
    "match": {
      "date": "20140101000000"
    }
  }
}

GET /t/_search
{
  "query": {
    "match": {
      "date": 20140101000000
    }
  }
}

[rjernst]

As Igor said before, part of the problem is date fields have this dual behavior, where it can accept a formatted date, but also an epoch timestamp. However, in this case, the value being searched is passed as a json number. The only way we could interpret that with the original supplied format would be to write the number as a string and then try to parse. I don't think we should do this.

We could however make a couple improvements:

• Add a flag to disable interpreting as an epoch time
• Throw an error if a number is passed in when this feature is disabled

[clintongormley]

Another possibility would be to add epoch as a format, then users can control the order in which the epoch format is tried, eg:

 "format": "epoch|YYYY-mm-dd"

vs

 "format": "YYYY-mm-dd|epoch"

[rjernst]

@clintongormley I really like that idea! I think we should do that, and change the docs to describe the default format to match the current behavior? Should we have a way to specify whether it is seconds or milliseconds epoch? We currently use the first example you gave, with milliseconds epoch.

[clintongormley]

@rjernst yes, i think supporting epoch seconds would be an excellent idea, eg Perl uses floating point seconds. So perhaps epoch_seconds and epoch_ms?