Whitespace tokenizer splits at 255 characters by default

[quilin]

Elasticsearch Version: 5.4.0, Build: 780f8c4/2017-04-28T17:43:27.229Z, JVM: 1.8.0_131

java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

OS Name Microsoft Windows 10 Enterprise
Version 10.0.14393 Build 14393
But the issue seems to reporoduce on some Linux machine (in AWS) as well

Description of the problem including expected versus actual behavior:
The false positive match against the analyzed field. Expected nothing to be found, but found a match instead.

Steps to reproduce:

1. Create the index
   curl -XPUT localhost:9200/test_index_bug {"settings":{"analysis":{"analyzer":{"caseInsensitiveKeyAnalyzer":{"type":"custom","filter":["lowercase"],"tokenizer":"whitespace"}}}},"mappings":{"ric":{"properties":{"id":{"type":"integer"},"ricName":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256},"ricName":{"type":"text","analyzer":"caseInsensitiveKeyAnalyzer"}}},"displayName":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256},"displayName":{"type":"text","analyzer":"caseInsensitiveKeyAnalyzer"}}},"exchangeCode":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"exchangeName":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"domain":{"type":"integer"},"peId":{"type":"integer"},"isAlias":{"type":"boolean"},"createdDate":{"type":"date"},"updatedDate":{"type":"date"}}}}}
2. Put the document
   curl -XPUT localhost:9200/test_index_bug/123 {"id":123,"publicId":"F","ricName":"F","createdDate":"2017-09-12T11:54:51.1421471Z","updatedDate":"2017-09-12T11:54:51.1421471Z","domain":123,"displayName":"test","exchangeNumber":123,"exchangeName":"test","exchangeCode":"test","peId":123,"isAlias":false}
3. Search for the document
   curl -XPOST localhost:9200/_search {"size":1,"query":{"bool":{"must":[{"bool":{"should":[{"match":{"ricName.ricName":{"query":"dgsdlruityhaekljeha;izudyvklaejrbt09834576hgadagyhrtertkghsldkjrhtweiorugysdokljagvalkjdrthwieuartyad78z967agyhdajklskjrntjre.skdauygusldbnalwkeruty57840w67yrukshgdajklsghkdajghslkduayhtweuiry68593470498576wiuergпавыапргцыкше8рполдваяыловраолпдфлгвангшфщыгывнаолдыоамлсичрядлывороалфдцоукрегкшвщышгапниоладчитоваирыолфdhsgjsksrtsrtkdghkdhgkdhkdgkhaasadjlknknkjsgdnm,.sdagnn,mhjlkwerthjlksdgahjklyoiugadsgnm.,hjlkyoiudbnjgadkbkjgadbhkjadgshkjdaghjksdahgshdagjkshgkjahskjlhgakjshgakjsdgdshaglkjsdhgakjdshgakjdslgf"}}}]}}]}}}

Expect nothing to be found, but instead, a hit with a single document is returned:

{ "took": 1, "timed_out": false, "_shards": { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 1, "max_score": 0.2876821, "hits": [ { "_index": "test_index_bug", "_type": "ric", "_id": "123", "_score": 0.2876821, "_source": { "id": 123, "publicId": "F", "ricName": "F", "createdDate": "2017-09-12T11:54:51.1421471Z", "updatedDate": "2017-09-12T11:54:51.1421471Z", "domain": 123, "displayName": "test", "exchangeNumber": 123, "exchangeName": "test", "exchangeCode": "test", "peId": 123, "isAlias": false } } ] } }

The length of the query string is 511 symbols. If I change the length or replace the last letter ("f" for anything but "F"), the response is empty as expected.

[cbuescher]

@quilin thanks for raising this, indeed looks weird. I was able to reproduce on 5.4.3, not sure if it is a bug yet but two things to note:

• I initially thought the nesting of the {"bool":{"must":[{"bool":{"should":[{"match":{"ricName.ricName": .... in your reproduction was part of the problem, but it turns out a simple must clause also exhibits the problem
• when I run this simplified query using the _validate API you can see that the query string gets split right before the "f" that matches your test document:

GET /test_index_bug/ric/_validate/query?explain=true
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "ricName.ricName": {
              "query": "dgsdlruityhaekljeha;izudyvklaejrbt09834576hgadagyhrtertkghsldkjrhtweiorugysdokljagvalkjdrthwieuartyad78z967agyhdajklskjrntjre.skdauygusldbnalwkeruty57840w67yrukshgdajklsghkdajghslkduayhtweuiry68593470498576wiuergпавыапргцыкше8рполдваяыловраолпдфлгвангшфщыгывнаолдыоамлсичрядлывороалфдцоукрегкшвщышгапниоладчитоваирыолфdhsgjsksrtsrtkdghkdhgkdhkdgkhaasadjlknknkjsgdnm,.sdagnn,mhjlkwerthjlksdgahjklyoiugaksgnm.,hjlkyoiudbnjgadkbkjgadbhkjadgshkjdaghjksdahgshdagjkshgkjahskjlhgakjshgakjsdgdshaglkjsdhgakjdshgakjdslgf"
            }
          }
        }
      ]
    }
  }
}

-->

{
  "valid": true,
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "explanations": [
    {
      "index": "test_index_bug",
      "valid": true,
      "explanation": "+(+(ricName.ricName:dgsdlruityhaekljeha;izudyvklaejrbt09834576hgadagyhrtertkghsldkjrhtweiorugysdokljagvalkjdrthwieuartyad78z967agyhdajklskjrntjre.skdauygusldbnalwkeruty57840w67yrukshgdajklsghkdajghslkduayhtweuiry68593470498576wiuergпавыапргцыкше8рполдваяыловраолпдфлгвангшфщы ricName.ricName:гывнаолдыоамлсичрядлывороалфдцоукрегкшвщышгапниоладчитоваирыолфdhsgjsksrtsrtkdghkdhgkdhkdgkhaasadjlknknkjsgdnm,.sdagnn,mhjlkwerthjlksdgahjklyoiugaksgnm.,hjlkyoiudbnjgadkbkjgadbhkjadgshkjdaghjksdahgshdagjkshgkjahskjlhgakjshgakjsdgdshaglkjsdhgakjdshgakjdslg 
ricName.ricName:f)) #_type:ric"
    }
  ]
}

That explains why removing any character or adding more characters doesn't produce a match. My question is why the analyzer (whould be whitespace here) splits the query string and if this is expected. Will mark as bug for now.

[cbuescher]

Okay, seems like the whitespace tokenizer has a maximum token length of 255 after which it starts a new token. This is what the _analysis output of your query string looks like:

{
  "tokens": [
    {
      "token": "dgsdlruityhaekljeha;izudyvklaejrbt09834576hgadagyhrtertkghsldkjrhtweiorugysdokljagvalkjdrthwieuartyad78z967agyhdajklskjrntjre.skdauygusldbnalwkeruty57840w67yrukshgdajklsghkdajghslkduayhtweuiry68593470498576wiuergпавыапргцыкше8рполдваяыловраолпдфлгвангшфщы",
      "start_offset": 0,
      "end_offset": 255,
      "type": "word",
      "position": 0
    },
    {
      "token": "гывнаолдыоамлсичрядлывороалфдцоукрегкшвщышгапниоладчитоваирыолфdhsgjsksrtsrtkdghkdhgkdhkdgkhaasadjlknknkjsgdnm,.sdagnn,mhjlkwerthjlksdgahjklyoiugaksgnm.,hjlkyoiudbnjgadkbkjgadbhkjadgshkjdaghjksdahgshdagjkshgkjahskjlhgakjshgakjsdgdshaglkjsdhgakjdshgakjdslg",
      "start_offset": 255,
      "end_offset": 510,
      "type": "word",
      "position": 1
    },
    {
      "token": "f",
      "start_offset": 510,
      "end_offset": 511,
      "type": "word",
      "position": 2
    }
  ]
}

This seems to be the default behaviour for many tokenizers and is e.g. documented for the Standard tokenizer with examples of how to override this default using the max_token_length parameter. I tried this approach for the whitespace tokenizer just now but wasn't succeful so far. Looking at the code it seems we don't support this configuration option. Changing from "bug" to discussion to get more feedback if we should support this. Also changing the title to reflect the cause of this issue.

[quilin]

That totally makes sense now. Thanks a lot!

[cbuescher]

@quilin out of curiosity: is this something you ran into in a "real" scenario or did this happen just playing around? I'm asking because I'd like to understand in which use cases tokens of this length might appear. I haven't seen this very often so far.

[cbuescher]

I think the current behaviour should be added to our documentation. Even if such long tokens are edge cases, we should allow overriding the "max_token_length" for the whitespace tokenizer like we do for several other tokenizers.

[quilin]

It was found by an accident. Like somebody fell on a keyboard and then found the result.
I agree that the "max_token_length" should be available for overriding, but it's my fault that I did not read the documentation thoroughly.