token_count type : add an option to count tokens (and not positions)

[fbaligand]

Elasticsearch version: 2.4.2 and 5.2.1

Description of the problem including expected versus actual behavior:
Currently, if I have a "token_count" field, based on an analyzer containing a stop filter, the indexed "token_count" field counts stop words.
It would be great to have an option to only count analysis result tokens.

Steps to reproduce:

1. I have this index configuration :

{
  "settings": {
    "index": {
      "analysis": {
        "analyzer": {
          "default": {
            "tokenizer": "standard",
            "filter": [
              "standard",
              "lowercase",
              "stop_words"
            ]
          }
        },
        "filter": {
          "stop_words": {
            "type": "stop",
            "stopwords": [
              "this", "is", "a"
            ]
          }
        }
      }
    }
  },
  "mappings": {
      "properties": {
        "mytext": {
        	"index": "analyzed",
         	"type": "string",
         	"analyzer": "default",
         	"fields": {
         		"length": {
         			"type": "token_count",
         			"analyzer": "default",
         			"store": "yes"
         		}
         	}
        }
      }
    }
  }
}

2. I index this document :

{
 "mytext": "this is a cat"
}

3. I make this search query :

GET _search
{
  "query": {
    "query_string": {
      "query": "mytext.length:1"
    }
  }
}

4. It should return 1 result. But it actually returns 0 result.

[clintongormley]

This limitation is documented:

Technically the token_count type sums position increments rather than counting tokens. This means that even if the analyzer filters out stop words they are included in the count.

[clintongormley]

@nik9000 how feasible would it be to change this behaviour?

[jpountz]

Maybe we could add a discount_overlaps option like similarities have.

[nik9000]

I'm not sure! It has been years since I looked at the APIs this is built on.

[fbaligand]

Is it possible to have an option so that :

• first, the field source content pass through all analyzer chain (char_filter, tokenizer, token filter),
• and at the end, we count the result tokens and store it in the indexed field

By this way, we are sure that token_count equals the analyzer tokens count

[jimczi]

@fbaligand that's the current behavior except that we count the number of positions created by the analyzer chain. The stop filter removes some token but increment the position on the token right after it in order to be able to build accurate phrase query.
I think that discount_overlaps is what we currently do by counting the positions and not the tokens. Though it would not solve this issue which is about position increment greater than 1.
enable_position_increments is probably what we're looking after (like query parsers have) so that we don't count position holes ?

[fbaligand]

@jimczi
Thanks a lot for this explanation ! I understand better now.
An option like enable_position_increments would be great !

[nik9000]

Yeah, like enable_position_increments: false to get the behavior that you are asking for. What we have is true.

[fbaligand]

That would be perfect !

[fbaligand]

Hi @clintongormley, @jpountz, @nik9000, @jimczi

I just submitted a PR (#24175) which adds the new boolean option "enable_position_increments", as described by @nik9000.
By default, option value is true (which means that positions increments are counted).
And if option is set to false, only result tokens (after full analysis) are counted.