null pointer exception with token_count data type and indexing a null value

[joshuar]

Elasticsearch version: 5.4.0

Plugins installed: X-Pack

JVM version (java -version):

        "version": "1.8.0_131",
        "vm_name": "OpenJDK 64-Bit Server VM",
        "vm_version": "25.131-b11",
        "vm_vendor": "Oracle Corporation",

OS version (uname -a if on a Unix-like system):

        "name": "Linux",
        "arch": "amd64",
        "version": "4.10.17-200.fc25.x86_64",

Fedora 25

Description of the problem including expected versus actual behavior:

It looks like there is an issue with the handling of null values for the token_count data type. Trying to index a document that has a multi-field with a token_count data type results in a null pointer exception. It seems like token_count should support null values?

Steps to reproduce:

1. Create a mapping using a token_count field:

PUT /test
{
  "mappings": {
    "doc": {
      "properties": {
        "test": {
          "type": "keyword",
          "fields": {
            "nr": {
              "type": "token_count",
              "analyzer": "default"
            }
          }
        }
      }
    }
  }
}

2. Add a doc without a null value works:

POST /test/doc
{
  "test": "1234 5678"
}

3. Add a doc with a null value throws an exception:

POST /test/doc
{
  "test": null
} 

Results in the following exception thrown back at the client:

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "failed to parse [test.nr]"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "failed to parse [test.nr]",
    "caused_by": {
      "type": "null_pointer_exception",
      "reason": null
    }
  },
  "status": 400
}

And I see the following thrown at DEBUG level in the Elasticsearch logs:

[2017-05-29T04:18:55,042][DEBUG][r.suppressed             ] path: /test/doc, params: {index=test, type=doc}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [test.nr]
	at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:298) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.mapper.FieldMapper$MultiFields.parse(FieldMapper.java:561) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:300) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:450) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.mapper.DocumentParser.parseNullValue(DocumentParser.java:595) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:392) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:373) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:93) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:66) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:277) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:536) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShard.prepareIndexOnPrimary(IndexShard.java:513) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.prepareIndexOperationOnPrimary(TransportShardBulkAction.java:450) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:458) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:143) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:113) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:69) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:939) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:908) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:322) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:264) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:888) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:885) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:147) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1658) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:897) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.access$400(TransportReplicationAction.java:93) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:281) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:260) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:252) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:245) ~[?:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:294) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$authorizeAsync$4(ServerTransportFilter.java:180) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:127) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:121) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:109) ~[?:?]
	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.authorizeAsync(ServerTransportFilter.java:182) ~[?:?]
	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$3(ServerTransportFilter.java:170) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:192) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$2(AuthenticationService.java:212) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:224) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:190) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:147) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:118) ~[?:?]
	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:144) ~[?:?]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:301) ~[?:?]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:627) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.4.0.jar:5.4.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.lang.NullPointerException
	at org.elasticsearch.index.mapper.TokenCountFieldMapper.parseCreateField(TokenCountFieldMapper.java:128) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:287) ~[elasticsearch-5.4.0.jar:5.4.0]
	... 58 more

[hlidek]

I've experienced the same NPE issue.

• Elasticsearch version: 5.3.0
• without plugins
• Java 1.8.0_121, Ubuntu 16.04.2 LTS

[fred84]

@clintongormley May I take this issue?

[bleskes]

@fred84 go for it. Thanks.