Cluster crash for case insensitive grok query

[dosera]

Elasticsearch version:
Version: 7.4.0, Build: default/rpm/22e1767283e61a198cb4db791ea66e3f11ab9910/2019-09-27T08:36:48.569419Z, JVM: 13

Plugins installed:
/

JVM version:
java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)

OS version:
Linux 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
When using the grok debugger with a case insensitive pattern (?i) in Kibana a request is sent to elasticsearch that crashes all nodes involved.

This is most likely related to #48899 but I would not expect the whole cluster to crash just because of a query.

Steps to reproduce:

1. Go to kibana's grok debugger
2. Use a grok pattern like ^%{DATA:before}(?<value>(?i)authorization[:=]\s?)(%{DATA:pass})([,]%{GREEDYDATA:after})?$
3. See that your cluster just crashed

Provide logs (if relevant):

[2019-11-19T12:31:30,348][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elk-kafkabox] fatal error in thread [Thread-5], exiting
java.lang.NoSuchFieldError: codeLen
        at org.joni.Analyser.expandCaseFoldStringAlt(Analyser.java:1521) ~[?:?]
        at org.joni.Analyser.expandCaseFoldString(Analyser.java:1586) ~[?:?]
        at org.joni.Analyser.setupTree(Analyser.java:1782) ~[?:?]
        at org.joni.Analyser.setupTree(Analyser.java:1762) ~[?:?]
        at org.joni.Analyser.setupTree(Analyser.java:1873) ~[?:?]
        at org.joni.Analyser.setupTree(Analyser.java:1883) ~[?:?]
        at org.joni.Analyser.setupTree(Analyser.java:1762) ~[?:?]
        at org.joni.Analyser.compile(Analyser.java:113) ~[?:?]
        at org.joni.Regex.<init>(Regex.java:159) ~[?:?]
        at org.joni.Regex.<init>(Regex.java:136) ~[?:?]
        at org.joni.Regex.<init>(Regex.java:122) ~[?:?]
        at org.elasticsearch.grok.Grok.<init>(Grok.java:106) ~[?:?]
        at org.elasticsearch.grok.Grok.<init>(Grok.java:86) ~[?:?]
        at org.elasticsearch.ingest.common.GrokProcessor.<init>(GrokProcessor.java:51) ~[?:?]
        at org.elasticsearch.ingest.common.GrokProcessor$Factory.create(GrokProcessor.java:161) ~[?:?]
        at org.elasticsearch.ingest.common.GrokProcessor$Factory.create(GrokProcessor.java:133) ~[?:?]
        at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:422) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:392) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:336) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.ingest.Pipeline.create(Pipeline.java:73) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.action.ingest.SimulatePipelineRequest.parse(SimulatePipelineRequest.java:166) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.action.ingest.SimulatePipelineTransportAction.doExecute(SimulatePipelineTransportAction.java:58) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.action.ingest.SimulatePipelineTransportAction.doExecute(SimulatePipelineTransportAction.java:35) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:153) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:129) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:396) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:685) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.simulatePipeline(AbstractClient.java:1129) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.rest.action.ingest.RestSimulatePipelineAction.lambda$prepareRequest$0(RestSimulatePipelineAction.java:54) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:108) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:222) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:295) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:166) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:322) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:372) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:301) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:69) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) ~[?:?]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[?:?]
        at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) ~[?:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:597) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:551) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Ingest)

[cbuescher]

Hi @dosera, I just tested this with the latest of the 7.4 branch which already includes #48899 and the grok pattern doesn't take down the cluster anymore. Maybe @henningandersen can confirm that the fix has been forward-ported to 7.5 and master (I couldn't find the patch upon quick search but maybe it wasn't needed on those branches). After that I think we can close this.

[dosera]

Hi @cbuescher , so with the latest you refer to 7.4.3 I guess aka 0a07043 ?
If so I guess it's fine, thanks.

[cbuescher]

I used 7023df0 which should include that commit, yes. I just want to wait to hear about the situation on the newer 7.x branches and master before closing here. The "jcodings" dependencies there are at version "1.0.44", but the test added in #48899 seems to be on those branches (e.g. 8835142), so the version might be okay there.

[henningandersen]

@cbuescher , yes, this was already fixed in 7.5 by #47374, so no changes were necessary in 7.5+. Will add a note to #48899 too.

[cbuescher]

Great, closing then.