Crashes after upgrading to 5.3.0

[lancemind]

Elasticsearch version:
{
"name" : "MkZ0lPb",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "t_EwPLkSRDWMrUAcaae9Uw",
"version" : {
"number" : "5.3.0",
"build_hash" : "3adb13b",
"build_date" : "2017-03-23T03:31:50.652Z",
"build_snapshot" : false,
"lucene_version" : "6.4.1"
},
"tagline" : "You Know, for Search"
}
Plugins installed: [
searchguard-5.3.0-11
S3
]

JVM version:
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

OS version:
Linux 3.13.0-112-generic #159-Ubuntu SMP Fri Mar 3 15:26:07 UTC 2017 x86_64 x86_64 x86_64
GNU/Linux
Description of the problem including expected versus actual behavior:
Upgraded to 5.3.0 today @ 12:00 PST and have had ES crash twice. Logs with stack trace attached.

Steps to reproduce:

1. Upgrade to ES 5.3.0

Provide logs (if relevant):
elasticsearch.log.gz

Describe the feature:

[jasontedor]

There's nowhere near enough information to reproduce this. Also, please remove Search Guard and see if the issue still reproduces as we will want a minimal reproduction anyway.

Since Elastic prefers to use GitHub only for verified bug reports and feature requests, I'm going to close this issue. If you need help digging through this, you can try the forum.

[floragunn]

java.lang.StackOverflowError within java.util.regex.Pattern in bulk thread, this indeed looks like a bug (either in ES or in Search Guard)

[2017-04-06T20:10:15,489][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[MkZ0lPb][bulk][T#1]], exiting
java.lang.StackOverflowError: null
	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658) ~[?:1.8.0_121]
	at java.util.regex.Pattern$Loop.match(Pattern.java:4785) ~[?:1.8.0_121]
	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717) ~[?:1.8.0_121]
	at java.util.regex.Pattern$BranchConn.match(Pattern.java:4568) ~[?:1.8.0_121]
	at java.util.regex.Pattern$CharProperty.match(Pattern.java:3777) ~[?:1.8.0_121]
	at java.util.regex.Pattern$Branch.match(Pattern.java:4604) ~[?:1.8.0_121]
	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658) ~[?:1.8.0_121]
	at java.util.regex.Pattern$Loop.match(Pattern.java:4785) ~[?:1.8.0_121]
	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717) ~[?:1.8.0_121]
	at java.util.regex.Pattern$BranchConn.match(Pattern.java:4568) ~[?:1.8.0_121]
	at java.util.regex.Pattern$CharProperty.match(Pattern.java:3777) ~[?:1.8.0_121]
	at java.util.regex.Pattern$Branch.match(Pattern.java:4604) ~[?:1.8.0_121]
	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658) ~[?:1.8.0_121]

Maybe related to #22812

[jimczi]

this indeed looks like a bug (either in ES or in Search Guard)

@floragunncom without a clear reproducible case we can't help you.
As @jasontedor suggested check first if disabling SearchGuard eliminates the problem.

[jasontedor]

I have a hunch what caused this. I think it's due to excessive backtracking from the regular expression used to extract warning values when de-duplicating warning headers and this was addressed by #24114. If you upgrade to 5.3.1, you should see this issue is fixed (if not, then it means you're experiencing a different issue).

[floragunn]

I came up with that idea three weeks ago ;-)

[jasontedor]

I don't understand what you're saying? The logs clearly have a stack overflow, the issue was where they were coming from, not that there was a stack overflow?