Run with -XX:-OmitStackTraceInFastThrow

[nik9000]

If a particular code path fails enough times the JVM can chose to omit the stack trace for it. This post does a better job of explaining it than I can. I vote that we disable this feature of the JVM because eating debugging information is bad, bad, bad. I've seen this happen in a few situations:

1. When someone sends a ton of failing documents.
2. When someone sends a script that NPEs on a ton of documents. This is especially bad because without the stack trace we can't track down whether the cause is something in the script engine or in the user's scripts. The script_stack that we so lovingly build is always empty.
3. In painless's tests when we test the script_stack. This is where I first encountered it.

I asked @danielmitterdorfer to do a benchmark run with the flag set and he said it didn't really change the performance of things that we measured. I expect it really helps speed up those cases I referenced above at the cost of losing all the nice debugging information that we find so useful. That being said, I don't think the benchmarking we have is really going to be testing the cases where this flag really shines. We don't typically benchmark the "I'm sending a million broken documents" use case. But maybe we don't care if we slow that process down? At lest, not if that speed sacrifices debugging information.

[ywelsch]

We discussed this on FixitFriday and think it's worth giving it a try for 6.0+ (which will give us plenty of time to test this change).