Escaped characters U+0000..001F in JSON not dealt with correctly

[mbert]

On Kubernetes container logs are formatted as JSON and redirected into log files in /var/log/containers. In this process the characters U+0000..001F are escaped. Hence, a tab character becomes "\u0009" (i.e. a string). This breaks recognition of stacktraces that contain tabs.

For Java the following change to the existing code works:

diff --git a/lib/fluent/plugin/exception_detector.rb b/lib/fluent/plugin/exception_detector.rb
index c83fcab..ed23c49 100644
--- a/lib/fluent/plugin/exception_detector.rb
+++ b/lib/fluent/plugin/exception_detector.rb
@@ -53,9 +53,9 @@ module Fluent
       rule(:start_state,
            /(?:Exception|Error|Throwable|V8 errors stack trace)[:\r\n]/,
            :java),
-      rule(:java, /^[\t ]+(?:eval )?at /, :java),
-      rule(:java, /^[\t ]*(?:Caused by|Suppressed):/, :java),
-      rule(:java, /^[\t ]*... \d+\ more/, :java)
+      rule(:java, /^(\\u0009|[\t ])+(?:eval )?at /, :java),
+      rule(:java, /^(\\u0009|[\t ])*(?:Caused by|Suppressed):/, :java),
+      rule(:java, /^(\\u0009|[\t ])*... \d+\ more/, :java)
     ].freeze
 
     PYTHON_RULES = [

[igorpeshansky]

Sorry, this fell through the cracks.

By the time the JSON log entry gets to this plugin, it should have been parsed and all escapes decoded into the actual characters (including tabs). If that doesn't happen, could we see a simple repro test case?

This plugin is designed to work with Stackdriver Error Reporting, which should recognize the resulting multiline message as a stack trace. I'm verifying this with the team right now, but I suspect that it won't recognize those Unicode escapes, even if the stack trace were joined into a single log entry, so your proposed fix won't work.

/cc @mbert @DandyDeveloper

[mbert]

I ran across this problem when trying out the EFK stack configuration from the Kubernetes addon on github. The exception detection plugin is configured in this configmap here: https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-configmap.yaml

I currently have no Kubernetes cluster to play with, if necessary I may have to set one up to reproduce this later this week.

[igorpeshansky]

@mbert Looks like the input configuration mentioned in #33 (comment) has been cargo-culted from the Stackdriver-specific config (https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-gcp/fluentd-gcp-configmap-old.yaml). There is no point in using this plugin without the data going into Stackdriver.

I would report this as a bug against kubernetes/kubernetes and ask them to remove this plugin from the ES config.

If you can reproduce the issue in a Kubernetes cluster configured to write to Stackdriver (using fluentd-gcp), please reopen and let's debug.