Uploader can't keep up

[joshuaspence]

I've noticed that the S3 uploader threads aren't able to keep up with the processing of uploading temporary files. This seemed to have happened when we change the prefix from %{+yyyy-MM-dd}/ to %{+yyyy-MM-dd}/%{type}/. From one of our production Logstash instances:

> find /tmp/logstash/ -type f | wc -l
539

I'm still investigating this, but I suspect that we need more upload workers? The specific problem here is that the logs don't give me any indication that something is going wrong.

[joshuaspence]

If I start and stop Logstash a few times, the temporary files seem to be successfully uploaded. If I leave Logstash running, however, the files in /tmp/logstash just seem to grow.

[joshuaspence]

> date; find /tmp/logstash/ -type f | wc -l
Mon Jan 16 16:36:57 EST 2017
15
> date; find /tmp/logstash/ -type f | wc -l
Mon Jan 16 16:44:19 EST 2017
37
> date; find /tmp/logstash/ -type f | wc -l
Mon Jan 16 16:46:41 EST 2017
54
> date; find /tmp/logstash/ -type f | wc -l
Mon Jan 16 16:53:46 EST 2017
75
> date; find /tmp/logstash/ -type f | wc -l
Mon Jan 16 16:57:25 EST 2017
92

[joshuaspence]

Here's a snippet of the DEBUG logs:

{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603529156,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603529882,"thread":"S3 output uploader, file: /tmp/logstash/3be35c60-2698-4dd0-afad-117d1e6f437d/us-east-1/2017-01-16/syslog/ls.s3.0622f12f-54d0-4cac-ab2a-48761304fc76.2017-01-16T16.46.part2.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/logstash/ls.s3.062b64c1-449a-40a1-bb76-d1a8f71339c0.2017-01-16T16.41.part1.txt","path":"/tmp/logstash/f4a704eb-f507-4a66-8836-5e2c3f444d57/us-east-1/2017-01-16/logstash/ls.s3.062b64c1-449a-40a1-bb76-d1a8f71339c0.2017-01-16T16.41.part1.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603529882,"thread":"S3 output uploader, file: /tmp/logstash/3be35c60-2698-4dd0-afad-117d1e6f437d/us-east-1/2017-01-16/syslog/ls.s3.0622f12f-54d0-4cac-ab2a-48761304fc76.2017-01-16T16.46.part2.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/f4a704eb-f507-4a66-8836-5e2c3f444d57/us-east-1/2017-01-16/logstash/ls.s3.062b64c1-449a-40a1-bb76-d1a8f71339c0.2017-01-16T16.41.part1.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603530003,"thread":"S3 output uploader, file: /tmp/logstash/f4a704eb-f507-4a66-8836-5e2c3f444d57/us-east-1/2017-01-16/logstash/ls.s3.062b64c1-449a-40a1-bb76-d1a8f71339c0.2017-01-16T16.41.part1.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/f4a704eb-f507-4a66-8836-5e2c3f444d57/us-east-1/2017-01-16/logstash/ls.s3.062b64c1-449a-40a1-bb76-d1a8f71339c0.2017-01-16T16.41.part1.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603544158,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603559160,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603574162,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603589164,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603604165,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603604165,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/s3/ls.s3.d02f297e-a5c0-4f94-9d09-b02966a0f670.2017-01-16T16.48.part2.txt","path":"/tmp/logstash/6dcad243-08ac-4247-bcba-ef31e2582844/us-east-1/2017-01-16/s3/ls.s3.d02f297e-a5c0-4f94-9d09-b02966a0f670.2017-01-16T16.48.part2.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603604166,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/6dcad243-08ac-4247-bcba-ef31e2582844/us-east-1/2017-01-16/s3/ls.s3.d02f297e-a5c0-4f94-9d09-b02966a0f670.2017-01-16T16.48.part2.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603604315,"thread":"S3 output uploader, file: /tmp/logstash/6dcad243-08ac-4247-bcba-ef31e2582844/us-east-1/2017-01-16/s3/ls.s3.d02f297e-a5c0-4f94-9d09-b02966a0f670.2017-01-16T16.48.part2.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/6dcad243-08ac-4247-bcba-ef31e2582844/us-east-1/2017-01-16/s3/ls.s3.d02f297e-a5c0-4f94-9d09-b02966a0f670.2017-01-16T16.48.part2.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603619179,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603634180,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603649182,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603664184,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603679186,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603679187,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/php-fpm/ls.s3.21127a73-1982-48b6-95f0-554002a6d11b.2017-01-16T16.49.part2.txt","path":"/tmp/logstash/2babe224-d14c-4d1f-833e-177c264d328d/us-east-1/2017-01-16/php-fpm/ls.s3.21127a73-1982-48b6-95f0-554002a6d11b.2017-01-16T16.49.part2.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603679187,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/2babe224-d14c-4d1f-833e-177c264d328d/us-east-1/2017-01-16/php-fpm/ls.s3.21127a73-1982-48b6-95f0-554002a6d11b.2017-01-16T16.49.part2.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603679297,"thread":"S3 output uploader, file: /tmp/logstash/2babe224-d14c-4d1f-833e-177c264d328d/us-east-1/2017-01-16/php-fpm/ls.s3.21127a73-1982-48b6-95f0-554002a6d11b.2017-01-16T16.49.part2.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/2babe224-d14c-4d1f-833e-177c264d328d/us-east-1/2017-01-16/php-fpm/ls.s3.21127a73-1982-48b6-95f0-554002a6d11b.2017-01-16T16.49.part2.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603694195,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603709197,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603724199,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603739200,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603754201,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603769203,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603771007,"thread":"S3 output uploader, file: /tmp/logstash/3be35c60-2698-4dd0-afad-117d1e6f437d/us-east-1/2017-01-16/syslog/ls.s3.0622f12f-54d0-4cac-ab2a-48761304fc76.2017-01-16T16.46.part2.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/mysql-slow-query/ls.s3.530aa6fe-345d-4e56-bf72-c5fd2646e0c4.2017-01-16T16.48.part2.txt","path":"/tmp/logstash/1d12a40a-1ad5-4d7f-955a-7b813aebecd3/us-east-1/2017-01-16/mysql-slow-query/ls.s3.530aa6fe-345d-4e56-bf72-c5fd2646e0c4.2017-01-16T16.48.part2.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603771007,"thread":"S3 output uploader, file: /tmp/logstash/3be35c60-2698-4dd0-afad-117d1e6f437d/us-east-1/2017-01-16/syslog/ls.s3.0622f12f-54d0-4cac-ab2a-48761304fc76.2017-01-16T16.46.part2.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/1d12a40a-1ad5-4d7f-955a-7b813aebecd3/us-east-1/2017-01-16/mysql-slow-query/ls.s3.530aa6fe-345d-4e56-bf72-c5fd2646e0c4.2017-01-16T16.48.part2.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603771396,"thread":"S3 output uploader, file: /tmp/logstash/1d12a40a-1ad5-4d7f-955a-7b813aebecd3/us-east-1/2017-01-16/mysql-slow-query/ls.s3.530aa6fe-345d-4e56-bf72-c5fd2646e0c4.2017-01-16T16.48.part2.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/1d12a40a-1ad5-4d7f-955a-7b813aebecd3/us-east-1/2017-01-16/mysql-slow-query/ls.s3.530aa6fe-345d-4e56-bf72-c5fd2646e0c4.2017-01-16T16.48.part2.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603771565,"thread":"S3 output uploader, file: /tmp/logstash/3be35c60-2698-4dd0-afad-117d1e6f437d/us-east-1/2017-01-16/syslog/ls.s3.0622f12f-54d0-4cac-ab2a-48761304fc76.2017-01-16T16.46.part2.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/nginx-access/ls.s3.006aa311-a945-4ba3-81ab-67efc6fd7ae8.2017-01-16T16.51.part3.txt","path":"/tmp/logstash/8b11bb9b-9cab-4615-a5a8-d9a7212163df/us-east-1/2017-01-16/nginx-access/ls.s3.006aa311-a945-4ba3-81ab-67efc6fd7ae8.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603771565,"thread":"S3 output uploader, file: /tmp/logstash/3be35c60-2698-4dd0-afad-117d1e6f437d/us-east-1/2017-01-16/syslog/ls.s3.0622f12f-54d0-4cac-ab2a-48761304fc76.2017-01-16T16.46.part2.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/8b11bb9b-9cab-4615-a5a8-d9a7212163df/us-east-1/2017-01-16/nginx-access/ls.s3.006aa311-a945-4ba3-81ab-67efc6fd7ae8.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603772901,"thread":"[main]>worker2","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/json/ls.s3.a52149ac-41da-436f-bbd2-8ca4407cb5ba.2017-01-16T16.51.part3.txt","path":"/tmp/logstash/e511d301-77ad-4421-bf0d-b166467d920c/us-east-1/2017-01-16/json/ls.s3.a52149ac-41da-436f-bbd2-8ca4407cb5ba.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603772901,"thread":"[main]>worker2","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/e511d301-77ad-4421-bf0d-b166467d920c/us-east-1/2017-01-16/json/ls.s3.a52149ac-41da-436f-bbd2-8ca4407cb5ba.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603773290,"thread":"[main]>worker2","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/telegraf/ls.s3.0ff969d7-851f-4e25-8b9e-bda9832c6460.2017-01-16T16.51.part3.txt","path":"/tmp/logstash/996671c0-4ecd-48f6-b247-dce0bce5129e/us-east-1/2017-01-16/telegraf/ls.s3.0ff969d7-851f-4e25-8b9e-bda9832c6460.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603773290,"thread":"[main]>worker2","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/996671c0-4ecd-48f6-b247-dce0bce5129e/us-east-1/2017-01-16/telegraf/ls.s3.0ff969d7-851f-4e25-8b9e-bda9832c6460.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603775689,"thread":"S3 output uploader, file: /tmp/logstash/8b11bb9b-9cab-4615-a5a8-d9a7212163df/us-east-1/2017-01-16/nginx-access/ls.s3.006aa311-a945-4ba3-81ab-67efc6fd7ae8.2017-01-16T16.51.part3.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/8b11bb9b-9cab-4615-a5a8-d9a7212163df/us-east-1/2017-01-16/nginx-access/ls.s3.006aa311-a945-4ba3-81ab-67efc6fd7ae8.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603775907,"thread":"S3 output uploader, file: /tmp/logstash/e511d301-77ad-4421-bf0d-b166467d920c/us-east-1/2017-01-16/json/ls.s3.a52149ac-41da-436f-bbd2-8ca4407cb5ba.2017-01-16T16.51.part3.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/e511d301-77ad-4421-bf0d-b166467d920c/us-east-1/2017-01-16/json/ls.s3.a52149ac-41da-436f-bbd2-8ca4407cb5ba.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603776102,"thread":"S3 output uploader, file: /tmp/logstash/996671c0-4ecd-48f6-b247-dce0bce5129e/us-east-1/2017-01-16/telegraf/ls.s3.0ff969d7-851f-4e25-8b9e-bda9832c6460.2017-01-16T16.51.part3.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/996671c0-4ecd-48f6-b247-dce0bce5129e/us-east-1/2017-01-16/telegraf/ls.s3.0ff969d7-851f-4e25-8b9e-bda9832c6460.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603777640,"thread":"S3 output uploader, file: /tmp/logstash/137be95c-79e4-4850-a5ef-c3714dd4db54/us-east-1/2017-01-16/varnishncsa/ls.s3.e22bbe59-e19f-4b02-bfc2-efd45b23e382.2017-01-16T16.36.part0.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/syslog/ls.s3.570d83c0-8df5-4424-96cc-8e34d1819c7f.2017-01-16T16.51.part3.txt","path":"/tmp/logstash/3ff87b15-b066-4fd1-8547-7c24058e6322/us-east-1/2017-01-16/syslog/ls.s3.570d83c0-8df5-4424-96cc-8e34d1819c7f.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603777641,"thread":"S3 output uploader, file: /tmp/logstash/137be95c-79e4-4850-a5ef-c3714dd4db54/us-east-1/2017-01-16/varnishncsa/ls.s3.e22bbe59-e19f-4b02-bfc2-efd45b23e382.2017-01-16T16.36.part0.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/3ff87b15-b066-4fd1-8547-7c24058e6322/us-east-1/2017-01-16/syslog/ls.s3.570d83c0-8df5-4424-96cc-8e34d1819c7f.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603777941,"thread":"S3 output uploader, file: /tmp/logstash/3ff87b15-b066-4fd1-8547-7c24058e6322/us-east-1/2017-01-16/syslog/ls.s3.570d83c0-8df5-4424-96cc-8e34d1819c7f.2017-01-16T16.51.part3.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/3ff87b15-b066-4fd1-8547-7c24058e6322/us-east-1/2017-01-16/syslog/ls.s3.570d83c0-8df5-4424-96cc-8e34d1819c7f.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603778218,"thread":"S3 output uploader, file: /tmp/logstash/137be95c-79e4-4850-a5ef-c3714dd4db54/us-east-1/2017-01-16/varnishncsa/ls.s3.e22bbe59-e19f-4b02-bfc2-efd45b23e382.2017-01-16T16.36.part0.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/varnishncsa/ls.s3.0ae61d3c-01f6-4995-bbfa-a44b19aaeaed.2017-01-16T16.51.part3.txt","path":"/tmp/logstash/50d54986-bd4b-486a-89cb-a4aa0025c9d5/us-east-1/2017-01-16/varnishncsa/ls.s3.0ae61d3c-01f6-4995-bbfa-a44b19aaeaed.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603778218,"thread":"S3 output uploader, file: /tmp/logstash/137be95c-79e4-4850-a5ef-c3714dd4db54/us-east-1/2017-01-16/varnishncsa/ls.s3.e22bbe59-e19f-4b02-bfc2-efd45b23e382.2017-01-16T16.36.part0.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/50d54986-bd4b-486a-89cb-a4aa0025c9d5/us-east-1/2017-01-16/varnishncsa/ls.s3.0ae61d3c-01f6-4995-bbfa-a44b19aaeaed.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603779218,"thread":"S3 output uploader, file: /tmp/logstash/50d54986-bd4b-486a-89cb-a4aa0025c9d5/us-east-1/2017-01-16/varnishncsa/ls.s3.0ae61d3c-01f6-4995-bbfa-a44b19aaeaed.2017-01-16T16.51.part3.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/50d54986-bd4b-486a-89cb-a4aa0025c9d5/us-east-1/2017-01-16/varnishncsa/ls.s3.0ae61d3c-01f6-4995-bbfa-a44b19aaeaed.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603780542,"thread":"S3 output uploader, file: /tmp/logstash/b820db22-3c6a-4a8e-9abe-3e60f3e3b81c/us-east-1/2017-01-16/syslog/ls.s3.8bf26151-4117-448d-a93d-7cd2eb516fc9.2017-01-16T16.41.part1.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/elb/ls.s3.2ab6891b-95a3-420d-bf85-adf848f03843.2017-01-16T16.51.part3.txt","path":"/tmp/logstash/8ae8f1aa-f012-4be6-9758-744a02d0bafe/us-east-1/2017-01-16/elb/ls.s3.2ab6891b-95a3-420d-bf85-adf848f03843.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603780543,"thread":"S3 output uploader, file: /tmp/logstash/b820db22-3c6a-4a8e-9abe-3e60f3e3b81c/us-east-1/2017-01-16/syslog/ls.s3.8bf26151-4117-448d-a93d-7cd2eb516fc9.2017-01-16T16.41.part1.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/8ae8f1aa-f012-4be6-9758-744a02d0bafe/us-east-1/2017-01-16/elb/ls.s3.2ab6891b-95a3-420d-bf85-adf848f03843.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603780973,"thread":"S3 output uploader, file: /tmp/logstash/8ae8f1aa-f012-4be6-9758-744a02d0bafe/us-east-1/2017-01-16/elb/ls.s3.2ab6891b-95a3-420d-bf85-adf848f03843.2017-01-16T16.51.part3.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/8ae8f1aa-f012-4be6-9758-744a02d0bafe/us-east-1/2017-01-16/elb/ls.s3.2ab6891b-95a3-420d-bf85-adf848f03843.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603781135,"thread":"S3 output uploader, file: /tmp/logstash/137be95c-79e4-4850-a5ef-c3714dd4db54/us-east-1/2017-01-16/varnishncsa/ls.s3.e22bbe59-e19f-4b02-bfc2-efd45b23e382.2017-01-16T16.36.part0.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/filebeat/ls.s3.72345d19-c5e9-4668-963f-8f4ed0c2fe62.2017-01-16T16.51.part3.txt","path":"/tmp/logstash/9fbcfa5a-c8dc-437f-933a-76b0b1a6527c/us-east-1/2017-01-16/filebeat/ls.s3.72345d19-c5e9-4668-963f-8f4ed0c2fe62.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603781135,"thread":"S3 output uploader, file: /tmp/logstash/137be95c-79e4-4850-a5ef-c3714dd4db54/us-east-1/2017-01-16/varnishncsa/ls.s3.e22bbe59-e19f-4b02-bfc2-efd45b23e382.2017-01-16T16.36.part0.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/9fbcfa5a-c8dc-437f-933a-76b0b1a6527c/us-east-1/2017-01-16/filebeat/ls.s3.72345d19-c5e9-4668-963f-8f4ed0c2fe62.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603781213,"thread":"S3 output uploader, file: /tmp/logstash/9fbcfa5a-c8dc-437f-933a-76b0b1a6527c/us-east-1/2017-01-16/filebeat/ls.s3.72345d19-c5e9-4668-963f-8f4ed0c2fe62.2017-01-16T16.51.part3.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/9fbcfa5a-c8dc-437f-933a-76b0b1a6527c/us-east-1/2017-01-16/filebeat/ls.s3.72345d19-c5e9-4668-963f-8f4ed0c2fe62.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603784206,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603784206,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/nginx-error/ls.s3.c764ab13-4e9d-49e5-a8bd-f50cf6a5d581.2017-01-16T16.51.part3.txt","path":"/tmp/logstash/fd259e06-8fad-4057-aa97-c39eab5a6857/us-east-1/2017-01-16/nginx-error/ls.s3.c764ab13-4e9d-49e5-a8bd-f50cf6a5d581.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603784206,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/fd259e06-8fad-4057-aa97-c39eab5a6857/us-east-1/2017-01-16/nginx-error/ls.s3.c764ab13-4e9d-49e5-a8bd-f50cf6a5d581.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603784210,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-17/telegraf/ls.s3.f260c78e-5db6-4d80-acf3-423497a702ea.2017-01-16T16.51.part3.txt","path":"/tmp/logstash/5fe39dfe-7a6b-45d3-99ac-8d0f8a138a1f/us-east-1/2017-01-17/telegraf/ls.s3.f260c78e-5db6-4d80-acf3-423497a702ea.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603784210,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/5fe39dfe-7a6b-45d3-99ac-8d0f8a138a1f/us-east-1/2017-01-17/telegraf/ls.s3.f260c78e-5db6-4d80-acf3-423497a702ea.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603784220,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/consul/ls.s3.0ded8ecf-1806-47e0-bc1d-2371a1c07bd3.2017-01-16T16.51.part3.txt","path":"/tmp/logstash/49915dda-8e4b-4ace-a04b-a6a7cb9bfc73/us-east-1/2017-01-16/consul/ls.s3.0ded8ecf-1806-47e0-bc1d-2371a1c07bd3.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603784221,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/49915dda-8e4b-4ace-a04b-a6a7cb9bfc73/us-east-1/2017-01-16/consul/ls.s3.0ded8ecf-1806-47e0-bc1d-2371a1c07bd3.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603784227,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/php/ls.s3.82093633-a289-4826-8cfe-f7829af875df.2017-01-16T16.51.part3.txt","path":"/tmp/logstash/f5ae7a62-5763-49d9-893c-d7d7bf5009fd/us-east-1/2017-01-16/php/ls.s3.82093633-a289-4826-8cfe-f7829af875df.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603784227,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/f5ae7a62-5763-49d9-893c-d7d7bf5009fd/us-east-1/2017-01-16/php/ls.s3.82093633-a289-4826-8cfe-f7829af875df.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603784316,"thread":"S3 output uploader, file: /tmp/logstash/fd259e06-8fad-4057-aa97-c39eab5a6857/us-east-1/2017-01-16/nginx-error/ls.s3.c764ab13-4e9d-49e5-a8bd-f50cf6a5d581.2017-01-16T16.51.part3.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/fd259e06-8fad-4057-aa97-c39eab5a6857/us-east-1/2017-01-16/nginx-error/ls.s3.c764ab13-4e9d-49e5-a8bd-f50cf6a5d581.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603784334,"thread":"S3 output uploader, file: /tmp/logstash/f5ae7a62-5763-49d9-893c-d7d7bf5009fd/us-east-1/2017-01-16/php/ls.s3.82093633-a289-4826-8cfe-f7829af875df.2017-01-16T16.51.part3.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/f5ae7a62-5763-49d9-893c-d7d7bf5009fd/us-east-1/2017-01-16/php/ls.s3.82093633-a289-4826-8cfe-f7829af875df.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603784337,"thread":"S3 output uploader, file: /tmp/logstash/f5ae7a62-5763-49d9-893c-d7d7bf5009fd/us-east-1/2017-01-16/php/ls.s3.82093633-a289-4826-8cfe-f7829af875df.2017-01-16T16.51.part3.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/heartbeat/ls.s3.811d6dc5-ac66-4cdb-a069-5ae83265ba66.2017-01-16T16.51.part3.txt","path":"/tmp/logstash/b06d03d4-b97e-4bfb-a1c5-acdd6d5ec880/us-east-1/2017-01-16/heartbeat/ls.s3.811d6dc5-ac66-4cdb-a069-5ae83265ba66.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603784337,"thread":"S3 output uploader, file: /tmp/logstash/f5ae7a62-5763-49d9-893c-d7d7bf5009fd/us-east-1/2017-01-16/php/ls.s3.82093633-a289-4826-8cfe-f7829af875df.2017-01-16T16.51.part3.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/b06d03d4-b97e-4bfb-a1c5-acdd6d5ec880/us-east-1/2017-01-16/heartbeat/ls.s3.811d6dc5-ac66-4cdb-a069-5ae83265ba66.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603784340,"thread":"S3 output uploader, file: /tmp/logstash/f5ae7a62-5763-49d9-893c-d7d7bf5009fd/us-east-1/2017-01-16/php/ls.s3.82093633-a289-4826-8cfe-f7829af875df.2017-01-16T16.51.part3.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/metric/ls.s3.441e644a-d6f6-48d3-9321-4115a118957c.2017-01-16T16.51.part3.txt","path":"/tmp/logstash/647de233-69de-4a84-8026-ba68a4bc529c/us-east-1/2017-01-16/metric/ls.s3.441e644a-d6f6-48d3-9321-4115a118957c.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603784340,"thread":"S3 output uploader, file: /tmp/logstash/f5ae7a62-5763-49d9-893c-d7d7bf5009fd/us-east-1/2017-01-16/php/ls.s3.82093633-a289-4826-8cfe-f7829af875df.2017-01-16T16.51.part3.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/647de233-69de-4a84-8026-ba68a4bc529c/us-east-1/2017-01-16/metric/ls.s3.441e644a-d6f6-48d3-9321-4115a118957c.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603784372,"thread":"S3 output uploader, file: /tmp/logstash/5fe39dfe-7a6b-45d3-99ac-8d0f8a138a1f/us-east-1/2017-01-17/telegraf/ls.s3.f260c78e-5db6-4d80-acf3-423497a702ea.2017-01-16T16.51.part3.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/5fe39dfe-7a6b-45d3-99ac-8d0f8a138a1f/us-east-1/2017-01-17/telegraf/ls.s3.f260c78e-5db6-4d80-acf3-423497a702ea.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603784406,"thread":"S3 output uploader, file: /tmp/logstash/49915dda-8e4b-4ace-a04b-a6a7cb9bfc73/us-east-1/2017-01-16/consul/ls.s3.0ded8ecf-1806-47e0-bc1d-2371a1c07bd3.2017-01-16T16.51.part3.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/49915dda-8e4b-4ace-a04b-a6a7cb9bfc73/us-east-1/2017-01-16/consul/ls.s3.0ded8ecf-1806-47e0-bc1d-2371a1c07bd3.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603784415,"thread":"S3 output uploader, file: /tmp/logstash/647de233-69de-4a84-8026-ba68a4bc529c/us-east-1/2017-01-16/metric/ls.s3.441e644a-d6f6-48d3-9321-4115a118957c.2017-01-16T16.51.part3.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/647de233-69de-4a84-8026-ba68a4bc529c/us-east-1/2017-01-16/metric/ls.s3.441e644a-d6f6-48d3-9321-4115a118957c.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603784469,"thread":"S3 output uploader, file: /tmp/logstash/b06d03d4-b97e-4bfb-a1c5-acdd6d5ec880/us-east-1/2017-01-16/heartbeat/ls.s3.811d6dc5-ac66-4cdb-a069-5ae83265ba66.2017-01-16T16.51.part3.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/b06d03d4-b97e-4bfb-a1c5-acdd6d5ec880/us-east-1/2017-01-16/heartbeat/ls.s3.811d6dc5-ac66-4cdb-a069-5ae83265ba66.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603799420,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603799420,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/log/ls.s3.a9f780fb-b4b3-4194-aa02-d7be1f8ac67e.2017-01-16T16.51.part3.txt","path":"/tmp/logstash/d459275f-493b-4d36-8be5-ab84403732ec/us-east-1/2017-01-16/log/ls.s3.a9f780fb-b4b3-4194-aa02-d7be1f8ac67e.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603799420,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/d459275f-493b-4d36-8be5-ab84403732ec/us-east-1/2017-01-16/log/ls.s3.a9f780fb-b4b3-4194-aa02-d7be1f8ac67e.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603799566,"thread":"S3 output uploader, file: /tmp/logstash/d459275f-493b-4d36-8be5-ab84403732ec/us-east-1/2017-01-16/log/ls.s3.a9f780fb-b4b3-4194-aa02-d7be1f8ac67e.2017-01-16T16.51.part3.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/d459275f-493b-4d36-8be5-ab84403732ec/us-east-1/2017-01-16/log/ls.s3.a9f780fb-b4b3-4194-aa02-d7be1f8ac67e.2017-01-16T16.51.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603814427,"thread":"S3 output uploader, file: /tmp/logstash/647de233-69de-4a84-8026-ba68a4bc529c/us-east-1/2017-01-16/metric/ls.s3.441e644a-d6f6-48d3-9321-4115a118957c.2017-01-16T16.51.part3.txt","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603829429,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603830278,"thread":"S3 output uploader, file: /tmp/logstash/137be95c-79e4-4850-a5ef-c3714dd4db54/us-east-1/2017-01-16/varnishncsa/ls.s3.e22bbe59-e19f-4b02-bfc2-efd45b23e382.2017-01-16T16.36.part0.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/logstash/ls.s3.cdab2557-b324-4683-89d2-5c1cf91f2da1.2017-01-16T16.52.part2.txt","path":"/tmp/logstash/2de0e0bb-e00d-4f8d-b257-59f8ea8c613f/us-east-1/2017-01-16/logstash/ls.s3.cdab2557-b324-4683-89d2-5c1cf91f2da1.2017-01-16T16.52.part2.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603830278,"thread":"S3 output uploader, file: /tmp/logstash/137be95c-79e4-4850-a5ef-c3714dd4db54/us-east-1/2017-01-16/varnishncsa/ls.s3.e22bbe59-e19f-4b02-bfc2-efd45b23e382.2017-01-16T16.36.part0.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/2de0e0bb-e00d-4f8d-b257-59f8ea8c613f/us-east-1/2017-01-16/logstash/ls.s3.cdab2557-b324-4683-89d2-5c1cf91f2da1.2017-01-16T16.52.part2.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603830464,"thread":"S3 output uploader, file: /tmp/logstash/2de0e0bb-e00d-4f8d-b257-59f8ea8c613f/us-east-1/2017-01-16/logstash/ls.s3.cdab2557-b324-4683-89d2-5c1cf91f2da1.2017-01-16T16.52.part2.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/2de0e0bb-e00d-4f8d-b257-59f8ea8c613f/us-east-1/2017-01-16/logstash/ls.s3.cdab2557-b324-4683-89d2-5c1cf91f2da1.2017-01-16T16.52.part2.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603844436,"thread":"S3 output uploader, file: /tmp/logstash/647de233-69de-4a84-8026-ba68a4bc529c/us-east-1/2017-01-16/metric/ls.s3.441e644a-d6f6-48d3-9321-4115a118957c.2017-01-16T16.51.part3.txt","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603859437,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603874439,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603889440,"thread":"S3 output uploader, file: /tmp/logstash/647de233-69de-4a84-8026-ba68a4bc529c/us-east-1/2017-01-16/metric/ls.s3.441e644a-d6f6-48d3-9321-4115a118957c.2017-01-16T16.51.part3.txt","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603904441,"thread":"S3 output uploader, file: /tmp/logstash/647de233-69de-4a84-8026-ba68a4bc529c/us-east-1/2017-01-16/metric/ls.s3.441e644a-d6f6-48d3-9321-4115a118957c.2017-01-16T16.51.part3.txt","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603919443,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603934445,"thread":"S3 output uploader, file: /tmp/logstash/647de233-69de-4a84-8026-ba68a4bc529c/us-east-1/2017-01-16/metric/ls.s3.441e644a-d6f6-48d3-9321-4115a118957c.2017-01-16T16.51.part3.txt","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603949446,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603964448,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603979453,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603979453,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/php-fpm/ls.s3.3cb7b248-dfb0-4938-a98e-66c455b08784.2017-01-16T16.54.part3.txt","path":"/tmp/logstash/f3c998d9-3865-47bc-8535-ab703f00f3bf/us-east-1/2017-01-16/php-fpm/ls.s3.3cb7b248-dfb0-4938-a98e-66c455b08784.2017-01-16T16.54.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603979453,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/f3c998d9-3865-47bc-8535-ab703f00f3bf/us-east-1/2017-01-16/php-fpm/ls.s3.3cb7b248-dfb0-4938-a98e-66c455b08784.2017-01-16T16.54.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603979587,"thread":"S3 output uploader, file: /tmp/logstash/f3c998d9-3865-47bc-8535-ab703f00f3bf/us-east-1/2017-01-16/php-fpm/ls.s3.3cb7b248-dfb0-4938-a98e-66c455b08784.2017-01-16T16.54.part3.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/f3c998d9-3865-47bc-8535-ab703f00f3bf/us-east-1/2017-01-16/php-fpm/ls.s3.3cb7b248-dfb0-4938-a98e-66c455b08784.2017-01-16T16.54.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603984943,"thread":"S3 output uploader, file: /tmp/logstash/b820db22-3c6a-4a8e-9abe-3e60f3e3b81c/us-east-1/2017-01-16/syslog/ls.s3.8bf26151-4117-448d-a93d-7cd2eb516fc9.2017-01-16T16.41.part1.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/s3/ls.s3.38d8bb56-3e3e-433a-8cf0-4d663fb0c551.2017-01-16T16.53.part3.txt","path":"/tmp/logstash/5b47f07a-2139-4cd6-8d87-01041ccfd363/us-east-1/2017-01-16/s3/ls.s3.38d8bb56-3e3e-433a-8cf0-4d663fb0c551.2017-01-16T16.53.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603984943,"thread":"S3 output uploader, file: /tmp/logstash/b820db22-3c6a-4a8e-9abe-3e60f3e3b81c/us-east-1/2017-01-16/syslog/ls.s3.8bf26151-4117-448d-a93d-7cd2eb516fc9.2017-01-16T16.41.part1.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/5b47f07a-2139-4cd6-8d87-01041ccfd363/us-east-1/2017-01-16/s3/ls.s3.38d8bb56-3e3e-433a-8cf0-4d663fb0c551.2017-01-16T16.53.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603985252,"thread":"S3 output uploader, file: /tmp/logstash/5b47f07a-2139-4cd6-8d87-01041ccfd363/us-east-1/2017-01-16/s3/ls.s3.38d8bb56-3e3e-433a-8cf0-4d663fb0c551.2017-01-16T16.53.part3.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/5b47f07a-2139-4cd6-8d87-01041ccfd363/us-east-1/2017-01-16/s3/ls.s3.38d8bb56-3e3e-433a-8cf0-4d663fb0c551.2017-01-16T16.53.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484603994459,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604009462,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604024465,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604035582,"thread":"S3 output uploader, file: /tmp/logstash/137be95c-79e4-4850-a5ef-c3714dd4db54/us-east-1/2017-01-16/varnishncsa/ls.s3.e22bbe59-e19f-4b02-bfc2-efd45b23e382.2017-01-16T16.36.part0.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-17/syslog/ls.s3.67bc18c4-a091-49b1-b811-b34b83f95968.2017-01-16T16.50.part1.txt","path":"/tmp/logstash/d47c98a6-d210-42f8-ab70-4c3a838d4557/us-east-1/2017-01-17/syslog/ls.s3.67bc18c4-a091-49b1-b811-b34b83f95968.2017-01-16T16.50.part1.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604035582,"thread":"S3 output uploader, file: /tmp/logstash/137be95c-79e4-4850-a5ef-c3714dd4db54/us-east-1/2017-01-16/varnishncsa/ls.s3.e22bbe59-e19f-4b02-bfc2-efd45b23e382.2017-01-16T16.36.part0.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/d47c98a6-d210-42f8-ab70-4c3a838d4557/us-east-1/2017-01-17/syslog/ls.s3.67bc18c4-a091-49b1-b811-b34b83f95968.2017-01-16T16.50.part1.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604035739,"thread":"S3 output uploader, file: /tmp/logstash/d47c98a6-d210-42f8-ab70-4c3a838d4557/us-east-1/2017-01-17/syslog/ls.s3.67bc18c4-a091-49b1-b811-b34b83f95968.2017-01-16T16.50.part1.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/d47c98a6-d210-42f8-ab70-4c3a838d4557/us-east-1/2017-01-17/syslog/ls.s3.67bc18c4-a091-49b1-b811-b34b83f95968.2017-01-16T16.50.part1.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604039467,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604054469,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604069471,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604071600,"thread":"S3 output uploader, file: /tmp/logstash/137be95c-79e4-4850-a5ef-c3714dd4db54/us-east-1/2017-01-16/varnishncsa/ls.s3.e22bbe59-e19f-4b02-bfc2-efd45b23e382.2017-01-16T16.36.part0.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/nginx-access/ls.s3.fa9b0ef9-4b0c-442d-87e8-efcf0a44faf9.2017-01-16T16.56.part4.txt","path":"/tmp/logstash/af77db64-0bd9-4842-a973-a4ec5fa36896/us-east-1/2017-01-16/nginx-access/ls.s3.fa9b0ef9-4b0c-442d-87e8-efcf0a44faf9.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604071601,"thread":"S3 output uploader, file: /tmp/logstash/137be95c-79e4-4850-a5ef-c3714dd4db54/us-east-1/2017-01-16/varnishncsa/ls.s3.e22bbe59-e19f-4b02-bfc2-efd45b23e382.2017-01-16T16.36.part0.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/af77db64-0bd9-4842-a973-a4ec5fa36896/us-east-1/2017-01-16/nginx-access/ls.s3.fa9b0ef9-4b0c-442d-87e8-efcf0a44faf9.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604073664,"thread":"[main]>worker2","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/json/ls.s3.10d71bbb-e9e4-459a-add1-78af9d6ae466.2017-01-16T16.56.part4.txt","path":"/tmp/logstash/45529e93-3813-4988-ab58-957b2b868d87/us-east-1/2017-01-16/json/ls.s3.10d71bbb-e9e4-459a-add1-78af9d6ae466.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604073664,"thread":"[main]>worker2","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/45529e93-3813-4988-ab58-957b2b868d87/us-east-1/2017-01-16/json/ls.s3.10d71bbb-e9e4-459a-add1-78af9d6ae466.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604073734,"thread":"S3 output uploader, file: /tmp/logstash/3be35c60-2698-4dd0-afad-117d1e6f437d/us-east-1/2017-01-16/syslog/ls.s3.0622f12f-54d0-4cac-ab2a-48761304fc76.2017-01-16T16.46.part2.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/telegraf/ls.s3.9bf2d2d7-8238-4d9d-a82d-aff4d7283023.2017-01-16T16.56.part4.txt","path":"/tmp/logstash/b151d0c6-2290-4bf1-8bdf-9087882d6b7e/us-east-1/2017-01-16/telegraf/ls.s3.9bf2d2d7-8238-4d9d-a82d-aff4d7283023.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604073734,"thread":"S3 output uploader, file: /tmp/logstash/3be35c60-2698-4dd0-afad-117d1e6f437d/us-east-1/2017-01-16/syslog/ls.s3.0622f12f-54d0-4cac-ab2a-48761304fc76.2017-01-16T16.46.part2.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/b151d0c6-2290-4bf1-8bdf-9087882d6b7e/us-east-1/2017-01-16/telegraf/ls.s3.9bf2d2d7-8238-4d9d-a82d-aff4d7283023.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604075959,"thread":"S3 output uploader, file: /tmp/logstash/af77db64-0bd9-4842-a973-a4ec5fa36896/us-east-1/2017-01-16/nginx-access/ls.s3.fa9b0ef9-4b0c-442d-87e8-efcf0a44faf9.2017-01-16T16.56.part4.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/af77db64-0bd9-4842-a973-a4ec5fa36896/us-east-1/2017-01-16/nginx-access/ls.s3.fa9b0ef9-4b0c-442d-87e8-efcf0a44faf9.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604076279,"thread":"S3 output uploader, file: /tmp/logstash/45529e93-3813-4988-ab58-957b2b868d87/us-east-1/2017-01-16/json/ls.s3.10d71bbb-e9e4-459a-add1-78af9d6ae466.2017-01-16T16.56.part4.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/45529e93-3813-4988-ab58-957b2b868d87/us-east-1/2017-01-16/json/ls.s3.10d71bbb-e9e4-459a-add1-78af9d6ae466.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604076434,"thread":"[main]>worker2","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/mysql-slow-query/ls.s3.bba8ddd5-5a25-4cb0-acf0-603d90ff124d.2017-01-16T16.56.part3.txt","path":"/tmp/logstash/a097f18a-289c-45f9-a415-7ec314d102ac/us-east-1/2017-01-16/mysql-slow-query/ls.s3.bba8ddd5-5a25-4cb0-acf0-603d90ff124d.2017-01-16T16.56.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604076434,"thread":"[main]>worker2","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/a097f18a-289c-45f9-a415-7ec314d102ac/us-east-1/2017-01-16/mysql-slow-query/ls.s3.bba8ddd5-5a25-4cb0-acf0-603d90ff124d.2017-01-16T16.56.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604076677,"thread":"S3 output uploader, file: /tmp/logstash/b151d0c6-2290-4bf1-8bdf-9087882d6b7e/us-east-1/2017-01-16/telegraf/ls.s3.9bf2d2d7-8238-4d9d-a82d-aff4d7283023.2017-01-16T16.56.part4.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/b151d0c6-2290-4bf1-8bdf-9087882d6b7e/us-east-1/2017-01-16/telegraf/ls.s3.9bf2d2d7-8238-4d9d-a82d-aff4d7283023.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604076715,"thread":"S3 output uploader, file: /tmp/logstash/a097f18a-289c-45f9-a415-7ec314d102ac/us-east-1/2017-01-16/mysql-slow-query/ls.s3.bba8ddd5-5a25-4cb0-acf0-603d90ff124d.2017-01-16T16.56.part3.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/a097f18a-289c-45f9-a415-7ec314d102ac/us-east-1/2017-01-16/mysql-slow-query/ls.s3.bba8ddd5-5a25-4cb0-acf0-603d90ff124d.2017-01-16T16.56.part3.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604077714,"thread":"S3 output uploader, file: /tmp/logstash/b820db22-3c6a-4a8e-9abe-3e60f3e3b81c/us-east-1/2017-01-16/syslog/ls.s3.8bf26151-4117-448d-a93d-7cd2eb516fc9.2017-01-16T16.41.part1.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/syslog/ls.s3.00133851-514b-47a0-b6f3-62cf050c518d.2017-01-16T16.56.part4.txt","path":"/tmp/logstash/568b1a82-0f93-4d1c-a965-babd6b038a6d/us-east-1/2017-01-16/syslog/ls.s3.00133851-514b-47a0-b6f3-62cf050c518d.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604077714,"thread":"S3 output uploader, file: /tmp/logstash/b820db22-3c6a-4a8e-9abe-3e60f3e3b81c/us-east-1/2017-01-16/syslog/ls.s3.8bf26151-4117-448d-a93d-7cd2eb516fc9.2017-01-16T16.41.part1.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/568b1a82-0f93-4d1c-a965-babd6b038a6d/us-east-1/2017-01-16/syslog/ls.s3.00133851-514b-47a0-b6f3-62cf050c518d.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604078086,"thread":"S3 output uploader, file: /tmp/logstash/568b1a82-0f93-4d1c-a965-babd6b038a6d/us-east-1/2017-01-16/syslog/ls.s3.00133851-514b-47a0-b6f3-62cf050c518d.2017-01-16T16.56.part4.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/568b1a82-0f93-4d1c-a965-babd6b038a6d/us-east-1/2017-01-16/syslog/ls.s3.00133851-514b-47a0-b6f3-62cf050c518d.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604078242,"thread":"S3 output uploader, file: /tmp/logstash/b820db22-3c6a-4a8e-9abe-3e60f3e3b81c/us-east-1/2017-01-16/syslog/ls.s3.8bf26151-4117-448d-a93d-7cd2eb516fc9.2017-01-16T16.41.part1.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/varnishncsa/ls.s3.9dd497bc-46ba-47ea-8de0-44b303319866.2017-01-16T16.56.part4.txt","path":"/tmp/logstash/c3b835fd-0a03-4838-82d2-a5b8b5ba6afb/us-east-1/2017-01-16/varnishncsa/ls.s3.9dd497bc-46ba-47ea-8de0-44b303319866.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604078243,"thread":"S3 output uploader, file: /tmp/logstash/b820db22-3c6a-4a8e-9abe-3e60f3e3b81c/us-east-1/2017-01-16/syslog/ls.s3.8bf26151-4117-448d-a93d-7cd2eb516fc9.2017-01-16T16.41.part1.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/c3b835fd-0a03-4838-82d2-a5b8b5ba6afb/us-east-1/2017-01-16/varnishncsa/ls.s3.9dd497bc-46ba-47ea-8de0-44b303319866.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604079301,"thread":"S3 output uploader, file: /tmp/logstash/c3b835fd-0a03-4838-82d2-a5b8b5ba6afb/us-east-1/2017-01-16/varnishncsa/ls.s3.9dd497bc-46ba-47ea-8de0-44b303319866.2017-01-16T16.56.part4.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/c3b835fd-0a03-4838-82d2-a5b8b5ba6afb/us-east-1/2017-01-16/varnishncsa/ls.s3.9dd497bc-46ba-47ea-8de0-44b303319866.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604082236,"thread":"S3 output uploader, file: /tmp/logstash/3be35c60-2698-4dd0-afad-117d1e6f437d/us-east-1/2017-01-16/syslog/ls.s3.0622f12f-54d0-4cac-ab2a-48761304fc76.2017-01-16T16.46.part2.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/filebeat/ls.s3.97255e24-9416-432a-a39a-286b2c180faf.2017-01-16T16.56.part4.txt","path":"/tmp/logstash/061be46e-aff6-4f3c-a877-6c74fe76160a/us-east-1/2017-01-16/filebeat/ls.s3.97255e24-9416-432a-a39a-286b2c180faf.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604082237,"thread":"S3 output uploader, file: /tmp/logstash/3be35c60-2698-4dd0-afad-117d1e6f437d/us-east-1/2017-01-16/syslog/ls.s3.0622f12f-54d0-4cac-ab2a-48761304fc76.2017-01-16T16.46.part2.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/061be46e-aff6-4f3c-a877-6c74fe76160a/us-east-1/2017-01-16/filebeat/ls.s3.97255e24-9416-432a-a39a-286b2c180faf.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604082346,"thread":"S3 output uploader, file: /tmp/logstash/061be46e-aff6-4f3c-a877-6c74fe76160a/us-east-1/2017-01-16/filebeat/ls.s3.97255e24-9416-432a-a39a-286b2c180faf.2017-01-16T16.56.part4.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/061be46e-aff6-4f3c-a877-6c74fe76160a/us-east-1/2017-01-16/filebeat/ls.s3.97255e24-9416-432a-a39a-286b2c180faf.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604084472,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604084473,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/php/ls.s3.5d54916c-60a6-4301-a5da-124b6d22c0fb.2017-01-16T16.56.part4.txt","path":"/tmp/logstash/dca41411-50f1-4cda-b973-062d6328e671/us-east-1/2017-01-16/php/ls.s3.5d54916c-60a6-4301-a5da-124b6d22c0fb.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604084473,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/dca41411-50f1-4cda-b973-062d6328e671/us-east-1/2017-01-16/php/ls.s3.5d54916c-60a6-4301-a5da-124b6d22c0fb.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604084477,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/nginx-error/ls.s3.2634336b-71d6-45d7-8d86-7cf269e6adf8.2017-01-16T16.56.part4.txt","path":"/tmp/logstash/3d621f4d-6252-4c48-a233-3d9bc8d87c94/us-east-1/2017-01-16/nginx-error/ls.s3.2634336b-71d6-45d7-8d86-7cf269e6adf8.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604084477,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/3d621f4d-6252-4c48-a233-3d9bc8d87c94/us-east-1/2017-01-16/nginx-error/ls.s3.2634336b-71d6-45d7-8d86-7cf269e6adf8.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604084488,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-17/telegraf/ls.s3.c75ffdd8-80a1-41ee-a30c-6cc132b3cd82.2017-01-16T16.56.part4.txt","path":"/tmp/logstash/7c71461c-5f33-44bc-9aaa-aacc8f4ecfcf/us-east-1/2017-01-17/telegraf/ls.s3.c75ffdd8-80a1-41ee-a30c-6cc132b3cd82.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604084488,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/7c71461c-5f33-44bc-9aaa-aacc8f4ecfcf/us-east-1/2017-01-17/telegraf/ls.s3.c75ffdd8-80a1-41ee-a30c-6cc132b3cd82.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604084491,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/consul/ls.s3.4a415a3b-2700-4154-ae3d-237835765f8d.2017-01-16T16.56.part4.txt","path":"/tmp/logstash/7ff61a5e-1981-477b-bf30-e5995492fae2/us-east-1/2017-01-16/consul/ls.s3.4a415a3b-2700-4154-ae3d-237835765f8d.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604084491,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/7ff61a5e-1981-477b-bf30-e5995492fae2/us-east-1/2017-01-16/consul/ls.s3.4a415a3b-2700-4154-ae3d-237835765f8d.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604084689,"thread":"S3 output uploader, file: /tmp/logstash/dca41411-50f1-4cda-b973-062d6328e671/us-east-1/2017-01-16/php/ls.s3.5d54916c-60a6-4301-a5da-124b6d22c0fb.2017-01-16T16.56.part4.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/dca41411-50f1-4cda-b973-062d6328e671/us-east-1/2017-01-16/php/ls.s3.5d54916c-60a6-4301-a5da-124b6d22c0fb.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604084733,"thread":"S3 output uploader, file: /tmp/logstash/7ff61a5e-1981-477b-bf30-e5995492fae2/us-east-1/2017-01-16/consul/ls.s3.4a415a3b-2700-4154-ae3d-237835765f8d.2017-01-16T16.56.part4.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/7ff61a5e-1981-477b-bf30-e5995492fae2/us-east-1/2017-01-16/consul/ls.s3.4a415a3b-2700-4154-ae3d-237835765f8d.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604084737,"thread":"S3 output uploader, file: /tmp/logstash/7ff61a5e-1981-477b-bf30-e5995492fae2/us-east-1/2017-01-16/consul/ls.s3.4a415a3b-2700-4154-ae3d-237835765f8d.2017-01-16T16.56.part4.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/heartbeat/ls.s3.37a366e0-3d99-4934-95fd-0eff11d25bc4.2017-01-16T16.56.part4.txt","path":"/tmp/logstash/beb88c15-d696-421e-81fc-3eb34c020ca4/us-east-1/2017-01-16/heartbeat/ls.s3.37a366e0-3d99-4934-95fd-0eff11d25bc4.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604084737,"thread":"S3 output uploader, file: /tmp/logstash/7ff61a5e-1981-477b-bf30-e5995492fae2/us-east-1/2017-01-16/consul/ls.s3.4a415a3b-2700-4154-ae3d-237835765f8d.2017-01-16T16.56.part4.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/beb88c15-d696-421e-81fc-3eb34c020ca4/us-east-1/2017-01-16/heartbeat/ls.s3.37a366e0-3d99-4934-95fd-0eff11d25bc4.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604084739,"thread":"S3 output uploader, file: /tmp/logstash/7ff61a5e-1981-477b-bf30-e5995492fae2/us-east-1/2017-01-16/consul/ls.s3.4a415a3b-2700-4154-ae3d-237835765f8d.2017-01-16T16.56.part4.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/elb/ls.s3.5bedc623-8fec-473a-9b4d-cfcc2e7459a1.2017-01-16T16.56.part4.txt","path":"/tmp/logstash/df43c4cb-15c1-4293-8a04-1ac3207b73e3/us-east-1/2017-01-16/elb/ls.s3.5bedc623-8fec-473a-9b4d-cfcc2e7459a1.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604084740,"thread":"S3 output uploader, file: /tmp/logstash/7ff61a5e-1981-477b-bf30-e5995492fae2/us-east-1/2017-01-16/consul/ls.s3.4a415a3b-2700-4154-ae3d-237835765f8d.2017-01-16T16.56.part4.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/df43c4cb-15c1-4293-8a04-1ac3207b73e3/us-east-1/2017-01-16/elb/ls.s3.5bedc623-8fec-473a-9b4d-cfcc2e7459a1.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604084758,"thread":"S3 output uploader, file: /tmp/logstash/3d621f4d-6252-4c48-a233-3d9bc8d87c94/us-east-1/2017-01-16/nginx-error/ls.s3.2634336b-71d6-45d7-8d86-7cf269e6adf8.2017-01-16T16.56.part4.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/3d621f4d-6252-4c48-a233-3d9bc8d87c94/us-east-1/2017-01-16/nginx-error/ls.s3.2634336b-71d6-45d7-8d86-7cf269e6adf8.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604084824,"thread":"S3 output uploader, file: /tmp/logstash/7c71461c-5f33-44bc-9aaa-aacc8f4ecfcf/us-east-1/2017-01-17/telegraf/ls.s3.c75ffdd8-80a1-41ee-a30c-6cc132b3cd82.2017-01-16T16.56.part4.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/7c71461c-5f33-44bc-9aaa-aacc8f4ecfcf/us-east-1/2017-01-17/telegraf/ls.s3.c75ffdd8-80a1-41ee-a30c-6cc132b3cd82.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604084867,"thread":"S3 output uploader, file: /tmp/logstash/beb88c15-d696-421e-81fc-3eb34c020ca4/us-east-1/2017-01-16/heartbeat/ls.s3.37a366e0-3d99-4934-95fd-0eff11d25bc4.2017-01-16T16.56.part4.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/beb88c15-d696-421e-81fc-3eb34c020ca4/us-east-1/2017-01-16/heartbeat/ls.s3.37a366e0-3d99-4934-95fd-0eff11d25bc4.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604085047,"thread":"S3 output uploader, file: /tmp/logstash/df43c4cb-15c1-4293-8a04-1ac3207b73e3/us-east-1/2017-01-16/elb/ls.s3.5bedc623-8fec-473a-9b4d-cfcc2e7459a1.2017-01-16T16.56.part4.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/df43c4cb-15c1-4293-8a04-1ac3207b73e3/us-east-1/2017-01-16/elb/ls.s3.5bedc623-8fec-473a-9b4d-cfcc2e7459a1.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604085050,"thread":"S3 output uploader, file: /tmp/logstash/df43c4cb-15c1-4293-8a04-1ac3207b73e3/us-east-1/2017-01-16/elb/ls.s3.5bedc623-8fec-473a-9b4d-cfcc2e7459a1.2017-01-16T16.56.part4.txt","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/metric/ls.s3.3d40259f-95bc-47a7-bbce-6766ba4f9fd0.2017-01-16T16.56.part4.txt","path":"/tmp/logstash/04b4a9fa-c3ca-40d5-a9c7-a66b32bfe086/us-east-1/2017-01-16/metric/ls.s3.3d40259f-95bc-47a7-bbce-6766ba4f9fd0.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604085050,"thread":"S3 output uploader, file: /tmp/logstash/df43c4cb-15c1-4293-8a04-1ac3207b73e3/us-east-1/2017-01-16/elb/ls.s3.5bedc623-8fec-473a-9b4d-cfcc2e7459a1.2017-01-16T16.56.part4.txt","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/04b4a9fa-c3ca-40d5-a9c7-a66b32bfe086/us-east-1/2017-01-16/metric/ls.s3.3d40259f-95bc-47a7-bbce-6766ba4f9fd0.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604085110,"thread":"S3 output uploader, file: /tmp/logstash/04b4a9fa-c3ca-40d5-a9c7-a66b32bfe086/us-east-1/2017-01-16/metric/ls.s3.3d40259f-95bc-47a7-bbce-6766ba4f9fd0.2017-01-16T16.56.part4.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/04b4a9fa-c3ca-40d5-a9c7-a66b32bfe086/us-east-1/2017-01-16/metric/ls.s3.3d40259f-95bc-47a7-bbce-6766ba4f9fd0.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604099532,"thread":"[main]>worker2","logEvent":{"message":"Rotate file","strategy":"LogStash::Outputs::S3::SizeAndTimeRotationPolicy","key":"us-east-1/2017-01-16/log/ls.s3.3051f7b8-d026-4bc8-a284-24b698295061.2017-01-16T16.56.part4.txt","path":"/tmp/logstash/e0059e95-fc73-4d6a-90b6-4f0f9bc4ab20/us-east-1/2017-01-16/log/ls.s3.3051f7b8-d026-4bc8-a284-24b698295061.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604099532,"thread":"[main]>worker2","logEvent":{"message":"Queue for upload","path":"/tmp/logstash/e0059e95-fc73-4d6a-90b6-4f0f9bc4ab20/us-east-1/2017-01-16/log/ls.s3.3051f7b8-d026-4bc8-a284-24b698295061.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604099627,"thread":"S3 output uploader, file: /tmp/logstash/e0059e95-fc73-4d6a-90b6-4f0f9bc4ab20/us-east-1/2017-01-16/log/ls.s3.3051f7b8-d026-4bc8-a284-24b698295061.2017-01-16T16.56.part4.txt","logEvent":{"message":"Removing temporary file","file":"/tmp/logstash/e0059e95-fc73-4d6a-90b6-4f0f9bc4ab20/us-east-1/2017-01-16/log/ls.s3.3051f7b8-d026-4bc8-a284-24b698295061.2017-01-16T16.56.part4.txt"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604100054,"thread":"S3 output uploader, file: /tmp/logstash/df43c4cb-15c1-4293-8a04-1ac3207b73e3/us-east-1/2017-01-16/elb/ls.s3.5bedc623-8fec-473a-9b4d-cfcc2e7459a1.2017-01-16T16.56.part4.txt","logEvent":{"message":"Periodic check for stale files"}}
{"level":"DEBUG","loggerName":"logstash.outputs.s3","timeMillis":1484604115058,"thread":"S3, Stale factory sweeper","logEvent":{"message":"Periodic check for stale files"}}

[ph]

@joshuaspence Can you add your S3 output specific configuration to this ticket?

There are a few things to take into consideration when using dynamic prefixes,
When you use a dynamic prefix, like these %{+yyyy-MM-dd}/ to %{+yyyy-MM-dd}/%{type}/, the s3 output will create a file per dynamic path created.

This mean that the rotation strategy, the default is time + size, will be applied to each file, if the velocity of the events isn't big it could mean the time will only be used to known when a file need to be send to S3, the default time value is 15 minutes, which may be big in your case that would result in the s3 output not keeping up.

On the other, if you generate a lot of files that could mean the worker cannot keep up, the default is half of your cores, so I suggest that you bump that value since theses thread will be mostly IO bound.

If I start and stop Logstash a few times, the temporary files seem to be successfully uploaded. If I leave Logstash running, however, the files in /tmp/logstash just seem to grow.

by default Logstash will start a recovery process that will push all the files found in the path to S3 and will block logstash until he was able to do so.

I suggest playing with theses settings: time_file, size_file, rotation_strategy and the number of workers and the bounded queue size.

Also thanks @joshuaspence for reporting all the issues you found, I pay special attention to them, and I really appreciate your reports.

[Jinkxed]

I was having something similar happen prior to upgrading my ELK stack. I wasn't using any type of dynamic prefix. I was just using something like staging-, uat-, prod- etc.

This issue would eventually run the server out of open files and I'd have to restart ELK.

[ph]

@sc-chad This was also with the s3 output 4.0.3 ?

[Jinkxed]

@ph I haven't been able to get 4.0.3 to work. I posted an issue in another thread. #121

[ph]

@sc-chad OK, I will take look at the other issue, the problem that you described in this issue was with what version of logstash and what version of the s3 output?

[Jinkxed]

@ph
logstash was 5.0.1 I believe
s3 output was 2.3.0

[ph]

@sc-chad much have changed in the version, especially the file descriptor handling, I wonder if the open file issues was not from another plugins.

The current master version is a complete rewrite and it not yet bundled in the official release of logstash

[Jinkxed]

@ph

Well I know the issue only started when we started using the s3 output plugin (previously).

I'll know more soon as I can get 4.0.3 to work if this issue is still there or not.

[joshuaspence]

Yeah this is likely a duplicate of #122.