Periodical dd-task-scheduler errors - Unable to create StatsD client - localhost:8125

[drobakowski]

Hi there,

I'm not sure if the following errors origins from this library or java-dogstatsd-client, but we're seeing a lot of dd-task-scheduler errors from Java lambdas like this:

[dd.trace 2022-04-18 22:16:49:700 +0000] [dd-task-scheduler] ERROR datadog.communication.monitor.DDAgentStatsDConnection - Unable to create StatsD client - localhost:8125

We are using version 1.4.4 of this library in combination with the latest serverless-datadog-plugin - v5.0.0 which uses the lambda layer v21 of datadog-lambda-extension and v4 of the dd-trace-java layer and java11. For instrumenting things we've followed the official DataDog documentation. The errors doesn't seem to be specific to the Serverless plugin, as we also get this errors when we integrate the layers and configs directly without the plugin. Also the log level doesn't seem to take effect, as we're seeing the logs unaggregated on INFO level, despite the fact that we have configured logLevel: warn or DD_LOG_LEVEL: WARN.

Our serverless.yml config looks like this:

...
provider:
  environment:
    DD_ENV: ${self:provider.stage}
    DD_SERVICE: ${self:service}
    DD_TRACE_SAMPLE_RATE: ${self:custom.datadogTracingSampleRate.${opt:stage, self:provider.stage}}
    DD_VERSION: ${git:sha1}
...
custom:
  datadogTracingSampleRate:
    dev: 1.0
    int: 0.8
    prod: 1.0
  ...
  datadog:
    addExtension: true
    apiKey: ${ssm:.../apiKey}
    site: datadoghq.eu

    flushMetricsToLogs: true
    addLayers: true
    logLevel: warn
    enableTags: true
    injectLogContext: true
    integrationTesting: false

Any ideas what we could do to fix the error?

[maxday]

Hey @drobakowski
Thanks for opening the issue, I was able to reproduce it on my end.
Can you try setting DD_RUNTIME_METRICS_ENABLED to false as en environment variable?

[drobakowski]

Hi @maxday, thanks for your reply! I've set the environment variable but unfortunately it didn't seem to help, getting the same errors. :/

[drobakowski]

Hi @maxday, after another fresh deploy with DD_RUNTIME_METRICS_ENABLED: false and some time the errors doesn't seem to occur anymore. :) So is this feature not supported with Java runtime and should we update the documentation and automatically configure this in the datadog-serverless-plugin for Java runtime?

[maxday]

Thanks for confirming @drobakowski
Yes I'll create a ticket to document that 💯

[drobakowski]

Hi @maxday,

after switching to the new Datadog Extension v22 with the release of serverless-plugin-datadog v5.0.1 we do see errors again with reference to StatsD client creation:

as well as the following:

DD_RUNTIME_METRICS_ENABLED: false is still set.

Do you maybe have any idea what we're doing wrong or what have changed?

Thanks and all the best
David