[Bug]: Stack overflow when calling unfiltered `tracing::event!` after `SdkLogProvider` is shutdown.

[lucamuscat]

What happened?

When integrating the opentelemetry crate with the tracing_subscriber crate, calling tracing::event! and its derivates after the SdkLoggerProvider has been called leads to a stack overflow due to BatchLogProcessor::emit using otel_warn! when trying to report the mpsc::TrySendError:Disconnected error.

It seems that under the hood, otel_warn! may call tracing::warn!, which will cause the warning reporting the BatchLogProcessor being shutdown to get fed back into the BatchLogProcessor due to the tracing subscriber being none the wiser, which in turn causes the same warning to be emitted again, leading to infinite recursion.

I have created a PR that fixes this issue by simply not calling otel_warn! in this edge case if the log record itself is emitted by the same otel_warn!. I will publish this PR after creating this issue.

OpenTelemetry API Version (i.e version of opentelemetry crate)

0.30.0

OpenTelemetry SDK Version (i.e version of opentelemetry_sdk crate)

0.30.0

What Exporter(s) are you seeing the problem on?

N/A

Relevant log output

thread 'test_no_stack_overflow_when_event_is_emitted_after_shutdown' has overflowed its stack
fatal runtime error: stack overflow, aborting

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[scottgerring]

I believe this is the same root issue as @cijothomas 's trying to fix in: #2877 and #3084 . We don't seem to have an issue open to capture this anymore @cijothomas - should we use this one, or re-open e.g. #761 ?

[lucamuscat]

I believe this is the same root issue as @cijothomas 's trying to fix in: #2877 and #3084 . We don't seem to have an issue open to capture this anymore @cijothomas - should we use this one, or re-open e.g. #761 ?

It does seem to be the case. #761 has been open for around 4 years now, with no definite fix at the moment.

Although my PR does not solve the issue at a fundamental level, it does prevent the stack overflow from occurring in this easily reproducible bug.

Now the question is whether we should temporarily fix this stack overflow issue until the root of the issue itself is solved, or leave a potential footgun (i.e., the stack overflow) in the API for an undetermined period of time.

Edit: I pushed a new PR (#3172) that suppresses telemetry emitted by BatchLogProcessor::emit, fixing the stack overflow issue with minimal changes.