java.lang.ArrayIndexOutOfBoundsException: Index 93 out of bounds for length 93 #9044
Description
Tracer Version(s)
1.50.0
Java Version(s)
21
JVM Vendor
Eclipse Adoptium / Temurin
Bug Report
After tracer got update to 1.50.0, there is an issue with the WebClient, I can see a lot of ReadTimeoutExceptions as well as these stacktraces:
java.lang.ArrayIndexOutOfBoundsException: Index 93 out of bounds for length 93
at datadog.trace.agent.core.util.PercentEscaper.nextEscapeIndex(PercentEscaper.java:240)
at datadog.trace.agent.core.util.PercentEscaper.escapeSlow(PercentEscaper.java:216)
at datadog.trace.agent.core.util.PercentEscaper.escape(PercentEscaper.java:150)
at datadog.trace.agent.core.util.PercentEscaper.escapeValue(PercentEscaper.java:115)
at datadog.trace.agent.core.baggage.BaggagePropagator.inject(BaggagePropagator.java:80)
at datadog.context.propagation.CompositePropagator.inject(CompositePropagator.java:18)
at datadog.trace.instrumentation.netty41.client.HttpClientRequestTracingHandler.write(HttpClientRequestTracingHandler.java:95)
at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:891)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:956)
at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1263)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
Requests fail around 90% of the time, but we were not able to identify what exactly triggers the failure.
I created a separate support request in DD with trace links, but this might be helpful for others facing similar behaviour
Expected Behavior
APM shouldn't affect the application on parsing failures
Reproduction Code
This is how the client was created
public static <V> V client(
WebClient.Builder builder,
ClientSettings settings,
Class<V> clientClass
) {
var client = builder
.clone()
.baseUrl(settings.baseUrl())
.clientConnector(
new ReactorClientHttpConnector(
HttpClient.create().compress(true).responseTimeout(settings.requestTimeout())
)
);
var adapter = WebClientAdapter.create(client.build());
var factory = HttpServiceProxyFactory.builderFor(adapter).build();
return factory.createClient(clientClass);
}