Netty 4.1.87 to 4.1.94 seriously regressed perf

[ejona86]

On July 6th a change to master seriously impacted our performance benchmarks.

The only interesting thing from that time is the Netty upgrade.

The prod team noticed as well, in b/290613465 . We have two candidate PRs as the cause for the slowdown, but still investigating.

[ejona86]

I've confirmed that netty/netty#13408 is the cause of the regression. These runs were done at 361616a (the upgrade commit) and 361616a^, so just the Netty version changed.

Benchmark                                                             (direct)  (transport)    Mode     Cnt           Score          Error  Units
4.1.87:
TransportBenchmark.streamingCallsMessageThroughput                       false        NETTY   thrpt      10      645802.206 ±    13270.777  ops/s
4.1.88:
TransportBenchmark.streamingCallsMessageThroughput                       false        NETTY   thrpt      10      654535.745 ±     7323.670  ops/s
4.1.93:
TransportBenchmark.streamingCallsMessageThroughput                       false        NETTY   thrpt      10      658665.726 ±    12600.729  ops/s
4.1.94 (two runs):
TransportBenchmark.streamingCallsMessageThroughput                       false        NETTY   thrpt      10      564743.743 ±     4025.113  ops/s
TransportBenchmark.streamingCallsMessageThroughput                       false        NETTY   thrpt      10      600532.355 ±    19314.619  ops/s
Custom 4.1.94 build with PR 13408 reverted:
TransportBenchmark.streamingCallsMessageThroughput                       false        NETTY   thrpt      10      653681.499 ±     5988.903  ops/s

[normanmaurer]

@ejona86 Hmm... This is interesting. I wonder why this could cause a regression...

[franz1981]

@ejona86 there is anything that was making uses of the specific classes I have removed/changed?
Like using reflection to do something specific that now fail?

Another reason could be that there is some condition specific to grpc which can cause small/normal caches to not be used anymore, that is something pretty easy to spot with flamegraphs

If there is a microbench, next week I can help to investigate

[ejona86]

netty/netty#13510 tries to address this. I will run the benchmarks and see how things fare.

I was totally with y'all; I didn't understand how the change caused such a regression. It looked like a plain code movement. We are scheduled to do our release build today, so I was moving fast to avoid reverting to 4.1.87. I only tracked it down late on a Friday and it is probably already fixed Monday morning!

[ejona86]

The benchmarks recovered after this change: https://grafana-dot-grpc-testing.appspot.com/?orgId=1&from=1688194800000&to=1690268399000

[normanmaurer]

@ejona86 thanks a lot for the quick test... Would still be nice if we could get a better understanding what exactly was the problem :) But at least its fixed