Netty channel hangs after sending a payload of size > 16 MB

[xiazuojie]

The max allowed size of a frame is 16 MB (according to this). If a payload is larger than 16 MB, encoding it to a frame will fail with an exception "Length is larger than 24 bits".

However, this exception is not properly handled by MonoSendMany. A single such exception (encoding happens inside applying MapFuseableSubscriber#mapper) will terminate the subscription. No payloads will be sent to netty channel after (due to subscription terminated).

In short, a single payload of size >= 16 MB will make the channel hang forever. And worse, there is no error log at all when this happens.

According to spec, large payloads should be fragmented to more frames. However, this feature is currently not implemented transparently.

[OlegDokuka]

I will take a look at that.

Is not that a reactor-netty bug?

cc/ @violetagg

[xiazuojie]

The feature used to be in rsocket-java (SendPublisher) , later moved to reactor-netty. I think it's more about RSocket Java not properly handling payloads > 16MB.

[violetagg]

I will take a look at that.

Is not that a reactor-netty bug?

cc/ @violetagg

Not seen so far.

[rstoyanchev]

According to spec, large payloads should be fragmented to more frames. However, this feature is currently not implemented transparently.

There is a fragmentation(..) setting on RSocketFactory and if you set that it should fragment. If you're keeping it at 0 intentionally (it is also the default) then yes it should be handled better.

[OlegDokuka]

Due to what I see, there are two issues:

1. Lack of documentation on what is the defaults in RSocket-Java and that there is no fragmentation by default. It means that all the frames that exit 16 MB size is illegal.

2. 1. We have a bug in TCP connection which tries to add frame length at the transport level and throws an exception because the encoded frame was not validated and just simply got encoded and sent (lack of Validation on RSocket level). Therefore, FrameLengthFlightweight does late validation and throws an exception
   2. We have a bug in WebSocket which sends that frame and then the responder is not able to reassemble it back

Note, in both (2,3) we get connection termination instead of logical stream termination with the proper message

[rstoyanchev]

Given #768 to validate payload length earlier, and #769 to document, what's actually left here?

Payloads with invalid length should not be able to get through, but arguably the transport layer should have independent validation. An additional check in WebSocket transports perhaps to prevent sending a frame that's too large? If that's all it is maybe this can become a task under #768 and be closed as "superseded"?

[OlegDokuka]

@rstoyanchev

An additional check in WebSocket transports

This check is already present, no need to put extra one