Rate-limiting for resend request responses

[philipwhiuk]

When a session reconnects mid-session it's possible for it to miss a high volume of messages.

At the application level it's possible for the application to self-throttle the release of messages.

It would be useful if resend-request responses could be optionally-throttled to only release messages at a given rate.

This would ideally take the form of a per-session configuration parameter e.g. ResendRequestMaxMessagesPerSecond.

An alternative is to implement it in the form of MINA filters, however this solution can lead to high memory usage because MINA will be forced to store in memory all the resend response messages while they are dequeued.

[chrjohn]

I think there can be problems with this approach because of the following reasons:

• When there is a high volume of messages and you throttle resends I assume it can take a long time until all resent messages have been received.
• Moreover, the receiving side needs to cache all messages until the ResendRequest has been satisfied which also could lead to an out-of-memory situation.
• Also you are probably going to generate even more messages while the resend is still running. So will the resend ever be satisfied?

So I am asking myself if this won't cause more problems than it will solve. But probably depends on the number of messages you are wanting to resend. What could possibly cause such a high volume of messages?

My approach would be to increase the socket receive buffer on the receiving side. ;)

[philipwhiuk]

So we’re already throttling the standard flow in our app code and caching messages. That’s not an issue.

The problem is that while this works fine in the connected case, when they lose connection we send something like HeartbeatInterval*2*ThrottleRate messages before we disconnect them.

Then when we reconnect QFJ unleashes the whole flow in a fraction of a second which often causes another disconnect.

As for messages sent while pending resend - I think we handle that in app code (or could do). Anyway twice throttle isn’t much of a problem vs 100*Throttle.

And yeah if I could fix the other side I would!

Yes, the rate I’m looking at is not high here - I’m talking about < 100 msgs as a throttle.

This might well be a custom one off use case. I just try to upstream as much as I can.

On a very related note - is there any good documentation of the threading model in QFJ?

[AbhinavAtul]

Would it be a good idea to customize the apache MINA IOSession#writeRequestQueue with a Deque and place response payloads for resend requests at the start of the Deque while regular response payloads at the end? The queue processor will consume from the start prioritizing resend data so that counter party apps catch up with a higher probability rather than being overwhelmed with messages that will can only be processed after resend requests are satisfied.

[AbhinavAtul]

Version 3.0.0-M2 of apache-mina makes the DefaultWriteQueue implement Deque by extending the ConcurrentLinkedDeque. apache-mina version bump should allow for an easier customization on prioritized scheduling of resend responses.

[chrjohn]

@AbhinavAtul the 3.0.0 branch of MINA is discontinued as far as I am aware. Current development is on 2.2.x