Consumer creation lagging and timeout #863
-
Describe the bugFor a serial creation of 300 named consumers for a single stream with pre-existing messages, each creation may take various seconds or even be timeout. If there is no existing messages of the stream prior the creation flow, it can be all good and finished very quickly in a few seconds. Please be noted, as long as there is 1 message prior the creation flow, it can be problematic. Changing the invterval of each consumer creation can mitigate or worse the issue for it to appear. Server be running on the same machine of the app or a separate one ( eg, docker). Attached the reproducing gradle app. The log of the failed case with 10ms interval: ./gradlew run
The log of failed case with 1 second interval: ./gradlew run
Reproduction stepsServer ver: 4.1.3 (brew), 4.1 (docker)
Expected behaviorAll 300 consumer should be created sucessfuly without lagging or timeout, just like the behavior when there is no existing msg of the stream. Additional contextNo response |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 5 replies
-
|
Thanks for the detailed analysis. I managed to reproduce the behavior locally. You should use the The example combines several factors that combined trigger this issue (255 consumers per connection by default, only 1 node, all consumers on the same, 300 consumers, which can be considered a significant amount for 1 connection). I would not consider this example completely realistic though. If you are testing RabbitMQ streams for a project, you are welcome to share more about your use case so we can discuss it. Nevertheless I identified a problem on the server side that makes the problem go away. It will require more testing, so should go in 4.2.1 as 4.2.0 is scheduled for new week and we are in code-freeze period. I'll update this discussion accordingly. |
Beta Was this translation helpful? Give feedback.
-
|
my project's case is like this: I have the appA publishes messages and the appB connects devices and consumes messages and pass the messages to the connectecd devices in the device's language ; I need keep the order of message for each deivce. I can't use a single share queue for all the devices as there will be the 'head of line' issue caused by device getting offline. Now I'm trying the solution to use stream, one consumer for each device with filter that is based on the deviceId. The solution works fine with limited number of devices, eg a few to dozens. while trying more devices, the reported issue started to appear. I need to support 50k+ devices, both appA and appB can be multiple instances ; still each instance needs to handle 10k+ devices. I noticed maxConsumersByConnection before raising the issue and it didn't seem to be relevant as the connections would be automatically created when needed, I assumed. I thought with 50k devices and max 255 consumer per conn, there would be about 200 connections which didn't seem to be unrealistic to me. I think the demo application is realitic. Let me know why it is not if the design of rabbitmq stream aims differerntly. I can't see any restriction/warning of using it in this way from the official document. Any suggestion on the solution will be appreciated very much. Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
if you want to simulate an IoT scenario each device would have their own connection (right?) so you'd need to set maxConsumersByConnection to 1 |
Beta Was this translation helpful? Give feedback.
-
|
This broker PR should help. It built the Using 1 consumer per connection should indeed be more realistic for your use case, for both the broker and client side. Testing for 50 K devices will require some tuning, I'm not sure this could be done on a laptop, you'd better off testing on a beefy setup, at least 1 client and 1 broker VMs. Server-side offset tracking may not be the best solution for 50 K consumer references, you could disable it by not setting any name on the consumers and use the subscription listener callback to implement offset tracking yourself, e.g. with a KV store. |
Beta Was this translation helpful? Give feedback.
-
If testing with 1 second interval, more errors after timeout:
If setting maxConsumersByConnection from 1 to 50, it seems ok but some consumers can take more and more time like couple of seconds, while getting the value close to the default max.
|
Beta Was this translation helpful? Give feedback.
-
|
We recommended to use 1 consumer per connection because this is a IoT scenario, so we assume each device will have its own VM, connection, etc. You should try to disable server-side offset tracking. Each started consumer gets the only message in the stream and store its offset. Offset tracking data is also stored in the stream and this triggers listeners on the server side. Long story short, it can keep connections busy and make the whole starting sequence difficult to sustain. Disabling it could tell us if it is the main source of problems. You can also experiment with You may also hit the limit of 1 stream at some point, so consider also partitioning your system into e.g. 10 or more streams. I don't expect one node running on a laptop to handle 50 K devices with 1 stream. A 3- or 5-node cluster, with several streams spread across the node seems more realistic. |
Beta Was this translation helpful? Give feedback.
Thanks for the detailed analysis. I managed to reproduce the behavior locally. You should use the
EnvironmentBuilder#maxConsumersByConnection(int)method to avoid having 255 consumers per connection. This default is OK for an application with consumers that does not consume from the same stream.The example combines several factors that combined trigger this issue (255 consumers per connection by default, only 1 node, all consumers on the same, 300 consumers, which can be considered a significant amount for 1 connection). I would not consider this example completely realistic though. If you are testing RabbitMQ streams for a project, you are welcome to share more about your use case so we…