Skip to content
This repository was archived by the owner on Nov 16, 2020. It is now read-only.

Commit a4ce73d

Browse files
Doco updates based on recent mailing list discussions/feedback
1 parent 032d9c9 commit a4ce73d

File tree

1 file changed

+49
-25
lines changed

1 file changed

+49
-25
lines changed

README.md

Lines changed: 49 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -20,48 +20,63 @@ of a publisher and a consumer:
2020

2121
![Sharding Overview](https://raw.githubusercontent.com/rabbitmq/rabbitmq-sharding/master/docs/sharded_queues.png)
2222

23-
As you can see in the graphic, the producers publishes a series of
23+
On the picture above the producers publishes a series of
2424
messages, those messages get partitioned to different queues, and then
25-
our consumer get messages from one of those queues. Therefore if you
26-
have a partition with 3 queues, then you will need to have at least 3
25+
our consumer get messages from one of those queues. Therefore if there is
26+
a partition with 3 queues, it is assumed that there are at least 3
2727
consumers to get all the messages from those queues.
2828

29-
## Auto-scaling ##
29+
Queues in RabbitMQ are [units of concurrency](http://www.rabbitmq.com/queues.html#runtime-characteristics)
30+
(and, if there are enough cores available, parallelism). This plugin makes
31+
it possible to have a single logical queue that is partitioned into
32+
multiple regular queues ("shards"). This trades off total ordering
33+
on the logical queue for gains in parallelism.
34+
35+
Message distribution between shards (partitioning) is achieved
36+
with a custom exchange type that distributes messages by applying
37+
a hashing function to the routing key.
3038

31-
One interesting property of this plugin, is that if you add more nodes
32-
to your RabbitMQ cluster, then the plugin will automatically create
33-
more shards in the new node. Say you had a shard with 4 queues in
34-
`node a` and `node b` just joined the cluster. The plugin will
35-
automatically create 4 queues in `node b` and join them to the shard
36-
partition. Already delivered messages _will not_ be rebalanced, but
37-
newly arriving messages will be partitioned to the new queues.
3839

39-
## Partitioning Messages ##
40+
## Messages Distribution Between Shards (Partitioning)
4041

41-
The exchanges that ship by default with RabbitMQ work in a "all or
42+
The exchanges that ship by default with RabbitMQ work in an "all or
4243
nothing" fashion, i.e: if a routing key matches a set of queues bound
4344
to the exchange, then RabbitMQ will route the message to all the
44-
queues in that set. Therefore for this plugin to work, we need to
45+
queues in that set. For this plugin to work it is necessary to
4546
route messages to an exchange that would partition messages, so they
46-
are routed to _at most_ one queue.
47+
are routed to _at most_ one queue (a subset).
4748

48-
The plugin provides a new exchange type `"x-modulus-hash"` that will use
49-
the traditional hashing technique applying to partition messages
50-
across queues.
49+
The plugin provides a new exchange type, `"x-modulus-hash"`, that will use
50+
a hashing function to partition messages routed to a logical queue
51+
across a number of regular queues (shards).
5152

5253
The `"x-modulus-hash"` exchange will hash the routing key used to
5354
publish the message and then it will apply a `Hash mod N` to pick the
5455
queue where to route the message, where N is the number of queues
5556
bound to the exchange. **This exchange will completely ignore the
5657
binding key used to bind the queue to the exchange**.
5758

58-
You could also use other exchanges that have similar behaviour like
59-
the _Consistent Hash Exchange_ or the _Random Exchange_. The first
60-
one has the advantage of shipping directly with RabbitMQ.
59+
There are other exchanges with similar behaviour:
60+
the _Consistent Hash Exchange_ or the _Random Exchange_.
61+
Those were designed with regular queues in mind, not this plugin, so `"x-modulus-hash"`
62+
is highly recommended.
63+
64+
If message partitioning is the only feature necessary and the automatic scaling
65+
of the number of shards (covered below) is not needed or desired, consider using
66+
[Consistent Hash Exchange](https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange)
67+
instead of this plugin.
68+
69+
70+
## Auto-scaling
71+
72+
One of the main properties of this plugin is that when a new node
73+
is added to the RabbitMQ cluster, then the plugin will automatically create
74+
more shards on the new node. Say there is a shard with 4 queues on
75+
`node a` and `node b` just joined the cluster. The plugin will
76+
automatically create 4 queues on `node b` and "join" them to the shard
77+
partition. Already delivered messages _will not_ be rebalanced but
78+
newly arriving messages will be partitioned to the new queues.
6179

62-
If _just need message partitioning_ but not the automatic queue
63-
creation provided by this plugin, then you can just use the
64-
[Consistent Hash Exchange](https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange).
6580

6681
## Consuming From a Sharded [Pseudo-]Queue ##
6782

@@ -109,14 +124,23 @@ this plugin.
109124
For load balancers, the "least connections" strategy is more likely to produce an even distribution compared
110125
to round robin and other strategies.
111126

127+
### How Evenly Will Messages Be Distributed?
128+
129+
As with many data distribution approaches based on a hashing function,
130+
even distribution between shards depends on the distribution (variability) of inputs,
131+
that is, routing keys. In other words the larger the set of routing keys is,
132+
the more even will message distribution between shareds be. If all messages had
133+
the same routing key, they would all end up on the same shard.
134+
135+
112136

113137
## Installing ##
114138

115139
### RabbitMQ 3.6.0 or later
116140

117141
As of RabbitMQ `3.6.0` this plugin is included into the RabbitMQ distribution.
118142

119-
Enable it with the following command:
143+
Like any other [RabbitMQ plugin](http://www.rabbitmq.com/plugins.html) it has to be enabled before it can be used:
120144

121145
```bash
122146
rabbitmq-plugins enable rabbitmq_sharding

0 commit comments

Comments
 (0)