@@ -20,48 +20,63 @@ of a publisher and a consumer:
20
20
21
21
![ Sharding Overview] ( https://raw.githubusercontent.com/rabbitmq/rabbitmq-sharding/master/docs/sharded_queues.png )
22
22
23
- As you can see in the graphic, the producers publishes a series of
23
+ On the picture above the producers publishes a series of
24
24
messages, those messages get partitioned to different queues, and then
25
- our consumer get messages from one of those queues. Therefore if you
26
- have a partition with 3 queues, then you will need to have at least 3
25
+ our consumer get messages from one of those queues. Therefore if there is
26
+ a partition with 3 queues, it is assumed that there are at least 3
27
27
consumers to get all the messages from those queues.
28
28
29
- ## Auto-scaling ##
29
+ Queues in RabbitMQ are [ units of concurrency] ( http://www.rabbitmq.com/queues.html#runtime-characteristics )
30
+ (and, if there are enough cores available, parallelism). This plugin makes
31
+ it possible to have a single logical queue that is partitioned into
32
+ multiple regular queues ("shards"). This trades off total ordering
33
+ on the logical queue for gains in parallelism.
34
+
35
+ Message distribution between shards (partitioning) is achieved
36
+ with a custom exchange type that distributes messages by applying
37
+ a hashing function to the routing key.
30
38
31
- One interesting property of this plugin, is that if you add more nodes
32
- to your RabbitMQ cluster, then the plugin will automatically create
33
- more shards in the new node. Say you had a shard with 4 queues in
34
- ` node a ` and ` node b ` just joined the cluster. The plugin will
35
- automatically create 4 queues in ` node b ` and join them to the shard
36
- partition. Already delivered messages _ will not_ be rebalanced, but
37
- newly arriving messages will be partitioned to the new queues.
38
39
39
- ## Partitioning Messages ##
40
+ ## Messages Distribution Between Shards (Partitioning)
40
41
41
- The exchanges that ship by default with RabbitMQ work in a "all or
42
+ The exchanges that ship by default with RabbitMQ work in an "all or
42
43
nothing" fashion, i.e: if a routing key matches a set of queues bound
43
44
to the exchange, then RabbitMQ will route the message to all the
44
- queues in that set. Therefore for this plugin to work, we need to
45
+ queues in that set. For this plugin to work it is necessary to
45
46
route messages to an exchange that would partition messages, so they
46
- are routed to _ at most_ one queue.
47
+ are routed to _ at most_ one queue (a subset) .
47
48
48
- The plugin provides a new exchange type ` "x-modulus-hash" ` that will use
49
- the traditional hashing technique applying to partition messages
50
- across queues.
49
+ The plugin provides a new exchange type, ` "x-modulus-hash" ` , that will use
50
+ a hashing function to partition messages routed to a logical queue
51
+ across a number of regular queues (shards) .
51
52
52
53
The ` "x-modulus-hash" ` exchange will hash the routing key used to
53
54
publish the message and then it will apply a ` Hash mod N ` to pick the
54
55
queue where to route the message, where N is the number of queues
55
56
bound to the exchange. ** This exchange will completely ignore the
56
57
binding key used to bind the queue to the exchange** .
57
58
58
- You could also use other exchanges that have similar behaviour like
59
- the _ Consistent Hash Exchange_ or the _ Random Exchange_ . The first
60
- one has the advantage of shipping directly with RabbitMQ.
59
+ There are other exchanges with similar behaviour:
60
+ the _ Consistent Hash Exchange_ or the _ Random Exchange_ .
61
+ Those were designed with regular queues in mind, not this plugin, so ` "x-modulus-hash" `
62
+ is highly recommended.
63
+
64
+ If message partitioning is the only feature necessary and the automatic scaling
65
+ of the number of shards (covered below) is not needed or desired, consider using
66
+ [ Consistent Hash Exchange] ( https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange )
67
+ instead of this plugin.
68
+
69
+
70
+ ## Auto-scaling
71
+
72
+ One of the main properties of this plugin is that when a new node
73
+ is added to the RabbitMQ cluster, then the plugin will automatically create
74
+ more shards on the new node. Say there is a shard with 4 queues on
75
+ ` node a ` and ` node b ` just joined the cluster. The plugin will
76
+ automatically create 4 queues on ` node b ` and "join" them to the shard
77
+ partition. Already delivered messages _ will not_ be rebalanced but
78
+ newly arriving messages will be partitioned to the new queues.
61
79
62
- If _ just need message partitioning_ but not the automatic queue
63
- creation provided by this plugin, then you can just use the
64
- [ Consistent Hash Exchange] ( https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange ) .
65
80
66
81
## Consuming From a Sharded [ Pseudo-] Queue ##
67
82
@@ -109,14 +124,23 @@ this plugin.
109
124
For load balancers, the "least connections" strategy is more likely to produce an even distribution compared
110
125
to round robin and other strategies.
111
126
127
+ ### How Evenly Will Messages Be Distributed?
128
+
129
+ As with many data distribution approaches based on a hashing function,
130
+ even distribution between shards depends on the distribution (variability) of inputs,
131
+ that is, routing keys. In other words the larger the set of routing keys is,
132
+ the more even will message distribution between shareds be. If all messages had
133
+ the same routing key, they would all end up on the same shard.
134
+
135
+
112
136
113
137
## Installing ##
114
138
115
139
### RabbitMQ 3.6.0 or later
116
140
117
141
As of RabbitMQ ` 3.6.0 ` this plugin is included into the RabbitMQ distribution.
118
142
119
- Enable it with the following command :
143
+ Like any other [ RabbitMQ plugin ] ( http://www.rabbitmq.com/plugins.html ) it has to be enabled before it can be used :
120
144
121
145
``` bash
122
146
rabbitmq-plugins enable rabbitmq_sharding
0 commit comments