Force smooth shard allocation

[nik9000]

So I have one index that will get tons more traffic then all the others. For that reason its pretty important that its shards get spread out reasonably evenly even in the face of things like rolling restarts, failed nodes, and the like. I'm pretty sure the distribution is reasonably random right now. Could Elasticsearch bias towards smoothly distributing the shards?

This came up because I started load testing Elasticsearch and saw this:

That is, elastic1003 isn't seeing any load at all and elastic105 and elastic1011 aren't shouldering their fair share.

On digging, I got this:

manybubbles@elastic1008:~$ curl -s 'localhost:9200/_cat/shards/enwiki_content_1395837666' | perl -pe 's/.*(elastic.*)/\1/' | sort | uniq -c 
      3 elastic1001 
      2 elastic1002 
      2 elastic1004 
      1 elastic1005 
      3 elastic1006 
      3 elastic1007 
      4 elastic1008 
      3 elastic1009 
      2 elastic1010 
      1 elastic1011 
      4 elastic1012 
      3 elastic1013 
      3 elastic1014 
      3 elastic1015 
      3 elastic1016 

Which shows that elastic1003 doesn't have any of the shards and elastic1005 and elastic1011 only have one each.

Annoying work around: I can use the allocation api to shift shards from the nodes with more to the nodes with fewer. It'd be much better if it were automatic.

[s1monw]

ok so if I understand you correctly you have an index that should be better balanced but other indices that are less "hot" prevent that from happening due to overall balance, is that what you are saying? did you look at the balancer options the index one could help you if you wanna raise it slowly you should see some re-balancing happening?

[s1monw]

thanks so much for improving the docs on this! @nik9000 out of curiosity did it help you to raise the settings? and if so can you share some of this experience?

[nik9000]

Sure! I slowly raised the index balance relative to the others until I got something that caused the shards on this index to balance out. This is where I ended up:

          "balance": {
            "shard": 0.195,
            "index": 0.8,
            "primary": 0.005
          }

Which balanced the shards out pretty well:

manybubbles@elastic1001:~$ curl -s 'localhost:9200/_cat/shards/enwiki_content_1395837666' | perl -pe 's/.*(elastic.*)/\1/' | sort | uniq -c
      3 elastic1001 
      2 elastic1002 
      2 elastic1003 
      2 elastic1004 
      2 elastic1005 
      2 elastic1006 
      3 elastic1007 
      3 elastic1008 
      3 elastic1009 
      2 elastic1010 
      2 elastic1011 
      3 elastic1012 
      3 elastic1013 
      3 elastic1014 
      2 elastic1015 
      3 elastic1016 

And the cluster is still relatively balanced any way:

manybubbles@elastic1001:~$ curl -s 'localhost:9200/_cat/shards' | perl -pe 's/.*(elastic.*)/\1/' | sort | uniq -c
    349 elastic1001 
    348 elastic1002 
    347 elastic1003 
    353 elastic1004 
    352 elastic1005 
    349 elastic1006 
    351 elastic1007 
    353 elastic1008 
    348 elastic1009 
    350 elastic1010 
    352 elastic1011 
    353 elastic1012 
    350 elastic1013 
    350 elastic1014 
    354 elastic1015 
    352 elastic1016 

I think I saw it balance my index's shards, then do the same for some other indexes.

I'm happy about how it turned out, especially because we're about to do another rolling restart on those machine to repartition them.

When I reran the performance test the picture was much prettier:

I still have more work to do to lower those but they are much better now that they are distributed.

[s1monw]

thanks for the insight nik! I am happy that the parameters worked for you. I think in the future we somehow need to take the "weight" of an index into account to make this less manual.

[nik9000]

I sent a half baked pr a while ago using a few decays to start automatically calculating weight. I'm sure it wasn't right but might have useful ideas. It'd be useful in, say, a log archiving scenario where you normally search and write to "current" indexes.

Even with something like that, you'd still want to let users override whatever it comes up with with because they might have some sense of future changes.

Even with a

Sent from my iPhone

On Apr 10, 2014, at 5:08 AM, Simon Willnauer notifications@github.com wrote:

thanks for the insight nik! I am happy that the parameters worked for you. I think in the future we somehow need to take the "weight" of an index into account to make this less manual.

—
Reply to this email directly or view it on GitHub.