How to handle broker decommissioning and partition rebalance?

[kchodnicki]

Typically it's a troublesome thing to do, how to handle it inside k8s?

[solsson]

I did a concept for something like that in #95. Do you think it's a useful approach? If so please close this ticket and comment there.

[kchodnicki]

@solsson ok, but how do you actually automate it? Each time a broker is created I want to do rebalancing. Same for broker graceful shutdown.

[solsson]

I guess you could listen to events such as those from #92 and execute jobs using the kube API. But you'll probably want some grace period, for example at graceful shutdown followed by restart.

How often will you scale up and down? Often enough to motivate the maintenance of such automation? Are you looking for something like HPA? I'm not sure StatefulSet supports that. If you have some custom automation in place to trigger scaling, maybe you can trigger rebalance from there too?

[kchodnicki]

I think I will have to provide API for scaling this, and rebalance on API call.
Thanks!