Dynamic period table capacity adjustment

[awh]

During the period from midnight to midnight + max chunk age, newly matured chunks require 2x the number of index entries to be written to the period tables because their contents span two day buckets. Provisioned capacity must be sized to cope with this nightly spike or the series flush queues start to back up:

However, doing so means that capacity is over-provisioned for the noon to midnight period. The table manager can be enhanced to step the capacity up automatically before midnight, and back down again after noon.

[tomwilkie]

Dupe of #318? @leth added metrics on per-tables used capacity in #460, which is the first step in dynamically adjusting provisioning.

[leth]

The scope of this could be a bit smaller; just focused on the midnight rollover, rather than general capacity?

[tomwilkie]

Thats up to the person who implements it I guess! I personally don't see how just focusing on the time around midnight make is much easier.

[awh]

I suspect the solution for #318 will be far more complex than adding if time > midnight & time < noon activeCapacity *= 2 to the table manager...

[bboreham]

Do we still need this issue now autoscaling is supported?

Edit: AWS autoscaling does nothing at all when you're not using the table, so we currently still have the problem that the table is over-provisioned from when it is created until when we really need it.

[bboreham]

I think the new metrics-based scaling in table-manager is coping ok with this, so closing this issue.