Responsible Memory Usage #562
Description
News: https://hackmd.io/pvj8fsC3QbSc7o54ZH4qJQ
There are memory leaks in the codebase. Not only do we need to fix them, we need a proper attitude to ensure that they do not happen again.
Thus we need to achieve three things:
-
A solid and well-documented approach to heap tracing.
-
To rewrite our optional allocations under an attitude of zero-cost optimisations and, in general, ensure that all dynamic allocations are bounded.
-
To integrate all dynamic allocations into our triad of reporting apparatus: prometheus, telemetry and informant.
Heap tracing
It should not take days of research to figure out how to determine what lines in the codebase are resulting in so much memory being allocated. We need a clear, well-maintained document on how to setup and run Substrate (and derivative clients like Polkadot) in order to get full traces of allocations.
There are several tools that may be used (heaptrack, jemalloc, Massif/Valgrind, ...) and it is not clear which are viable, and/or sensible.
Zero-cost optimisations
This is principle a little like the "pay for what you use" mantra of Zero-Cost Abstractions that Rust gives. Essentially, it should be possible to "turn off" or reduce non-mandatory memory allocations to a large degree. This would come at a potentially large performance cost, but we will assume for now that under some circumstances this is acceptable, and in any case, having the ability to tune how much memory is allowed to be allocated in each of the various dynamic subsystems is useful.
We need to introduce CLI options such that, at the extreme end, a user (or, developer) can minimise expected memory usage. In many cases, this memory usage (given by buffer sizes, cache limits, queue capacities and pre-allocation values) is hard-coded. In other cases, it's unbounded. It should, at all times, be bounded whose value is given by the user to be configured over the CLI.
It should be configurable for all dynamically allocating subsystems. Every buffer, queue and cache. Every time something it allocated and kept around for later use. Every allocation that isn't strictly required for normal operation. If a buffer or queue is automatically grown under some circumstance, it should be instantly shrunk again once the need is gone.
One example would be the 1GB that is given to rocks db's cache size - it should be possible to lower this, all the way to zero, ideally. There are, I'm sure, many examples.
Items that allocate large amounts but that are used transiently (such the memory slab for wasm execution) should be configurable so that they remain allocated only as long as they are being actively used. This will mean that every execution will need to allocate afresh, but we don't care.
The end result of this effort is to be able to lower the expected amount of memory used in stable-state (i.e. when not importing or authoring) to around 250 MB, even with dozens of peers.
Footprint reporting
In addition to this, every dynamically allocated queue, buffer and cache should be continuously reporting its size and footprint out to telemetry, informant and/or prometheus. We should obviously begin with the lower-hanging fruit - those that we suspect of being bad behavers, but eventually we should have an attitude of reporting everything.
We probably want to do this only in a certain mode of operation, to avoid performance costs under normal operation.
Related: paritytech/substrate#4679