Revisit backoff grace period computation #188
Description
The current exponential backoff mechanism can lead to multiple Lambda environments being synchronously stuck, as they will all attempt to try to reconnect to a potentially unresponsive APM Server at the same time.
The following picture is a bar chart illustrating the duration of several invocations over 10 minutes, coupled with a slow APM Server. There is a very visible synchronous pattern of multiple Lambda environments stuck trying to query the APM Server at the same time, leading to the creation of new environments and increasing the cold start rate.
Immediate mitigation actions to be attempted are:
- Increase the initial grace period, to avoid the creation of a lot of Lambda environments at the start
- Increase the jitter to make the grace period duration more random, and assign a specific jitter value to each instance of the extension at cold start
- Perform a high-throughput benchmark over a long period of time to monitor the environment scaling.