Test and handle case when APM server doesn't respond

[estolfo]

We need to handle the case when the APM server doesn't respond to extension requests. The extension should never hold up the runtime so we'll opt for dropping the APM data that failed to send instead of retrying.

See issue #130 for a possible related example

• Handling case when channel is full
• Exponential backoff when sending to APM server, so that it's not hitting the same timeout repeatedly

[felixbarny]

When the queue is full, we should still send a success status code to the agents, even if we end up dropping the data. Otherwise, the agent will do the exponential backoff and start buffering events in its internal queue. This can lead to the agent being unable to enqueue and process flush events, therefore blocking the lambda.

[jlvoiseux]

Work towards closing this issue has been done in c7384e0. Modifications include:

• Addition of a Timeout config option for the HTTP client performing APM queries (see issue):
  https://github.com/jlvoiseux/apm-aws-lambda/blob/c7384e08f52c3aa0c8612c2bc98b9c5f6ebc1c00/apm-lambda-extension/extension/process_env.go#L68-L72
  https://github.com/jlvoiseux/apm-aws-lambda/blob/c7384e08f52c3aa0c8612c2bc98b9c5f6ebc1c00/apm-lambda-extension/main.go#L71-L74

• Drop events when the channel is full:
  https://github.com/jlvoiseux/apm-aws-lambda/blob/c7384e08f52c3aa0c8612c2bc98b9c5f6ebc1c00/apm-lambda-extension/extension/route_handlers.go#L102-L107

[jlvoiseux]

Following discussions with @felixbarny and @estolfo, we decided to implement the following backoff mechanism in the extension:

• Upon detection of a sending error in extension.PostToAPMServer() (used in both background and syncflush strategies) we enter a backoff state.
• Upon backoff state entry, we start to wait for an amount of time specified in the Transport specification. During this time interval, nothing is sent to the APM server, either in background or through the FlushAPMData() function enabled by the syncflush strategy.
• When the backoff time interval is elapsed, we try again to send data (either through the background goroutine or the flush function, whichever comes first). If another failure is detected, the reconnectCount counter is incremented, and we remain in backoff state, but this time with a larger time interval. If data is successfully sent to the APM Server, the counter is reset and we go out of backoff mode.
• The backoff state, the associated timer and the reconnection counter are tracked at the global extension level so that both the background data sending goroutine and the flush function are affected by this strategy in a consistent fashion.

[jlvoiseux]

Closed after merging #148