Handling SIGTERM for Distributed and Standalone Mode

[AdheipSingh]

• Handling of SIGTERM. This is needed for readiness probe to work, so whenever kubelet sends a SIGTERM to pod, readiness handler response unavailable, this way service will send no traffic to this pod.

• This PR also makes sure actix drains HTTP connection before exiting.

how to stop traffic. The current issue is service is still sending traffic to terminating pods. We added pre-stop hooks to sleep for kube proxy and core dns cache to clear, which worked at times, but failed too. So now, we want to make sure connections are drained.
Handling SIGTERM
1. Kubelet sends SIGTERM to pod when we issue delete pod
2. Actix hijacks the call ( thread is running ) and tweaks a global flag SIGNAL_RECEIVED to true.
3.Readiness Probe handler reads the flag each time, it is called.
4. on pod readiness probe is set, kubelet will call probe every 5 seconds.
5. Readiness Probe will return unavailable, and pod will go 0/1 unavailable.
6. Service at this point will route to other available pods.
7. After 120 seconds pod will terminate, during that period ingestor will s3 sync ( TODO: once this is working on k8s i will add in SIGTERM handling itself )

Description

---

This PR has:

• been tested to ensure log ingestion and log query works.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added documentation for new or modified features or behaviors.

[AdheipSingh]

• I need to build the image from Dockerfile and test in cluster once again. I have been testing using Dockerfile.debug image.
  This will also need changes in existing helm charts. To add readiness probe.
  Please suggest if you want to add in same PR or new one.
  cc @nitisht @nikhilsinhaparseable

[nitisht]

Please suggest if you want to add in same PR or new one.

Let's add in the same PR

[AdheipSingh]

@nitisht any specific reason for adding shell. Is it because image is distroless, i have been using Dockerfile.debug, so just want to confirm ?

[nitisht]

Yes, because distroless doesn't have any other shell.

[AdheipSingh]

Hmm, but do we need a shell ? I mean wasn't shell for ${hostname}, container and app PID should be same.
I'lll check once if needed. If you see CMD will trigger the command itself.

[nitisht]

Sure, we can remove in that case

[nitisht]

Yes, because distroless doesn't have any other shell.

[AdheipSingh]

Please suggest if you want to add in same PR or new one.

Let's add in the same PR

Done

[AdheipSingh]

TODO: to be tested with Dockerfile changes in cluster.

[AdheipSingh]

Dockefile changes are testing. No shell copy needed.