Ecosystem: Support liveness and readiness probes

[ja20222]

As a user of NKG
I want NKG to have liveness and readiness probes
So that I can leverage better automation with NKG utilizing the knowledge that a particular NKG pod is ready to serve traffic.

Acceptance

• The NKG pod is only labeled as "ready" when nginx is ready to serve traffic.
  • Stale configuration should not prevent the probe from reporting a "ready" status.
• Liveness and/or readiness probes are exposed for both the nginx data container and NKG controller container as needed.
• A manual test is performed to verify that our upgrade method minimizes downtime using the probes appropriately.

[mpstefan]

Blocked by #664

[brianehlert]

Bringing forward from conversation: configuration change should not cause ready status to 'flap' as that bypasses how NGINX works with new workers and bleeding old sessions. Ready status based on configuration also sets the potential case of having a greater traffic impact and giving a poor user experience.

[sjberman]

Current approach that I'm going to take is a single readiness probe on the controller that returns true if it has written out its initial nginx config (or has nothing to do on startup).

For the future of when we separate our control plane and data plane, this approach will have to change. We'll have to have probes on both the control plane and data plane. The control plane probe will only be concerned with the control plane being ready, and the data plane probe will be concerned with nginx being ready. We can determine what that looks like at that point.

The reason I'm skipping the more robust solution right now is that it will take a bit of refactoring in our code, and may end up changing when we separate things anyway.

@pleshakov Thoughts on this?