Kubernetes readiness probe endpoint returning 404

[chadlwilson]

There appears to be some change in behaviour for the Kubernetes-oriented readiness group endpoint on 2.3.2 compared to 2.3.1.

For a service that has no external dependencies (and only readinessState in the health group), the /actuator/health/readiness endpoint is returning a 404.

Configuration we are using:

management.server.port=9083
management.health.probes.enabled=true
management.endpoints.enabled-by-default=false
management.endpoint.info.enabled=true
management.endpoint.health.enabled=true
management.endpoint.health.show-details=always
management.endpoint.health.group.liveness.include=livenessState,diskSpace,refreshScope
management.endpoint.health.group.readiness.include=readinessState
management.endpoint.health.group.liveness.show-details=always
management.endpoint.health.group.readiness.show-details=always
management.endpoints.web.exposure.include=health

Expected Behaviour
We expect this to just return 200 with { "status": "UP" }

Actual Behaviour

$ http http://localhost:9083/actuator/health/readiness
HTTP/1.1 404 Not Found

Full health call:

$ http http://localhost:9083/actuator/health
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/json
Date: Sat, 25 Jul 2020 06:27:55 GMT
Transfer-Encoding: chunked

{
    "components": {
        "discoveryComposite": {
            "components": {
                "discoveryClient": {
                    "description": "Discovery Client not initialized",
                    "status": "UNKNOWN"
                }
            },
            "description": "Discovery Client not initialized",
            "status": "UNKNOWN"
        },
        "diskSpace": {
            "details": {
                "exists": true,
                "free": 287311962112,
                "threshold": 10485760,
                "total": 499963174912
            },
            "status": "UP"
        },
        "livenessStateProbeIndicator": {
            "status": "UP"
        },
        "ping": {
            "status": "UP"
        },
        "reactiveDiscoveryClients": {
            "components": {
                "Simple Reactive Discovery Client": {
                    "description": "Discovery Client not initialized",
                    "status": "UNKNOWN"
                }
            },
            "description": "Discovery Client not initialized",
            "status": "UNKNOWN"
        },
        "readinessStateProbeIndicator": {
            "status": "UP"
        },
        "refreshScope": {
            "status": "UP"
        }
    },
    "groups": [
        "liveness",
        "readiness"
    ],
    "status": "UP"
}

This may relate to #22107.

[chadlwilson]

After a bit more digging, I'm not really sure why or whether it was intended, however the issue seems to be that readinessState has become readinessStateProbeIndicator (and same for livenessState) so the old configuration was not correctly including the indicator at all, leaving the readiness group empty.

This seems to work as expected.

management.endpoint.health.group.liveness.include=livenessStateProbeIndicator,diskSpace,refreshScope
management.endpoint.health.group.readiness.include=readinessStateProbeIndicator

[bclozel]

Yes this is an unintended side effect of #22107. The workaround you're mentioning is the right one in the meantime.

Thanks for raising this issue!

[chadlwilson]

No problem - feel free to re-title it as appropriate.

Unfortunately this is a transparently breaking change for many people, they probably won't realise the probe status isn't being included in the status in addition to, say, db, redis etc because including a non-existent indicator in a group doesn't seem to fail startup :(

[bclozel]

I've tagged this issue as a regression.

I'm really sorry for letting in that one.

[OrangeDog]

Does this cover the fact that they are listed under groups at /health, but then don't actually exist?