Unable to see container's metrics about external volumes attached (k8s persistent volumes)

[eedugon]

Hello,

We are having a discussion at "prometheus-operator" project, and based on @brancz suggestion, we are raising the topic here, as our problem seems related with cadvisor in some way.
(this is the original thread, just in case: #prometheus-operator/prometheus-operator#485)

In a kubernetes + prometheus-operator environment we are not able to get any kind of metric about persistent volumes that are attached to containers in pods, and we don't really understand why.

Keeping out of the topic that Kubernetes should probably provide information about them, because PVs are actually K8s resources, we are wondering how we can't find any information at cadvisor/kubelet level metrics.

For example, a container might show the following disks (by df -h directly in a bash of the container):

# df -h
overlay 154G 12G 136G 9% /
/dev/xvdbg       20G   46M   19G   1% /var/lib/zookeeper (THIS IS THE PERSISTENT VOLUME)
/dev/xvda1 154G 7.2G 140G 5% /etc/hosts
...
...

In that output, the "154G" device is the main disk of the container owner (k8s node), and we have no problem finding metrics about that disk at all levels.
But the other device (/dev/xvdbg) is actually the persistent volume that is mounted in the container, and there's no track of it at all in the metrics.

container_fs_usage_bytes information about that pod/container has only:
container_fs_usage_bytes{container_name="kafka",device="/dev/xvda1",.....}
Which represents actually the physical disk of the k8s node, owning the container.
But there's noting else about /dev/xvdbg, and that's what we are looking for...

Do you have any suggestion, idea or explanation for this behavior? Are we missing something at configuration level to make kubelet reporting this?

Note1: At node level, lsblk shows the disk of the PV:

admin@node$ lsblk 
NAME    MAJ:MIN   RM  SIZE RO TYPE MOUNTPOINT
xvda    202:0      0  164G  0 disk 
`-xvda1 202:1      0  164G  0 part /
xvdbg   202:14848  0   20G  0 disk /var/lib/kubelet/pods/ae6c1d55-6d7a-11e7-bc61-068b6da15381/volumes/kubernetes.io~aws-ebs

# and it's also mounted
/dev/xvdbg       20G   46M   19G   1% /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2a/vol-0bafa0be2b20de87a

In the node we have cadvisor/kubelet metrics and also node_exporter.... none of them are reporting what we are looking for. I suspect we are missing some kind of mapping or configuration somewhere, because the disk is there...

Thanks very much in advance, any help will be appreciated,
Eduardo
(+ @hartmut-pq)

[dashpole]

The current scope of cAdvisor is restricted to monitoring containers. It provides the kubelet with container metrics only, and has no concept of volumes, pods, or any other higher-level Kubernetes API objects. The kubelet has a built-in cadvisor, which monitors containers in kubernetes. The kubelet currently takes the container metrics provided by cAdvisor, and combines them with volume metrics, and kubernetes-specific metadata (e.g. container->pod mappings) to produce the summary API, which is exposed by the kubelet at 10255/stats/summary.

The underlying issue you are raising here is that the kubelet does not expose this via prometheus.

[hartmut-pq]

Hi @dashpole so you're saying all relevant information is available / provided by cAdvisor already but not mapped/interpreted by the kubelet->summary implementation yet?

[dashpole]

No, cAdvisor does not provide volume metrics, and has no concept of volumes or pods, which is why there are no volume metrics exposed via prometheus.
The kubelet itself collects and exposes volume metrics via the summary API, but doesnt expose it in any other way (e.g. prometheus)

[brancz]

Interesting, thanks for the insight @dashpole. Seems like we should maybe work towards exposing the stats API as Prometheus metrics by the kubelet.

[dashpole]

Would just adding volume metrics to the kubelet's prometheus endpoint be sufficient?

[eedugon]

Hi @dashpole ,

Yes, the information we are looking for is available in :10255/stats/summary, and as you said, that's not translated to any kind of metric (that was actually the issue to me).

I don't want to ask for anything out of the scope for any component, that's the reason my question is more or less.... "who do you think should be responsible of translating that into a prometheus metric"?

Your proposal "just adding volume metrics to the kubelet's prometheus endpoint" is exactly what I was looking for, but i don't know if by kubelet or whom (controller-manager?).

I also fully agree that cadvisor should know nothing about k8s concepts like persistent volumes, but from container perspective, if a docker has a filesystem mounted, I expect prometheus metrics to be exported about that filesystem, which is actually very important to track.

Let me know your view and thanks in advance!
Edu

[hartmut-pq]

@brancz so would this be escalated with kubernetes/kubernetes then?
May this be closely related to the works/proposal for 1.8? (prometheus-operator/prometheus-operator#485 (comment))
Who's going to push it to the next round then? :-)

[brancz]

@gnufied told me there is something already in the works, maybe he can comment on where we should go next.

[gnufied]

@brancz yes I was talking about this proposal - kubernetes/community#855 cc @jingxu97

[brancz]

@gnufied I don't want to hijack that PR so I'll ask here. Do you think it's reasonable to also add all of those metrics as metrics exposed in the /metrics endpoint of the kubelet?

[dashpole]

I think it is reasonable. We should also have a discussion on the relationship between kubernetes and prometheus. Seems odd to have some metrics on the cadvisor port, and others on the kubelet port. I think ideally, the prometheus endpoint should mirror the information provided by the kubelet's http endpoints (e.g summary API).

[dashpole]

I guess my biggest concern/question is that I don't know how prometheus deals with metrics changing. What about a metric (name, format, labels) can change across a release without causing disruption? Prometheus doesnt look like it has versioning.

[hartmut-pq]

anyone pls correct my if I'm wrong - but @dashpole prometheus merely reads what's there - and it's entirely up to the prometheus consumers do deal with changed labels/data... The major usage basically are prometheus alert rules and grafana to visualise based on the metrics.

[hartmut-pq]

@dashpole the proposal you mentioned will simply expose the storage metrics - and seem to aim to map/provide PVCReference.
Most benefit in terms of monitoring - if you have a quick look at the original comment/issue here prometheus-operator/prometheus-operator#485 (comment) would be to be allowed to alert if PVs are running out of storage...