Not all /sys/block devices will have a "dev" file

[mattysweeps]

I see the following error message in kubelet logs:

Jun 00 00:00:00 my-worker-node kubelet[XXXXXXX]: E0000 00:00:00.000000 2571634 info.go:99] Failed to get disk map: open /sys/block/nvme0c0n1/dev: no such file or directory

This error message matches cadvisor here:

cadvisor/machine/info.go

Line 104 in b7e6727

klog.Errorf("Failed to get disk map: %v", err)

Looking for usages of /dev this is the culprit:

cadvisor/utils/sysfs/sysfs.go

Line 208 in b7e6727

dev, err := os.ReadFile(path.Join(blockDir, name, "/dev"))

Full Stack:

• cadvisor/utils/sysfs/sysfs.go

  Line 208 in b7e6727

  dev, err := os.ReadFile(path.Join(blockDir, name, "/dev"))

• cadvisor/utils/sysinfo/sysinfo.go

  Line 63 in b7e6727

  dev, err := sysfs.GetBlockDeviceNumbers(name)

• cadvisor/machine/info.go

  Line 104 in b7e6727

  klog.Errorf("Failed to get disk map: %v", err)

According to linux kernel source: the device name scheme means the device is in subsystem 0, controller 0, namespace 1
https://elixir.bootlin.com/linux/latest/source/drivers/nvme/host/core.c#L4361

This is a device for internal kernel usage only, and it's flagged as such with GENHD_FL_HIDDEN.

Therefore, it's probably wrong that cadvisor is trying to read this device. It should probably be ignored with a similar rule as here:

cadvisor/utils/sysinfo/sysinfo.go

Line 57 in b7e6727

if strings.HasPrefix(name, "loop") || strings.HasPrefix(name, "ram") || strings.HasPrefix(name, "sr") {

I'm not an NVMe expert. I've been told that this type of device might only be present when multipathing is enabled.
I'm also not sure what machine/info.go is doing with the information it gets from the block devices, so I'm not sure the impact here, other than there's a lot of errors in my kubelet log output.

Note that for _, disk := range disks { the err is return directly from the disk range, meaning other disks might not be gathered or inspected.

[matthewei]

Hi, @mattysweeps I meet the same issue, is there any progress? could you share me some info?

[mattysweeps]

Hi @matthewei

I'm not currently investigating a fix. The code change is simple, but I'm not sure how to properly test it.

[rogerwegmanneit]

I got this error message by using mayastor 2.4 with microk8s. And you?

[ringerc]

It looks like this was fixed by 88bfa0d ~2y ago in #3260

A subsequent fix #3522 solved a problem where the device was not always correctly mapped; see #3522 (comment) . This could lead to device="" outputs, colliding metrics, and wild results for metrics.

But there's also an underlying bug in how cadvisor exposes these metrics that means this might be trigger-able in other ways, possibly including when exposing metrics on newly hot-plugged devices. See #3693