Nodes stats API CPU usage value is -1

[yskim8830]

Elasticsearch Version

7.10.2-oss

Installed Plugins

No response

Java Version

15.0.1

OS Version

Ubuntu 22.04.1 LTS

Problem Description

Hello, I have a question.
I have installed elasticsearch on azure. OS is Ubuntu.
And look up the cpu usage.
I ran '_cat/nodes?format=json' , and the cpu value is -1.
can not understand.
I browsed the documentation (https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html)

(integer) Recent CPU usage for the whole system, or -1 if not supported.

In what cases is this not supported?

Steps to Reproduce

GET _cat/nodes?format=json

Logs (if relevant)

No response

[DaveCTurner]

Thanks very much for your interest in Elasticsearch.

This appears to be a user question, and we'd like to direct these kinds of things to the Elasticsearch forum. If you can stop by there, we'd appreciate it. This allows us to use GitHub for verified bug reports, feature requests, and pull requests. Moreover your question relates to a version that is no longer supported.

There's an active community in the forum that should be able to help get an answer to your question. As such, I hope you don't mind that I close this.

[BBQigniter]

I think this issue can be reopened @DaveCTurner

We currently see the same issue on our ECK-managed ELK-cluster version 8.4.1 (on our test-system which is setup similarly it's not happening)

All our Elasticsearch instances show cpu -1 if I call for example GET _cat/nodes?v in Kibana.

The problem suddenly seems to have started at about midnight yesterday 06.12.2022

A node/pod restart didn't help and I'll check if I can find some hint why this suddenly happens

[BBQigniter]

hmm, I wasn't able to find anything obvious on the nodes or in the Elasticsearch node/pod logs.

could it be that this bug has something to with #90319 even though I was not able to find some similar log-message?

[DaveCTurner]

The CPU value comes pretty much directly from the JDK's OperatingSystemMXBean#getCpuLoad() method which AIUI ultimately reads information from /proc/stat (or the containers equivalent) and returns -1 on a failure. Unfortunately the JDK doesn't give us any information about the nature of the failure. However, given that nothing in Elasticsearch changed at this time, and restarting Elasticsearch doesn't help, it sounds like something has changed in your environment instead.

Does the problem persist if you restart the whole host rather than just the affected pods?

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[BBQigniter]

nothing really changed OS-wise afaik

Does the problem persist if you restart the whole host rather than just the affected pods?

A reboot of the host helps - just tried it and CPU stats are showing now for the nodes/pods running on that rebooted node

maybe this can help for debugging - got some of those proc-values, they are each from one pod and their sub processes related to the pod

# cat proc/[pid]/stat from one pod where we get cpu usage -1

221607 (containerd-shim) S 1 221607 2658 0 -1 1077944320 134168 9059653 0 0 948 703 3497 3476 20 0 13 0 415066449 729735168 3729 18446744073709551615 4194304 8722262 140735344045440 140735344044952 4610275 0 1002055680 0 2143420159 18446744073709551615 0 0 17 2 0 0 0 0 0 13582336 13764896 21200896 140735344049733 140735344049894 140735344049894 140735344050135 0

221628 (pause) S 221607 221628 221628 0 -1 4202752 821 0 0 0 0 1 0 0 20 0 1 0 415066455 978944 1 18446744073709551615 4194304 4703985 140727784534480 140727784533688 4442753 0 0 0 81922 18446744071996920076 0 0 17 46 0 0 0 0 0 4856992 4879056 23351296 140727784538017 140727784538024 140727784538024 140727784538097 0

222448 (tini) S 221607 222448 222448 0 -1 1077944576 952 0 0 0 3 9 0 0 20 0 1 0 415067883 2551808 96 18446744073709551615 4194304 4210500 140725988866032 140725988864960 140596471157039 0 0 3145728 0 18446744071996916454 0 0 17 39 0 0 0 0 0 6311440 6313355 34439168 140725988870946 140725988871005 140725988871005 140725988880366 0

222483 (java) S 222448 222483 222448 0 -1 1077944576 27450 18085 0 0 356 131 7 7 20 0 16 0 415067893 3059519488 21015 18446744073709551615 94762231468032 94762231471784 140721030445552 140721030427984 140580531637463 0 0 0 16800975 18446744073709551615 0 0 17 39 0 0 0 0 0 94762231479544 94762231480344 94762245988352 140721030449501 140721030449916 140721030449916 140721030459346 0

222679 (java) S 222483 222483 222448 0 -1 1077944320 11394808 685 8074 0 76264 12144 0 0 20 0 314 0 415067986 1523783335936 3129388 18446744073709551615 94862709989376 94862709993128 140737155636880 140737155619312 140223925181655 0 4 0 16800975 18446744073709551615 0 0 17 34 0 0 0 0 0 94862710000888 94862710001688 94862721220608 140737155647036 140737155648199 140737155648199 140737155657682 0

222755 (controller) S 222679 222483 222448 0 -1 4202752 1256 0 0 0 0 0 0 0 20 0 4 0 415068103 110759936 1043 18446744073709551615 94805063438336 94805063535373 140731503384352 140731503381536 140435283203020 0 4 4096 268469480 18446744073709551615 0 0 17 42 0 0 0 0 0 94805063555032 94805063558096 94805087289344 140731503390506 140731503390586 140731503390586 140731503390632 0

222530 (tini) S 221607 222530 222530 0 -1 1077944576 959 0 0 0 2 9 0 0 20 0 1 0 415067907 2551808 96 18446744073709551615 4194304 4210500 140732777364736 140732777363664 140102443786543 0 0 3145728 0 18446744071996916454 0 0 17 1 0 0 0 0 0 6311440 6313355 33427456 140732777368300 140732777368373 140732777368373 140732777377770 0

222544 (filebeat) S 222530 222544 222530 0 -1 1077944576 45259 0 0 0 1254 379 0 0 20 0 39 0 415067912 3832725504 35593 18446744073709551615 94757982494720 94758100303164 140722729481664 140722729480720 94758027300067 0 0 0 2143420159 18446744073709551615 0 0 17 13 0 0 0 0 0 94758102403592 94758144445952 94758159704064 140722729487078 140722729487110 140722729487110 140722729496539 0

and from the just rebooted node

# cat proc/[pid]/stat from one pod where we get cpu usage is reported correctly after a reboot

8801 (containerd-shim) S 1 8801 2641 0 -1 1077944320 41210 2982665 0 0 405 284 1638 1267 20 0 12 0 15453 729735168 3165 18446744073709551615 4194304 8722262 140724682261376 140724682260888 4610275 0 1002055680 0 2143420159 18446744073709551615 0 0 17 28 0 0 0 0 0 13582336 13764896 19427328 140724682264133 140724682264294 140724682264294 140724682264535 0

8822 (pause) S 8801 8822 8822 0 -1 4202752 819 0 0 0 0 1 0 0 20 0 1 0 15458 978944 1 18446744073709551615 4194304 4703985 140735669901120 140735669900328 4442753 0 0 0 81922 18446744072487653644 0 0 17 26 0 0 0 0 0 4856992 4879056 31166464 140735669907361 140735669907368 140735669907368 140735669907441 0

10520 (tini) S 8801 10520 10520 0 -1 1077944576 952 0 0 0 1 3 0 0 20 0 1 0 17015 2551808 97 18446744073709551615 4194304 4210500 140733505775856 140733505774784 140517726753071 0 0 3145728 0 18446744072487650022 0 0 17 5 0 0 0 0 0 6311440 6313355 13033472 140733505784612 140733505784671 140733505784671 140733505794030 0

10534 (java) S 10520 10534 10520 0 -1 1077944576 25320 18186 0 0 262 54 7 11 20 0 16 0 17025 3059519488 21166 18446744073709551615 94519970144256 94519970148008 140734133156288 140734133138720 139681108339927 0 0 0 16800975 18446744073709551615 0 0 17 35 0 0 0 0 0 94519970155768 94519970156568 94519982010368 140734133160287 140734133160702 140734133160702 140734133170130 0

10676 (java) S 10534 10534 10520 0 -1 1077944320 6371590 686 126770 0 200464 16888 0 0 20 0 337 0 17113 178659020800 9843254 18446744073709551615 94380764372992 94380764376744 140735627557456 140735627539888 139814241893591 0 4 0 16800975 18446744073709551615 0 0 17 40 0 0 0 0 0 94380764384504 94380764385304 94380771102720 140735627564603 140735627565769 140735627565769 140735627575250 0

10815 (controller) S 10676 10534 10520 0 -1 4202752 1256 0 0 0 0 0 0 0 20 0 4 0 17261 110759936 1043 18446744073709551615 94020444704768 94020444801805 140731775959456 140731775956640 140643054023628 0 4 4096 268469480 18446744073709551615 0 0 17 35 0 0 0 0 0 94020444821464 94020444824528 94020473704448 140731775962922 140731775963002 140731775963002 140731775963048 0

10587 (tini) S 8801 10587 10587 0 -1 1077944576 959 0 0 0 2 4 0 0 20 0 1 0 17067 2551808 96 18446744073709551615 4194304 4210500 140722992578272 140722992577200 140252072542511 0 0 3145728 0 18446744072487650022 0 0 17 15 0 0 0 0 0 6311440 6313355 31371264 140722992581357 140722992581430 140722992581430 140722992590826 0

10602 (filebeat) S 10587 10602 10587 0 -1 1077944576 37699 0 1 0 993 246 0 0 20 0 38 0 17073 3755061248 35088 18446744073709551615 94408615620608 94408733429052 140724300903728 140724300902784 94408660425955 0 0 0 2143420159 18446744073709551615 0 0 17 11 0 0 0 0 0 94408735529480 94408777571840 94408797970432 140724300913383 140724300913415 140724300913415 140724300922843 0

As we use also prometheus to get metrics of the pods etc. I quickly checked them too - there we do not see any strange CPU usage values.

[DaveCTurner]

I think it uses /proc/stat, not /proc/[pid]/stat. Can you share this file from a bad node?

[BBQigniter]

sorry, not possible anymore :( as I already have rebooted all affected nodes.