KEP-3008: QoS-class resources #3004

marquiz · 2021-10-07T14:14:49Z

One-line PR description:
New KEP for adding class-based resources to CRI protocol

Issue link: QoS-class resources #3008

Other comments:

k8s-ci-robot · 2021-10-07T14:14:59Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

marquiz · 2021-10-07T14:15:19Z

/cc @kad @haircommander

kikisdeliveryservice · 2021-10-08T21:24:23Z

Hi @marquiz

All KEP PRs must have an open issue in k/enhancements (this repo). Please open an issue and fill it out completely and rename this PR to KEP-issue number but in the title of this PR and in your README.md

Thanks!

marquiz · 2021-10-11T06:49:38Z

All KEP PRs must have an open issue in k/enhancements (this repo). Please open an issue and fill it out completely and rename this PR to KEP-issue number but in the title of this PR and in your README.md

Thanks for the guidance @kikisdeliveryservice. Done

k8s-triage-robot · 2022-01-17T12:46:28Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

marquiz · 2022-02-01T09:59:53Z

/retitle KEP-3008: Class-based resources

Addressing review feedback from thocking.

k8s-ci-robot · 2024-02-01T19:04:19Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: marquiz
Once this PR has been reviewed and has the lgtm label, please assign johnbelamaric for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

keps/prod-readiness/OWNERS
keps/sig-node/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

marquiz · 2024-02-01T19:04:35Z

Updated details on cluster autoscaler

- bump versions in kep.yaml - fix typos - added a use case for per-container OOM kill behavior

marquiz · 2024-04-10T10:00:15Z

Small update, added per-container OOM kill behavior as a use case. There is a growing list of use cases for Kubernetes-managed QoS resources with links to ongoing efforts (e.g. swap and OOM kill)

mkarrmann · 2024-04-11T01:34:11Z

@marquiz As the person who's been most actively pushing for the OOM config option and just found about this KEP, I strongly agree that this is a great fit! This design is seems awesome and is a really nice generalization of the specific problem I was hoping to have solved!

kannon92 · 2024-04-11T22:37:41Z

keps/sig-node/3008-qos-class-resources/README.md

+container whose memory limit is higher than the requests, but that is treated
+by the node as `Guaranteed`.
+
+Taking this idea further, QoS-class resources could also make it possible to


Maybe kubernetes/kubernetes#78848 is a good tangigle use case to mention.

Thanks @kannon92 for the reference, I'll add it to the proposal. This is already speculated in Splitting Pod QoS Class section but I wasn't aware of this open issue.

kannon92 · 2024-04-11T22:39:47Z

keps/sig-node/3008-qos-class-resources/README.md

+QoS-class resource. Likely benefits of using the QoS-class resources mechanism
+would be to be able to set per-namespace defaults with LimitRanges and allow
+permission-control to high-priority classes with ResourceQuotas.
+


Another idea is smarter oom setups (systemd-oomd for example).

systemd-oomd allows one to set settings for oom (psi based pressure and swap usage) on a cgroup slice. I could envision a future where one can create a QoS and that gives knobs to control systemd-oomd for more aggressive oom.

https://fedoraproject.org/wiki/Changes/EnableSystemdOomd

I was playing around with this and I explored the ideas of setting different knobs based on the existing kubepods slices.

Interesting. So you think a about pre-defined oom kill classes (in contrast to e.g. exposing every possible systemd-oomd knob to the user). At first thought, this sounds like a good fit. I'll add this to the proposal too, as a possible future use case.

kannon92 · 2024-04-11T22:52:18Z

keps/sig-node/3008-qos-class-resources/README.md

+that kubelet could evict a running pod that request QoS-class resources that
+are no more available on the node. This should be relatively straightforward to
+implement as kubelet knows what QoS-class resources are available on the node
+and also monitors all running pods.


Some areas that eviction manager is starting to show its age.

If one wants to add more disks to kubelet, we lose the ability for eviction manager to monitor it.

PSI based eviction

Swap based eviction

Moving things out of root partition (logs come in mind)

I've been thinking that eventually we are going to need some kind of pluggable eviction manager based on certain resources. It should be general but I haven't really made much progress on it. wanted to throw this out there as it may be worth considering.

@kannon92 thanks for the idea. Do you have any PoC implementation on this? If you have, it would be nice to wire it up to my code and see how it works.

adrianmoisey · 2024-04-17T17:20:35Z

keps/sig-node/3008-qos-class-resources/README.md

+
+<!-- References -->
+[intel-rdt]: https://www.intel.com/content/www/us/en/architecture-and-technology/resource-director-technology.html
+[linux-resctrl]: https://www.kernel.org/doc/html/latest/x86/resctrl.html


This URL now 404s

New link:

Suggested change

[linux-resctrl]: https://www.kernel.org/doc/html/latest/x86/resctrl.html

[linux-resctrl]: https://docs.kernel.org/arch/x86/resctrl.html

Madhu-1 · 2024-06-26T11:51:41Z

Hey @marquiz, it's great to see that this is making progress! Do you have any updates on the timeline for when this version is planned to be released? The KEP indicates it's planned for 1.31, but I was wondering if it might end up being pushed to 1.32 or later instead.

k8s-triage-robot · 2024-09-24T12:26:24Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

kad · 2024-09-24T14:36:46Z

/remove-lifecycle stale

k8s-triage-robot · 2024-12-23T15:02:54Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

kad · 2024-12-24T18:58:19Z

/remove-lifecycle stale

k8s-triage-robot · 2025-04-07T11:39:21Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

kad · 2025-04-07T12:05:31Z

/remove-lifecycle stale

k8s-triage-robot · 2025-07-06T13:01:43Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 7, 2021

k8s-ci-robot requested review from haircommander, kad, dchen1107 and derekwaynecarr October 7, 2021 14:15

k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/node Categorizes an issue or PR as relevant to SIG Node. labels Oct 7, 2021

marquiz mentioned this pull request Oct 11, 2021

QoS-class resources #3008

Open

4 tasks

marquiz changed the title ~~KEP-NNN: Class-based resources in CRI~~ KEP-3008: Class-based resources in CRI Oct 11, 2021

marquiz force-pushed the devel/kep-class-based-resources branch from 3c8bfb5 to ead1a96 Compare October 11, 2021 06:48

marquiz force-pushed the devel/kep-class-based-resources branch 3 times, most recently from ff20ea2 to 5f2e9fe Compare October 19, 2021 11:46

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 17, 2022

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Feb 1, 2022

k8s-ci-robot changed the title ~~KEP-3008: Class-based resources in CRI~~ KEP-3008: Class-based resources Feb 1, 2022

marquiz marked this pull request as ready for review February 11, 2022 13:18

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 11, 2022

marquiz force-pushed the devel/kep-class-based-resources branch 3 times, most recently from 8786f80 to bf746cc Compare February 22, 2022 14:39

marquiz added 2 commits January 26, 2024 18:49

KEP-3008: small update to summary

1c6b529

KEP-3008: update cluster autoscaler

251b0ca

Addressing review feedback from thocking.

liggitt removed this from the v1.30 milestone Mar 14, 2024

KEP-3008: update

fc082b2

- bump versions in kep.yaml - fix typos - added a use case for per-container OOM kill behavior

marquiz mentioned this pull request Apr 10, 2024

kubelet: new kubelet config option for disabling group oom kill kubernetes/kubernetes#122813

Closed

mkarrmann mentioned this pull request Apr 11, 2024

[Feature Proposal]: Ability to Configure Whether cgroupv2's group OOMKill is Used at the Pod Level kubernetes/kubernetes#124253

Closed

kannon92 reviewed Apr 11, 2024

View reviewed changes

adrianmoisey reviewed Apr 17, 2024

View reviewed changes

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 24, 2024

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 24, 2024

alicefr mentioned this pull request Oct 24, 2024

The VM Support to set Blkiotune kubevirt/kubevirt#13100

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 23, 2024

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 24, 2024

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 7, 2025

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 7, 2025

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 6, 2025

	[linux-resctrl]: https://www.kernel.org/doc/html/latest/x86/resctrl.html
	[linux-resctrl]: https://docs.kernel.org/arch/x86/resctrl.html

KEP-3008: QoS-class resources #3004

Are you sure you want to change the base?

KEP-3008: QoS-class resources #3004

Conversation

marquiz commented Oct 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Oct 7, 2021

Uh oh!

marquiz commented Oct 7, 2021

Uh oh!

kikisdeliveryservice commented Oct 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marquiz commented Oct 11, 2021

Uh oh!

k8s-triage-robot commented Jan 17, 2022

Uh oh!

marquiz commented Feb 1, 2022

Uh oh!

k8s-ci-robot commented Feb 1, 2024

Uh oh!

marquiz commented Feb 1, 2024

Uh oh!

marquiz commented Apr 10, 2024

Uh oh!

mkarrmann commented Apr 11, 2024

Uh oh!

kannon92 Apr 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marquiz Apr 15, 2024

Choose a reason for hiding this comment

Uh oh!

kannon92 Apr 11, 2024

Choose a reason for hiding this comment

Uh oh!

kannon92 Apr 11, 2024

Choose a reason for hiding this comment

Uh oh!

marquiz Apr 15, 2024

Choose a reason for hiding this comment

Uh oh!

kannon92 Apr 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marquiz Apr 15, 2024

Choose a reason for hiding this comment

Uh oh!

adrianmoisey Apr 17, 2024

Choose a reason for hiding this comment

Uh oh!

yedayak Jan 7, 2025

Choose a reason for hiding this comment

Uh oh!

Madhu-1 commented Jun 26, 2024

Uh oh!

k8s-triage-robot commented Sep 24, 2024

Uh oh!

kad commented Sep 24, 2024

Uh oh!

k8s-triage-robot commented Dec 23, 2024

Uh oh!

kad commented Dec 24, 2024

Uh oh!

k8s-triage-robot commented Apr 7, 2025

Uh oh!

kad commented Apr 7, 2025

Uh oh!

k8s-triage-robot commented Jul 6, 2025

Uh oh!

Uh oh!

marquiz commented Oct 7, 2021 •

edited

Loading

kikisdeliveryservice commented Oct 8, 2021 •

edited

Loading

kannon92 Apr 11, 2024 •

edited

Loading

kannon92 Apr 11, 2024 •

edited

Loading