KEP-3008: QoS-class resources #3004

marquiz · 2021-10-07T14:14:49Z

One-line PR description:
New KEP for adding class-based resources to CRI protocol

Issue link: QoS-class resources #3008

Other comments:

k8s-ci-robot · 2021-10-07T14:14:59Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

marquiz · 2021-10-07T14:15:19Z

/cc @kad @haircommander

kikisdeliveryservice · 2021-10-08T21:24:23Z

Hi @marquiz

All KEP PRs must have an open issue in k/enhancements (this repo). Please open an issue and fill it out completely and rename this PR to KEP-issue number but in the title of this PR and in your README.md

Thanks!

marquiz · 2021-10-11T06:49:38Z

All KEP PRs must have an open issue in k/enhancements (this repo). Please open an issue and fill it out completely and rename this PR to KEP-issue number but in the title of this PR and in your README.md

Thanks for the guidance @kikisdeliveryservice. Done

k8s-triage-robot · 2022-01-17T12:46:28Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

marquiz · 2022-02-01T09:59:53Z

/retitle KEP-3008: Class-based resources

marquiz · 2024-01-11T13:17:34Z

I'm trying to here capture here a summary of the PR evolution and status. The PR has been reviewed by numerous people and it has so many comments/conversations that it becomes impossible for me to summarize everything in a nice way.

Bits that are marked as unresolved as of today:

CRI detail: whether to use/not use common type for pod and container level QoS resources, (discussion)
What prefixi policy (k8s.io/ vs. example.com/ vs unprefixed) to use for QoS resource names (e.g. this discussion, this discussion)
Naming, some discussions/comments on that:
- PrioritizedResources proposed by @logicalhan here
- qosAssignments by @sftim here
- @thockin was not pleased but didn't suggest any alternative(s)
- comment from @pohly
Everything under "Future work" should be considered WIP or UNRESOLVED
- Separate API resource(s) describing (e.g. this comment), listed under "Future work" as a possible extension

Some previous concerns/shortcomings that have later been addressed:

pod-level vs. container-level QoS (discussion)
"official"/well-known/canonical names of Qos resources (e.g. this discussion), addressed in the proposal under Consts section (the details are still UNRESOLVED)
defaults, i.e. "what do I get if I request nothing" (e.g. discussion, another discussion), covered in the proposal e.g. under "Implicit defaults" and "PodStatus" sections
"Exotic" use cases (e.g. this comment and response), "Possible future scenarios" section under "User stories"
Class capacity (e.g. comment, another comment) was added. It is now possible to limit the number of users of classes.
Per runtime class QoS (comment), was listed as a non-goal of this KEP

thockin

Ran out of time for this session, more later.

thockin · 2024-01-25T00:13:33Z

keps/sig-node/3008-qos-class-resources/README.md

+to other Kubernetes resource types (i.e. native resources such as `cpu` and
+`memory` or extended resources) because you can assign that resource to a
+particular container. However, QoS-class resources are also different from
+those other resources because they are used to assign a _class identifier_,


Do we have concrete examples of things where we want to apply QoS, but which REALLY don't feel like "resources"?

While "QoS" is somewhat jargon-y, it's pretty well understood in my experience. Alternately, if this is just about names, we could look for similar words?

"banded resources" ? It sort of implies linearness of bands

"tiered resources" ? Also implies linearity, which I don't think is necessarily true

"categorized" ?

"qualitative" ?

keps/sig-node/3008-qos-class-resources/README.md

thockin · 2024-01-25T00:42:04Z

keps/sig-node/3008-qos-class-resources/README.md

+       Name QoSResourceName
+       // Allowed classes.
+       Classes []string
+       // Capacity is the hard limit for usage of the class.


I don't love that this is punted with no real idea how we might solve it

thockin · 2024-01-25T00:47:10Z

keps/sig-node/3008-qos-class-resources/README.md

+
+#### Cluster autoscaler
+
+The cluster autoscaler support will be extended to support QoS-class resources.


I find this paragraph vague and unconvincing. :)

Autoscaling has really become a requirement for many users, and this text doesn't explain to me how the autoscalers (we now have more than one!) are able to reason about "if I make a new node of shape X, it will satisfy this pod". We left this for "later" in other efforts and it really bit us in the butt.

If you think you have an answer, can you please flesh this out? If you don't think you have an answer, we need to think about it.

@johnbelamaric @mwielgus

Fair point @thockin (especially in the light of the recent DRA activities). I admit that this is vague and lacking details.

I will get back to this. My plan is now to have a PoC of a cluster autoscaler to have confidence that it can be done and write out the details here.

In case it wasn't clear, my comment was not meant to be rude.

@thockin no offence taken 😅

I took me a while to get back to this as I was not really familiar with the cluster autoscaler details. I wanted to try it out in practice and I did do a PoC implementation based on a kind cluster with CAPD deployed and running cluster-autoscaler against that. Scaling up node groups with one (or more) existing nodes worked pretty much out-of-the-box (after building cluster-autoscaler against my PoC kubernetes scheduler implementation). Because all of the scheduling logic (of QoS resources) is in k/k scheduler itself, the autoscaler correctly runs simulations and can determine how many nodes in what nodegroups need to be scaled up (based on the QoS resources available in existing nodes), also providing log messages why certain node group isn't suitable. What is missing is the provider-specific mechanism to inform the cluster autoscaler about QoS-resources of empty node groups but I wouldn't expect that to be an insurmountable problem (follow along the lines what is done for extended resources). I updated the KEP accordingly. WDYT?

aojea · 2024-01-25T08:22:30Z

keps/sig-node/3008-qos-class-resources/README.md

+Why should this KEP _not_ be implemented?
+-->
+
+## Alternatives


Can this not be done with existing device plugins framework and CDI https://github.com/cncf-tags/container-device-interface/blob/main/SPEC.md#cdi-json-specification?

@aojea I cannot think how CDI could satisfy all the requirements here 🤔 First there's the Kubernetes API (for which CDI brings no avail). Then, complicating the device plugin API for the QoS-resources hasn't even been in considered tbh (instead, we try to make kubelet's life easier)

Addressing review feedback from thocking.

k8s-ci-robot · 2024-02-01T19:04:19Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: marquiz
Once this PR has been reviewed and has the lgtm label, please assign johnbelamaric for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

keps/prod-readiness/OWNERS
keps/sig-node/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

marquiz · 2024-02-01T19:04:35Z

Updated details on cluster autoscaler

- bump versions in kep.yaml - fix typos - added a use case for per-container OOM kill behavior

marquiz · 2024-04-10T10:00:15Z

Small update, added per-container OOM kill behavior as a use case. There is a growing list of use cases for Kubernetes-managed QoS resources with links to ongoing efforts (e.g. swap and OOM kill)

mkarrmann · 2024-04-11T01:34:11Z

@marquiz As the person who's been most actively pushing for the OOM config option and just found about this KEP, I strongly agree that this is a great fit! This design is seems awesome and is a really nice generalization of the specific problem I was hoping to have solved!

kannon92 · 2024-04-11T22:37:41Z

keps/sig-node/3008-qos-class-resources/README.md

+container whose memory limit is higher than the requests, but that is treated
+by the node as `Guaranteed`.
+
+Taking this idea further, QoS-class resources could also make it possible to


Maybe kubernetes/kubernetes#78848 is a good tangigle use case to mention.

Thanks @kannon92 for the reference, I'll add it to the proposal. This is already speculated in Splitting Pod QoS Class section but I wasn't aware of this open issue.

kannon92 · 2024-04-11T22:39:47Z

keps/sig-node/3008-qos-class-resources/README.md

+QoS-class resource. Likely benefits of using the QoS-class resources mechanism
+would be to be able to set per-namespace defaults with LimitRanges and allow
+permission-control to high-priority classes with ResourceQuotas.
+


Another idea is smarter oom setups (systemd-oomd for example).

systemd-oomd allows one to set settings for oom (psi based pressure and swap usage) on a cgroup slice. I could envision a future where one can create a QoS and that gives knobs to control systemd-oomd for more aggressive oom.

https://fedoraproject.org/wiki/Changes/EnableSystemdOomd

I was playing around with this and I explored the ideas of setting different knobs based on the existing kubepods slices.

Interesting. So you think a about pre-defined oom kill classes (in contrast to e.g. exposing every possible systemd-oomd knob to the user). At first thought, this sounds like a good fit. I'll add this to the proposal too, as a possible future use case.

kannon92 · 2024-04-11T22:52:18Z

keps/sig-node/3008-qos-class-resources/README.md

+that kubelet could evict a running pod that request QoS-class resources that
+are no more available on the node. This should be relatively straightforward to
+implement as kubelet knows what QoS-class resources are available on the node
+and also monitors all running pods.


Some areas that eviction manager is starting to show its age.

If one wants to add more disks to kubelet, we lose the ability for eviction manager to monitor it.

PSI based eviction

Swap based eviction

Moving things out of root partition (logs come in mind)

I've been thinking that eventually we are going to need some kind of pluggable eviction manager based on certain resources. It should be general but I haven't really made much progress on it. wanted to throw this out there as it may be worth considering.

@kannon92 thanks for the idea. Do you have any PoC implementation on this? If you have, it would be nice to wire it up to my code and see how it works.

adrianmoisey · 2024-04-17T17:20:35Z

keps/sig-node/3008-qos-class-resources/README.md

+
+<!-- References -->
+[intel-rdt]: https://www.intel.com/content/www/us/en/architecture-and-technology/resource-director-technology.html
+[linux-resctrl]: https://www.kernel.org/doc/html/latest/x86/resctrl.html


This URL now 404s

Madhu-1 · 2024-06-26T11:51:41Z

Hey @marquiz, it's great to see that this is making progress! Do you have any updates on the timeline for when this version is planned to be released? The KEP indicates it's planned for 1.31, but I was wondering if it might end up being pushed to 1.32 or later instead.

k8s-triage-robot · 2024-09-24T12:26:24Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

kad · 2024-09-24T14:36:46Z

/remove-lifecycle stale

k8s-triage-robot · 2024-12-23T15:02:54Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

kad · 2024-12-24T18:58:19Z

/remove-lifecycle stale

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 7, 2021

k8s-ci-robot requested review from haircommander, kad, dchen1107 and derekwaynecarr October 7, 2021 14:15

k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/node Categorizes an issue or PR as relevant to SIG Node. labels Oct 7, 2021

marquiz mentioned this pull request Oct 11, 2021

QoS-class resources #3008

Open

4 tasks

marquiz changed the title ~~KEP-NNN: Class-based resources in CRI~~ KEP-3008: Class-based resources in CRI Oct 11, 2021

marquiz force-pushed the devel/kep-class-based-resources branch from 3c8bfb5 to ead1a96 Compare October 11, 2021 06:48

marquiz force-pushed the devel/kep-class-based-resources branch 3 times, most recently from ff20ea2 to 5f2e9fe Compare October 19, 2021 11:46

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 17, 2022

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Feb 1, 2022

k8s-ci-robot changed the title ~~KEP-3008: Class-based resources in CRI~~ KEP-3008: Class-based resources Feb 1, 2022

marquiz marked this pull request as ready for review February 11, 2022 13:18

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 11, 2022

marquiz force-pushed the devel/kep-class-based-resources branch 2 times, most recently from eefa0e3 to 8786f80 Compare February 22, 2022 14:38

marquiz force-pushed the devel/kep-class-based-resources branch from c43c5d7 to f2001a4 Compare January 9, 2024 17:00

KEP-3008: PRR: add details on rollback failures

7214ab4

marquiz mentioned this pull request Jan 16, 2024

Node memory swap support #2400

Open

60 tasks

thockin reviewed Jan 25, 2024

View reviewed changes

aojea reviewed Jan 25, 2024

View reviewed changes

marquiz added 2 commits January 26, 2024 18:49

KEP-3008: small update to summary

1c6b529

KEP-3008: update cluster autoscaler

251b0ca

Addressing review feedback from thocking.

liggitt removed this from the v1.30 milestone Mar 14, 2024

KEP-3008: update

fc082b2

- bump versions in kep.yaml - fix typos - added a use case for per-container OOM kill behavior

marquiz mentioned this pull request Apr 10, 2024

kubelet: new kubelet config option for disabling group oom kill kubernetes/kubernetes#122813

Closed

mkarrmann mentioned this pull request Apr 11, 2024

[Feature Proposal]: Ability to Configure Whether cgroupv2's group OOMKill is Used at the Pod Level kubernetes/kubernetes#124253

Closed

kannon92 reviewed Apr 11, 2024

View reviewed changes

adrianmoisey reviewed Apr 17, 2024

View reviewed changes

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 24, 2024

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 24, 2024

alicefr mentioned this pull request Oct 24, 2024

The VM Support to set Blkiotune kubevirt/kubevirt#13100

Open

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 23, 2024

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KEP-3008: QoS-class resources #3004

KEP-3008: QoS-class resources #3004

marquiz commented Oct 7, 2021 •

edited

Loading

k8s-ci-robot commented Oct 7, 2021

marquiz commented Oct 7, 2021

kikisdeliveryservice commented Oct 8, 2021 •

edited

Loading

marquiz commented Oct 11, 2021

k8s-triage-robot commented Jan 17, 2022

marquiz commented Feb 1, 2022

marquiz commented Jan 11, 2024

thockin left a comment

thockin Jan 25, 2024

thockin Jan 25, 2024

thockin Jan 25, 2024

marquiz Jan 26, 2024

thockin Jan 30, 2024

marquiz Feb 1, 2024

aojea Jan 25, 2024

marquiz Jan 26, 2024

k8s-ci-robot commented Feb 1, 2024

marquiz commented Feb 1, 2024

marquiz commented Apr 10, 2024

mkarrmann commented Apr 11, 2024

kannon92 Apr 11, 2024 •

edited

Loading

marquiz Apr 15, 2024

kannon92 Apr 11, 2024

kannon92 Apr 11, 2024

marquiz Apr 15, 2024

kannon92 Apr 11, 2024 •

edited

Loading

marquiz Apr 15, 2024

adrianmoisey Apr 17, 2024

Madhu-1 commented Jun 26, 2024

k8s-triage-robot commented Sep 24, 2024

kad commented Sep 24, 2024

k8s-triage-robot commented Dec 23, 2024

kad commented Dec 24, 2024


		#### Cluster autoscaler

		The cluster autoscaler support will be extended to support QoS-class resources.

KEP-3008: QoS-class resources #3004

Are you sure you want to change the base?

KEP-3008: QoS-class resources #3004

Conversation

marquiz commented Oct 7, 2021 • edited Loading

k8s-ci-robot commented Oct 7, 2021

marquiz commented Oct 7, 2021

kikisdeliveryservice commented Oct 8, 2021 • edited Loading

marquiz commented Oct 11, 2021

k8s-triage-robot commented Jan 17, 2022

marquiz commented Feb 1, 2022

marquiz commented Jan 11, 2024

thockin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Feb 1, 2024

marquiz commented Feb 1, 2024

marquiz commented Apr 10, 2024

mkarrmann commented Apr 11, 2024

kannon92 Apr 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kannon92 Apr 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Madhu-1 commented Jun 26, 2024

k8s-triage-robot commented Sep 24, 2024

kad commented Sep 24, 2024

k8s-triage-robot commented Dec 23, 2024

kad commented Dec 24, 2024

marquiz commented Oct 7, 2021 •

edited

Loading

kikisdeliveryservice commented Oct 8, 2021 •

edited

Loading

kannon92 Apr 11, 2024 •

edited

Loading

kannon92 Apr 11, 2024 •

edited

Loading