[containerd] Support containerd v2.0.x #11845

mzaian · 2025-01-01T12:49:38Z

What type of PR is this?

/kind container-managers
/kind feature

What this PR does / why we need it:
Add hashes for containerd versions 2.0.[0-2]
containerd 2.0.0 release notes https://github.com/containerd/containerd/releases/tag/v2.0.0
containerd 2.0.1 release notes https://github.com/containerd/containerd/releases/tag/v2.0.1
containerd 2.0.2 release notes https://github.com/containerd/containerd/releases/tag/v2.0.2
Update runc binary to v1.2.4
Make containerd 2.0.2 default
Set containerd_limit_open_file_num to 1048576

Which issue(s) this PR fixes:

Fixes #11836

Does this PR introduce a user-facing change?:

Make containerd 2.0.2 default
Update runc binary to v1.2.4
Set containerd_limit_open_file_num to 1048576 so it's configurable.

k8s-ci-robot · 2025-01-01T12:49:46Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mzaian

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mzaian]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mzaian · 2025-01-01T12:51:09Z

I did not remove LimitNOFILE from containerd.service.j2 which is removed from the 2.0.x release. There is a discussion here: containerd/containerd#8924 about that.

mzaian · 2025-01-01T12:51:15Z

/ok-to-test

VannTen · 2025-01-01T23:59:34Z

I don't believe we should do this (differ from upstream in this way). I read most of the containerd discussion; and the change seems correct to me. There is the question of systemd pre v240 where LimitNOfile would need an explicit value, but appart from that it seems sane.

yankay · 2025-01-02T01:40:27Z

Hi @mzaian,

I do not recommend upgrading the default version of containerd to v2 for several reasons:

Breaking Changes: Containerd v2 introduces significant breaking changes that require migration. Currently, most Kubernetes users are still on v1.x.
Upstream Dependency: The upstream Kubernetes project does not have any plans to migrate to containerd v2 at this time, as they are still relying on v1.x. You can check the details here.

Given the magnitude of this change, I suggest that Kubespray should consider supporting both containerd v2 and v1 both, while keeping v1.x as the default. Once the upstream project transitions to v2, Kubespray can then update the default to v2.

/hold (for more discuss)

mzaian · 2025-01-02T07:28:56Z

Hi @mzaian,

I do not recommend upgrading the default version of containerd to v2 for several reasons:

Breaking Changes: Containerd v2 introduces significant breaking changes that require migration. Currently, most Kubernetes users are still on v1.x.

Upstream Dependency: The upstream Kubernetes project does not have any plans to migrate to containerd v2 at this time, as they are still relying on v1.x. You can check the details here.

Given the magnitude of this change, I suggest that Kubespray should consider supporting both containerd v2 and v1 both, while keeping v1.x as the default. Once the upstream project transitions to v2, Kubespray can then update the default to v2.

/hold (for more discuss)

Hi @yankay

As we align with the upstream Kubernetes. Let's keep this on-hold for a future Kubespray release then.

VannTen · 2025-01-03T16:47:20Z

- Upstream Dependency: The upstream Kubernetes project does not have any plans to migrate to containerd v2 at this time, as they are still relying on v1.x. You can check the details [here](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.32.md?plain=1#L618).

I don't think this is correct, kubernetes support containerd 2.0 since 1.30 [1] I think the containerd releases prefixed with api are about the containerd grpc API [2] not containerd itself. Defaulting to 2.0 2 kubernetes releases after it's supported does not seems that much of a problem IMO [1]: https://github.com/containerd/containerd/blob/main/RELEASES.md#kubernetes-support [2]: https://github.com/containerd/containerd/tree/api/v1.7.19/api

yankay · 2025-01-07T05:49:07Z

Upstream Dependency: The upstream Kubernetes project does not have any plans to migrate to containerd v2 at this time, as they are still relying on v1.x. You can check the details here.
I don't think this is correct, kubernetes support containerd 2.0 since 1.30 [1] I think the containerd releases prefixed with api are about the containerd grpc API [2] not containerd itself. Defaulting to 2.0 2 kubernetes releases after it's supported does not seems that much of a problem IMO [1]: https://github.com/containerd/containerd/blob/main/RELEASES.md#kubernetes-support [2]: https://github.com/containerd/containerd/tree/api/v1.7.19/api

Thanks @VannTen

/unhold

mzaian · 2025-01-07T06:57:32Z

Upstream Dependency: The upstream Kubernetes project does not have any plans to migrate to containerd v2 at this time, as they are still relying on v1.x. You can check the details here.
I don't think this is correct, kubernetes support containerd 2.0 since 1.30 [1] I think the containerd releases prefixed with api are about the containerd grpc API [2] not containerd itself. Defaulting to 2.0 2 kubernetes releases after it's supported does not seems that much of a problem IMO [1]: https://github.com/containerd/containerd/blob/main/RELEASES.md#kubernetes-support [2]: https://github.com/containerd/containerd/tree/api/v1.7.19/api

Thanks @VannTen

/unhold

Ok good then we can proceed and this will be out with the next release. So we have time to test and fix any issues when they occur.

yankay · 2025-01-07T07:31:16Z

Yes @mzaian 🎉

There some bugs with containerd v2.0 and kubenretes , eg:

containerd 2.0.0 kubernetes bug containerd/containerd#11117

VannTen · 2025-01-07T08:47:02Z

So what about the LimitNOFile change ?

Also, maybe we should use a action required release note and link to containerd 2.0 release ?

mzaian · 2025-01-07T08:49:55Z

So what about the LimitNOFile change ?

Also, maybe we should use a action required release note and link to containerd 2.0 release ?

I will add it again and add a release not for it.

VannTen

Regarding the service file, should we template based on containerd version ?
Aka keep Limit* on 1.7 ?

VannTen · 2025-01-07T14:01:08Z

roles/container-engine/containerd/templates/containerd.service.j2

@@ -30,7 +30,6 @@ RestartSec=5
 # in the kernel. We recommend using cgroups to do container-local accounting.
 LimitNPROC={{ containerd_limit_proc_num }}
 LimitCORE={{ containerd_limit_core }}
-LimitNOFILE={{ containerd_limit_open_file_num }}


I think on systemd v240 we should use 1024:524288 as explained in containerd/containerd#8924 (comment)
(Rhel 8 and derivatives are on 239 if I believe Repology)

Not sure how to check for systemd version though 🤔

How about leaving this out of this PR and I can work separately in investigating this for all other supported operating systems.

Hi @mzaian,

Would it be better to retain LimitNOFILE={{ containerd_limit_open_file_num }} and adjust the default value of containerd_limit_open_file_num to 1048576, similar to what was done in kubernetes/kops#16329?
This way, users can still customize containerd_limit_open_file_num, and avoid to use infinity

Thank you!

I'd rather follow containerd on that. In particular, this means running a pod locally with an untweaked config (in podman or docker ) could have a different behavior than in a kubespray cluster (if running into the issues when the soft limit is high).

I'd still default to upstream in that case, for the reasons above.

I'm fine with both options? @yankay wdyt?

From the release note of containerd 2.0 https://github.com/containerd/containerd/blob/main/docs/containerd-2.0.md#limitnofile-configuration-has-been-removed , it's a break change.

Administrators on platforms running versions less than systemd 240 should explicitly configure LimitNOFILE=1024:524288 or risk falling back to the kernel default of 4096.

The Kops's option is to keep it not break.

So my idea is to open a new issue #11918 to config the LimitNOFILE, the kubespray can make the suggestion by the release note, like configuire it to LimitNOFILE=1024:524288 when the system version < 240.

Does it OK to merge the PR :-)

That's not completely accurate regarding kops, since they change it from infinity to 1048576.
Again, I still thinks we should defer to upstream default (and systemd>240, for that matter) and mention it in a action required note, linking in particular to the containerd discussion.

Regarding systemd version, since it's the default after >240, we could also use LimitNOFILE=1024:524288 on all systemd version, it simply won't have an effect.

For the release note, if we go that route, I would use something like :

action required LimitNOFile is now 1024:524288 for containerd (upstream defaults). If you relied on a the previous `infinity` value, set `containerd_limit_open_file_num` to a reasonable value (1048576 is the value used by kops, for instance). In particular, somet For more details, see the containerd discussion <link to the PR on containerd> and in particular this comment https://github.com/containerd/containerd/pull/8924#issuecomment-1868165195

That's not completely accurate regarding kops, since they change it from infinity to 1048576. Again, I still thinks we should defer to upstream default (and systemd>240, for that matter) and mention it in a action required note, linking in particular to the containerd discussion.

Regarding systemd version, since it's the default after >240, we could also use LimitNOFILE=1024:524288 on all systemd version, it simply won't have an effect.

For the release note, if we go that route, I would use something like :

action required LimitNOFile is now 1024:524288 for containerd (upstream defaults). If you relied on a the previous `infinity` value, set `containerd_limit_open_file_num` to a reasonable value (1048576 is the value used by kops, for instance). In particular, somet For more details, see the containerd discussion <link to the PR on containerd> and in particular this comment https://github.com/containerd/containerd/pull/8924#issuecomment-1868165195

Hi @VannTen,

I agree that we should stay consistent with the upstream. In this PR, either removing LimitNOFILE or setting LimitNOFILE=1048576 would be acceptable.

Then, I would like to submit another PR to automatically determine the systemd version to set this value, rather than relying solely on the release notes. #11918 What do you think?

roles/container-engine/containerd/templates/containerd.service.j2

mzaian · 2025-01-07T17:29:05Z

Regarding the service file, should we template based on containerd version ? Aka keep Limit* on 1.7 ?

I can do this but first I will check if they cherry-picked this change to 1.7.x

yankay · 2025-01-20T02:32:27Z

Thanks @mzaian

Does the PR need any more changes before we can merge it? 🙂

mzaian · 2025-01-20T11:00:49Z

Thanks @mzaian

Does the PR need any more changes before we can merge it? 🙂

I just added v2.0.2 and the updated runc binary to v1.2.4

k8s-ci-robot requested review from cyclinder and MrFreezeex January 1, 2025 12:49

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jan 1, 2025

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Jan 1, 2025

mzaian requested review from yankay, VannTen and tico88612 January 1, 2025 12:52

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 2, 2025

pacoxu mentioned this pull request Jan 3, 2025

CI migration Plan for containerd v2.0 kubernetes/test-infra#34063

Open

mzaian force-pushed the support-containerd-2.0.x branch 2 times, most recently from 037b6e9 to fba3798 Compare January 6, 2025 09:56

yankay mentioned this pull request Jan 6, 2025

Migrate the containerd config to deprecate the CRI registry properties #11861

Open

mzaian force-pushed the support-containerd-2.0.x branch from fba3798 to b730cf6 Compare January 6, 2025 13:31

yankay removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 7, 2025

mzaian force-pushed the support-containerd-2.0.x branch from b730cf6 to 5c8dc3e Compare January 7, 2025 13:40

VannTen reviewed Jan 7, 2025

View reviewed changes

mzaian force-pushed the support-containerd-2.0.x branch from 5c8dc3e to 4954219 Compare January 8, 2025 08:47

VannTen mentioned this pull request Jan 9, 2025

Deprecate support for RedHat 8 family hosts #11872

Open

mzaian force-pushed the support-containerd-2.0.x branch from 4954219 to 8ee2058 Compare January 20, 2025 11:00

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jan 20, 2025

mzaian requested a review from VannTen January 20, 2025 11:00

mzaian force-pushed the support-containerd-2.0.x branch from 8ee2058 to 4c91901 Compare January 22, 2025 11:29

mzaian mentioned this pull request Jan 22, 2025

Bump nerdctl to v2.0.3 #11913

Open

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 23, 2025

[containerd] Support containerd v2.0.x

8a0ecd5

mzaian force-pushed the support-containerd-2.0.x branch from 4c91901 to 8a0ecd5 Compare January 23, 2025 08:08

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 23, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[containerd] Support containerd v2.0.x #11845

[containerd] Support containerd v2.0.x #11845

mzaian commented Jan 1, 2025 •

edited

Loading

k8s-ci-robot commented Jan 1, 2025

mzaian commented Jan 1, 2025

mzaian commented Jan 1, 2025

VannTen commented Jan 1, 2025 via email

yankay commented Jan 2, 2025 •

edited

Loading

mzaian commented Jan 2, 2025

VannTen commented Jan 3, 2025 via email

yankay commented Jan 7, 2025 •

edited

Loading

mzaian commented Jan 7, 2025

yankay commented Jan 7, 2025 •

edited

Loading

VannTen commented Jan 7, 2025

mzaian commented Jan 7, 2025

VannTen left a comment

VannTen Jan 7, 2025

mzaian Jan 7, 2025

yankay Jan 8, 2025 •

edited

Loading

VannTen Jan 9, 2025

VannTen Jan 22, 2025

mzaian Jan 22, 2025

yankay Jan 23, 2025 •

edited

Loading

VannTen Jan 23, 2025

yankay Jan 23, 2025

mzaian commented Jan 7, 2025

yankay commented Jan 20, 2025

mzaian commented Jan 20, 2025

[containerd] Support containerd v2.0.x #11845

Are you sure you want to change the base?

[containerd] Support containerd v2.0.x #11845

Conversation

mzaian commented Jan 1, 2025 • edited Loading

k8s-ci-robot commented Jan 1, 2025

mzaian commented Jan 1, 2025

mzaian commented Jan 1, 2025

VannTen commented Jan 1, 2025 via email

yankay commented Jan 2, 2025 • edited Loading

mzaian commented Jan 2, 2025

VannTen commented Jan 3, 2025 via email

yankay commented Jan 7, 2025 • edited Loading

mzaian commented Jan 7, 2025

yankay commented Jan 7, 2025 • edited Loading

VannTen commented Jan 7, 2025

mzaian commented Jan 7, 2025

VannTen left a comment

Choose a reason for hiding this comment

VannTen Jan 7, 2025

Choose a reason for hiding this comment

mzaian Jan 7, 2025

Choose a reason for hiding this comment

yankay Jan 8, 2025 • edited Loading

Choose a reason for hiding this comment

VannTen Jan 9, 2025

Choose a reason for hiding this comment

VannTen Jan 22, 2025

Choose a reason for hiding this comment

mzaian Jan 22, 2025

Choose a reason for hiding this comment

yankay Jan 23, 2025 • edited Loading

Choose a reason for hiding this comment

VannTen Jan 23, 2025

Choose a reason for hiding this comment

yankay Jan 23, 2025

Choose a reason for hiding this comment

mzaian commented Jan 7, 2025

yankay commented Jan 20, 2025

mzaian commented Jan 20, 2025

mzaian commented Jan 1, 2025 •

edited

Loading

yankay commented Jan 2, 2025 •

edited

Loading

yankay commented Jan 7, 2025 •

edited

Loading

yankay commented Jan 7, 2025 •

edited

Loading

yankay Jan 8, 2025 •

edited

Loading

yankay Jan 23, 2025 •

edited

Loading