|
Bugzilla – Full Text Bug Listing |
| Summary: | [kubeadm1.27] Multiple Issues before `kubeadm init` runs successfully | ||
|---|---|---|---|
| Product: | [openSUSE] openSUSE Tumbleweed | Reporter: | Egbert Eich <eich> |
| Component: | Containers | Assignee: | Priyanka Saggu <priyanka.saggu> |
| Status: | NEW --- | QA Contact: | E-mail List <qa-bugs> |
| Severity: | Normal | ||
| Priority: | P5 - None | CC: | dcermak, eich |
| Version: | Current | ||
| Target Milestone: | --- | ||
| Hardware: | All | ||
| OS: | openSUSE Tumbleweed | ||
| Whiteboard: | |||
| Found By: | Development | Services Priority: | |
| Business Priority: | Blocker: | --- | |
| Marketing QA Status: | --- | IT Deployment: | --- |
I was partly wrong with my assessment in item 1. in above description: It is perfectly fine to use crio runtime - however, it needs to be enabled and running at the time `kubeadm init` runs. Thus, this should be set in a postinstall script. NOTE: before doing so it should be checked if another container runtime (containerd) is already enabled (or even running). It it is running, nothing should be done (ie crio should not be touched). If it is just enabled, it should be started. Hey Egbert, no updates just yet -- just writing to acknowledge the ticket. I'm discussing re-packaging kubernetes, will use the issues raised in the bugzilla as feedback, for the same! Once I've any implementation updates, will post here. Thanks again! (In reply to Egbert Eich from comment #0) > 2. kubelet.service does not start: > It complains about the option `--container-runtime` not being known. This > option is pulled in from /etc/sysconfig/kubelet. It needs to be deleted > from > this file. Could you please paste the contents of that file here? (In reply to Dan Čermák from comment #3) > (In reply to Egbert Eich from comment #0) > > > 2. kubelet.service does not start: > > It complains about the option `--container-runtime` not being known. This > > option is pulled in from /etc/sysconfig/kubelet. It needs to be deleted > > from > > this file. > > Could you please paste the contents of that file here? Do you mean before or after I've manually edited and fixed it? Since I don't have a backup copy of the 'before' version, I'd have to go thru the entire exercise of installing k8s and initializing a controller again. I'd suggest you folks try this yourself. Actually, since the sysconfig files are backed up by 'fillup-templates', here is the content of /usr/share/fillup-templates/sysconfig.kubelet: KUBELET_EXTRA_ARGS="--container-runtime=remote --container-runtime-endpoint=unix:///var/run/crio/crio.sock --runtime-request-timeout=15m --cgroup-driver=systemd -v=2" This is the content of /etc/sysconfig/kubelet - after I've manually fixed it: KUBELET_VER=1.27 KUBELET_EXTRA_ARGS="--container-runtime-endpoint=unix:///var/run/crio/crio.sock --runtime-request-timeout=14m --cgroup-driver=systemd -v=2" To somewhat lift the mystery around /etc/sysconfig/kubelet: 1. The 'fillup template' is from cri-o-kubeadm-criconfig-1.24.3 2. cri-o-kubeadm-criconfig-1.24.3 provides 'kubernetes-kubeadm-criconfig' - unversioned. 3. It seems like cri-o is released in tandem with kubernetes (the versions are in sync). Therefore, 1. cri-o should be updated in tandem with kubernetes. 2. Dependencies between cri-o and kubernetes should be set so that only matching versions are installed. I've used patterns-kubernetes-kubeadm to install all of the goodness required by kubeadm, this only has versioned dependencies. (In reply to Egbert Eich from comment #6) > I've used patterns-kubernetes-kubeadm to install all of the goodness > required by kubeadm, this only has versioned dependencies. I meant to say 'unversioned' dependencies. Hey Egbert, I've raised the following SRs: - cri-o ~ https://build.opensuse.org/request/show/1108439 - kubernetes1.27 ~ https://build.opensuse.org/request/show/1108445 - containerd ~ https://build.opensuse.org/request/show/1108423 With these changes, I was able to have a successful `kubeadm init` run! # Although - before `kubeadm init`, I had to manually load the kernel modules (doing that in %post scriptlet didn't work, it complained for modprobe not found. Nor did adding it as `ExecStartPre=` in 10-kubeadm.conf work. I'll explore more options) > sudo modprobe overlay && sudo modprobe br_netfilter > > # with cri-o as CRI > sudo kubeadm init --kubernetes-version 1.27.4 --cri-socket=unix:///var/run/crio/crio.sock > > # with containerd as CRI > sudo kubeadm init --kubernetes-version 1.27.4 --cri-socket=unix:///var/run/containerd/containerd.sock --- > 3. It seems like cri-o is released in tandem with kubernetes (the versions are > in sync). > Therefore, > 1. cri-o should be updated in tandem with kubernetes. > 2. Dependencies between cri-o and kubernetes should be set so that only matching > versions are installed. I agree ^. I'll fix the crio/kubernetes version dependency part. Also, I'm yet to fix - "updating the `KUBELET_VER=x.yz` in `/etc/sysconfig/kubelet` file" on a new kubernetes version installation (after an existing kubernetes installation).
---
Also, this just fixes the kubernetes 1.27 install, the previous versions still need fixing.
There's drift between kubeadm and kubelet/kubelet-common versions
> $ sudo zypper in kubernetes1.26-kubeadm
>
> ...
> Resolving package dependencies...
>
> The following 6 NEW packages are going to be installed:
> cri-o cri-o-kubeadm-criconfig kubernetes1.25-kubelet kubernetes1.26-kubeadm kubernetes1.27-kubelet kubernetes1.27-kubelet-common
(In reply to Priyanka Saggu from comment #8) > Hey Egbert, I've raised the following SRs: > > - cri-o ~ https://build.opensuse.org/request/show/1108439 > - kubernetes1.27 ~ https://build.opensuse.org/request/show/1108445 > - containerd ~ https://build.opensuse.org/request/show/1108423 > > > With these changes, I was able to have a successful `kubeadm init` run! Cool! We should actually get the OpenQA tests going once there is a version which works in manual testing. > > # Although - before `kubeadm init`, I had to manually load the kernel > modules > (doing that in %post scriptlet didn't work, it complained for modprobe not > found. Nor did adding it as `ExecStartPre=` in 10-kubeadm.conf work. I'll > explore more options) Ok, did you specify the full path to modprobe (ie /sbin/modprobe)? Usually, %pre/%post install scripts do so. The same is true with `ExecStartPre=`: grepping through /usr/lib/systemd/system I find multiple examples with: `ExecStartPre=-/sbin/modprobe ...` so it should be expected to work. (In reply to Priyanka Saggu from comment #9) > Also, I'm yet to fix - "updating the `KUBELET_VER=x.yz` in > `/etc/sysconfig/kubelet` file" on a new kubernetes version installation > (after an existing kubernetes installation). Looks like the `fillup` semantics should support this: `/bin/fillup -x /etc/sysconfig/kubelet /usr/share/fillup-templates/sysconfig.kubelet-kubernetes1.27` Unfortunately, there is no convenience macro for this - like the SUSE-bespoke %fillup_only macro. SUSE-RU-2023:4341-1: An update that contains one feature and has two fixes can now be installed. Category: recommended (moderate) Bug References: 1213829, 1214406 Jira References: PED-5839 Sources used: openSUSE Leap 15.4 (src): kubernetes1.27-1.27.6-150400.9.3.2 openSUSE Leap 15.5 (src): kubernetes1.27-1.27.6-150400.9.3.2 Containers Module 15-SP4 (src): kubernetes1.27-1.27.6-150400.9.3.2 Containers Module 15-SP5 (src): kubernetes1.27-1.27.6-150400.9.3.2 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination. SUSE-RU-2023:4366-1: An update that contains one feature and has two fixes can now be installed. Category: recommended (moderate) Bug References: 1213829, 1214406 Jira References: PED-5839 Sources used: openSUSE Leap 15.4 (src): kubernetes1.26-1.26.9-150400.9.3.2 openSUSE Leap 15.5 (src): kubernetes1.26-1.26.9-150400.9.3.2 Containers Module 15-SP4 (src): kubernetes1.26-1.26.9-150400.9.3.2 Containers Module 15-SP5 (src): kubernetes1.26-1.26.9-150400.9.3.2 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination. SUSE-RU-2023:4365-1: An update that contains one feature and has one fix can now be installed. Category: recommended (moderate) Bug References: 1214406 Jira References: PED-5839 Sources used: Containers Module 15-SP4 (src): kubernetes1.25-1.25.14-150400.9.3.2 Containers Module 15-SP5 (src): kubernetes1.25-1.25.14-150400.9.3.2 openSUSE Leap 15.4 (src): kubernetes1.25-1.25.14-150400.9.3.2 openSUSE Leap 15.5 (src): kubernetes1.25-1.25.14-150400.9.3.2 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination. SUSE-RU-2023:4498-1: An update that has one fix can now be installed. Category: recommended (moderate) Bug References: 1214406 Sources used: openSUSE Leap 15.4 (src): kubernetes1.24-1.24.17-150400.9.11.2 Containers Module 15-SP4 (src): kubernetes1.24-1.24.17-150400.9.11.2 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination. SUSE-RU-2023:4497-1: An update that has one fix can now be installed. Category: recommended (moderate) Bug References: 1214406 Sources used: openSUSE Leap 15.5 (src): kubernetes1.24-1.24.17-150500.3.10.2 Containers Module 15-SP5 (src): kubernetes1.24-1.24.17-150500.3.10.2 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination. |
1. The Kubernetes package on TW seems to require cri-o as runtime dependency, however, it requires containerd. Thus, - cri-o should not be installed as a dependency - containerd had to be installed enabled and started (containerd doesn't have an rccontainerd link) 2. kubelet.service does not start: It complains about the option `--container-runtime` not being known. This option is pulled in from /etc/sysconfig/kubelet. It needs to be deleted from this file. It is not clear who has created this file and which package owns it. The rpm database does not know it. Since it is pulled in by ..../systemd/system/kubelet.service.d/10-kubeadm.conf I suspect it comes from kubernetes1.27-kubeadem but it's hard to tell where it is created - at least not in a post-install script. It would be good if the file was '%ghost' in the %files section of the package it originates from. The same is true for the environment file /var/lib/kubelet/kubeadm-flags.env which is written by `kubeadm init`. 3. 'kubeadm init' checks if the kubelet service is enabled. 4. Moreover, it requires a reboot to start all dependent services and load all required kernel modules: - br_netfilter needs to be loaded - containerd needs to be started This was fine for kubic these packages were originally meant for as this had transactional updates and required a reboot. On a 'regular' system where this is not the case, these modules should probably be loaded and services be started from a 'postinstall' script.