Bugzilla – Bug 1218694
[k8s,kube-flannel] YAML File to set up Flannel Network Add-On is outdated
Last modified: 2024-02-26 11:24:44 UTC
After successfully setting up k8s (using `kubeadm init`), the coredns pod remains in a 'Pending' state: NAMESPACE NAME READY STATUS RESTARTS AG kube-system coredns-86ccd44ff8-24h9w 0/1 Pending 0 12 kube-system coredns-86ccd44ff8-rz5vq 0/1 Pending 0 12 kube-system etcd-k8s-head 1/1 Running 1 12 kube-system kube-apiserver-k8s-head 1/1 Running 1 12 kube-system kube-controller-manager-k8s-head 1/1 Running 1 12 kube-system kube-proxy-qprnj 1/1 Running 0 12 kube-system kube-scheduler-k8s-head 1/1 Running 1 13 This is due to a not yet installed network plugin. When installing `kubeadm` thru 'patterns-kubernetes-kubeadm', the package 'flannel-k8s-yaml' is installed alongside which contains a yaml file to set up Flannel as network plugin. Running `kubectl apply -f /usr/share/k8s-yaml/flannel/kube-flannel.yaml` using the supplied file fails, however after: | clusterrole.rbac.authorization.k8s.io/flannel created | clusterrolebinding.rbac.authorization.k8s.io/flannel created | serviceaccount/flannel created | configmap/kube-flannel-cfg created | daemonset.apps/kube-flannel-ds created with the error: | error: resource mapping not found for name: "psp.flannel.unprivileged" namespace: "" from "/usr/share/k8s-yaml/flannel/kube-flannel.yaml": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1" ensure CRDs are installed first When using the yaml file from https://github.com/flannel-io/flannel/blob/master/Documentation/kube-flannel.yml (wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml) it works, however, the flannel pod is still cycling thru a CrashLoopBackOff - it fails with (from log): | E0110 11:46:48.988799 1 main.go:332] Error registering network: failed to acquire lease: node "k8s-head" pod cidr not assigned
Thanks for the ticket, @Egbert Eich! For the crashing flannel pods: flannel, by default, expects the podCIDR to be set to "10.244.0.0/16"[2] at the time of "kubeadm init": > sudo kubeadm init --pod-network-cidr 10.244.0.0/16 (Or in case of custom podCIDR, same needs to be updated in the kube-flannel.yaml[1] file as well: ``` net-conf.json: | { "Network": "<customCIDR>", "Backend": { "Type": "vxlan" } } ``` ) [1] https://github.com/flannel-io/flannel/tree/master?tab=readme-ov-file#deploying-flannel-with-kubectl [2] https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml ----- For the coreDNS pods, they get stuck in "pending/containerCreating" state due to missing CNI plugins in /opt/cni/bin path. ``` > k describe pod coredns-86ccd44ff8-hz2lj -n kube-system Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 14s default-scheduler Successfully assigned kube-system/coredns-86ccd44ff8-hz2lj to 5 Warning FailedCreatePodSandBox 13s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "de201303e9a5802dbae988d668bed2640315da55b4654c2cdda3e5cffeec8ab4": plugin type="loopback" failed (add): failed to find plugin "loopback" in path [/opt/cni/bin] Normal SandboxChanged 13s kubelet Pod sandbox changed, it will be killed and re-created. ``` Flannel doesn't add these CNI binaries into the /opt/cni/bin path[3] Adding the binaries manually in the required path, fixes it: > mkdir -p /opt/cni/bin > curl -O -L https://github.com/containernetworking/plugins/releases/download/v1.4.0/cni-plugins-linux-amd64-v1.4.0.tgz > sudo tar -C /opt/cni/bin -xzf cni-plugins-linux-amd64-v1.4.0.tgz [3]https://github.com/flannel-io/flannel/issues/1715 I'll send an update to kube-flannel package for above.
The flaw of k8s is that it is extremely complex and this seems to be getting worse every time I look at it. The solution above may be acceptable for Tumbleweed, but it is not great: letting people download binaries from some place on the internet is not why people use distributions - it is definitely a security issue. If these plugins are required, they should be packaged and the package should be required by patterns-kubernetes-kubeadm. Pre-installing these plugins removes some flexibility, but the user who wants this can do things the hard way. k3s and RKE2 also make this choice for the user - and people love it.
cni-plugins[1] is already packaged and is being used, jfyi [1] - https://build.opensuse.org/package/show/openSUSE:Factory/cni-plugins
(In reply to Danish Prakash from comment #3) > cni-plugins[1] is already packaged and is being used, jfyi > > [1] - https://build.opensuse.org/package/show/openSUSE:Factory/cni-plugins Indeed! This installs the plugins into /usr/lib(exec)/cni. I'm not sure if they are found there, though, as kubelet seems to expect them in /opt/cni/bin/. Otherwise, the 'cni-plugins' package should probably be added to the dependencies somewhere. I'll probably give this a test tonight. @Pryanka, Danish, thank you for all the hints!
Thanks for pointing to the "cni-plugins" package, @Danish Prakash. > Indeed! This installs the plugins into /usr/lib(exec)/cni. > I'm not sure if they are found there, though, as kubelet seems to expect them in /opt/cni/bin/. > Otherwise, the 'cni-plugins' package should probably be added to the dependencies somewhere. @Egbert Eich, changing "bin_dir" value in "/etc/containerd/config.toml", from "/opt/cni/bin" -> "/use/libexec/cni" will change the search path for cni plugins. ``` [plugins."io.containerd.grpc.v1.cri".cni] bin_dir = "/usr/libexec/cni" ``` But then the default upstream "kube-flannel.yml" will also need updates in the init-containers[1] that install "flannel" binary to /opt/cni/bin [1] https://github.com/flannel-io/flannel/blob/master/Documentation/kube-flannel.yml#L140-L205 Will patching the "kube-flannel.yml" file for ^ in the flannel package help? Or something else will be more ideal?
Factory flannel package is up-to-date with upstream now - https://build.opensuse.org/request/show/1143027