Bug 1218694

Thanks for the ticket, @Egbert Eich!

For the crashing flannel pods:

flannel, by default, expects the podCIDR to be set to "10.244.0.0/16"[2] at the time of "kubeadm init":

> sudo kubeadm init --pod-network-cidr 10.244.0.0/16

(Or in case of custom podCIDR, same needs to be updated in the kube-flannel.yaml[1] file as well:

```
  net-conf.json: |
    {
      "Network": "<customCIDR>",
      "Backend": {
        "Type": "vxlan"
      }
    }
```
)

[1] https://github.com/flannel-io/flannel/tree/master?tab=readme-ov-file#deploying-flannel-with-kubectl
[2] https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml


-----

For the coreDNS pods, they get stuck in "pending/containerCreating" state due to missing CNI plugins in /opt/cni/bin path.

```
> k describe pod coredns-86ccd44ff8-hz2lj -n kube-system

Events:
  Type     Reason                  Age   From               Message
  ----     ------                  ----  ----               -------
  Normal   Scheduled               14s   default-scheduler  Successfully assigned kube-system/coredns-86ccd44ff8-hz2lj to 5
  Warning  FailedCreatePodSandBox  13s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "de201303e9a5802dbae988d668bed2640315da55b4654c2cdda3e5cffeec8ab4": plugin type="loopback" failed (add): failed to find plugin "loopback" in path [/opt/cni/bin]
  Normal   SandboxChanged          13s   kubelet            Pod sandbox changed, it will be killed and re-created.
```

Flannel doesn't add these CNI binaries into the /opt/cni/bin path[3]

Adding the binaries manually in the required path, fixes it:

> mkdir -p /opt/cni/bin
> curl -O -L https://github.com/containernetworking/plugins/releases/download/v1.4.0/cni-plugins-linux-amd64-v1.4.0.tgz
> sudo tar -C /opt/cni/bin -xzf cni-plugins-linux-amd64-v1.4.0.tgz

[3]https://github.com/flannel-io/flannel/issues/1715

I'll send an update to kube-flannel package for above.

The flaw of k8s is that it is extremely complex and this seems to be getting worse every time I look at it.
The solution above may be acceptable for Tumbleweed, but it is not great: letting people download binaries from some place on the internet is not why people use distributions - it is definitely a security issue.
 If these plugins are required, they should be packaged and the package should be required by patterns-kubernetes-kubeadm. Pre-installing these plugins removes some flexibility, but the user who wants this can do things the hard way.
k3s and RKE2 also make this choice for the user - and people love it.

cni-plugins[1] is already packaged and is being used, jfyi

[1] - https://build.opensuse.org/package/show/openSUSE:Factory/cni-plugins

(In reply to Danish Prakash from comment #3)
> cni-plugins[1] is already packaged and is being used, jfyi
> 
> [1] - https://build.opensuse.org/package/show/openSUSE:Factory/cni-plugins

Indeed! This installs the plugins into /usr/lib(exec)/cni. 
I'm not sure if they are found there, though, as kubelet seems to expect them in /opt/cni/bin/.
Otherwise, the 'cni-plugins' package should probably be added to the dependencies 
somewhere.
I'll probably give this a test tonight.

@Pryanka, Danish, thank you for all the hints!

Thanks for pointing to the "cni-plugins" package, @Danish Prakash.

> Indeed! This installs the plugins into /usr/lib(exec)/cni. 
> I'm not sure if they are found there, though, as kubelet seems to expect them in /opt/cni/bin/.
> Otherwise, the 'cni-plugins' package should probably be added to the dependencies somewhere.


@Egbert Eich, changing "bin_dir" value in "/etc/containerd/config.toml", from "/opt/cni/bin" -> "/use/libexec/cni" will change the search path for cni plugins.

```
    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/usr/libexec/cni"

```

But then the default upstream "kube-flannel.yml" will also need updates in the 
init-containers[1] that install "flannel" binary to /opt/cni/bin


[1] https://github.com/flannel-io/flannel/blob/master/Documentation/kube-flannel.yml#L140-L205


Will patching the "kube-flannel.yml" file for ^ in the flannel package help? Or something else will be more ideal?

Factory flannel package is up-to-date with upstream now - https://build.opensuse.org/request/show/1143027