Forgot to mention - this problem is biting TW as well, of course.

Hi eich@suse.com, I've submitted PR[1] to bump/add the release version markers for v1.18 - v1.27, aiming to update https://kubic.opensuse.org/release.

Once PR[1] is merged, I'll check if the problem with downloading container images still persists at registry.opensuse.org/kubic/kube-* and update accordingly. Thanks!

[1] https://github.com/kubic-project/kubic-o-o/pull/47

@Priyanka - thanks for looking into this so swiftly :)

I've seen that you've removed the leading 'v' from the version to make it more compatible with upstream.

I do wonder if this cannot be automated - i.e. generated by a script using registry data and triggered by OBS when a new container for kubic/* gets published. 

One more note: the 'upstream' version of this (from https://dl.k8s.io/release) does not have a trailing newline. I did not see any issues with it, though.

@Priyanka - since Richard has responded - after a gentle prod - I was able to test it. 

I'm afraid we are not quite there, yet:

* Issues on Leap 15.5:
  Since we advertise version 1.24.16, we should have the containers for it.
  Yet, 
    registry.opensuse.org/kubic/kube-apiserver:v1.24.16
    registry.opensuse.org/kubic/kube-controller-manager:v1.24.16
    registry.opensuse.org/kubic/kube-scheduler:v1.24.16
    registry.opensuse.org/kubic/kube-proxy:v1.24.16
  are not available on the registry, yet.

* Issues on TW (with latest kubernetes 1.27.4:
  Here the following containers are missing from the registry:
  registry.opensuse.org/kubic/pause:3.9
  registry.opensuse.org/kubic/etcd:3.5.7-0

There's also an issue with /usr/bin/kubelet (-> separate ticket).

Thanks!

eich@suse.com, thanks a lot for testing and listing all the issues - really helpful!

I'll look into how to build/push the new missing container image versions on the registry (including pause & etcd).

> There's also an issue with /usr/bin/kubelet (-> separate ticket).

Ack. Will take a look. Thanks again!

(In reply to Priyanka Saggu from comment #5)
> eich@suse.com, thanks a lot for testing and listing all the issues - really
> helpful!
> 
> I'll look into how to build/push the new missing container image versions on
> the registry (including pause & etcd).
> 

That's much appreciated, thank you!

> > There's also an issue with /usr/bin/kubelet (-> separate ticket).
> 
> Ack. Will take a look. Thanks again!

I had some time left so I've pushed a fix for this, maybe you can have a look. Thank you!

@Priyanka:
I've looked into this a bit - it seems like devel:kubic:containers/kubic-pause-image needs the 3.9 tag in:
https://build.opensuse.org/package/view_file/devel:kubic:containers/kubic-pause-image/kubic-pause-image.kiwi?expand=1

I'm not sure what happened to the rest, 
devel:kubic/etcd-for-k8s1.27 should probably be updated it looks like the package 
openSUSE:Factory/kubernetes needs to be updated to the new version of etcd:
# etcdversion - version of etcd
%define etcdversion 3.5.7
# etcdversionminus1 - version of etcd for versionminus1
%define etcdversionminus1 3.5.6
and possibly others.
I'm not sure why containers for the newer version of kubernetes1.24 are not being published, though.

It looks like we only publish containers for the latest version of kubernetes sub-packages - ie, once kubernetes1.N becomes available, we stop publishing containers for kubernetes1.N-1 sub-packages while upstream seems to support multiple versions of kubernetes.
This explains why 
    registry.opensuse.org/kubic/kube-apiserver:v1.24.16
    registry.opensuse.org/kubic/kube-controller-manager:v1.24.16
    registry.opensuse.org/kubic/kube-scheduler:v1.24.16
    registry.opensuse.org/kubic/kube-proxy:v1.24.16
and for this matter any kubernetes subpackages of kubernetes beyond 1.24.3.

To mitigate this, we most likely need to have separate versioned container packages 
like:
 openSUSE:Containers:Tumbleweed/kubic-kube-apiserver-image1.N
   with:
         <package name="kubernetes1.N-apiserver"/>
 openSUSE:Containers:Tumbleweed/kube-controller-manager1.N
   with:
         <package name="kubernetes1.N-controller-manager"/>
 openSUSE:Containers:Tumbleweed/kubic/kube-scheduler1.N
   with:
        <package name="kubernetes1.N-scheduler"/>
 openSUSE:Containers:Tumbleweed/kube-proxy1.N
   with:
        <package name="kubernetes1.N-proxy"/>
Also the currently 'unversioned' dependency package would be required 'per version'.
An easier way would be to not announce the latest 'upstream' versions through  https://kubic.opensuse.org/release/stable-1.<N>.txt but the latest for which a container is available. However, if the later releases were security-related, this may not be the way to go.
Unfortunately. right now, the efforts to maintain separate code stream of kubernetes are largely in vain as the registry does not require the needed containers. It would still allow users to build their own containers - but who would seriously consider that?

Hello eich@suse.com, apologies for the delay. I got caught up in resolving another issue upstream.

Addressing below each point separately, but just want to mention that I am also relatively new to Kubernetes packaging with O/IBS. So, I have been going back and forth to gather context & still be missing information fully.

> It looks like we only publish containers for the latest version of kubernetes sub-packages - ie, once kubernetes1.N becomes available, we stop publishing containers for kubernetes1.N-1 sub-packages while upstream seems to support multiple versions of kubernetes.

Based on my conversation with the folks maintaining automation behind openSUSE registry images, current situation is:

We are build/push-ing container images for the *last two patch release versions* of the *latest k8s minor releases -> N and N-1*, to registry.opensuse.org.

So, as of now, for N (v1.27), we have the following image tags available
(same for kube-proxy, kube-controller-manager, & kube-scheduler):

    registry.opensuse.org/kubic/kube-apiserver:v1.27.4
    registry.opensuse.org/kubic/kube-apiserver:v1.27.3

And for N-1 (v1.26), we have:

    registry.opensuse.org/kubic/kube-apiserver:v1.26.7
    registry.opensuse.org/kubic/kube-apiserver:v1.26.6

source kiwi files for ^ container images are here:
[1] https://build.opensuse.org/package/show/devel:kubic:containers/kubic-kube-apiserver-image
[2] https://build.opensuse.org/package/show/devel:kubic:containers/kubic-kube-apiserver-minus1-image

we currently don't have any new tags available for k8s versions beyond N-1 (v1.26), because the above source kiwi files are no longer tracking them.

---

Regarding `kubic-pause-image`, yes, it seems that adding another tag (3.9) in kiwi file[3] is required, and versions of kuberetes-pause[4] package need to be bumped (but I'm still digging further on pause image).

[3] https://build.opensuse.org/package/view_file/devel:kubic:containers/kubic-pause-image/kubic-pause-image.kiwi?expand=1
[4] https://build.opensuse.org/package/show/devel:kubic/kubernetes-pause

---

As for `devel:kubic/etcd-for-k8s1.27`, it appears to be up-to-date based on the dependencies[5] from upstream project. The meta (unversioned) kubernetes packages have the etcd values[5][6] set properly. 

[5] etcdversion (wrt v1.27.4) - https://github.com/kubernetes/kubernetes/blob/v1.27.4/build/dependencies.yaml#L63
[6] etcdversionminus1 (wrt v1.26.7) -https://github.com/kubernetes/kubernetes/blob/v1.26.7/build/dependencies.yaml#L56

---

I totally agree that the current situation of maintaining multiple versions of Kubernetes packaging in OBS needs improvement on multiple levels, to make it barely usable.

For starters, one suggestion I received is to consider moving the k8s container images to registry.suse.com and utilizing `BCI-dockerfile-generator`[7] for that purpose.
I am currently exploring this option.

[7] https://github.com/SUSE/BCI-dockerfile-generator

Hello Priyanka, let me also address the different topics individually (for brevity reasons I've deleted some lines) -

(In reply to Priyanka Saggu from comment #9)
> Hello eich@suse.com, apologies for the delay. I got caught up in resolving
> another issue upstream.
> 
> Addressing below each point separately, but just want to mention that I am
> also relatively new to Kubernetes packaging with O/IBS. So, I have been
> going back and forth to gather context & still be missing information fully.

That's fine. I've been with this company for quite a while, now, and have
handled a lot of difficult packaging issues and still don't know everything
- this is true in particular around building containers.
 
> we currently don't have any new tags available for k8s versions beyond N-1
> (v1.26), because the above source kiwi files are no longer tracking them.

Exactly, and here lies the problem: we still update older code streams (ie k8s minor versions) - must likely to fix security vulnerabilities - but we do not push updated containers for these. So, despite the effort that went into maintaining all these different code streams - they are not consumable (or if they were, users would potentially still get vulnerable containers).
This disadvantages the openSUSE Leap users in particular since they are stuck with
kubernetes1.24.

> ---
> 
> Regarding `kubic-pause-image`, yes, it seems that adding another tag (3.9)
> in kiwi file[3] is required, and versions of kuberetes-pause[4] package need
> to be bumped (but I'm still digging further on pause image).
> 
> [3]
> https://build.opensuse.org/package/view_file/devel:kubic:containers/kubic-
> pause-image/kubic-pause-image.kiwi?expand=1
> [4] https://build.opensuse.org/package/show/devel:kubic/kubernetes-pause

In my understanding, only the former needs an additional 'Tag' while the latter may
remain the same: pause is really a simple application - and it seems we have our own, which has seen fewer updates than upstream. I have not done any research on
'pause' - would it make sense to sync (the sources) with upstream?

> ---
> 
> As for `devel:kubic/etcd-for-k8s1.27`, it appears to be up-to-date based on
> the dependencies[5] from upstream project. The meta (unversioned) kubernetes
> packages have the etcd values[5][6] set properly. 
> 
> [5] etcdversion (wrt v1.27.4) -
> https://github.com/kubernetes/kubernetes/blob/v1.27.4/build/dependencies.
> yaml#L63
> [6] etcdversionminus1 (wrt v1.26.7)
> -https://github.com/kubernetes/kubernetes/blob/v1.26.7/build/dependencies.
> yaml#L56
> 

Indeed. There may have been a different issue involved that is unrelated to the package and the available containers that I was not able to get the correct version - or I was just confused.

> ---
> 
> I totally agree that the current situation of maintaining multiple versions
> of Kubernetes packaging in OBS needs improvement on multiple levels, to make
> it barely usable.
> 
> For starters, one suggestion I received is to consider moving the k8s
> container images to registry.suse.com and utilizing
> `BCI-dockerfile-generator`[7] for that purpose.
> I am currently exploring this option.
> 
> [7] https://github.com/SUSE/BCI-dockerfile-generator

Using BCI-dockerfile-generator might be an option, but I'm skeptical about moving to registry.suse.com:
1. It may be a good idea to keep enterprise and openSUSE activities separate and 
   let the former be derived from the latter. This gives openSUSE more 
   independence.
2. registry.opensuse.com comes with a lot more strings attached. For once, you need 
   to build in IBS.
3. At least currently, SLE users do not get any of the kubernetes packages (except 
   kubernetesXX-client) - not even through PackageHub. Thus, none of them will be 
   able to consume these containers.
4. There's no browsable interface for registry.suse.de - at least I haven't found 
   one, yet.
5. I don't see how this would resolve the problems at hand - we would still have to 
   fix them, wouldn't we?
Using registry.suse.com should not be a prerequisite to use the dockerfile generator, so you may still use it while staying on registry.opensuse.org.

(In reply to Egbert Eich from comment #10)
> > ---
> > 
> > I totally agree that the current situation of maintaining multiple versions
> > of Kubernetes packaging in OBS needs improvement on multiple levels, to make
> > it barely usable.
> > 
> > For starters, one suggestion I received is to consider moving the k8s
> > container images to registry.suse.com and utilizing
> > `BCI-dockerfile-generator`[7] for that purpose.
> > I am currently exploring this option.
> > 
> > [7] https://github.com/SUSE/BCI-dockerfile-generator
> 
> Using BCI-dockerfile-generator might be an option, but I'm skeptical about
> moving to registry.suse.com:
> 1. It may be a good idea to keep enterprise and openSUSE activities separate
> and 
>    let the former be derived from the latter. This gives openSUSE more 
>    independence.
> 2. registry.opensuse.com comes with a lot more strings attached. For once,
> you need 
>    to build in IBS.
> 3. At least currently, SLE users do not get any of the kubernetes packages
> (except 
>    kubernetesXX-client) - not even through PackageHub. Thus, none of them
> will be 
>    able to consume these containers.
> 4. There's no browsable interface for registry.suse.de - at least I haven't
> found 
>    one, yet.
> 5. I don't see how this would resolve the problems at hand - we would still
> have to 
>    fix them, wouldn't we?
> Using registry.suse.com should not be a prerequisite to use the dockerfile
> generator, so you may still use it while staying on registry.opensuse.org.

bci-dockerfile-generator is *not* bound to publishing to registry.suse.com. In fact, all the bci/ prefixed container images on registry.opensuse.org have been produced by this project and are build on Tumbleweed. It is really just a tool to automatically create containers on OBS.

(In reply to Dan Čermák from comment #11)

> > Using registry.suse.com should not be a prerequisite to use the dockerfile
> > generator, so you may still use it while staying on registry.opensuse.org.
> 
> bci-dockerfile-generator is *not* bound to publishing to registry.suse.com.
> In fact, all the bci/ prefixed container images on registry.opensuse.org
> have been produced by this project and are build on Tumbleweed. It is really
> just a tool to automatically create containers on OBS.

Thanks for confirming! This allows us to decouple things and get the problem at hand moving.

Thank you, Egbert and Dan, for providing clarification.

Based on the discussion, following are my next steps:

- Update the Kubernetes meta (unversioned) package by adding `*minus-two` and `*minus-three` packages for all server packages (kube-apiserver, kube-scheduler, kube-proxy, kube-controller) and respective etcd-for-k8s*, coredns-for-k8s* packages.

  This update will then make the latest four Kubernetes minor releases (supported upstream) available in the meta package itself.

- Add a new tag (3.9) to the `kubic-pause-image` kiwi source. Have decided to update the `kubernetes-pause` package to align it with the pause image version wrt latest k8s version.

- Add two more kiwi sources - `kubic-kube-*-minus2-image` and `kubic-kube-*-minus3-image`, which will then start building/pushing container images tags for the `n-2` and `n-3` k8s minor versions as well.

- Manually test `kubeadm init`.

  If the test is successful, proceed with the next steps.
  If not, address and fix any issues that arise.

---

After completing above steps, I will collaborate with a member from the OpenQA team to re-enable the single node kubeadm testing[1].

[1] https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/5ae356b8e8a720276a9b329832837dce92c33772/tests/console/kubeadm.pm

---

Then - explore how to enhance the overall Kubernetes packaging for future improvements.

Updates:

> - Update the Kubernetes meta (unversioned) package by adding `*minus-two` and `*minus-three` packages for all server packages (kube-apiserver, kube-scheduler, kube-proxy, kube-controller) and respective etcd-for-k8s*, coredns-for-k8s* packages.

[1] SR: https://build.opensuse.org/request/show/1102700

---

> - Add a new tag (3.9) to the `kubic-pause-image` kiwi source. Have decided to update the `kubernetes-pause` package to align it with the pause image version wrt latest k8s version.

[2] SR: https://build.opensuse.org/request/show/1102701
[3] SR: https://build.opensuse.org/request/show/1102685

> - Add two more kiwi sources - `kubic-kube-*-minus2-image` and `kubic-kube-*-minus3-image`, which will then start building/pushing container images tags for the `n-2` and `n-3` k8s minor versions as well.



[4] kubic-kube-apiserver-minus2-image ~ https://build.opensuse.org/request/show/1102680
[5] kube-kubic-apiserver-minus3-image ~ https://build.opensuse.org/request/show/1102679

[6] kubic-kube-controller-manager-minus2-image ~ https://build.opensuse.org/request/show/1102683
[7] kubic-kube-controller-manager-minus3-image ~ https://build.opensuse.org/request/show/1102682

[8] kubic-kube-scheduler-minus2-image ~ https://build.opensuse.org/request/show/1102694
[9] kubic-kube-scheduler-minus3-image ~ https://build.opensuse.org/request/show/1102695

[10] kubic-kube-proxy-minus2-image ~ https://build.opensuse.org/request/show/1102697
[11] kubic-kube-proxy-minus3-image ~ https://build.opensuse.org/request/show/1102696

[12] kubic-etcd-minus2-image ~ https://build.opensuse.org/request/show/1102699
[13] kubic-etcd-minus3-image ~ https://build.opensuse.org/request/show/1102698

[14] kubic-coredns-minus2-image ~ https://build.opensuse.org/request/show/1102693
[15] kubic-coredns-minus3-image ~ https://build.opensuse.org/request/show/1102692

With below Factory SR(s) merged now, hopefully (FWIU), it will start triggering container image build/publish:


- kubic-kube-apiserver-minus2-image ~ https://build.opensuse.org/request/show/1102853
- kubic-kube-apiserver-minus3-image ~ https://build.opensuse.org/request/show/1102854
- kubic-kube-controller-manager-minus2-image ~ https://build.opensuse.org/request/show/1102855
- kubic-kube-controller-manager-minus3-image ~ https://build.opensuse.org/request/show/1102856
- kubic-kube-proxy-minus2-image ~ https://build.opensuse.org/request/show/1102857
- kubic-kube-proxy-minus3-image ~ https://build.opensuse.org/request/show/1102858
- kubic-kube-scheduler-minus2-image ~ https://build.opensuse.org/request/show/1102859
- kubic-kube-scheduler-minus3-image ~ https://build.opensuse.org/request/show/1102860
- kubic-etcd-minus2-image ~ https://build.opensuse.org/request/show/1102861
- kubic-etcd-minus3-image ~ https://build.opensuse.org/request/show/1102862
- kubic-coredns-minus2-image ~ https://build.opensuse.org/request/show/1102863
- kubic-coredns-minus3-image ~ https://build.opensuse.org/request/show/1102864


I'll check if the new tags start to show up by tomorrow my morning (Aug 9, 2023 - 10:00 AM IST) & report back. Thanks!

Update:

New container image tags w.r.t to the latest Kubernetes patch version releases (v1.24 - v1.27) are now available on registry.opensuse.org. Thanks!