Bug 1215903 (broken, cloud, digitalocean, init, MicroOS:, on)

Summary: MicroOS: current openstack image not allowing ssh keys to be added on digitalocean
Product: [openSUSE] openSUSE Tumbleweed Reporter: Johannes Hölzel <freelancing>
Component: MicroOSAssignee: Forgotten User u0-bnvADNc <forgotten_u0-bnvADNc>
Status: NEW --- QA Contact: E-mail List <qa-bugs>
Severity: Major    
Priority: P1 - Urgent CC: k.kienemann, sb56637
Version: Current   
Target Milestone: ---   
Hardware: x86-64   
OS: Mac   
URL: https://download.opensuse.org/tumbleweed/appliances/openSUSE-MicroOS.x86_64-OpenStack-Cloud.qcow2
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Johannes Hölzel 2023-10-03 12:58:54 UTC
Since downloading the latest image from the microos page or updating my 2 month old image using zypper dup, the host is ignoring its SSH Keys.

What happened:
- The host becomes unavaiable since the ssh keys are invalidated:
Permission denied (publickey,keyboard-interactive). 

What have you tried:

Different cloud-init setups with adding a user, adding extra keys and more like these in various configurations:

--------------------------------------------------------------
cloud key provided SSH Key + 
#cloud-config
ssh_authorized_keys:
  - ssh-ed25519 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/P5 vscode@d015ae7a3097
ssh_deletekeys: false
disable_root: true
users:
  - name: deploy
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh-authorized-keys:
      - ssh-ed25519 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/P5 vscode@d015ae7a3097
deploy@...: Permission denied (publickey,keyboard-interactive).   
root@...: Permission denied (publickey,keyboard-interactive).		  
--------------------------------------------------------------

sometimes even with minor achievements like these:

--------------------------------------------------------------
cloud ssh key + 
#cloud-config
ssh_authorized_keys:
  - ssh-ed25519 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/P5 vscode@d015ae7a3097
ssh_deletekeys: true
users:
  - name: deploy
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh-authorized-keys:
      - ssh-ed25519 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/P5 vscode@d015ae7a3097

----> 
deploy can login root can not.
root@...: Permission denied (publickey,keyboard-interactive).<-- added twice.
-------------------------------------------------------------------


But no permanent success has been achieved.

What am I trying to do:
I am using cloud-init in combination with terraform to provision servers on digitalocean, yet even only providing the samples above has not been successfull.

When did the change start happening:
Over this weekend. Last week i as happily playing with the auto updating functionalties for a production workload, this morning i returned to unavailable machines. This week it even occurs if i freshly download an image from the microOS portal.

Source of my image:
https://en.opensuse.org/Portal:MicroOS
Comment 1 Johannes Hölzel 2023-10-04 06:58:32 UTC
I also took the liberty to check the so called "digialocean image" which was working even worse than the regular opentsack one.

( https://download.opensuse.org/tumbleweed/appliances/openSUSE-MicroOS.x86_64-DigitalOcean-Cloud.qcow2 )

Simply using the image without cloud-init returned this

```
ssh root@...
(root@164.90.130.104) "System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8)."
Password: 
```

Whilst just providing another ssh key with this:

```
ssh_authorized_keys:
  - ssh-ed25519 x/P5 x@x
```

left me with

```
ssh: connect to host x.x.x.x port 22: Connection refused    
```

Help please, i dont know what else to do at this point
Comment 2 Klaas Kienemann 2023-10-06 07:05:53 UTC
Same here - I've rolled back to MicroOS 20230927 and deactivated automatic transactional updates:

systemctl --now disable transactional-update.timer

sshd log output:

"User root not allowed because account is locked"

I suspect it is related to the libssh-config update in MicroOS 20230929. AFAIK cloud-init writes custom ssh configuration to /etc/ssh/sshd_config and libssh (libssh_server.config) does not parse this file any more.

https://build.opensuse.org/request/show/1113627

I would like to enable automatic transactional updates back again. Are there any workarounds for this?
Comment 3 Johannes Hölzel 2023-10-06 08:05:27 UTC
I think there is even more to it. I have currently adapted all my systems to use cloud-init but i can not even get rke2 to work at all.

I checked the audit logs for selinux denies but there were none.

Something has seriously crippled root and currently there is no way to run rke2 rootless...

my current workaround is to deploy a deploy user with cloud-init and the old version then updated right with dup after.

```
users:
  - name: ${var.deploy_user}
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - ${var.SSH_pub_key}
```
Comment 4 Johannes Hölzel 2023-10-06 09:50:28 UTC
so root is locked and the update seems to uninstall container-selinux and consecutively rke2-selinux.

I added both manually into my toolchain and now im up and running again.
So to sum up the workaround is:

- deploy deploy user with cloud-init
- install selinux packages separatetly

I guess pulling container selinux is fair when using the "Base System" image.
Also i have seen that there has been a new iso provided on the 4th so i will check that now too.

If anything this shows that we need a dedicated changelog for MicroOS and need to stop treating it as a component of tumbleweed.
Comment 5 S. B. 2023-12-16 17:55:18 UTC
The `openSUSE-MicroOS.x86_64-OpenStack-Cloud.qcow2` image is working fine with my SSH keys on DigitalOcean.