Bug 1217220 - [SELinux] current MicroOS doesn't allow ssh login
Summary: [SELinux] current MicroOS doesn't allow ssh login
Status: RESOLVED FIXED
: 1217343 1217697 1217729 (view as bug list)
Alias: None
Product: openSUSE Leap Micro
Classification: openSUSE
Component: Base (show other bugs)
Version: 5.4
Hardware: Other Other
: P1 - Urgent : Major
Target Milestone: ---
Assignee: Andreas Schwab
QA Contact: E-mail List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-11-16 10:23 UTC by Johannes Segitz
Modified: 2024-01-29 13:08 UTC (History)
17 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---
jsegitz: needinfo? (systemd-maintainers)
santiago.zarate: SHIP_STOPPER? (rtsvetkov)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Johannes Segitz 2023-11-16 10:23:38 UTC
While investigating another SELinux issue I installed
openSUSE-MicroOS-DVD-x86_64-Snapshot20231114-Media.iso
with default settings. After installation and adding
PermitRootLogin yes
to the sshd config root still can't log in

setenforce 0
allows to log in. No visible AVCs. Disabling dontaudit yields
type=AVC msg=audit(1700130079.518:309): avc:  denied  { read } for  pid=1178 comm="sshd" name="shadow" dev="overlay" ino=20716 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:shadow_t:s0 tclass=file permissive=1
type=AVC msg=audit(1700130079.518:310): avc:  denied  { open } for  pid=1178 comm="sshd" path="/etc/shadow" dev="overlay" ino=20716 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:shadow_t:s0 tclass=file permissive=1
type=AVC msg=audit(1700130079.518:311): avc:  denied  { getattr } for  pid=1178 comm="sshd" path="/etc/shadow" dev="overlay" ino=20716 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:shadow_t:s0 tclass=file permissive=1
type=AVC msg=audit(1700130079.691:323): avc:  denied  { noatsecure } for  pid=1179 comm="sshd" scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=process permissive=1
type=AVC msg=audit(1700130079.691:324): avc:  denied  { siginh } for  pid=1179 comm="bash" scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=process permissive=1

upon successful login.

I'll be away in a couple of minutes an likely will not be able to work on this today
Comment 1 Filippo Bonazzi 2023-11-16 15:17:31 UTC
I can't reproduce this on a machine where I have a public key setup for root and UsePAM.

Let me try to disable either individually and combined and see if anything breaks
Comment 2 Filippo Bonazzi 2023-11-16 15:24:20 UTC
Yup, this happens only when root tries to login using a password (allowed by sshd since PermitRootLogin is set to yes). Makes sense that it would try to access shadow_t in that case.

UsePAM appears to be unrelated (this failure or success happens regardless of setting).
Comment 3 Filippo Bonazzi 2023-11-16 15:44:34 UTC
There is no difference in the output of

$ sesearch -A -s sshd_t -t shadow_t
allow domain file_type:blk_file map; [ domain_can_mmap_files ]:True
allow domain file_type:chr_file map; [ domain_can_mmap_files ]:True
allow domain file_type:file map; [ domain_can_mmap_files ]:True
allow domain file_type:lnk_file map; [ domain_can_mmap_files ]:True
allow sshd_t file_type:dir { getattr open search };
allow sshd_t file_type:filesystem getattr;

between an old MicroOS where this works (20230908) and a fully updated one where this doesn't work.

There are no permissive domains on the old system either.

Could it be an actual change in ssh or an authentication layer?
Comment 4 Johannes Segitz 2023-11-16 15:50:27 UTC
I tried with
Information for package selinux-policy-targeted:
------------------------------------------------
Repository     : @System
Name           : selinux-policy-targeted
Version        : 20231012-0

and there I see the same behavior. This must be a change in ssh or some other component.

sshd_t shouldn't need access to shadow_t, this is mediated. I'll look deeper into this today as this is pretty serious
Comment 5 Johannes Segitz 2023-11-16 16:50:35 UTC
Also happens after zypper dup from
https://download.opensuse.org/history/20231019/tumbleweed/repo/non-oss/
so this has been present for a while, which is suprising
Comment 6 Filippo Bonazzi 2023-11-16 16:53:33 UTC
Not too surprising if you consider that root login with a password is disabled by default in the *micro*s and ALP. So not many people would notice
Comment 7 Johannes Segitz 2023-11-17 14:34:54 UTC
I don't think so, that's still a pretty common config even when you have to change the default. But it seems like this is only broken for new installations, not when you update, so that probably explains it. I'm currently trying to find the reason for this
Comment 8 Johannes Segitz 2023-11-17 15:33:20 UTC
openSUSE-MicroOS-DVD-x86_64-Snapshot20230921-Media.iso works
openSUSE-MicroOS-DVD-x86_64-Snapshot20231114-Media.iso fails
so let's see what it is :)
Comment 9 Johannes Segitz 2023-11-17 16:16:00 UTC
It's a glibc update that triggers this bug

works: glibc-2.38-2.1.x86_64
fails: glibc-2.38-6.1.x86_64

I think the change is:

Mon Oct  9 08:50:18 UTC 2023 - Andreas Schwab <schwab@suse.de>

- Add systemd also to gshadow lookups (jsc#PED-5188)
- For SLE continue to use nsswitch.conf without systemd

I'll look more into this on Monday and CC'd Andreas
Comment 10 Harshvardhan Sharma 2023-11-20 11:19:44 UTC
Hi Team, facing the same issue on ALP Dolomite 1.0 Milestone 5, root login works when I only install the package 'openssh-server-config-rootlogin' and reboot or setenforce 0.
Comment 11 Johannes Segitz 2023-11-22 09:51:01 UTC
That is expected behavior. Root login is disabled by default by default. The problem here is that it doesn't work even with
PermitRootLogin yes
in the sshd config
Comment 12 Johannes Segitz 2023-11-22 10:41:56 UTC
Removing the systemd entries from nsswitch.conf allows logging in again. Strangely this does not occur on Tumbleweed even though the nsswitch settings are the same.

I think this is probably more of a systemd issue. I'll have a look upstream to see how this is handled and if there's already an issue
Comment 13 Johannes Segitz 2023-11-22 16:02:27 UTC
After playing with this a bit more the issue can be trigger with
shadow:         compat systemd
in 
/etc/nsswitch.conf

Weirdly ssh login still works if sshd is started in debug mode on the command line.

sshd always tries to read /etc/shadow, so the AVC is misleading
Comment 14 Johannes Segitz 2023-11-22 17:18:11 UTC
I don't understand the behavior of the system. AFAIU
shadow:         compat systemd
means: Try compat first, upon success end here. If this didn't succeed try systemd.

So it doesn't make sense that
shadow:         compat
works and 
shadow:         compat systemd
doesn't work for a normal system user that will be found via compat.

@Andreas: Can you please enlighten me? I don't think that this is (only) a SELinux issue.
Comment 15 Andreas Schwab 2023-11-23 09:50:57 UTC
I have no idea what systemd is doing.
Comment 16 Johannes Segitz 2023-11-23 12:31:47 UTC
It's mainly an issue with the behavior of glibc, although I'm not sure if it's a bug or intended functionality.

man nsswitch.conf

       The STATUS value is matched against the result of the lookup function called by the preceding service specification, and can be one of:

           success     No error occurred and the requested entry is returned.  The default action for this  condition
                       is "return".

So my expectation is: If a service is working on it's own then specifying an additional service doesn't result in failure. But even when I specify explicitly
shadow:         compat [success=return] systemd
the behavior is still not the same as
shadow:         compat

I've traced it down in PAM to pam_modutil_getspnam:
 54         /* make the re-entrant call to get the spwd structure */
 55         errno = 0;
 56         status = getspnam_r(user, buffer,
 57                             sizeof(struct spwd) + (char *) buffer,
 58                             length, &result);
 59         if (!status && (result == buffer)) {

If only compat is set the call to getspnam_r returns an error code. Higher in the call chain this then results in PAM_UNIX_RUN_HELPER being returned, which causes pam to call into _unix_run_verify_binary to run unix_chkpwd.

If systemd is also set then getspnam_r succeeds, but it returns the entry from systemd, which prevents root from logging in, since they have "!*" for the root entry without any additional config.

OTOH this seems like a glibc bug to me, since compat should still work as before. Then again this seems like a special case and might be an issue with PAM or systemd. 

Setting needinfo for Andreas and the systemd maintainers.
Comment 17 Filippo Bonazzi 2023-11-24 08:48:12 UTC
*** Bug 1217343 has been marked as a duplicate of this bug. ***
Comment 18 Johannes Segitz 2023-11-27 09:35:02 UTC
I checked on Fedora and they reverted this
https://github.com/authselect/authselect/commit/1e78f7e048747024a846fd22d68afc6993734e92

Assigning to Andreas to have this removed from the glibc packaging. This is rather urgent as it breaks root login on ALP.

Keeping the needinfo for the systemd maintainers, as this is something that should probably be fixed on their side later on.
Comment 19 Johannes Segitz 2023-12-01 13:26:30 UTC
*** Bug 1217729 has been marked as a duplicate of this bug. ***
Comment 20 Cathy Hu 2023-12-01 13:39:21 UTC
*** Bug 1217697 has been marked as a duplicate of this bug. ***
Comment 22 Dan Čermák 2023-12-04 17:28:45 UTC
https://build.opensuse.org/request/show/1130827
Comment 23 OBSbugzilla Bot 2023-12-05 10:35:04 UTC
This is an autogenerated message for OBS integration:
This bug (1217220) was mentioned in
https://build.opensuse.org/request/show/1130945 Factory / glibc
Comment 24 Frederic Crozat 2023-12-05 11:10:30 UTC
The upstream bug seems to be https://github.com/systemd/systemd/issues/20299
Comment 27 Felix Niederwanger 2023-12-14 14:28:01 UTC
Bug is still present in SLE-Micro.s390x-6.0-Default-qcow-Build6.1.qcow2
Comment 28 Wayne Chen 2023-12-15 08:33:33 UTC
For me, I can login to installed vm as root using ssh key. VM installed using SLE-Micro.aarch64-6.0-Default-qcow-Build6.1.qcow2 and Ignition.
Comment 29 Wayne Chen 2023-12-15 08:41:01 UTC
(In reply to Wayne Chen from comment #28)
> For me, I can login to installed vm as root using ssh key. VM installed
> using SLE-Micro.aarch64-6.0-Default-qcow-Build6.1.qcow2 and Ignition.

It should be SLE-Micro.x86_64-6.0-Default-qcow-Build6.1.qcow2
Comment 30 Johannes Segitz 2023-12-15 08:43:57 UTC
(In reply to Felix Niederwanger from comment #27)
Does this build have the updated glibc from the comment above?
Comment 31 Felix Niederwanger 2023-12-15 10:21:37 UTC
I checked again SLE-Micro.x86_64-6.0-Default-qcow-Build6.1.qcow2 and root ssh login still fails. From ausearch:

> time->Fri Dec 15 10:18:55 2023
> type=USER_LOGIN msg=audit(1702635535.543:170): pid=1341 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login acct="root" exe="/usr/sbin/sshd" hostname=? addr=fee8::1 terminal=ssh res=failed'

Wayne Chen, can you please ensure, that you have SELinux enforcing turned on?
Comment 33 Filippo Bonazzi 2023-12-15 11:16:26 UTC
> For me, I can login to installed vm as root using ssh key

SSH login using a SSH key has never been broken in the context of this issue, and works normally. This issue is about root login with a password.
Comment 38 Martin Loviska 2024-01-12 14:00:25 UTC
Hello, 

I had to remove selinux kernel parameters in order to enable ssh access for root. Also, I have set *PermitRootLogin yes* in /etc/ssh/sshd_config.
This is crucial for testing zKVM images in openQA as the backend relies on it.

#### Tested image 

SLE-Micro.s390x-6.0-Default-qcow-Build8.48.qcow2

#### Packages

glibc-2.38-3.2.s390x
libssh-config-0.10.5-3.76.s390x
libssh4-0.10.5-3.76.s390x
libssh2-1-1.11.0-2.4.s390x

### Kernel params

root=UUID=9f3ab8ba-ee8f-42a5-8bdf-d4e2e45f5194 rd.timeout=60 rd.retry=45 hvc_iucv=8 TERM=dumb
Comment 39 Johannes Segitz 2024-01-15 07:47:32 UTC
Current version for ALP is glibc-2.38-3.4. Probably the image wasn't refreshed. Please try again with the next release

Dolomite 1.0 M8 & SLE Micro Public Beta release : Jan 30
Comment 41 Jose Lausuch 2024-01-29 13:08:43 UTC
With the latest build 10.15 I was able to login with root and password after installing openssh-server-config-rootlogin package without touching selinux policy.

> ssh root@192.168.1.107
> (root@192.168.1.107) Password: 
> Last login: Mon Jan 29 12:57:57 UTC 2024 on tty1
> Activate the web console with: systemctl enable --now cockpit.socket
> 
> Have a lot of fun...
> localhost:~ # 
> localhost:~ # sestatus 
> SELinux status:                 enabled
> SELinuxfs mount:                /sys/fs/selinux
> SELinux root directory:         /etc/selinux
> Loaded policy name:             targeted
> Current mode:                   enforcing
> Mode from config file:          enforcing
> Policy MLS status:              enabled
> Policy deny_unknown status:     allowed
> Memory protection checking:     actual (secure)
> Max kernel policy version:      33
> localhost:~ # 
> localhost:~ # rpm -q glibc
> glibc-2.38-3.15.x86_64
> localhost:~ # rpm -q libssh-config
> libssh-config-0.10.5-3.81.x86_64
> localhost:~ # rpm -q libssh4
> libssh4-0.10.5-3.81.x86_64
> localhost:~ # rpm -q openssh-server-config-rootlogin
> openssh-server-config-rootlogin-9.3p2-3.24.x86_64

So I consider this as fixed for ALP.