Bug 1215109 - Cockpit does not work with SELinux enforcing mode
Summary: Cockpit does not work with SELinux enforcing mode
Status: VERIFIED FIXED
Alias: None
Product: PUBLIC SUSE Linux Enterprise Micro 5.5
Classification: openSUSE
Component: Base (show other bugs)
Version: unspecified
Hardware: Other Other
: P1 - Urgent : Major
Target Milestone: ---
Assignee: Johannes Segitz
QA Contact: Jiri Srain
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-09-07 08:16 UTC by Jose Lausuch
Modified: 2024-05-29 11:39 UTC (History)
6 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jose Lausuch 2023-09-07 08:16:04 UTC
Environment:
> localhost:~ # sestatus 
> SELinux status:                 enabled
> SELinuxfs mount:                /sys/fs/selinux
> SELinux root directory:         /etc/selinux
> Loaded policy name:             targeted
> Current mode:                   enforcing
> Mode from config file:          enforcing
> Policy MLS status:              enabled
> Policy deny_unknown status:     allowed
> Memory protection checking:     requested (insecure)
> Max kernel policy version:      33


> localhost:~ # rpm -q cockpit
> cockpit-298-150500.1.1.x86_6

Latest image of SLE Micro 5.5 in staging, more concretely:
- SLE-Micro.x86_64-5.5.0-Default-Build3.11.raw.xz
- SLE-Micro-5.5-DVD-x86_64-Build19.1-Media1.iso


How to reproduce:
Start cockpit socket:
> localhost:~ # systemctl start cockpit.socket
> localhost:~ # systemctl status cockpit.socket
> ● cockpit.socket - Cockpit Web Service Socket
>      Loaded: loaded (/usr/lib/systemd/system/cockpit.socket; disabled; vendor preset: disabled)
>      Active: active (running) since Thu 2023-09-07 07:55:32 UTC; 11min ago
>    Triggers: ● cockpit.service
>        Docs: man:cockpit-ws(8)
>      Listen: [::]:9090 (Stream)
>     Process: 1999 ExecStartPost=/usr/share/cockpit/motd/update-motd  localhost (code=exited, status=0/SUCCESS)
>     Process: 2006 ExecStartPost=/bin/ln -snf active.motd /run/cockpit/motd (code=exited, status=0/SUCCESS)
>       Tasks: 0
>      CGroup: /system.slice/cockpit.socket
> 
> Sep 07 07:55:32 localhost.localdomain systemd[1]: Starting Cockpit Web Service Socket...
> Sep 07 07:55:32 localhost.localdomain systemd[1]: Listening on Cockpit Web Service Socket.

At this point, the socket service keeps active until you try to access the URL (locally or via external browser):
> localhost:~ # curl localhost:9090
> curl: (56) Recv failure: Connection reset by peer
> 
> localhost:~ # systemctl status cockpit.socket
> × cockpit.socket - Cockpit Web Service Socket
>      Loaded: loaded (/usr/lib/systemd/system/cockpit.socket; disabled; vendor preset: disabled)
>      Active: failed (Result: service-start-limit-hit) since Thu 2023-09-07 07:54:22 UTC; 3s ago
>    Triggers: ● cockpit.service
>        Docs: man:cockpit-ws(8)
>      Listen: [::]:9090 (Stream)
>     Process: 1935 ExecStartPost=/usr/share/cockpit/motd/update-motd  localhost (code=exited, status=0/SUCCESS)
>     Process: 1942 ExecStartPost=/bin/ln -snf active.motd /run/cockpit/motd (code=exited, status=0/SUCCESS)
>     Process: 1972 ExecStopPost=/bin/ln -snf inactive.motd /run/cockpit/motd (code=exited, status=0/SUCCESS)
> 
> Sep 07 07:54:04 localhost.localdomain systemd[1]: Starting Cockpit Web Service Socket...
> Sep 07 07:54:04 localhost.localdomain systemd[1]: Listening on Cockpit Web Service Socket.
> Sep 07 07:54:22 localhost.localdomain systemd[1]: cockpit.socket: Failed with result 'service-start-limit-hit'.
> 


This is what journal is showing:
> localhost:~ # journalctl --no-pager --quiet -p err  -o short-precise
> [...]
> Sep 07 07:49:45.836762 localhost.localdomain systemd[1]: Failed to start Cockpit Web Service.
> Sep 07 07:54:22.458599 localhost.localdomain systemd[1]: Failed to start Cockpit Web Service.
> Sep 07 07:54:22.509894 localhost.localdomain systemd[1]: Failed to start Cockpit Web Service.
> Sep 07 07:54:22.562947 localhost.localdomain systemd[1]: Failed to start Cockpit Web Service.
> Sep 07 07:54:22.616730 localhost.localdomain systemd[1]: Failed to start Cockpit Web Service.
> Sep 07 07:54:22.669806 localhost.localdomain systemd[1]: Failed to start Cockpit Web Service.
> Sep 07 07:54:22.671311 localhost.localdomain systemd[1]: Failed to start Cockpit Web Service.
> Sep 07 07:54:26.898093 localhost.localdomain setroubleshoot[1973]: SELinux is preventing cockpit-certifi from create access on the lnk_file cert.source. For complete SELinux messages run: sealert -l a0b1ff36-48d7-4ac5-8d14-310b50fa6c9f
> Sep 07 07:54:26.907831 localhost.localdomain setroubleshoot[1973]: SELinux is preventing cockpit-certifi from create access on the lnk_file cert.source. For complete SELinux messages run: sealert -l a0b1ff36-48d7-4ac5-8d14-310b50fa6c9f
> Sep 07 07:54:26.914469 localhost.localdomain setroubleshoot[1973]: SELinux is preventing cockpit-certifi from create access on the lnk_file cert.source. For complete SELinux messages run: sealert -l a0b1ff36-48d7-4ac5-8d14-310b50fa6c9f
> Sep 07 07:54:26.920824 localhost.localdomain setroubleshoot[1973]: SELinux is preventing cockpit-certifi from create access on the lnk_file cert.source. For complete SELinux messages run: sealert -l a0b1ff36-48d7-4ac5-8d14-310b50fa6c9f
> Sep 07 07:54:26.928835 localhost.localdomain setroubleshoot[1973]: SELinux is preventing cockpit-certifi from create access on the lnk_file cert.source. For complete SELinux messages run: sealert -l a0b1ff36-48d7-4ac5-8d14-310b50fa6c9f
> Sep 07 07:55:52.599700 localhost.localdomain setroubleshoot[2028]: SELinux is preventing cockpit-certifi from create access on the lnk_file cert.source. For complete SELinux messages run: sealert -l a0b1ff36-48d7-4ac5-8d14-310b50fa6c9f
> Sep 07 07:55:52.608688 localhost.localdomain setroubleshoot[2028]: SELinux is preventing cockpit-certifi from create access on the file cert. For complete SELinux messages run: sealert -l b533b1c0-cbf9-407e-a2c8-a2e6c620798d
> Sep 07 07:55:52.688573 localhost.localdomain setroubleshoot[2028]: SELinux is preventing cockpit-certifi from write access on the file /run/cockpit/tls/server/cert. For complete SELinux messages run: sealert -l e28abef2-b060-4150-b813-0c4659d5eb2a
> Sep 07 07:55:52.698329 localhost.localdomain setroubleshoot[2028]: SELinux is preventing cockpit-certifi from setattr access on the file cert. For complete SELinux messages run: sealert -l b5d70862-f16d-4093-8f91-2ce649da9d34




If I set the policy to permissive it works:
> localhost:~ # setenforce permissive
> localhost:~ # sestatus 
> SELinux status:                 enabled
> SELinuxfs mount:                /sys/fs/selinux
> SELinux root directory:         /etc/selinux
> Loaded policy name:             targeted
> Current mode:                   permissive
> Mode from config file:          enforcing
> Policy MLS status:              enabled
> Policy deny_unknown status:     allowed
> Memory protection checking:     requested (insecure)
> Max kernel policy version:      33
> localhost:~ # systemctl start cockpit.socket
> localhost:~ # systemctl status cockpit.socket
> ● cockpit.socket - Cockpit Web Service Socket
>      Loaded: loaded (/usr/lib/systemd/system/cockpit.socket; disabled; vendor preset: disabled)
>      Active: active (listening) since Thu 2023-09-07 07:55:32 UTC; 11s ago
>    Triggers: ● cockpit.service
>        Docs: man:cockpit-ws(8)
>      Listen: [::]:9090 (Stream)
>     Process: 1999 ExecStartPost=/usr/share/cockpit/motd/update-motd  localhost (code=exited, status=0/SUCCESS)
>     Process: 2006 ExecStartPost=/bin/ln -snf active.motd /run/cockpit/motd (code=exited, status=0/SUCCESS)
>       Tasks: 0
>      CGroup: /system.slice/cockpit.socket
> 
> Sep 07 07:55:32 localhost.localdomain systemd[1]: Starting Cockpit Web Service Socket...
> Sep 07 07:55:32 localhost.localdomain systemd[1]: Listening on Cockpit Web Service Socket.
> 
> localhost:~ # curl localhost:9090
> <!DOCTYPE html>
> <html>
> 
> <head>
>   <title>Loading...</title>
>   <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
> [...]


There is a workaround to have cockpit running in enforcing:
- Set the policy to permissive
- Start cockpit socket
- Set the policy to enforcing without stopping cockpit
- Cockpit should be active and working
However, whenever you restart the service it will fail again as described.
Comment 1 Gayane Osipyan 2023-09-07 15:55:59 UTC
Hi ,
For some reason i am not able reproduce this .I did following 
i have installed SLE-Micro-5.5-DVD-x86_64-Build19.1-Media1.iso (from https://download.suse.de/ibs/SUSE:/SLE-15-SP5:/Update:/Products:/Micro55:/Staging:/A/images/iso/ ) and updated cockpit to latest version 
rpm -q cockpit
cockpit-298-150500.1.1.x86_64

sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     requested (insecure)
Max kernel policy version:      33


with selinux enabled i am able access cockpit web UI .

curl localhost:9090
<!DOCTYPE html>
<html>

systemctl status cockpit.socket
● cockpit.socket - Cockpit Web Service Socket
     Loaded: loaded (/usr/lib/systemd/system/cockpit.socket; disabled; vendor preset: disabled)
     Active: active (listening) since Thu 2023-09-07 15:37:25 UTC; 7min ago
   Triggers: ● cockpit.service
       Docs: man:cockpit-ws(8)
     Listen: [::]:9090 (Stream)
    Process: 1173 ExecStartPost=/usr/share/cockpit/motd/update-motd  localhost (code=exited, status=0/SUCCESS)
    Process: 1180 ExecStartPost=/bin/ln -snf active.motd /run/cockpit/motd (code=exited, status=0/SUCCESS)
      Tasks: 0
     CGroup: /system.slice/cockpit.socket

Sep 07 15:37:25 localhost.localdomain systemd[1]: Starting Cockpit Web Service Socket...
Sep 07 15:37:25 localhost.localdomain systemd[1]: Listening on Cockpit Web Service Socket.
Comment 3 Jose Lausuch 2023-09-08 07:25:30 UTC
I have tried now with the latest iso: SLE-Micro-5.5-DVD-x86_64-Build19.2-Media1.iso

I enabled SElinux during Yast installation in enforcing (it was pre-selected by default), then I installed cockpit afterwards, reboot and did the following: 


> localhost:~ # rpm -q cockpit
> cockpit-298-150500.1.1.x86_64
> 
> localhost:~ # sestatus 
> SELinux status:                 enabled
> SELinuxfs mount:                /sys/fs/selinux
> SELinux root directory:         /etc/selinux
> Loaded policy name:             targeted
> Current mode:                   enforcing
> Mode from config file:          enforcing
> Policy MLS status:              enabled
> Policy deny_unknown status:     allowed
> Memory protection checking:     requested (insecure)
> Max kernel policy version:      33
> 
> localhost:~ # systemctl status cockpit.socket
> ○ cockpit.socket - Cockpit Web Service Socket
>      Loaded: loaded (/usr/lib/systemd/system/cockpit.socket; disabled; vendor preset: disabled)
>      Active: inactive (dead)
>    Triggers: ● cockpit.service
>        Docs: man:cockpit-ws(8)
>      Listen: [::]:9090 (Stream)
> 
> localhost:~ # systemctl start cockpit.socket
> localhost:~ # systemctl status cockpit.socket
> ● cockpit.socket - Cockpit Web Service Socket
>      Loaded: loaded (/usr/lib/systemd/system/cockpit.socket; disabled; vendor preset: disabled)
>      Active: active (listening) since Fri 2023-09-08 07:22:15 UTC; 1s ago
>    Triggers: ● cockpit.service
>        Docs: man:cockpit-ws(8)
>      Listen: [::]:9090 (Stream)
>     Process: 1196 ExecStartPost=/usr/share/cockpit/motd/update-motd  localhost (code=exited, status=0/SUCCESS)
>     Process: 1203 ExecStartPost=/bin/ln -snf active.motd /run/cockpit/motd (code=exited, status=0/SUCCESS)
>       Tasks: 0
>      CGroup: /system.slice/cockpit.socket
> 
> Sep 08 07:22:15 localhost.localdomain systemd[1]: Starting Cockpit Web Service Socket...
> Sep 08 07:22:15 localhost.localdomain systemd[1]: Listening on Cockpit Web Service Socket.
> 
> localhost:~ # curl localhost:9090
> curl: (56) Recv failure: Connection reset by peer
> 
> localhost:~ # systemctl status cockpit.socket
> × cockpit.socket - Cockpit Web Service Socket
>      Loaded: loaded (/usr/lib/systemd/system/cockpit.socket; disabled; vendor preset: disabled)
>      Active: failed (Result: service-start-limit-hit) since Fri 2023-09-08 07:22:21 UTC; 1s ago
>    Triggers: ● cockpit.service
>        Docs: man:cockpit-ws(8)
>      Listen: [::]:9090 (Stream)
>     Process: 1196 ExecStartPost=/usr/share/cockpit/motd/update-motd  localhost (code=exited, status=0/SUCCESS)
>     Process: 1203 ExecStartPost=/bin/ln -snf active.motd /run/cockpit/motd (code=exited, status=0/SUCCESS)
>     Process: 1245 ExecStopPost=/bin/ln -snf inactive.motd /run/cockpit/motd (code=exited, status=0/SUCCESS)
> 
> Sep 08 07:22:15 localhost.localdomain systemd[1]: Starting Cockpit Web Service Socket...
> Sep 08 07:22:15 localhost.localdomain systemd[1]: Listening on Cockpit Web Service Socket.
> Sep 08 07:22:21 localhost.localdomain systemd[1]: cockpit.socket: Failed with result 'service-start-limit-hit'.
>
Comment 5 Jan Zerebecki 2023-09-08 14:51:36 UTC
Does relabeling inside a transaction fix this?

Might there be a change in Factory that fixes this that we need to include? The selinux packages in Factory have a few changes that were not included.
Comment 6 Filippo Bonazzi 2023-09-11 06:06:43 UTC
Please attach the AVCs when you report a SELinux bug. These are the guidelines https://en.opensuse.org/openSUSE:Bugreport_SELinux

I am now trying to reproduce the issue
Comment 7 Gayane Osipyan 2023-09-11 06:23:37 UTC
(In reply to Filippo Bonazzi from comment #6)
> Please attach the AVCs when you report a SELinux bug. These are the
> guidelines https://en.opensuse.org/openSUSE:Bugreport_SELinux
> 
> I am now trying to reproduce the issue

Here are AVC from my env .

time->Fri Sep  8 15:21:51 2023
type=AVC msg=audit(1694186511.420:82): avc:  denied  { create } for  pid=1720 comm="cockpit-certifi" name="cert.source" scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:cockpit_var_run_t:s0 tclass=lnk_file permissive=1
----
time->Fri Sep  8 15:21:51 2023
type=AVC msg=audit(1694186511.420:83): avc:  denied  { create } for  pid=1720 comm="cockpit-certifi" name="cert" scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:cockpit_var_run_t:s0 tclass=file permissive=1
----
time->Fri Sep  8 15:21:51 2023
type=AVC msg=audit(1694186511.420:84): avc:  denied  { write } for  pid=1720 comm="cockpit-certifi" path="/run/cockpit/tls/server/cert" dev="tmpfs" ino=1517 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:cockpit_var_run_t:s0 tclass=file permissive=1
----
time->Fri Sep  8 15:21:51 2023
type=AVC msg=audit(1694186511.420:85): avc:  denied  { setattr } for  pid=1720 comm="cockpit-certifi" name="cert" dev="tmpfs" ino=1517 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:cockpit_var_run_t:s0 tclass=file permissive=1
Comment 9 Filippo Bonazzi 2023-09-11 06:34:02 UTC
Is it right that cockpit is not installed by default in SLEM 5.5?

Did you install it manually? What cockpit packages did you select?
Comment 10 Filippo Bonazzi 2023-09-11 07:31:50 UTC
Ok, I used the prebuilt qcow2 image and I was able to reproduce the issue.

During the investigation I realised that Cockpit ships its own SELinux policy module, outside of the system SELinux policy, so this is not for us in the security team. This is something for the cockpit maintainer to look into.

I can add that this issue does not happen on SLEM 5.4 (cockpit 251.3-150400.2.1) nor on MicroOS (298-2.1).
Comment 11 Jose Lausuch 2023-09-11 10:11:43 UTC
> Is it right that cockpit is not installed by default in SLEM 5.5?
> Did you install it manually? What cockpit packages did you select?
It is installed in the pre-installed image (raw, qcow, ...). I think it's not in the ISO, you need to select it manually, but after installation it behaves the same way. I installed the pattern itself:
> transactional-update pkg in patterns-microos-cockpit
> [...]
> The following 61 NEW packages are going to be installed:
>   checkpolicy cockpit cockpit-bridge cockpit-networkmanager cockpit-podman cockpit-selinux cockpit-storaged cockpit-system cockpit-tukit cockpit-ws cryptsetup distribution-logos-SLE girepository-1_0 glib-networking gptfdisk gsettings-desktop-schemas libatasmart-utils libatasmart4 libbd_btrfs2 libbd_crypto2 libbd_fs2 libbd_loop2 libbd_lvm2 libbd_mdraid2 libbd_part2 libbd_swap2 libbd_utils2 libblockdev libblockdev2 libbytesize1 libduktape206 libgirepository-1_0-1 libjson-glib-1_0-0 liblvm2cmd2_03 libmpfr6 libpolkit-agent-1-0 libpolkit-gobject-1-0 libpwquality1 libudisks2-0 libudisks2-0_btrfs libudisks2-0_lvm2 lvm2 mdadm patterns-microos-cockpit policycoreutils-python-utils polkit python3-audit python3-dasbus python3-dbus-python python3-extras python3-gobject python3-libxml2 python3-policycoreutils python3-rpm python3-setools python3-systemd setroubleshoot-plugins setroubleshoot-server tukitd udisks2 wallpaper-branding-SLE
> 
> The following NEW pattern is going to be installed:
>   microos-cockpit
> 


This is new issue happening lately due to some version bump of cockpit. Jiri can give more details about it. 

Here some test run in openQA with the AVCs:
https://openqa.suse.de/tests/12078156#step/cockpit_service/92
Comment 12 Johannes Segitz 2023-09-12 07:24:49 UTC
https://build.suse.de/request/show/307367 should fix this
Comment 14 Jan Zerebecki 2023-09-12 13:35:33 UTC
Johannes, do we need to forward port the selinux-policy rpm macro change to Factory?

I'm unsure if we should upstream the cockpit libexec patch. Will it generally still be the case on more systems? Will this remain like this in SLE?
Comment 15 Johannes Segitz 2023-09-12 13:50:02 UTC
I already have added this to home:jsegitz:branches:security:SELinux and sent it to security:SELinux and Factory once I'm done with some other changes

You can try this, but our behavior is wrong so they might not take it. The files should be in libexec, so I would keep this as a SLE only patch. 

And yes, this will not be backported to SLE AFAIUU
Comment 16 Jose Lausuch 2023-09-15 15:10:46 UTC
Confirmed to work with latest images (RC).
https://openqa.suse.de/tests/12127942#step/cockpit_service/1
Resolving.
Comment 19 Maintenance Automation 2023-10-19 16:30:12 UTC
SUSE-RU-2023:4134-1: An update that has one fix can now be installed.

Category: recommended (important)
Bug References: 1215109
Sources used:
SUSE Linux Enterprise Micro for Rancher 5.4 (src): cockpit-251.3-150400.6.4.1
SUSE Linux Enterprise Micro 5.4 (src): cockpit-251.3-150400.6.4.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 20 Martin Loviska 2024-05-29 11:39:42 UTC
.