Bug 1186158

Running containers like traefik fail on MicroOS due to SELinux. I am running:

# head -2 /etc/os-release 
NAME="openSUSE MicroOS"
# VERSION="20210520"

as of this morning and I am seeing the following selinux denied in /var/log/audit/audit.log

type=AVC msg=audit(1621669323.589:1091): avc:  denied  { read write } for  pid=4872 comm="entrypoint.sh" path="/dev/null" dev="tmpfs" ino=5 scontext=system_u:
system_r:container_t:s0:c587,c659 tcontext=system_u:object_r:container_file_t:s0:c587,c659 tclass=chr_file permissive=1
type=AVC msg=audit(1621669323.593:1092): avc:  denied  { open } for  pid=4872 comm="entrypoint.sh" path="/dev/null" dev="tmpfs" ino=5 scontext=system_u:system_r:container_t:s0:c587,c659 tcontext=system_u:object_r:container_file_t:s0:c587,c659 tclass=chr_file permissive=1
type=AVC msg=audit(1621669323.613:1093): avc:  denied  { read } for  pid=4885 comm="traefik" name="hpage_pmd_size" dev="sysfs" ino=3207 scontext=system_u:system_r:container_t:s0:c587,c659 tcontext=system_u:object_r:sysfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1621669323.613:1094): avc:  denied  { open } for  pid=4885 comm="traefik" path="/sys/kernel/mm/transparent_hugepage/hpage_pmd_size" dev="sysfs" ino=3207 scontext=system_u:system_r:container_t:s0:c587,c659 tcontext=system_u:object_r:sysfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1621669323.701:1095): avc:  denied  { create } for  pid=4885 comm="traefik" scontext=system_u:system_r:container_t:s0:c587,c659 tcontext=system_u:system_r:container_t:s0:c587,c659 tclass=netlink_route_socket permissive=1
type=AVC msg=audit(1621669323.701:1096): avc:  denied  { bind } for  pid=4885 comm="traefik" scontext=system_u:system_r:container_t:s0:c587,c659 tcontext=system_u:system_r:container_t:s0:c587,c659 tclass=netlink_route_socket permissive=1
type=AVC msg=audit(1621669323.701:1097): avc:  denied  { nlmsg_read } for  pid=4885 comm="traefik" scontext=system_u:system_r:container_t:s0:c587,c659 tcontext=system_u:system_r:container_t:s0:c587,c659 tclass=netlink_route_socket permissive=1
type=AVC msg=audit(1621669323.701:1098): avc:  denied  { getattr } for  pid=4885 comm="traefik" scontext=system_u:system_r:container_t:s0:c587,c659 tcontext=system_u:system_r:container_t:s0:c587,c659 tclass=netlink_route_socket permissive=1
type=AVC msg=audit(1621669323.777:1099): avc:  denied  { create } for  pid=4872 comm="traefik" scontext=system_u:system_r:container_t:s0:c587,c659 tcontext=system_u:system_r:container_t:s0:c587,c659 tclass=tcp_socket permissive=1
type=AVC msg=audit(1621669323.777:1100): avc:  denied  { setopt } for  pid=4872 comm="traefik" scontext=system_u:system_r:container_t:s0:c587,c659 tcontext=system_u:system_r:container_t:s0:c587,c659 tclass=tcp_socket permissive=1
type=AVC msg=audit(1621669323.777:1101): avc:  denied  { bind } for  pid=4872 comm="traefik" scontext=system_u:system_r:container_t:s0:c587,c659 tcontext=system_u:system_r:container_t:s0:c587,c659 tclass=tcp_socket permissive=1
type=AVC msg=audit(1621669323.777:1102): avc:  denied  { node_bind } for  pid=4872 comm="traefik" saddr=::1 scontext=system_u:system_r:container_t:s0:c587,c659 tcontext=system_u:object_r:node_t:s0 tclass=tcp_socket permissive=1
type=AVC msg=audit(1621669323.777:1103): avc:  denied  { name_bind } for  pid=4872 comm="traefik" src=80 scontext=system_u:system_r:container_t:s0:c587,c659 tcontext=system_u:object_r:http_port_t:s0 tclass=tcp_socket permissive=1
type=AVC msg=audit(1621669323.777:1104): avc:  denied  { net_bind_service } for  pid=4872 comm="traefik" capability=10  scontext=system_u:system_r:container_t:s0:c587,c659 tcontext=system_u:system_r:container_t:s0:c587,c659 tclass=capability permissive=1
type=AVC msg=audit(1621669323.777:1105): avc:  denied  { listen } for  pid=4872 comm="traefik" lport=80 scontext=system_u:system_r:container_t:s0:c587,c659 tcontext=system_u:system_r:container_t:s0:c587,c659 tclass=tcp_socket permissive=1
type=AVC msg=audit(1621669323.777:1106): avc:  denied  { getattr } for  pid=4872 comm="traefik" lport=80 scontext=system_u:system_r:container_t:s0:c587,c659 tcontext=system_u:system_r:container_t:s0:c587,c659 tclass=tcp_socket permissive=1
type=AVC msg=audit(1621669323.777:1107): avc:  denied  { accept } for  pid=4872 comm="traefik" lport=443 scontext=system_u:system_r:container_t:s0:c587,c659 tcontext=system_u:system_r:container_t:s0:c587,c659 tclass=tcp_socket permissive=1
type=AVC msg=audit(1621669323.781:1108): avc:  denied  { watch } for  pid=4872 comm="traefik" path="/etc/traefik" dev="overlay" ino=93 scontext=system_u:system_r:container_t:s0:c587,c659 tcontext=unconfined_u:object_r:etc_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1621669323.781:1109): avc:  denied  { write } for  pid=4872 comm="traefik" name="podman.sock" dev="tmpfs" ino=1239 scontext=system_u:system_r:container_t:s0:c587,c659 tcontext=system_u:object_r:var_run_t:s0 tclass=sock_file permissive=1
type=AVC msg=audit(1621669323.781:1110): avc:  denied  { connectto } for  pid=4872 comm="traefik" path="/run/podman/podman.sock" scontext=system_u:system_r:container_t:s0:c587,c659 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=unix_stream_socket permissive=1
type=SERVICE_START msg=audit(1621669323.805:1111): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=podman comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'

Using podman 3.2.0 and starting the docker API as a regular user, i.e. systemctl --user start podman yield the same issues that SELinux forbids reading /run/user/1000/podman/podman.sock The socket is readable from outside a container, but from the inside SELinux is preventing access to it. Setting SELinux to permissive fixes the access issue.

(In reply to G.M. Venekamp from comment #2)
> Using podman 3.2.0 and starting the docker API as a regular user, i.e.
> systemctl --user start podman yield the same issues that SELinux forbids
> reading /run/user/1000/podman/podman.sock The socket is readable from
> outside a container, but from the inside SELinux is preventing access to it.

That's exatlx what SELinux should do and is designed for, so everything is working. Accessing the podman.sock from inside a container is really dangerous security wise.

If you really need that, you need to learn SELinux and adjust the policy for your use case (or look if somebody has already a solution documented somewhere).
But we will not allow this by default.

Yes, that is of course correct for a non-privileged user. However, is that also the case for root? The use-case here is traefik and dynamic configuration. If you want to use that feature, traefik needs access to the docker API, i.e. /run/podman/podman.sock. SELinux is currently blocking access from within the traefik container.

On second thought I might not agree. I would agree if I expose a root owned podman.sock to a rootless container. I am not doing that. When I start podman like so: systemctl --user start podman, and thus I am creating a socket under /run/user/<uid>/podman/podman.sock. This socket is owned by the unprivileged user. Also this socket would not gain access to root owned containers. As far as I understand, rootless and rootfull are separated with podman. Therefore I would argue that the scope of the security is limited to that of an ordinary user. Hence it would be okay to expose the podman.sock to containers owned to that same user.

Inside a rootless container, I would not be able to gain any elevation or anything otherwise. Inside a rootless container I cannot do more then on the outside. Therefore SELinux should not block read access. I don't think you ever write to the docker API and blocking write access would make sense.

Is my reasoning flawed in any way and if so, how?

Hi,

I am trying reproduce this can you please give more information why we want to reach podman.sock and also if possible more details about a container.
By default it is not allowed and we would have to have either privilege container or container without SELinux protection.

The use case that I have and think others might as well is the following: I'd like to run my containers with as little privileges as possible. This is for security reasons. Therefore I run my containers as an ordinary user and not as root. Podman has decent support for this. In order to support multiple containers I'd like to use a proxy. My choice is traefik. Since I run my containers as an unprivileged user I must run traefik as the same user, because of namespaces. One feature of traefik is dynamic configuration of containers and it needs access to /var/run/podman/podman.sock Now, security wise you probabely don't want to give traefik direct access to that socket. You put something inbetween that restricts traefik's access, i.e. read only to subset of the API. However, this proxy would require the same level of access to the socket.

In short: traefik could use the podman.sock to learn about running containers. However, because container running as an unprivileged user rerequire traefik to be run as the same unprivileged user it must be able to access podman.sock as the unprivileged user.

One thing to note though is that if I would have another privileged user, that user will not be able to access the namespaces of the other unprivileged user. Just like root and non-root users cannot use each other namespaces, unprivileged user cannot use them amongst themselve as well.

If you run containers as an unprivileged user, and you run as root: podman container ls, you will see container created as root and not any containers owned by other users.

Hi Gayane Osipyan,

Did you get all information you needed?

Hi,
Sorry for delayed reply.

I did following steps on my setup started podman as non root.
 systemctl --user start  podman.service
and then created traefik 
podman run -d --restart=always -p 8080:8080 -p 80:80 -p 444:443 -v ${traefik-dir}/:/etc/traefik/ -v /run/user/<uid>/podman/podman.sock:/run/user/<uid>/podman/podman.sock  -v ${traefik-dir}/traefik.log:/traefik.log --name traefik traefik

with selinux enabled I am able to access dashboard 

Also for debugging for me i find helpful to use setenforce 0 and look for denials with  ausearch -m avc

I'll try to replicate the issue in a clean VM and report back. Might be the case that the SELinux rules have been altered, such that this issues has been mitigated already.

I have created a new VM and installed MicroOS (20220401) in it. Chose the container platform version of MicroOS and left SELinux on 'enforce' Started podman as non root like so:

systemctl --user enable --now podman.socket

I think you should use socket here and not service, but in reality it amount to the same I think.

Next I started traefik like this:

podman run \
  --restart=always \
  --memory=1024m \
  --memory-reservation=512m \
  --network=container-net \
  --volume /run/user/1000/podman/podman.sock:/var/run/podman.sock:ro,z \
  --volume /home/gerben/containers/traefik/rootfs/etc/traefik:/etc/traefik:Z \
  --publish 8080:80 \
  --publish 8443:443 \
  --name traefik \
  traefik:v2.5.6

Once started I can see a number of the following messages:

time="2022-04-04T12:02:19Z" level=debug msg="FIXME: Got an status-code for which error does not match any expected type!!!: -1" module=api status_code=-1
time="2022-04-04T12:02:19Z" level=error msg="Failed to retrieve information of the docker client and server host: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/podman.sock: Get \"http://%2Fvar%2Frun%2Fpodman.sock/v1.24/version\": dial unix /var/run/podman.sock: connect: permission denied" providerName=docker

Checking the audit log with 'ausearch -m AVC' yields a number of the following messages:

type=AVC msg=audit(1649064605.263:115): avc:  denied  { connectto } for  pid=1582 comm="traefik" path="/run/user/1000/podman/podman.sock" scontext=system_u:system_r:container_t:s0:c140,c679 tcontext=unconfined_u:system_r:container_runtime_t:s0-s0:c0.c1023 tclass=unix_stream_socket permissive=0

Telling me SELinux is blocking access to the file.

After having set SELinux to permissive (setenforce 0) and starting the the traefik container as above, I do not see any of those permission denied messages. Of course I see the following message in the audit log now:

type=AVC msg=audit(1649098620.131:640): avc:  denied  { connectto } for  pid=7640 comm="traefik" path="/run/user/1000/podman/podman.sock" scontext=system_u:system_r:container_t:s0:c535,c853 tcontext=unconfined_u:system_r:container_runtime_t:s0-s0:c0.c1023 tclass=unix_stream_socket permissive=1


The traefik dashboard will show up regardless of having access to the socket or not. Traefik uses the docker API to check if there are containers it should proxy for. When starting a container you can use labels to pass info to traefik. Traefik needs the docker API to be able to read those labels. There is not need for Traefik to use the Docker API for displaying its own dashboard.

Hi,

Thanks for details after doing some research and also replicating this issue on non microos setup with SElinux enabled my understanding is following

1. File provider should be used for rootless podman container+traefik .(For my testing purposes it worked)
providers:
  file:
    filename: /path/to/config/dynamic_conf.yml

2. SElinux policy can be adjusted locally but seems its strongly not recommended(you can use udica which will generate policy for specific container)

3. run Traefik from a rootfull container.

One suggestion I found is to run traefik non-rootfull and use the tcp exposed docker-socket .The dockerproxy has to run rootfull .(Haven't tested this )

Are there any updates ?Should we close bug ?

Please reopen if needed.