Bugzilla – Bug 1199871
Rootless podman logging broken by default
Last modified: 2024-07-18 12:01:39 UTC
On OpenSUSE Leap 15.x normal users cannot access `journalctl` without being root/sudo This breaks the default logging driver for podman. While this is not a problem for rootful podman (because it can access journalctl), it is a problem for rootless podman. By default on a clean OpenSUSE Leap system, if you: - sudo zypper in podman - sudo usermod --add-subuids 100000-101000 --add-subgids 100000-101000 $USER - podman run --rm -ti --name test alpine:latest sh If I try to get the logs of the `test` container: :~> podman logs test Error: initial journal cursor: failed to get cursor: cannot assign requested address This is fixed by changing the logging driver to something else. For example setting: [containers] log_driver = "k8s-file" [engine] events_logger = "file" in ~/.config/containers/containers.conf, and recreating the container, fixes the issue.
Possible solutions could be to either - default the system configuration do use the file logging driver or - allow normal users to use journalctl by default or - inject this configuration by default on new users in their home (so that rootless podman works)
(In reply to Luca Di Maio from comment #0) > On OpenSUSE Leap 15.x normal users cannot access `journalctl` without being > root/sudo > > This breaks the default logging driver for podman. > Yes. And, at least on my system, this is a problem on Tumbleweed and/or MicroOS as well. > This is fixed by changing the logging driver to something else. > For example setting: > > [containers] > log_driver = "k8s-file" > [engine] > events_logger = "file" > > in ~/.config/containers/containers.conf, and recreating the container, fixes > the issue. > Right, and when doing this, `podman logs <name_of_rootless_container>` works. However, `podman log -f <name_of_rootless_container>` starts to misbehave. In fact, I do see the container's logs, but I don't get the traditional "follow" behavior. I just get the output printed on the terminal, and then I'm back to the prompt, while I'd expect it to "hang", and to continue see the logs as they come, until interrupted. On the other hand, `podman log -f <containername> seems to work as expected, for rootful containers (at least here). Bug 1199790, and a couple of issues filed against distrobox upstream (https://github.com/89luca89/distrobox/issues/287 and https://github.com/89luca89/distrobox/issues/286) are probably all related to this.
(In reply to Dario Faggioli from comment #2) > However, `podman log -f <name_of_rootless_container>` starts to misbehave. > > In fact, I do see the container's logs, but I don't get the traditional > "follow" behavior. I just get the output printed on the terminal, and then > I'm back to the prompt, while I'd expect it to "hang", and to continue see > the logs as they come, until interrupted. > > On the other hand, `podman log -f <containername> seems to work as expected, > for rootful containers (at least here). > > Bug 1199790, and a couple of issues filed against distrobox upstream > (https://github.com/89luca89/distrobox/issues/287 and > https://github.com/89luca89/distrobox/issues/286) are probably all related > to this. It works for me on Leap after the fix But when no new logs are present, it just exits without follow This behavior is in common with podman on Fedora also, which uses normal journald driver
adding the user to system-journal group solve the issue.
I'm not sure if this is the same bug, but for me, when using current distrobox master, I can't create a rootless contain on SLE15-SP5: $ ./distrobox create --name tumbleweed-pkg --image registry.opensuse.org/opensuse/distrobox-packaging:latest Creating 'tumbleweed-pkg' using image registry.opensuse.org/opensuse/distrobox-packaging:latest Resource limits are not supported and ignored on cgroups V1 rootless systems [ OK ] Distrobox 'tumbleweed-pkg' successfully created. To enter, run: distrobox enter tumbleweed-pkg At this point, ok, but when I try to enter the container: $ ./distrobox enter tumbleweed-pkg WARN[0000] Failed to mount subscriptions, skipping entry in /etc/containers/mounts.conf: open /etc/zypp/credentials.d/SCCcredentials: permission denied Starting container... Error: using --follow with the journald --log-driver but without the journald --events-backend (file) is not supported (In reply to Walddys Emmanuel Dorrejo Céspedes from comment #4) > adding the user to system-journal group solve the issue. Also I couldn't find this group on SLE15-SP5... am I missing something? Nowadays that SLE and openSUSE share the same codebase, I expect this behavior to be also present on Leap 15.5, right? Thanks!
(In reply to Marcos de Souza from comment #5) > Also I couldn't find this group on SLE15-SP5... am I missing something? > Nowadays that SLE and openSUSE share the same codebase, I expect this > behavior to be also present on Leap 15.5, right? I believe the solution to your case is to change the log driver, you can set log_driver and events_logger as suggested in the previous comment. This bug is mentioned in https://distrobox.it/compatibility/#host-distros.
(In reply to Alexandre Vicenzi from comment #6) > (In reply to Marcos de Souza from comment #5) > > > Also I couldn't find this group on SLE15-SP5... am I missing something? > > Nowadays that SLE and openSUSE share the same codebase, I expect this > > behavior to be also present on Leap 15.5, right? > > I believe the solution to your case is to change the log driver, you can set > log_driver and events_logger as suggested in the previous comment. > > This bug is mentioned in https://distrobox.it/compatibility/#host-distros. Indeed, if I create a file in my hope directory like stated in comment 1: > [containers] > log_driver = "k8s-file" > [engine] > events_logger = "file" > > in ~/.config/containers/containers.conf, and recreating the container, fixes > the issue. It worked with minimal warnings: mpdesouza@dogmatix:~> distrobox create --name mpdesouza-lp --image registry.opensuse.org/opensuse/distrobox-packaging: latest Creating 'mpdesouza-lp' using image registry.opensuse.org/opensuse/distrobox-packaging:latest Resource limits are no t supported and ignored on cgroups V1 rootless systems [ OK ] Distrobox 'mpdesouza-lp' successfully created. To enter, run: distrobox enter mpdesouza-lp mpdesouza@dogmatix:~> distrobox enter mpdesouza-lp WARN[0000] Failed to mount subscriptions, skipping entry in /etc/containers/mounts.conf: open /etc/zypp/credentials.d/ SCCcredentials: permission denied Starting container... [ OK ] Installing basic packages... [ OK ] Setting up devpts mounts... [ OK ] Setting up read-only mounts... [ OK ] Setting up read-write mounts... [ OK ] Setting up host's sockets integration... [ OK ] Integrating host's themes, icons, fonts... [ OK ] Setting up package manager exceptions... [ OK ] Setting up rpm exceptions... [ OK ] Setting up distrobox profile... [ OK ] Setting up sudo... [ OK ] Setting up user groups... [ OK ] Setting up kerberos integration... [ OK ] Setting up user's group list... [ OK ] Setting up existing user... [ OK ] Setting up user home... [ OK ] Ensuring user's access... [ OK ] Container Setup Complete! It worked fine for me. Thanks!
(In reply to Marcos de Souza from comment #7) > Creating 'mpdesouza-lp' using image > registry.opensuse.org/opensuse/distrobox-packaging:latest Resource limits > are no > t supported and ignored on cgroups V1 rootless systems Know behavior, the user has no permission to set resource limits. Should not affect container behavior. > WARN[0000] Failed to mount subscriptions, skipping entry in > /etc/containers/mounts.conf: open /etc/zypp/credentials.d/ > SCCcredentials: permission denied Worth checking if the file exists and if it can be mounted in a container. The guest container won't be able to access the host SCC subscription. Is the host using SELinux?
Regarding the original issue, Podman still uses `log_driver = "journald"`. The command `podman logs` does not fail anymore, but it does not print out any of the container logs on a rootless env, on a rootfull env it does work as expected. Aleksa, anything we can do about the Podman package? or should this be fixed somewhere else?
The command `podman logs -f` does fail with the: using --follow with the journald --log-driver but without the journald --events-backend (file) is not supported It is the same issue seen by Marcos in distrobox.
This has been fixed for a while in Tumbleweed and recently on Leap 15.6. The SR that introduced the fix is https://build.opensuse.org/request/show/1167004. This SR changed the default log_drive from journald to unset. -log_driver = "journald" +#log_driver = "k8s-file" The fix is not present in Leap 15.5 because libcontainers-common is older than the one in 15.6 and does not include this change. Danish, can we match 15.5 and 15.6 libcontainers-common? Podman version is already the same. 15.6 has: ~> rpm -q podman libcontainers-common podman-4.9.5-150500.3.15.1.x86_64 libcontainers-common-20240408-150600.1.1.noarch 15.5 has: ~> rpm -q podman libcontainers-common podman-4.9.5-150500.3.15.1.x86_64 libcontainers-common-20240206-150500.4.9.2.noarch