Bugzilla – Bug 1222078
Default /etc/containers/storage.conf causes Podman to fail
Last modified: 2024-03-27 14:30:37 UTC
Hi, upon package updates on some of our Tumbleweed hosts an /etc/containers/storage.conf file is placed, which causes Podman to no longer operate: ``` gitlab-runner1 (gitlab runner):~ # systemctl status podman × podman.service - Podman API Service Loaded: loaded (/usr/lib/systemd/system/podman.service; disabled; preset: disabled) Active: failed (Result: exit-code) since Wed 2024-03-27 02:03:08 UTC; 12h ago Duration: 44ms TriggeredBy: × podman.socket Docs: man:podman-system-service(1) Process: 7659 ExecStart=/usr/bin/podman $LOGGING system service (code=exited, status=125) Main PID: 7659 (code=exited, status=125) CPU: 71ms Mar 27 02:03:08 gitlab-runner1 systemd[1]: Starting Podman API Service... Mar 27 02:03:08 gitlab-runner1 systemd[1]: Started Podman API Service. Mar 27 02:03:08 gitlab-runner1 podman[7659]: time="2024-03-27T02:03:08Z" level=info msg="/usr/bin/podman filtering at log level info" Mar 27 02:03:08 gitlab-runner1 podman[7659]: time="2024-03-27T02:03:08Z" level=info msg="Using sqlite as database backend" Mar 27 02:03:08 gitlab-runner1 podman[7659]: Error: database graph driver "" does not match our graph driver "overlay": database configuration mismatch ``` I tried simply injecting blank data into the file to avoid it being replaced during updates, however this does not work: ``` gitlab-runner2 (gitlab runner):~ # echo > /etc/containers/storage.conf gitlab-runner2 (gitlab runner):~ # podman system reset -f WARN[0000] The storage 'driver' option should be set in /etc/containers/storage.conf. A driver was picked automatically. ERRO[0000] runroot must be set ``` The only solution I found to work for us is `rm /etc/containers/storage.conf` and `podman system reset -f` - afterwards the service starts again. The same results are observed simply using `podman` directly - Podman on these hosts is just primarily used for socket connectivity, hence we notice the service failure first. In case it is relevant, we use the following configuration: ``` gitlab-runner2 (gitlab runner):~ # grep -Ev '^#|^$' /etc/containers/containers.conf [containers] default_capabilities = [ "CHOWN", "DAC_OVERRIDE", "FOWNER", "FSETID", "KILL", "NET_BIND_SERVICE", "SETFCAP", "SETGID", "SETPCAP", "SETUID", "SYS_CHROOT" ] default_sysctls = [ "net.ipv4.ping_group_range=0 0", ] init_path = "/usr/bin/catatonit" log_driver = "journald" [secrets] [secrets.opts] [network] network_backend = "netavark" cni_plugin_dirs = ["/usr/libexec/cni"] default_network = "podman_ipv6" default_subnet = "2a07:de40:b27e:4002::/64" [engine] image_copy_tmp_dir="/tmp" [engine.runtimes] [engine.volume_plugins] [machine] ``` It might also be noteworthy that Podman is only used rootful and that the issue already exists since quite some time, but I only now got to report it (it seems storage.conf/libcontainers-common is not updated too often hence it's only a nuisance every once in a while). Cheers, Georg