Bug 1222078 - Default /etc/containers/storage.conf causes Podman to fail
Summary: Default /etc/containers/storage.conf causes Podman to fail
Status: NEW
Alias: None
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Containers (show other bugs)
Version: Current
Hardware: x86-64 openSUSE Tumbleweed
: P5 - None : Normal (vote)
Target Milestone: ---
Assignee: Containers Team
QA Contact: E-mail List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-03-27 14:28 UTC by Georg Pfuetzenreuter
Modified: 2024-03-27 14:30 UTC (History)
1 user (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Georg Pfuetzenreuter 2024-03-27 14:28:28 UTC
Hi,

upon package updates on some of our Tumbleweed hosts an /etc/containers/storage.conf file is placed, which causes Podman to no longer operate:

```
gitlab-runner1 (gitlab runner):~ # systemctl status podman
× podman.service - Podman API Service
     Loaded: loaded (/usr/lib/systemd/system/podman.service; disabled; preset: disabled)
     Active: failed (Result: exit-code) since Wed 2024-03-27 02:03:08 UTC; 12h ago
   Duration: 44ms
TriggeredBy: × podman.socket
       Docs: man:podman-system-service(1)
    Process: 7659 ExecStart=/usr/bin/podman $LOGGING system service (code=exited, status=125)
   Main PID: 7659 (code=exited, status=125)
        CPU: 71ms

Mar 27 02:03:08 gitlab-runner1 systemd[1]: Starting Podman API Service...
Mar 27 02:03:08 gitlab-runner1 systemd[1]: Started Podman API Service.
Mar 27 02:03:08 gitlab-runner1 podman[7659]: time="2024-03-27T02:03:08Z" level=info msg="/usr/bin/podman filtering at log level info"
Mar 27 02:03:08 gitlab-runner1 podman[7659]: time="2024-03-27T02:03:08Z" level=info msg="Using sqlite as database backend"
Mar 27 02:03:08 gitlab-runner1 podman[7659]: Error: database graph driver "" does not match our graph driver "overlay": database configuration mismatch
```

I tried simply injecting blank data into the file to avoid it being replaced during updates, however this does not work:

```
gitlab-runner2 (gitlab runner):~ # echo > /etc/containers/storage.conf
gitlab-runner2 (gitlab runner):~ # podman system reset -f
WARN[0000] The storage 'driver' option should be set in /etc/containers/storage.conf. A driver was picked automatically.
ERRO[0000] runroot must be set
```

The only solution I found to work for us is `rm /etc/containers/storage.conf` and `podman system reset -f` - afterwards the service starts again.

The same results are observed simply using `podman` directly - Podman on these hosts is just primarily used for socket connectivity, hence we notice the service failure first.

In case it is relevant, we use the following configuration:

```
gitlab-runner2 (gitlab runner):~ # grep -Ev '^#|^$' /etc/containers/containers.conf
[containers]
default_capabilities = [
  "CHOWN",
  "DAC_OVERRIDE",
  "FOWNER",
  "FSETID",
  "KILL",
  "NET_BIND_SERVICE",
  "SETFCAP",
  "SETGID",
  "SETPCAP",
  "SETUID",
  "SYS_CHROOT"
]
default_sysctls = [
  "net.ipv4.ping_group_range=0 0",
]
init_path = "/usr/bin/catatonit"
log_driver = "journald"
[secrets]
[secrets.opts]
[network]
network_backend = "netavark"
cni_plugin_dirs = ["/usr/libexec/cni"]
default_network = "podman_ipv6"
default_subnet = "2a07:de40:b27e:4002::/64"
[engine]
image_copy_tmp_dir="/tmp"
[engine.runtimes]
[engine.volume_plugins]
[machine]
```

It might also be noteworthy that Podman is only used rootful and that the issue already exists since quite some time, but I only now got to report it (it seems storage.conf/libcontainers-common is not updated too often hence it's only a nuisance every once in a while).

Cheers,
Georg