Bug 1222450

Summary: Error: crun: opening file `memory.max` for writing: No such file or directory: OCI runtime attempted to invoke a command that was not found
Product: [openSUSE] openSUSE Tumbleweed Reporter: Miika Alikirri <miika.alikirri>
Component: BasesystemAssignee: systemd maintainers <systemd-maintainers>
Status: NEW --- QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None CC: asarai, miika.alikirri
Version: Current   
Target Milestone: ---   
Hardware: x86-64   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: output of podman info

Description Miika Alikirri 2024-04-08 10:12:12 UTC
Created attachment 874124 [details]
output of podman info

On a fresh installation of tumbleweed (version 20240403), podman fails to run as a non-root user when `--memory` option is set.

running:
```
podman run --rm --memory 4M tumbleweed echo it works
```

prints out:
```
Error: crun: opening file `memory.max` for writing: No such file or directory: OCI runtime attempted to invoke a command that was not found
```

And running it as sudo runs it correctly and prints:
```
it works
```
Comment 1 Aleksa Sarai 2024-05-06 10:07:49 UTC
It seems that your machine doesn't have the memory controller enabled on cgroupv2 (in fact it seems to only have pids enabled). What is the contents of /sys/fs/cgroup/cgroup.controllers?

  cgroupControllers:
  - pids
  cgroupManager: systemd
  cgroupVersion: v2

Because systemd only allows unprivileged users to manage cgroupv2 cgroups, crun can't do whatever cgroupv1 fallback it is presumably doing when you run podman as root.

(FWIW, I can't reproduce this at all on my Tumbleweed machine. It's not a fresh install though.)
Comment 2 Aleksa Sarai 2024-05-06 10:09:15 UTC
(In reply to Aleksa Sarai from comment #1)
> What is the contents of /sys/fs/cgroup/cgroup.controllers?

And also /sys/fs/cgroup/cgroup.subtree_control
Comment 3 Miika Alikirri 2024-05-06 12:18:20 UTC
(In reply to Aleksa Sarai from comment #2)
> (In reply to Aleksa Sarai from comment #1)
> > What is the contents of /sys/fs/cgroup/cgroup.controllers?
> 
> And also /sys/fs/cgroup/cgroup.subtree_control

I downloaded a latest snapshot and did nothing but install podman on it and got the same result.

Here are the file contents:

$ cat /sys/fs/cgroup/cgroup.subtree_control
pids
$ cat /sys/fs/cgroup/cgroup.controllers
cpuset cpu io memory hugetlb pids rdma misc
Comment 4 Miika Alikirri 2024-05-21 09:06:51 UTC
Any progress on this? Is there anything I can do to help?
Comment 5 Aleksa Sarai 2024-06-06 04:34:45 UTC
So, the issue is that systemd is not enabling all of the controllers at the root (subtree_control is missing a bunch of controllers on your system), resulting in the container not being able to start because the controllers are not available. This is a little surprising because crun/runc creates a systemd transient unit that has all of the restrictions listed.

I can reproduce this somewhat on my machine -- depending on the running services, the set of subtree_control-enabled controllers varies. If I start Docker and run a container, all of the controllers are enabled and using podman works. But if you stop the Docker service, the cpu controller is disabled and so --cpu-shares no longer works.

This seems like a systemd issue to me. Why is a TransientUnit with all of the relevant restrictions applied not sufficient to get systemd to enable the needed controllers?
Comment 6 Chenzi Cao 2024-07-16 16:25:01 UTC
I assign it to systemd maintainers to take a look at this, please feel free to reassign whenever necessary, thanks.