Bug 1222499 - runc v1.2.0~rc1 breaks distrobox (and maybe more)
Summary: runc v1.2.0~rc1 breaks distrobox (and maybe more)
Status: NEW
Alias: None
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Containers (show other bugs)
Version: Current
Hardware: Other Other
: P5 - None : Normal (vote)
Target Milestone: ---
Assignee: Aleksa Sarai
QA Contact: E-mail List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-04-09 05:46 UTC by Frederic Crozat
Modified: 2024-05-06 09:15 UTC (History)
5 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Frederic Crozat 2024-04-09 05:46:14 UTC
Since runc was updated to v1.2.0~rc1 last week in TW, distrobox (and therefore podman), when forcing runc engine, is no longer able to start container rootless.

Error is :
Error: unable to start container "988cc7b326ef04596a3d14ccc41bbdb43c74a6420b5e476b5da00b4e381f0d6f": runc: runc create failed: unable to start container process: error during container init: error mounting "/" to rootfs at "/run/host": mount dst=/run/host, dstFd=/proc/thread-self/fd/8, flags=0x5020: operation not permitted: OCI permission denied"

~/.config/containers/containers.conf
[containers]

[engine]
runtime="runc"
Comment 1 Eduardo Minguez 2024-04-22 09:27:45 UTC
This happened to me as well on microos
Comment 2 Aleksa Sarai 2024-04-22 14:45:04 UTC
I can't reproduce this on my TW machine. I'm not familiar with distrobox, is

 % DBX_CONTAINER_MANAGER="podman" distrobox create foo
 % DBX_CONTAINER_MANAGER="podman" distrobox enter foo

a reasonable way of testing for the issue?

Upstream runc has integration tests for rootless containers, and as one of the upstream maintainers I develop runc on openSUSE and have not seen any issues using runc (1.2.0~rc1 or during its development) on openSUSE, so it seems more likely there is something about distrobox's configuration which started breaking with a newer runc version.
Comment 3 Frederic Crozat 2024-04-30 09:16:44 UTC
Reproducer from Aeon (ie Tumbleweed MicroOS):

DBX_CONTAINER_MANAGER="podman" distrobox create --image fedora:latest --name fedora -a "--runtime runc"

DBX_CONTAINER_MANAGER="podman" distrobox enter fedora


this will fail at "enter" command.

However:

distrobox rm -f fedora (to do the cleanup)


DBX_CONTAINER_MANAGER="podman" distrobox create --image fedora:latest --name fedora -a "--runtime crun"

DBX_CONTAINER_MANAGER="podman" distrobox enter fedora

will work nicely
Comment 4 Luca Di Maio 2024-04-30 09:43:41 UTC
I can reproduce that yes

This seems also (from a quick test) NOT related to selinux.

It seems like runc has some permission enforcement, probably related
to some breaking changes introduced upstream:

https://github.com/opencontainers/runc/commit/d4b670fca6d0ac606777376440ffe49686ce15f4

---

Will need to find a fix (in my upstream) because mounting /:/run/host is an integral part of distrobox, and likely of any toolbox implementation (be it MicroOS, CoreOS, Fedora and so on)

Can we link this ticket to an issue upstream?
Comment 5 Luca Di Maio 2024-04-30 10:02:55 UTC
After a further investigation, it seems it really is related to that breaking change

mounting as ":rslave" something that is :ro, will output that error

---

Changing the code for mounting the rootfs into this:

	for rootdir in /*; do
		if findmnt --notruncate --noheadings --list --output OPTIONS --target "${rootdir}" |
			tr ',' '\n' | grep -q "^ro$"; then

			result_command="${result_command}
				--volume ${rootdir}:/run/host/${rootdir}:ro,rslave"
		else
			result_command="${result_command}
				--volume ${rootdir}:/run/host/${rootdir}:rslave"
		fi
	done

Will indeed fix the problem as the different :ro mountpoints are correctly mounted as such:

--volume /bin:/run/host/bin:ro,rslave
--volume /boot:/run/host/boot:ro,rslave
--volume /dev:/run/host/dev:rslave
--volume /etc:/run/host/etc:rslave
--volume /home:/run/host/home:rslave
--volume /lib:/run/host/lib:ro,rslave
--volume /lib64:/run/host/lib64:ro,rslave
--volume /mnt:/run/host/mnt:ro,rslave
--volume /opt:/run/host/opt:rslave
--volume /proc:/run/host/proc:rslave
--volume /root:/run/host/root:rslave
--volume /run:/run/host/run:rslave
--volume /sbin:/run/host/sbin:ro,rslave
--volume /srv:/run/host/srv:rslave
--volume /sys:/run/host/sys:rslave
--volume /tmp:/run/host/tmp:rslave
--volume /usr:/run/host/usr:ro,rslave
--volume /var:/run/host/var:rslave

This is a fix that will be added on (my) upstream
Comment 6 Luca Di Maio 2024-05-01 11:04:58 UTC
Fixed in commit f031b8e8176c9c8a877389591c4732d2eeee78ea

Will be part of next release
Comment 7 Luca Di Maio 2024-05-01 17:05:06 UTC
Fix has been released
https://github.com/89luca89/distrobox/releases/tag/1.7.2.0
Comment 8 Eduardo Minguez 2024-05-03 13:25:37 UTC
Any hints on when this fix will land on Tumbleweed?
Comment 9 Aleksa Sarai 2024-05-06 09:15:15 UTC
(In reply to Luca Di Maio from comment #5)
> After a further investigation, it seems it really is related to that
> breaking change
> 
> mounting as ":rslave" something that is :ro, will output that error

runc treats propagation flags and mount options separately, this is odd. I'll see if I can come up with a test using runc directly (I don't know what podman is doing to the mount options before giving them to runc).

> ---
> 
> Changing the code for mounting the rootfs into this:
> 
> 	for rootdir in /*; do
> 		if findmnt --notruncate --noheadings --list --output OPTIONS --target
> "${rootdir}" |
> 			tr ',' '\n' | grep -q "^ro$"; then
> 
> 			result_command="${result_command}
> 				--volume ${rootdir}:/run/host/${rootdir}:ro,rslave"
> 		else
> 			result_command="${result_command}
> 				--volume ${rootdir}:/run/host/${rootdir}:rslave"
> 		fi
> 	done

rro,rslave might be a less painful way of doing this. runc 1.1.0 and later support this though it requires Linux 5.12.