Bug 1221759

Summary:	systemd fails to switch root: os-release file is missing.
Product:	[openSUSE] openSUSE Tumbleweed	Reporter:	Andreas Schwab <schwab>
Component:	Basesystem	Assignee:	Adrian Schröter <adrian>
Status:	NEW ---	QA Contact:	E-mail List <qa-bugs>
Severity:	Major
Priority:	P5 - None	CC:	dimstar, dleuenberger, fbui, guillaume.gardet, slawek
Version:	Current
Target Milestone:	---
Hardware:	Other
OS:	Other
Whiteboard:
Found By:	---	Services Priority:
Business Priority:		Blocker:	---
Marketing QA Status:	---	IT Deployment:	---

Description Andreas Schwab 2024-03-20 13:37:56 UTC

This is a regression from the current tumbleweed snapshot (it works when building against openSUSE:Factory/snapshot).
The only two differences between the kernel-obs-build initrds (from kernel-obs-build-6.7.9-1.1 and kernel-obs-build-6.8.1-1.1) are systemd (v254.9 vs v255.3) and the kernel.

$ osc build --noservice standard x86_64 --vm-type qemu -j2 --clean
Building aaa_base.spec for standard/x86_64
Getting buildconfig from server and store to /space/osc/openSUSE:Factory/aaa_base/.osc/_buildconfig-standard-x86_64
Getting buildinfo from server and store to /space/osc/openSUSE:Factory/aaa_base/.osc/_buildinfo-standard-x86_64.xml
Updating cache of required packages
0.0% cache miss. 39/39 dependencies cached.

Skipping verification of package signatures
Writing build configuration
Running build
VM_ROOT: /var/tmp/build-root/openSUSE:Factory/standard/x86_64/img, VM_SWAP: /var/tmp/build-root/openSUSE:Factory/standard/x86_64/swap
Creating ext4 filesystem on /var/tmp/build-root/openSUSE:Factory/standard/x86_64/img
tune2fs 1.47.0 (5-Feb-2023)
Setting maximal mount count to -1
logging output to /var/tmp/build-root/openSUSE:Factory/standard/x86_64/.build.log...
[    0s] Using BUILD_ROOT=/var/tmp/build-root/openSUSE:Factory/standard/x86_64/.mount
[    0s] Using BUILD_ARCH=x86_64:i686:i586:i486:i386
[    0s] Doing qemu build in /var/tmp/build-root/openSUSE:Factory/standard/x86_64/img
[    0s] 
[    1s] 
[    1s] achrid-1 started "build aaa_base.spec" at Wed Mar 20 13:11:40 UTC 2024.
[    1s] 
[    1s] 
[    1s] processing recipe /space/osc/openSUSE:Factory/aaa_base/aaa_base.spec ...
[    1s] running changelog2spec --target rpm --file /space/osc/openSUSE:Factory/aaa_base/aaa_base.spec
[    1s] init_buildsystem --configdir /usr/lib/build/configs --cachedir /var/cache/build --prepare --clean --rpmlist /tmp/rpmlist.xjc128k6 /space/osc/openSUSE:Factory/aaa_base/aaa_base.spec ...
[    1s] unpacking preinstall image openSUSE:Factory/standard/preinstallimage-base [ca63a53d27093320ebf62ec5cf06d696]
[    4s] 
[    4s] copying packages...
[    4s] reordering...cycle: libpython3_11-1_0 -> python311-base
[    4s]   breaking dependency python311-base -> libpython3_11-1_0
[    4s] done
[    4s] Detected virtio-serial support
[   10s] booting qemu...
[   10s] Using virtio-serial support
[   10s] /usr/bin/qemu-system-x86_64 -nodefaults -no-reboot -nographic -vga none -cpu qemu64 -runas qemu -net none -kernel /var/tmp/build-root/openSUSE:Factory/standard/x86_64/.mount/boot/kernel -initrd /var/tmp/build-root/openSUSE:Factory/standard/x86_64/.mount/boot/initrd -append root=/dev/disk/by-id/virtio-0 rootfstype=ext4 rootflags=noatime mitigations=off nmi_watchdog=0 rw rd.driver.pre=binfmt_misc oops=panic panic=1 quiet elevator=noop console=hvc0 init=/.build/build -m 4096 -drive file=/var/tmp/build-root/openSUSE:Factory/standard/x86_64/img,format=raw,if=none,id=disk,cache=unsafe -device virtio-blk,drive=disk,serial=0 -drive file=/var/tmp/build-root/openSUSE:Factory/standard/x86_64/swap,format=raw,if=none,id=swap,cache=unsafe -device virtio-blk,drive=swap,serial=1 -device virtio-serial,max_ports=2 -device virtconsole,chardev=virtiocon0 -chardev stdio,id=virtiocon0 -chardev socket,id=monitor,server=on,wait=off,path=/var/tmp/build-root/openSUSE:Factory/standard/x86_64/img.qemu/monitor -mon chardev=monitor,mode=readline -smp 2
[   21s] [    9.370485][    T1] systemd[1]: Failed to start Virtual Console Setup.
[   21s] [FAILED] Failed to start Virtual Console Setup.
[   26s] [   14.555504] dracut-pre-udev[290]: sh: line 1: /usr/lib/module-init-tools/unblacklist: No such file or directory
[   39s] [FAILED] Failed to start Virtual Console Setup.
[   41s] [FAILED] Failed to start Switch Root.
[   43s] 
[   43s] Generating "/run/initramfs/rdsosreport.txt"
[   43s] 
[   43s] 
[   43s] Entering emergency mode. Exit the shell to continue.
[   43s] Type "journalctl" to view system logs.
[   43s] You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
[   43s] after mounting them and attach it to a bug report.
[   43s] 
[   43s] 
[   44s] Press Enter for maintenance
[   48s] (or press Control-D to continue): 
:/root# systemctl
[   71s]   UNIT                                                                           LOAD   ACTIVE SUB     DESCRIPTION                                                             
[   71s]   sys-devices-pci0000:00-0000:00:02.0-virtio0-block-vda.device                   loaded active plugged /sys/devices/pci0000:00/0000:00:02.0/virtio0/block/vda
[   71s]   sys-devices-pci0000:00-0000:00:03.0-virtio1-block-vdb.device                   loaded active plugged /sys/devices/pci0000:00/0000:00:03.0/virtio1/block/vdb
[   71s]   sys-devices-pci0000:00-0000:00:04.0-virtio2-virtio\x2dports-vport2p0.device    loaded active plugged /sys/devices/pci0000:00/0000:00:04.0/virtio2/virtio-ports/vport2p0
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.0-tty-ttyS0.device   loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.0/tty/ttyS0
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.1-tty-ttyS1.device   loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.1/tty/ttyS1
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.10-tty-ttyS10.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.10/tty/ttyS10
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.11-tty-ttyS11.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.11/tty/ttyS11
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.12-tty-ttyS12.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.12/tty/ttyS12
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.13-tty-ttyS13.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.13/tty/ttyS13
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.14-tty-ttyS14.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.14/tty/ttyS14
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.15-tty-ttyS15.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.15/tty/ttyS15
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.16-tty-ttyS16.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.16/tty/ttyS16
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.17-tty-ttyS17.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.17/tty/ttyS17
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.18-tty-ttyS18.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.18/tty/ttyS18
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.19-tty-ttyS19.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.19/tty/ttyS19
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.2-tty-ttyS2.device   loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.2/tty/ttyS2
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.20-tty-ttyS20.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.20/tty/ttyS20
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.21-tty-ttyS21.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.21/tty/ttyS21
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.22-tty-ttyS22.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.22/tty/ttyS22
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.23-tty-ttyS23.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.23/tty/ttyS23
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.24-tty-ttyS24.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.24/tty/ttyS24
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.25-tty-ttyS25.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.25/tty/ttyS25
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.26-tty-ttyS26.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.26/tty/ttyS26
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.27-tty-ttyS27.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.27/tty/ttyS27
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.28-tty-ttyS28.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.28/tty/ttyS28
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.29-tty-ttyS29.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.29/tty/ttyS29
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.3-tty-ttyS3.device   loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.3/tty/ttyS3
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.30-tty-ttyS30.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.30/tty/ttyS30
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.31-tty-ttyS31.device loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.31/tty/ttyS31
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.4-tty-ttyS4.device   loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.4/tty/ttyS4
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.5-tty-ttyS5.device   loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.5/tty/ttyS5
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.6-tty-ttyS6.device   loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.6/tty/ttyS6
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.7-tty-ttyS7.device   loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.7/tty/ttyS7
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.8-tty-ttyS8.device   loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.8/tty/ttyS8
[   71s]   sys-devices-platform-serial8250-serial8250:0-serial8250:0.9-tty-ttyS9.device   loaded active plugged /sys/devices/platform/serial8250/serial8250:0/serial8250:0.9/tty/ttyS9
[   71s]   sys-devices-virtual-tty-hvc0.device                                            loaded active plugged /sys/devices/virtual/tty/hvc0
[   71s]   sys-devices-virtual-tty-hvc1.device                                            loaded active plugged /sys/devices/virtual/tty/hvc1
[   71s]   sys-devices-virtual-tty-hvc2.device                                            loaded active plugged /sys/devices/virtual/tty/hvc2
[   71s]   sys-devices-virtual-tty-hvc3.device                                            loaded active plugged /sys/devices/virtual/tty/hvc3
[   71s]   sys-devices-virtual-tty-hvc4.device                                            loaded active plugged /sys/devices/virtual/tty/hvc4
[   71s]   sys-devices-virtual-tty-hvc5.device                                            loaded active plugged /sys/devices/virtual/tty/hvc5
[   71s]   sys-devices-virtual-tty-hvc6.device                                            loaded active plugged /sys/devices/virtual/tty/hvc6
[   71s]   sys-devices-virtual-tty-hvc7.device                                            loaded active plugged /sys/devices/virtual/tty/hvc7
[   71s]   sys-module-fuse.device                                                         loaded active plugged /sys/module/fuse                                                        
[   71s]   -.mount                                                                        loaded active mounted Root Mount
[   71s]   sysroot.mount                                                                  loaded active mounted /sysroot                                                                
[   71s]   init.scope                                                                     loaded active running System and Service Manager                                              
[   71s]   emergency.service                                                              loaded active running Emergency Shell
[   71s] ● initrd-switch-root.service                                                     loaded failed failed  Switch Root
[   71s]   systemd-journald.service                                                       loaded active running Journal Service
[   71s] ● systemd-vconsole-setup.service                                                 loaded failed failed  Virtual Console Setup                                                   
[   71s]   -.slice                                                                        loaded active active  Root Slice
[   71s]   system.slice                                                                   loaded active active  System Slice                                                            
[   71s]   systemd-journald-dev-log.socket                                                loaded active running Journal Socket (/dev/log)
[   71s]   systemd-journald.socket                                                        loaded active running Journal Socket                                                          
[   71s]   emergency.target                                                               loaded active active  Emergency Mode
[   71s]   initrd-fs.target                                                               loaded active active  Initrd File Systems
[   71s]   initrd-root-fs.target                                                          loaded active active  Initrd Root File System
[   71s]   initrd-switch-root.target                                                      loaded active active  Switch Root
[   71s] 
[   71s] Legend: LOAD   → Reflects whether the unit definition was properly loaded.
[   71s]         ACTIVE → The high-level unit activation state, i.e. generalization of SUB.
[   71s]         SUB    → The low-level unit activation state, values depend on unit type.
[   71s] 
[   71s] 59 loaded units listed. Pass --all to see loaded but inactive units, too.
[   71s] To show all installed unit files use 'systemctl list-unit-files'.
[   80s] :/root# journalctl -u initrd-switch-root.service
[   80s] Mar 20 13:12:20 localhost systemd[1]: Starting Switch Root...
[   80s] Mar 20 13:12:21 localhost @ystemctl[383]: Failed to switch root: Specified switch root path '/sysroot' does not seem to be an OS tree. os-release file is missing.
[   80s] Mar 20 13:12:21 localhost systemd[1]: initrd-switch-root.service: Main process exited, code=exited, status=1/FAILURE
[   80s] Mar 20 13:12:21 localhost systemd[1]: initrd-switch-root.service: Failed with result 'exit-code'.
[   80s] Mar 20 13:12:21 localhost systemd[1]: Failed to start Switch Root.
[   80s] Mar 20 13:12:21 localhost systemd[1]: initrd-switch-root.service: Triggering OnFailure= dependencies.

Comment 1 Franck Bui 2024-03-21 16:18:27 UTC

(In reply to Andreas Schwab from comment #0)
> [   80s] Mar 20 13:12:20 localhost systemd[1]: Starting Switch Root...
> [   80s] Mar 20 13:12:21 localhost @ystemctl[383]: Failed to switch root:
> Specified switch root path '/sysroot' does not seem to be an OS tree.
> os-release file is missing.

Dominique fixed a similar issue recently: https://github.com/openSUSE/obs-build/commit/c295ac72c80dc1f4107c5c1ef703061b7b080869

The fix has probably not reached tumbleweed yet.

Dominique, do you confirm ?

Comment 2 Dominique Leuenberger 2024-03-25 10:20:32 UTC

are you running 'osc build' on an up-to-date TW system or a Leap/SLE system?

We are aware that 'build' would need to go out as maintenance update to SLE/Leap in order to support TW/kvm builds there. On TW, the build package needs to be up-to-date

> rpm -q build
build-20240313-1.1.noarch

* Mi Mär 13 2024 Adrian Schröter <adrian@suse.de>
- Add a hack for systemd 255.3, creating an empty /etc/os-release
  if missing after preinstall.

Comment 3 Franck Bui 2024-04-15 14:16:32 UTC

Andreas, can you provide some feedback to Dominique otherwise I'll assume that your problem is solved somehow and will close the bug.

Thanks.

Comment 4 Sławomir Lach 2024-04-25 09:08:24 UTC

Same bug probably on OBS installed on OpenSUSE Leap 15.5, see: https://forums.opensuse.org/t/open-build-service-bledy/173502/10 (if you known Polish, sorry). If you need some more info/translation, I will try to deliver it to you.

Comment 5 Dominique Leuenberger 2024-06-05 09:35:24 UTC

The systemd side of things are in place, OBS-build is generally fixed, but not yet released as maintenance update to CODE15

Re-assigning to Adrian, hopefully we can get a build update out soon™