Bug 1213595

Summary: Live ISOs fail to boot due to wrong by-label links
Product: [openSUSE] openSUSE Distribution Reporter: Sehsuvar SELMANOGLU <sehsuvarselmanoglu>
Component: BasesystemAssignee: Marcus Schäfer <marcus.schaefer>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Major    
Priority: P5 - None CC: fbui, fvogt, lubos.kocman, ms, sehsuvarselmanoglu
Version: Leap 15.5   
Target Milestone: ---   
Hardware: x86-64   
OS: Other   
URL: https://github.com/OSInside/kiwi/issues/2338
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: 7z File for some logs

Description Sehsuvar SELMANOGLU 2023-07-24 12:00:52 UTC
Created attachment 868398 [details]
7z File for some logs

Hi,
I downloaded OpenSUSE Leap ISO (openSUSE-Leap-15.5-KDE-Live-x86_64-Build10.40-Media.iso) at 08/06/2023 (as soon as released) then I “dd” to USB stick and plugged to computer and it worked fine. At first boot, it did create the persistent storage cow (boot.log : [   22.002982] dracut-initqueue[877]: mke2fs 1.46.4 (18-Aug-2021) )

Few days later I download again this time build number was different (openSUSE-Leap-15.5-KDE-Live-x86_64-Build10.57-Media.iso). At first boot, it did not create persistent storage. dracut says (boot.log : [   21.489260] dracut-initqueue[769]: Warning: Falling back to temporary overlay)
same as the new release (openSUSE-Leap-15.5-KDE-Live-x86_64-Build10.73-Media.iso)

Any release after 10.40 does not creates persistent storage (neither 10.53 nor 10.73) 


I tried with different CPU (AMD, Core Duo, I5) and different USB stick (4GB 8GB
16GB and 32GB) even with different port (USB2 USB3) with different USB s and so on. No matter what I tried don't work. Releases after 10.40 (10.57 or 10.73) do 
not creates the persistent storage.

So I open discussion on the forums.opensuse.org and someone suggested that 
I need to open bug report. and he/she wrote "It is not something you can fix without recreating initrd. The script that is used here comes from kiwi dracut module.".

I also add the 7zip file for each ISO "....10.40", "...10.57", "...10.73" that contains some log files.

"boot.log" : "sudo cp /var/log/boot.log"
"boot.msg" : "cp /var/log/boot.msg"
"dmesg.txt" : Output of "dmesg" right after the first boot
"fdisk.txt" : 0utput of "fdisk -l" right after first boot
"journal.txt" : Output of "sudo journalctl --no-pager --full -b"

Thank you very much.
Comment 1 Fabian Vogt 2023-07-26 13:43:39 UTC
Confirmed, this is also visible in openQA: https://openqa.opensuse.org/tests/3456098

[    9.243802] dracut-initqueue[882]: mount: /run/overlay/overlayfs: can't find LABEL="cow".
[    9.254383] dracut-initqueue[887]: This disk is currently in use - repartitioning is probably a bad idea.
[    9.254420] dracut-initqueue[887]: It's recommended to umount all file systems, and swapoff all swap
[    9.254442] dracut-initqueue[887]: partitions on this disk.
[    9.254459] dracut-initqueue[887]: The device contains 'iso9660' signature and it may remain on the device. It is recommended to wipe the device with wipefs(8) or fdisk --wipe, in order to avoid possible collisions.
[    9.291225] dracut-initqueue[900]: mke2fs 1.46.4 (18-Aug-2021)
[    9.291387] dracut-initqueue[900]: /dev/sda3 is apparently in use by the system; will not make a filesystem here!
[    9.291630] dracut-initqueue[813]: Warning: Failed to setup persistent write space !
[    9.291667] dracut-initqueue[813]: Warning: Falling back to temporary overlay

The test passes though because it doesn't (yet) verify persistence, only that everything works.

Probably a bug in kiwi, I'll collect some info to confirm that.
Comment 2 Fabian Vogt 2023-07-26 14:57:25 UTC
According to blkid, both /dev/sda and /dev/sda1 have LABEL="openSUSE_Leap_15.5_KDE_Live" and udev links sda into /dev/disk/by-label. This means that the whole disk is mounted, not just the first partition.

Can it be avoided that the label is also visible on /dev/sda?

Any idea what could cause this behaviour change?
At least in the current TW snapshot it appears to be working.
Comment 3 Sehsuvar SELMANOGLU 2023-12-17 21:02:23 UTC
Since release of Build12.x the bug disappered it it working now (Build13.8)
Comment 4 Sehsuvar SELMANOGLU 2023-12-17 21:04:26 UTC
openSUSE-Leap-15.5-KDE-Live-x86_64-Build13.8-Media.iso works fine
Comment 5 Fabian Vogt 2024-03-07 15:57:44 UTC
Looks like this is broken again somehow, all openQA runs with 15.5 and 15.6 isos booted off USB fail: https://openqa.opensuse.org/tests/3992142#step/finish_desktop/2

I booted the system with "debug console=ttyS0 systemd.log_target=console rd.udev.log_level=debug", fortunately it still fails. Log is attached.

It is visible that udev prefers the by-label link to sda instead of sda1 for some reason, even though https://bugzilla.suse.com/show_bug.cgi?id=1213185 implies that the fix for https://github.com/systemd/systemd/issues/28468 got backported.

dev-disk-by\x2duuid-2024\x2d02\x2d21\x2d09\x2d57\x2d45\x2d00.device: Device dev-disk-by\x2duuid-2024\x2d02\x2d21\x2d09\x2d57\x2d45\x2d00.device appeared twice with different sysfs paths /sys/devices/pci0000:00/0000:00:09.0/usb4/4-1/4-1:1.0/host3/target3:0:0/3:0:0:0/block/sda and /sys/devices/pci0000:00/0000:00:09.0/usb4/4-1/4-1:1.0/host3/target3:0:0/3:0:0:0/block/sda/sda1, ignoring the latter.
dev-disk-by\x2dlabel-openSUSE_Leap_15.5_Rescue_CD.device: Device dev-disk-by\x2dlabel-openSUSE_Leap_15.5_Rescue_CD.device appeared twice with different sysfs paths /sys/devices/pci0000:00/0000:00:09.0/usb4/4-1/4-1:1.0/host3/target3:0:0/3:0:0:0/block/sda and /sys/devices/pci0000:00/0000:00:09.0/usb4/4-1/4-1:1.0/host3/target3:0:0/3:0:0:0/block/sda/sda1, ignoring the latter.
Comment 6 OBSbugzilla Bot 2024-03-27 17:35:01 UTC
This is an autogenerated message for OBS integration:
This bug (1213595) was mentioned in
https://build.opensuse.org/request/show/1162956 Factory / python-kiwi
Comment 9 Franck Bui 2024-03-28 10:09:18 UTC
After some investigation (and offline discussions) it appears that the problem lives in kiwi. It operates on the disk containing the live OS (to create a third partition) and this appears to generates some REMOVE uevents for the existing partitions. This has the side effect to temporarily modify the by-label symlink.

Reassigning to Marcus.
Comment 10 Marcus Schäfer 2024-03-28 10:36:48 UTC
Yep and the fix for this was merged upstream and package updates for Factory/TW
ALP and SLES/Leap has been created.

Thanks for your help in debugging the issue
Comment 11 Lubos Kocman 2024-04-03 08:10:22 UTC
Many thanks!
Comment 13 Marcus Schäfer 2024-04-15 13:33:13 UTC
Should be fixed now
Comment 14 Maintenance Automation 2024-04-22 12:30:07 UTC
SUSE-RU-2024:1371-1: An update that has four fixes can now be installed.

Category: recommended (moderate)
Bug References: 1207128, 1213595, 1219074, 1221915
Maintenance Incident: [SUSE:Maintenance:33219](https://smelt.suse.de/incident/33219/)
Sources used:
openSUSE Leap 15.5 (src):
 python-kiwi-9.24.43-150100.3.76.1
SUSE Linux Enterprise Micro for Rancher 5.3 (src):
 python-kiwi-9.24.43-150100.3.76.1
SUSE Linux Enterprise Micro 5.3 (src):
 python-kiwi-9.24.43-150100.3.76.1
SUSE Linux Enterprise Micro for Rancher 5.4 (src):
 python-kiwi-9.24.43-150100.3.76.1
SUSE Linux Enterprise Micro 5.4 (src):
 python-kiwi-9.24.43-150100.3.76.1
SUSE Linux Enterprise Micro 5.5 (src):
 python-kiwi-9.24.43-150100.3.76.1
Development Tools Module 15-SP5 (src):
 python-kiwi-9.24.43-150100.3.76.1
SUSE Linux Enterprise Micro 5.1 (src):
 python-kiwi-9.24.43-150100.3.76.1
SUSE Linux Enterprise Micro 5.2 (src):
 python-kiwi-9.24.43-150100.3.76.1
SUSE Linux Enterprise Micro for Rancher 5.2 (src):
 python-kiwi-9.24.43-150100.3.76.1
openSUSE Leap Micro 5.3 (src):
 python-kiwi-9.24.43-150100.3.76.1
openSUSE Leap Micro 5.4 (src):
 python-kiwi-9.24.43-150100.3.76.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 17 Maintenance Automation 2024-07-16 12:31:14 UTC
SUSE-RU-2024:1371-2: An update that has four fixes can now be installed.

Category: recommended (moderate)
Bug References: 1207128, 1213595, 1219074, 1221915
Maintenance Incident: [SUSE:Maintenance:33219](https://smelt.suse.de/incident/33219/)
Sources used:
SUSE Linux Enterprise Micro 5.5 (src):
 python-kiwi-9.24.43-150100.3.76.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.