Bugzilla – Bug 1219289
[Build 50.1] multipath btrfs i/o error on both Leap 15.6 and SLES 15 SP6
Last modified: 2024-03-12 22:51:25 UTC
[ 19.580062][ T1541] Btrfs loaded, assert=on, zoned=yes, fsverity=yes YpuUt-0- N06xI-0- A99Hf-0- KdUfq-0- T_TZ1-0- X5qi2-0- 4U~aA-0- tzby~-0- 5RNha-0- FCJd_-0- [ 542.229523][ T1] reboot: Restarting system [ 4.763467][ T39] I/O error, dev dm-0, sector 26624 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 4.764673][ T39] I/O error, dev dm-0, sector 6281672 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 2 [ 4.765835][ T39] I/O error, dev dm-0, sector 27461752 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 4.774008][ T39] I/O error, dev dm-0, sector 26624 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 [ 4.775816][ T39] Buffer I/O error on dev dm-2, logical block 1024, async page read [ 4.777179][ T39] I/O error, dev dm-0, sector 27461752 op 0x0:(READ) flags 0x0 phys_seg 8 prio class 2 [ 4.778832][ T39] Buffer I/O error on dev dm-3, logical block 120, async page read [ 4.779942][ T39] Buffer I/O error on dev dm-3, logical block 121, async page read [ 4.780861][ T39] Buffer I/O error on dev dm-3, logical block 122, async page read [ 4.781779][ T39] Buffer I/O error on dev dm-3, logical block 123, async page read [ 4.782713][ T39] Buffer I/O error on dev dm-3, logical block 124, async page read [ 4.783632][ T39] Buffer I/O error on dev dm-3, logical block 125, async page read [ 4.784551][ T39] Buffer I/O error on dev dm-3, logical block 126, async page read [ 4.785468][ T39] Buffer I/O error on dev dm-3, logical block 127, async page read [ 4.787739][ T39] I/O error, dev dm-0, sector 6281672 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 [ 4.788877][ T39] BTRFS error (device dm-2): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 0, rd 1, flush 0, corrupt 0, gen 0 [ 4.806655][ T764] I/O error, dev dm-0, sector 5776032 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 2 [ 4.808202][ T39] I/O error, dev dm-0, sector 5776032 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 [ 4.809547][ T39] BTRFS error (device dm-2): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 0, rd 2, flush 0, corrupt 0, gen 0 [ 4.826490][ T39] I/O error, dev dm-0, sector 5776032 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 [ 4.827621][ T39] BTRFS error (device dm-2): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 0, rd 3, flush 0, corrupt 0, gen 0 [ 4.840281][ T725] I/O error, dev dm-0, sector 6667344 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2 [ 4.841501][ T39] BTRFS error (device dm-2): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 0, rd 4, flush 0, corrupt 0, gen 0 [ 4.851372][ T39] BTRFS error (device dm-2): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 0, rd 5, flush 0, corrupt 0, gen 0 [ 4.854475][ T39] BTRFS error (device dm-2): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 0, rd 6, flush 0, corrupt 0, gen 0 [ 4.856205][ T39] BTRFS error (device dm-2): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 0, rd 7, flush 0, corrupt 0, gen 0 [ 4.859367][ T39] BTRFS error (device dm-2): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 0, rd 8, flush 0, corrupt 0, gen 0 [ 4.861346][ T39] BTRFS error (device dm-2): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 0, rd 9, flush 0, corrupt 0, gen 0 [ 4.862902][ T39] BTRFS error (device dm-2): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 0, rd 10, flush 0, corrupt 0, gen 0 [ 5.638429][ T175] BTRFS error (device dm-2): failed to run delayed ref for logical 187138048 num_bytes 16384 type 182 action 1 ref_mod 1: -5 [ 5.639984][ T175] BTRFS: error (device dm-2: state A) in btrfs_run_delayed_refs:2127: errno=-5 IO failure [ 35.198650][ T175] I/O error, dev dm-0, sector 6395984 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 2 [ 35.201357][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 2, rd 349, flush 0, corrupt 0, gen 0 [ 35.205056][ T175] I/O error, dev dm-0, sector 6399680 op 0x1:(WRITE) flags 0x100000 phys_seg 2 prio class 2 [ 35.207572][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 3, rd 349, flush 0, corrupt 0, gen 0 [ 35.211061][ T175] I/O error, dev dm-0, sector 6399872 op 0x1:(WRITE) flags 0x100000 phys_seg 5 prio class 2 [ 35.213086][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 4, rd 349, flush 0, corrupt 0, gen 0 [ 35.216132][ T175] I/O error, dev dm-0, sector 6401328 op 0x1:(WRITE) flags 0x100000 phys_seg 4 prio class 2 [ 35.218191][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 5, rd 349, flush 0, corrupt 0, gen 0 [ 35.221271][ T175] I/O error, dev dm-0, sector 6416888 op 0x1:(WRITE) flags 0x100000 phys_seg 5 prio class 2 [ 35.223343][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 6, rd 349, flush 0, corrupt 0, gen 0 [ 35.226447][ T175] I/O error, dev dm-0, sector 6484096 op 0x1:(WRITE) flags 0x100000 phys_seg 24 prio class 2 [ 35.228524][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 7, rd 349, flush 0, corrupt 0, gen 0 [ 35.231635][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 8, rd 349, flush 0, corrupt 0, gen 0 [ 35.234760][ T175] I/O error, dev dm-0, sector 6484544 op 0x1:(WRITE) flags 0x100000 phys_seg 10 prio class 2 [ 35.236854][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 9, rd 349, flush 0, corrupt 0, gen 0 [ 35.240709][ T175] I/O error, dev dm-0, sector 6484832 op 0x1:(WRITE) flags 0x100000 phys_seg 4 prio class 2 [ 35.243366][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 10, rd 349, flush 0, corrupt 0, gen 0 [ 35.247280][ T175] I/O error, dev dm-0, sector 6485168 op 0x1:(WRITE) flags 0x100000 phys_seg 10 prio class 2 [ 35.249895][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 11, rd 349, flush 0, corrupt 0, gen 0 [ 35.253811][ T175] I/O error, dev dm-0, sector 6485456 op 0x1:(WRITE) flags 0x100000 phys_seg 10 prio class 2 [ 40.318690][ T175] I/O error, dev dm-0, sector 6399872 op 0x1:(WRITE) flags 0x100000 phys_seg 4 prio class 2 [ 40.321502][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 15, rd 349, flush 0, corrupt 0, gen 0 [ 40.325400][ T175] I/O error, dev dm-0, sector 6401328 op 0x1:(WRITE) flags 0x100000 phys_seg 6 prio class 2 [ 40.327985][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 16, rd 349, flush 0, corrupt 0, gen 0 [ 40.331497][ T175] I/O error, dev dm-0, sector 6416888 op 0x1:(WRITE) flags 0x100000 phys_seg 5 prio class 2 [ 40.333752][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 17, rd 349, flush 0, corrupt 0, gen 0 [ 40.336850][ T175] I/O error, dev dm-0, sector 6417312 op 0x1:(WRITE) flags 0x100000 phys_seg 5 prio class 2 [ 40.338918][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 18, rd 349, flush 0, corrupt 0, gen 0 [ 40.342008][ T175] I/O error, dev dm-0, sector 6433488 op 0x1:(WRITE) flags 0x100000 phys_seg 6 prio class 2 [ 40.344086][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 19, rd 349, flush 0, corrupt 0, gen 0 [ 40.347223][ T175] I/O error, dev dm-0, sector 6452392 op 0x1:(WRITE) flags 0x100000 phys_seg 6 prio class 2 [ 40.349316][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 20, rd 349, flush 0, corrupt 0, gen 0 [ 40.352483][ T175] I/O error, dev dm-0, sector 6484096 op 0x1:(WRITE) flags 0x100000 phys_seg 29 prio class 2 [ 40.354611][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 21, rd 349, flush 0, corrupt 0, gen 0 [ 40.357856][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 22, rd 349, flush 0, corrupt 0, gen 0 [ 40.361028][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 23, rd 349, flush 0, corrupt 0, gen 0 [ 40.364214][ T175] I/O error, dev dm-0, sector 6484544 op 0x1:(WRITE) flags 0x100000 phys_seg 8 prio class 2 [ 40.366316][ T175] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 24, rd 349, flush 0, corrupt 0, gen 0 [ 40.369490][ T175] I/O error, dev dm-0, sector 6484832 op 0x1:(WRITE) flags 0x100000 phys_seg 20 prio class 2 [ 40.371617][ T175] I/O error, dev dm-0, sector 6485168 op 0x1:(WRITE) flags 0x100000 phys_seg 12 prio class 2 [ 54.119264][ T1075] I/O error, dev dm-0, sector 1446568 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 2 [ 54.199114][ T39] I/O error, dev dm-0, sector 1446568 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 [ 54.203343][ T39] BTRFS error (device dm-2: state EA): bdev /dev/mapper/0QEMU_QEMU_HARDDISK_hd0-part2 errs: wr 33, rd 350, flush 0, corrupt 0, gen 0 ## Observation openQA test in scenario sle-15-SP6-Online-x86_64-multipath@64bit fails in [first_boot](https://openqa.suse.de/tests/13364459/modules/first_boot/steps/6) ## Test suite description Testsuite maintained at https://gitlab.suse.de/qa-maintenance/qam-openqa-yml. Maintainer: QE Yast Test installation on machine with virtual multipath hardware. Only tests succesful detection of multipath and installation. No functional testing of multipath itself. ## Reproducible Fails since (at least) Build [46.40](https://openqa.suse.de/tests/13241300) Happens on Leap 15.6 as well, so very well reproducible. Restarting test resulted into the same error. https://openqa.opensuse.org/tests/3898697#step/first_boot/18 ## Further details Always latest result in this scenario: [latest](https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Online&machine=64bit&test=multipath&version=15-SP6)
Eugenio mentioned that test has not been run in 15 SP6 44.1 (Beta2). So it's quite possible that the issue was already there. We've only noticed the issue on Leap 15.6 Build597.1. Leap 15.6 build from 7 days ago haven't had this issue, but was broken for other reasons https://openqa.opensuse.org/tests/overview?distri=opensuse&version=15.6&build=590.1&groupid=50
Can you check whether it's a kernel regression? That is, compare the results between both 6.14.0-150600-4 and 6.14.0-150600.5 kernels. Also, test with the very latest SLE15-SP6 KOTD would be appreciated.
I did ask Jan Stehlik to find someone from within QE, multipath testing is outside my possibilities.
(In reply to Takashi Iwai from comment #2) > Can you check whether it's a kernel regression? That is, compare the > results between both 6.14.0-150600-4 and 6.14.0-150600.5 kernels. Last working fine build was 45.1 with kernel 6.14.0-150600-4. First failing build was 47.2 with kernel 6.14.0-150600-4. Kernel was not changed at all, it must be something else, maybe in installer? I will try to look on it more, but test fails in installation scenario (no qcow2 image was produced).
I managed to recreate last working 15-SP6 build 45.1 in openQA and created image which I imported into my virtual multipath setup. Issue is not related to kernel, multipath, btrfs directly. Issue is in sg3_utils. It is enough just to upgrade sg3_utils (and nothing else) and system will fail with reported error. Version upgrade: sg3_utils 1.47+13.75d23ac-150400.3.6.1 -> 1.48+4.fa9618a-150600.1.1 Here is at least log during installation (not sure if it is usefull): (1/1) Installing: sg3_utils-1.48+4.fa9618a-150600.1.1.x86_64 ............................................................................................................................................................................................................................................................[done] %posttrans(sg3_utils-1.48+4.fa9618a-150600.1.1.x86_64) script output: dracut[I]: Executing: /usr/bin/dracut --kver=6.4.0-150600.4-default -f dracut[I]: Module 'systemd-networkd' will not be installed, because command 'networkctl' could not be found! dracut[I]: Module 'systemd-networkd' will not be installed, because command '/usr/lib/systemd/systemd-networkd' could not be found! dracut[I]: Module 'systemd-networkd' will not be installed, because command '/usr/lib/systemd/systemd-networkd-wait-online' could not be found! dracut[I]: Module 'systemd-pcrphase' will not be installed, because command '/usr/lib/systemd/systemd-pcrphase' could not be found! dracut[I]: Module 'systemd-portabled' will not be installed, because command 'portablectl' could not be found! dracut[I]: Module 'systemd-portabled' will not be installed, because command '/usr/lib/systemd/systemd-portabled' could not be found! dracut[I]: Module 'systemd-repart' will not be installed, because command 'systemd-repart' could not be found! dracut[I]: Module 'systemd-resolved' will not be installed, because command 'resolvectl' could not be found! dracut[I]: Module 'systemd-resolved' will not be installed, because command '/usr/lib/systemd/systemd-resolved' could not be found! dracut[I]: Module 'dbus-broker' will not be installed, because command 'dbus-broker' could not be found! dracut[I]: Module 'rngd' will not be installed, because command 'rngd' could not be found! dracut[I]: Module 'connman' will not be installed, because command 'connmand' could not be found! dracut[I]: Module 'connman' will not be installed, because command 'connmanctl' could not be found! dracut[I]: Module 'connman' will not be installed, because command 'connmand-wait-online' could not be found! dracut[I]: Module 'network-manager' will not be installed, because command 'NetworkManager' could not be found! dracut[I]: 62bluetooth: Could not find any command of '/usr/lib/bluetooth/bluetoothd /usr/libexec/bluetooth/bluetoothd'! dracut[I]: Module 'dmraid' will not be installed, because command 'dmraid' could not be found! dracut[I]: Module 'dmsquash-live-ntfs' will not be installed, because command 'ntfs-3g' could not be found! dracut[I]: Module 'pcsc' will not be installed, because command 'pcscd' could not be found! dracut[I]: Module 'tpm2-tss' will not be installed, because command 'tpm2' could not be found! dracut[I]: Module 'iscsi' will not be installed, because command 'iscsi-iname' could not be found! dracut[I]: Module 'iscsi' will not be installed, because command 'iscsiadm' could not be found! dracut[I]: Module 'iscsi' will not be installed, because command 'iscsid' could not be found! dracut[I]: Module 'nvmf' will not be installed, because command 'nvme' could not be found! dracut[I]: Module 'nvmf' will not be installed, because command 'jq' could not be found! dracut[I]: Module 'biosdevname' will not be installed, because command 'biosdevname' could not be found! dracut[I]: Module 'memstrack' will not be installed, because command 'memstrack' could not be found! dracut[I]: memstrack is not available dracut[I]: If you need to use rd.memdebug>=4, please install memstrack and procps-ng dracut[I]: Module 'squash' will not be installed, because command 'mksquashfs' could not be found! dracut[I]: Module 'squash' will not be installed, because command 'unsquashfs' could not be found! dracut[I]: Module 'systemd-pcrphase' will not be installed, because command '/usr/lib/systemd/systemd-pcrphase' could not be found! dracut[I]: Module 'systemd-portabled' will not be installed, because command 'portablectl' could not be found! dracut[I]: Module 'systemd-portabled' will not be installed, because command '/usr/lib/systemd/systemd-portabled' could not be found! dracut[I]: Module 'systemd-repart' will not be installed, because command 'systemd-repart' could not be found! dracut[I]: Module 'systemd-resolved' will not be installed, because command 'resolvectl' could not be found! dracut[I]: Module 'systemd-resolved' will not be installed, because command '/usr/lib/systemd/systemd-resolved' could not be found! dracut[I]: Module 'dbus-broker' will not be installed, because command 'dbus-broker' could not be found! dracut[I]: Module 'rngd' will not be installed, because command 'rngd' could not be found! dracut[I]: Module 'connman' will not be installed, because command 'connmand' could not be found! dracut[I]: Module 'connman' will not be installed, because command 'connmanctl' could not be found! dracut[I]: Module 'connman' will not be installed, because command 'connmand-wait-online' could not be found! dracut[I]: Module 'network-manager' will not be installed, because command 'NetworkManager' could not be found! dracut[I]: 62bluetooth: Could not find any command of '/usr/lib/bluetooth/bluetoothd /usr/libexec/bluetooth/bluetoothd'! dracut[I]: Module 'dmraid' will not be installed, because command 'dmraid' could not be found! dracut[I]: Module 'dmsquash-live-ntfs' will not be installed, because command 'ntfs-3g' could not be found! dracut[I]: Module 'pcsc' will not be installed, because command 'pcscd' could not be found! dracut[I]: Module 'tpm2-tss' will not be installed, because command 'tpm2' could not be found! dracut[I]: Module 'iscsi' will not be installed, because command 'iscsi-iname' could not be found! dracut[I]: Module 'iscsi' will not be installed, because command 'iscsiadm' could not be found! dracut[I]: Module 'iscsi' will not be installed, because command 'iscsid' could not be found! dracut[I]: Module 'nvmf' will not be installed, because command 'nvme' could not be found! dracut[I]: Module 'nvmf' will not be installed, because command 'jq' could not be found! dracut[I]: Module 'memstrack' will not be installed, because command 'memstrack' could not be found! dracut[I]: memstrack is not available dracut[I]: If you need to use rd.memdebug>=4, please install memstrack and procps-ng dracut[I]: Module 'squash' will not be installed, because command 'mksquashfs' could not be found! dracut[I]: Module 'squash' will not be installed, because command 'unsquashfs' could not be found! dracut[I]: *** Including module: systemd *** dracut[I]: *** Including module: systemd-initrd *** dracut[I]: *** Including module: i18n *** dracut[I]: *** Including module: drm *** dracut[I]: *** Including module: plymouth *** dracut[I]: *** Including module: btrfs *** dracut[I]: *** Including module: dm *** dracut[I]: *** Including module: kernel-modules *** dracut[I]: *** Including module: kernel-modules-extra *** dracut[I]: *** Including module: multipath *** dracut[I]: *** Including module: qemu *** dracut[I]: *** Including module: resume *** dracut[I]: *** Including module: rootfs-block *** dracut[I]: *** Including module: suse-btrfs *** dracut[I]: *** Including module: suse-xfs *** dracut[I]: *** Including module: terminfo *** dracut[I]: *** Including module: udev-rules *** dracut[I]: *** Including module: virtiofs *** dracut[I]: *** Including module: dracut-systemd *** dracut[I]: *** Including module: haveged *** dracut[I]: *** Including module: usrmount *** dracut[I]: *** Including module: base *** dracut[I]: *** Including module: fs-lib *** dracut[I]: *** Including module: shutdown *** dracut[I]: *** Including module: suse *** dracut[I]: *** Including module: suse-initrd *** dracut[I]: *** Including modules done *** dracut[I]: *** Installing kernel module dependencies *** dracut[I]: *** Installing kernel module dependencies done *** dracut[I]: *** Resolving executable dependencies *** dracut[I]: *** Resolving executable dependencies done *** dracut[I]: *** Hardlinking files *** dracut[I]: *** Hardlinking files done *** dracut[I]: *** Generating early-microcode cpio image *** dracut[I]: *** Constructing GenuineIntel.bin *** dracut[I]: *** Store current command line parameters *** dracut[I]: Stored kernel commandline: dracut[I]: rd.driver.pre=btrfs dracut[I]: rd.driver.pre=scsi_dh_alua rd.driver.pre=scsi_dh_emc rd.driver.pre=scsi_dh_rdac rd.driver.pre=dm_multipath dracut[I]: resume=UUID=15d401e6-a69c-4db6-b6b1-c02c10d6d21d dracut[I]: root=UUID=120d4cdb-e95e-45a0-8bc6-38796ed22e48 rootfstype=btrfs rootflags=rw,relatime,discard=async,space_cache=v2,subvolid=272,subvol=/@/.snapshots/6/snapshot,subvol=@/.snapshots/6/snapshot dracut[I]: *** Stripping files *** dracut[I]: *** Stripping files done *** dracut[I]: *** Creating image file '/boot/initrd-6.4.0-150600.4-default' *** dracut[I]: *** Creating initramfs image file '/boot/initrd-6.4.0-150600.4-default' done *** Running post-transaction scripts ........................................................................................................................................................................................................................................................................................[done] CommitResult (total 1, done 1, error 0, skipped 0, updateMessages 0) Checking for running processes using deleted libraries... Do you need additional info? I can easily reproduce it, just system is usually unusable after fail to get additional logs.
So something for Martin? Adding to Cc.
Yes, this one is for me. While I try to reproduce it, can you provide full boot logs (possibly with udev.log-priority=debug) and a supportconfig (taken before the update of course)?
Created attachment 872367 [details] supportconfig
Created attachment 872368 [details] boot-log Supportconfig before the update is attached. Serial console boot log with error after the update is attached now.
I couldn't reproduce the issue on my test system so far.
I have reason to assume that this is fixed with the latest sg3_utils submission (IBS sr#319778). Can you please test that? If that doesn't help, please read on... (In reply to Petr Cervinka from comment #9) > Created attachment 872368 [details] > boot-log Hm I would need a complete log. The serial console only shows the high-level messages. Could you use "rd.shell" to collect a journalctl output from the failed boot? I can see that one of the SCSI disks is failing: > [ 10.115316][ T823] device-mapper: multipath: 254:0: Failing path 8:0. > [ 10.182339][ C1] I/O error, dev dm-0, sector 134112 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 2 But I don't see why, and I also don't see Can you set "verbosity 3" in /etc/multipath.conf, rebuild the initrd, and create another boot log with "udev.log-priority=debug"?
From udev.txt: P: /devices/pci0000:00/0000:00:02.2/0000:03:00.0/virtio2/host0/target0:0:0/0:0:0:2/block/sda S: disk/by-id/scsi-0QEMU_QEMU_HARDDISK_1 E: SCSI_IDENT_LUN_VENDOR=1 E: ID_SERIAL=0QEMU_QEMU_HARDDISK_1 This is probably the reason for the failure. The qemu / libvirt setup should make sure that we have better device identifiers available than SCSI-0 ("vendor specific"). See jsc#PED-6226. The mentioned SR (comment 11) will change the defaults such that these identifiers are supported again. But sooner or later we won't support them any more.
(In reply to Martin Wilck from comment #11) > I have reason to assume that this is fixed with the latest sg3_utils > submission (IBS sr#319778). Can you please test that? I tested following sg3_utils on same image and qemu setup: libsgutils2-1_48-2 1.48+9.7626b4d-150600.6.1 x86_64 Plain RPM files cache obs://build.suse.de/home:mwilck sg3_utils 1.48+9.7626b4d-150600.6.1 x86_64 Plain RPM files cache obs://build.suse.de/home:mwilck It works fine after the update. > If that doesn't help, please read on... I guess, further troubleshooting steps are not needed. (In reply to Martin Wilck from comment #10) > I couldn't reproduce the issue on my test system so far. Could you please share details about your setup? (In reply to Martin Wilck from comment #12) > The mentioned SR (comment 11) will change the defaults such that these > identifiers are supported again. But sooner or later we won't support them > any more. I'm curious about proper setup. If this feature will go away soon or later, we need to adapt core of openQA, which handles multipath setup in qemu. (Not just my quick reproduction setup with serial 1).
(In reply to Petr Cervinka from comment #13) > Could you please share details about your setup? I used iSCSI with a SLE15-SP5 LIO server, which sets proper device IDs. Perhaps you want to share details about *your* setup (e.g. qemu command line, libvirt XML) so that I can point out what to improve.
(In reply to Martin Wilck from comment #14) > (In reply to Petr Cervinka from comment #13) > > I used iSCSI with a SLE15-SP5 LIO server, which sets proper device IDs. > > Perhaps you want to share details about *your* setup (e.g. qemu command > line, libvirt XML) so that I can point out what to improve. Thank you Martin. Just want to update this bug, that we discussed further details on slack and we should change openQA multipath setup for the future (SLE backward compatibility can be removed in the future).
In general, I recommend to replace the "serial" element by "wwn" in libvirt/qemu based setups.
This is an autogenerated message for OBS integration: This bug (1219289) was mentioned in https://build.opensuse.org/request/show/1146256 Factory / multipath-tools
This is an autogenerated message for OBS integration: This bug (1219289) was mentioned in https://build.opensuse.org/request/show/1146517 Factory / multipath-tools
This is an autogenerated message for OBS integration: This bug (1219289) was mentioned in https://build.opensuse.org/request/show/1146796 Factory / multipath-tools
multipath-tools-0.9.8~1+82+suse.dcd98a3 has landed in SLE15-SP6 GA. Closing.
A SR mentioning this bug was successfully integrated into Build 58.1. Please, if needed, set it to RESOLVED FIXED for QE to verify it.
Hello Martin, could you please clarify whether this btrfs "warning" is related to the issue or not? Many thanks https://openqa.opensuse.org/tests/4007059#step/force_scheduled_tasks/11 Feel free to close if you see this as a separate issue.
No, that's something different.