Bug 1223609

Summary: kernel panic - unable to mount root fs on unknown-block - first boot into 15.6 RC from a simple 15.5 via zypper dup
Product: [openSUSE] openSUSE Distribution Reporter: andreas bittner <abittner>
Component: Upgrade ProblemsAssignee: dracut maintainers <dracut-maintainers>
Status: RESOLVED DUPLICATE QA Contact: Jiri Srain <jsrain>
Severity: Major    
Priority: P5 - None CC: antonio.feijoo
Version: Leap 15.6   
Target Milestone: ---   
Hardware: x86-64   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: zypper log and zypp log extract from dup 15.5 to 15.6RC

Description andreas bittner 2024-05-01 11:48:28 UTC
simple 15.5 x64, on rather old amd hardware but worked fine in 15.5

system has come a long way with upgrades from previous leaps
with a single sata HDD
completely updated in 15.5 before
then zypper via releasever=15.6  and zypper ref and zypper dup, download-in-advance.

upgraded thousands of packages, then rebooted. then never came back alive. looking at the machine physically, I see it in a kernel panic error message telling me

(copy paste from a photo/picture, text OCR, sorry for errs)

---------

0.63410711

T11 Kernel panic not syncing: UPS: Unable to mount root fs on unknown-block(0,0)

711 CPU: 2 PID: 1 Соми: змарриг/6 Not tainted 6.4.0-150600.16-default #1 SLE15-SP6 38db936c6f7776667f67c71a8b5f453a65a8919f

0.63428110

T1] Hardware name: Systen manufacturer Systen Product Name/F2A85-U PRO, BIOS 6503 07/10/2014

0.63452011

0.63470611

T11 Call Trace:

T1] <TASK>

0.63476831 8.63482311

711 dump_stack_lul+0x44/8x69

0.63491710

T11 panic+0x189/8x2c0

8.63499211

711 mount_root_generic+0x114/9x110

0.63508810 8.63517311

T11 ? mount root+6x146/9x190

0.63528811

T11 7 pfx_ignore_unknown_bootoption+0x10/0x10 T11 prepare namespace+8x1a6/8x200

0.63538210

Til kernel init_freeable+0x25c/8x298

8.63540110 0.63557410

Til?pix kernel_init+0x10/8x10

T11 kernel Init+0x16/0x130 fron_fork+0x2c/0x50

8.63565511 Til ret

TLI TASK

0.635739][

6.63585511

Til Kernel Offset: 0x1f660000 from 0xffffffff81000000 (relocation range: 6xffffffff80000000-0xreeeeeerbeeeeeee)

6.63667411

T1]-[end Kernel panic not syncing: UFS: Unable to mount root fs on unknown-block(0,0) 1

-----------

grub2 bootmenu appears fine, I didnt try safe modes or other stuff there just yet. I tried another boot (ctrl alt del) but it ends up in the smiliar or same kernel panic, sometimes other cpu cores shown.

thanks.
Comment 1 andreas bittner 2024-05-01 14:06:26 UTC
being at the machine, physically, I can select from the grub2 menu at boot, apparently the older suse leap 15.5 kernels, although the lines show as leap 15.6 but with those 5.something kernels.

i can boot the .52 kernel at the end, that was the last one this then-15.5 machine has gotten (i think some .59 kernel for 15.5 came out yesterday or so I remember), just fine with 15.6 apparently.

i took the machine off from the network and it sits now standalone on the desk. I guess I could extract some zypper log files or something if anyone points me to?
Comment 2 andreas bittner 2024-05-01 15:15:40 UTC
Created attachment 874601 [details]
zypper log and zypp log extract from dup 15.5 to 15.6RC

tar with xz compress, log extract from affected system. attached
Comment 3 andreas bittner 2024-05-01 16:12:47 UTC
using usb install media with 15.6 RC .iso file, booting up into upgrade mode and letting it upgrade, selects 8 packages only and about 200megabytes or such and installs along at the end when saving settings and bootloader stuff, it fails apparently with the same errors as in zypper logs given, that it cannot complete suse mkinitrd or something methinks. something like that.

this didnt help. dunno whats wrong here the 15.5 was working a long time fine also kernel updates and opensuse patches via zypper up and all.

maybe i try a fresh 15.5 and upgrade that to 15.6rc on this hardware? cant be that buggy?

p.s. i have had 15.6rc installing normally fresh into a virtualbox simple setup though.
Comment 4 andreas bittner 2024-05-01 17:49:33 UTC
a test with:

a clean 15.5 on a simple sata disk (ssd though) with mbr partitioning and all updates inside the normally working 15.5

and then zypper dup into 15.6

turns out the upgraded 15.6 working and booting normally.

so I guess I am back to looking into why the upgrader or zypper dup messed up the kernel or initrd or whatever and booting parts?

:(
Comment 5 andreas bittner 2024-05-03 12:14:29 UTC
today I have tried to fix this machine via the 15.6rc media rescue bootentry, mounting / and /boot and adding sys dev proc and run  and then chroot into the system

I have tried to look into grub , grub_installdevice  and those parts and it looks fine, the disk UUID and names and partitions etc all exist and make sense and are correct.

inside chroot: firing up the yast on the commandline and making it re-install the boot setting rewrite MBR or bootstuff into the whole disk and the partition of /boot and all finishes without erros being shown from inside yast.

the system doesnt boot after this either.

rescue once again and then mkinitrd installs kind of multiple big blocks worth of kernel stuff (with log output) and at the end with the 6.4.0 kernel entry it seems to fail at the dracut step and prints the error to the screen, and this error seems to be in the log files I have provided as an attachment earlier.

--------------

zypp-history-log-0000.txt-# dracut[E]: FAILED: /usr/lib/dracut/dracut-install -D /var/tmp/dracut.7LMEwP/initramfs -N ^i2o_scsi$ --kerneldir /lib/modules/5.14.21-150500.55.49-default/ -m pata_atiixp ahci ata_generic sata_nv pata_amd fan amd74xx ide_pci_generic reiserfs edd
zypp-history-log-0000.txt:# dracut[F]: installkernel failed in module suse-initrd
zypp-history-log-0000.txt-# warning: %postun(kernel-default-extra-5.14.21-150500.55.49.1.x86_64) scriptlet failed, exit status 1

---------------

it seems that I can not make this hosed system to boot any more even after repair attempts with the 6.4.x kernel entry

thanks.
Comment 6 Antonio Feijoo 2024-05-03 12:47:40 UTC
(In reply to andreas bittner from comment #5)
> --------------
> 
> zypp-history-log-0000.txt-# dracut[E]: FAILED:
> /usr/lib/dracut/dracut-install -D /var/tmp/dracut.7LMEwP/initramfs -N
> ^i2o_scsi$ --kerneldir /lib/modules/5.14.21-150500.55.49-default/ -m
> pata_atiixp ahci ata_generic sata_nv pata_amd fan amd74xx ide_pci_generic
> reiserfs edd
> zypp-history-log-0000.txt:# dracut[F]: installkernel failed in module
> suse-initrd
> zypp-history-log-0000.txt-# warning:
> %postun(kernel-default-extra-5.14.21-150500.55.49.1.x86_64) scriptlet
> failed, exit status 1
> 
> ---------------

This issue has already been reported and the fix is in progress: https://github.com/openSUSE/dracut/pull/328

tl;dr: your system probably has some leftover legacy kernel modules that were tried to be added using a `INITRD_MODULES=` entry in /etc/sysconfig/kernel or `# SUSE INITRD:` comment in any modprobe config file. The fix restores the old behavior to avoid breaking the build of the initrd.
Comment 7 andreas bittner 2024-05-04 09:34:45 UTC
thanks a lot for that feedback. I did have INITRD_MODULES line with some values for it in the /etc/sysconfig/kernel   file all right.

I uncommented the whole line, rescue then chroot  and then mkinitrd

it did then rebuild the last block/entry as well for the 6.4.0 kernel just as well without errors this time.

system boots now fine in 15.6rc with the first default 6.4.0 kernel entry just as well.

do we not need this INITRD_MODULES variable any more at all? in mine there are such things as reiserfs and PIIX4 or those old chipset intel controllers and what not.

thanks.
Comment 8 andreas bittner 2024-05-04 13:25:52 UTC
was curious about /etc/sysconfig/kernel  file and on my test from scratch 15.5 and upgrade to 15.6rc there isnt even such a file at all any more

thanks.
Comment 9 Antonio Feijoo 2024-05-13 06:27:25 UTC
(In reply to andreas bittner from comment #7)
> do we not need this INITRD_MODULES variable any more at all? in mine there
> are such things as reiserfs and PIIX4 or those old chipset intel controllers
> and what not.

No, nowadays if you need to manually add some kernel modules in your initrd that are not automatically included, you have to use the configuration options provided by dracut (see man dracut.conf.5).

(In reply to andreas bittner from comment #8)
> was curious about /etc/sysconfig/kernel  file and on my test from scratch
> 15.5 and upgrade to 15.6rc there isnt even such a file at all any more
> 
> thanks.

It's a legacy way used to configure some parts of the system. At least for configuring your initrd it has not been used for quite some time now, but I don't know if others still use it.

(In reply to Antonio Feijoo from comment #6)
> This issue has already been reported and the fix is in progress:
> https://github.com/openSUSE/dracut/pull/328

The fix has been applied and will be available on the next dracut version. Thanks!

*** This bug has been marked as a duplicate of bug 1223467 ***