Bug 1223468 - XFS bigtime not supported by grub
Summary: XFS bigtime not supported by grub
Status: RESOLVED WORKSFORME
Alias: None
Product: PUBLIC SUSE Linux Enterprise Desktop 15 SP6
Classification: openSUSE
Component: Other (show other bugs)
Version: unspecified
Hardware: i686 Other
: P2 - High : Normal
Target Milestone: ---
Assignee: Bootloader Maintainers
QA Contact: Lubos Kocman
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-04-29 07:05 UTC by Mark Overmeer
Modified: 2024-04-30 00:40 UTC (History)
7 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---
rtsvetkov: needinfo? (bootloader-maintainers)
mark: SHIP_STOPPER?


Attachments
install log (3.68 MB, text/plain)
2024-04-29 10:48 UTC, Mark Overmeer
Details
01-probed.xml extracted from the y2log (28.24 KB, text/xml)
2024-04-29 11:15 UTC, Stefan Hundhammer
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mark Overmeer 2024-04-29 07:05:05 UTC
When installing LEAP 15.6RC1 on a freshly formatted XFS partition, GRUB cannot boot: it cannot mount the file-system because "unknown XFS feature 0x8".

As far as I can see, 15.6 comes with the old grub 2.02, which does not support big timestamps.  Newer versions of grub do support it.

In the current state, the installation process does support the XFS it formatted itself for / (/boot), which leaves an unbootable system.
Comment 2 Stefan Hundhammer 2024-04-29 07:31:15 UTC
Mark, so you probably added that feature as an extra option for filesystem creation for that partition? Because if that was a new default, we'd get dozens of similar bug reports, but so far, this is the only one AFAICs.
Comment 3 Stefan Hundhammer 2024-04-29 07:33:22 UTC
-> component "Bootloader" which doesn't seem to exist for this product.
Comment 4 Mark Overmeer 2024-04-29 08:28:31 UTC
(In reply to Stefan Hundhammer from comment #2)
> Mark, so you probably added that feature as an extra option for filesystem
> creation for that partition? Because if that was a new default, we'd get
> dozens of similar bug reports, but so far, this is the only one AFAICs.

Yes, you can disable it with "-O bigtime=0".  The feature is available since Linux kernel 5.10.  Maybe, this shows because I boot via MBR, which probably is getting pretty rare.

Fact is, that I install Leap 15.6 (Beta/RC1) with the default and the patch-level of yesterday, and I do not get a bootable set-up.  Probably because Grub 2.02 is too old.
Comment 5 Stefan Hundhammer 2024-04-29 09:11:33 UTC
Since Leap 15.6 is the openSUSE version of SLE-15 SP6, I don't think we can or want to move to a newer Grub2 version at this time.

IIRC (Michael Chang, please correct me if I am wrong), we have so many SUSE patches on top of that version for continued support of features that SLE business customers requested over time that it would be a major piece of work to port all of them to a newer Grub2 version; for another, SLE-15 SP6 is now in the release candidate state where that kind of change would be a major risk for the release.

But if that option is now a default in XFS, something needs to be done urgently. And that might very well be to change that default back to remain compatible.
Comment 6 Stefan Hundhammer 2024-04-29 09:16:03 UTC
@Radoslav: If this "bigtime" option is now a default for XFS, this is probably a problem for SLE-15 SP6 as well.
Comment 7 Stefan Hundhammer 2024-04-29 09:17:32 UTC
Mark, please attach y2logs generated with the supplied 'save_y2logs' script so we can have a closer look at this.

See also

  https://en.opensuse.org/openSUSE:Report_a_YaST_bug#Attachments_-_y2logs,_hwinfo_etc.
Comment 8 Radoslav Tzvetkov 2024-04-29 09:23:58 UTC
Beni, team, how far is this critical for SP6?
Comment 9 Stefan Hundhammer 2024-04-29 09:25:25 UTC
Related:


https://www.suse.com/support/kb/doc/?id=000020881
(SLE-12 SP5)

"Code updates were made to the kernel, grub2 and related programs, in order to allow imported xfs filesystems created with the 'bigtime' and 'inobtcount' attributes to be mounted in read-only mode.
...
...at least 2.02-146.1 or later...
"
Comment 10 Anthony Iliopoulos 2024-04-29 09:33:27 UTC
(In reply to Stefan Hundhammer from comment #5)

> But if that option is now a default in XFS, something needs to be done
> urgently. And that might very well be to change that default back to remain
> compatible.

The bigtime feature was intentionally enabled by default in SP6 since we have to support the customers that need to prepare for Y2038. We already have backwards compatibility with the last two SPs (so the SLE15-SP4 kernel can mount xfs filesystems with the feature switched on).

Also, grub2 even in SP4 is recent enough that implements bigtime and can mount partitions with the feature enabled.

I do not know where the error message "unknown XFS feature 0x8" is coming from, it doesn't appear to be a grub error, but I am sure that Michael can help you diagnose this further.

BTW, SP6 comes with grub-2.12 which is the latest tagged release, not grub-2.02.
Comment 11 Michael Chang 2024-04-29 10:02:38 UTC
Hi Mark,

The grub package in leap 15.6 should indeed be updated to the latest version, 2.12, not 2.02. Could you please provide more details on how you encountered version 2.02?

Additionally, the XFS bigtime feature support was added to grub in version 2.06 back in 2021. Since we have been using this feature for quite some time, it's unusual for this issue to arise now. Any further information you can share would be helpful in resolving this.
Comment 12 Mark Overmeer 2024-04-29 10:47:36 UTC
(In reply to Anthony Iliopoulos from comment #10)
> (In reply to Stefan Hundhammer from comment #5)
> BTW, SP6 comes with grub-2.12 which is the latest tagged release, not
> grub-2.02.

Guys, you are much more experienced in debugging this than I.  I am really impressed by the amount of traffic this generated.

I only use the Leap 15.6 Beta DVD to try to boot and fix my system. I must withdraw my conclusion that it was a grub2-xfs issue.  Sometimes, it is a bit confusing what happens in recovery processes.  I will be more careful now.

Back to what I really saw after a full installation of the 15.6beta1 DVD on a freshly formatted XFS, so without update: grub complained it could not load a file in /boot/... When using the "boot installed Linux", I was able to get into the installation and perform a system upgrade to get all latest patches.  Via YaST, I rewrite the bootloader but it had no effect.

Attached my y2log, as requested.
I will rerun my installation procedure, to reproduce filename it complains about.  Anything I have to ask grub to collect additional information?
Comment 13 Mark Overmeer 2024-04-29 10:48:55 UTC
Created attachment 874551 [details]
install log
Comment 14 Stefan Hundhammer 2024-04-29 11:07:25 UTC
No XFS in that y2log. But nevermind; it appears clear now that this option is now the default in 15.6.
Comment 15 Stefan Hundhammer 2024-04-29 11:15:07 UTC
Created attachment 874553 [details]
01-probed.xml   extracted from the y2log

XFS is mounted as /.

But IIRC this needs a separate BIOS-boot partition for Grub2 to embed.
Comment 16 Mark Overmeer 2024-04-29 11:34:38 UTC
(In reply to Stefan Hundhammer from comment #15)
> XFS is mounted as /.
> But IIRC this needs a separate BIOS-boot partition for Grub2 to embed.

Not for Leap 15.2.  I boot via MBR.
Comment 17 Stefan Hundhammer 2024-04-29 12:34:15 UTC
Yes, but the MBR can only hold a small part of the bootloader: Only stage1. Modern bootloaders need more than the ~400 bytes that are available in the MBR (minus the partition table). Grub2 has a stage2 that needs more disk space. For ext4 / ext2 and some other filesytem types, it can use the unused blocks between the first FS block and where the FS actually starts, but XFS does not have such an unused gap.
Comment 18 Michael Chang 2024-04-29 12:39:45 UTC
(In reply to Mark Overmeer from comment #13)
> Created attachment 874551 [details]
> install log

According to the log, the menu entry 'UEFI Firmware Settings' [1] in grub.cfg indicates a UEFI system, as it would not typically appear otherwise. However, it appears that your installation was made to legacy boot mode to the MBR [2], which does not seem consistent with the target platform.

> 2024-04-28 14:25:33 <1> moon(4395) [Ruby] bootloader/grub2base.rb(read):130
> grub sections: ["openSUSE Leap 15.6 Beta", "openSUSE Leap 15.6 Beta, with Linux
> 6.4.0-150600.10-default", "openSUSE Leap 15.6 Beta, with Linux
> 6.4.0-150600.10-default (recovery mode)", "openSUSE Leap 15.2 (on /dev/sda1)",
> "openSUSE Leap 15.2 (on /dev/sda1)", "openSUSE Leap 15.2, with Linux
> 5.3.18-lp152.98-default (on /dev/sda1)", "openSUSE Leap 15.2, with Linux
> 5.3.18-lp152.98-default (recovery mode) (on /dev/sda1)", "openSUSE Leap 15.2,
> with Linux 5.3.18-lp152.75-default (on /dev/sda1)", "openSUSE Leap 15.2, with
> Linux 5.3.18-lp152.75-default (recovery mode) (on /dev/sda1)", "UEFI Firmware
> Settings"]
> 
> 2024-04-28 14:33:51 <1> moon(31190) [Ruby] bootloader/grub2base.rb(read):130
> grub sections: ["openSUSE Leap 15.6", "openSUSE Leap 15.6, with Linux
> 6.4.0-150600.16-default", "openSUSE Leap 15.6, with Linux
> 6.4.0-150600.16-default (recovery mode)", "openSUSE Leap 15.6, with Linux
> 6.4.0-150600.10-default", "openSUSE Leap 15.6, with Linux
> 6.4.0-150600.10-default (recovery mode)", "openSUSE Leap 15.2 (on /dev/sda1)",
> "openSUSE Leap 15.2 (on /dev/sda1)", "openSUSE Leap 15.2, with Linux
> 5.3.18-lp152.98-default (on /dev/sda1)", "openSUSE Leap 15.2, with Linux
> 5.3.18-lp152.98-default (recovery mode) (on /dev/sda1)", "openSUSE Leap 15.2,
> with Linux 5.3.18-lp152.75-default (on /dev/sda1)", "openSUSE Leap 15.2, with
> Linux 5.3.18-lp152.75-default (recovery mode) (on /dev/sda1)", "UEFI Firmware
> Settings"]
> 
> 2024-04-28 14:51:04 <1> moon(3673) [Ruby] bootloader/grub2base.rb(read):130
> grub sections: ["openSUSE Leap 15.6", "openSUSE Leap 15.6, with Linux
> 6.4.0-150600.16-default", "openSUSE Leap 15.6, with Linux
> 6.4.0-150600.16-default (recovery mode)", "openSUSE Leap 15.6, with Linux
> 6.4.0-150600.10-default", "openSUSE Leap 15.6, with Linux
> 6.4.0-150600.10-default (recovery mode)", "openSUSE Leap 15.2 (on /dev/sda1)",
> "openSUSE Leap 15.2 (on /dev/sda1)", "openSUSE Leap 15.2, with Linux
> 5.3.18-lp152.98-default (on /dev/sda1)", "openSUSE Leap 15.2, with Linux
> 5.3.18-lp152.98-default (recovery mode) (on /dev/sda1)", "openSUSE Leap 15.2,
> with Linux 5.3.18-lp152.75-default (on /dev/sda1)", "openSUSE Leap 15.2, with
> Linux 5.3.18-lp152.75-default (recovery mode) (on /dev/sda1)", "UEFI Firmware
> Settings"]
> 
> 2024-04-28 15:02:31 <1> moon(6809) [Ruby] bootloader/grub2base.rb(read):130
> grub sections: ["openSUSE Leap 15.6", "openSUSE Leap 15.6, with Linux
> 6.4.0-150600.16-default", "openSUSE Leap 15.6, with Linux
> 6.4.0-150600.16-default (recovery mode)", "openSUSE Leap 15.6, with Linux
> 6.4.0-150600.10-default", "openSUSE Leap 15.6, with Linux
> 6.4.0-150600.10-default (recovery mode)", "openSUSE Leap 15.2 (on /dev/sda1)",
> "openSUSE Leap 15.2 (on /dev/sda1)", "openSUSE Leap 15.2, with Linux
> 5.3.18-lp152.98-default (on /dev/sda1)", "openSUSE Leap 15.2, with Linux
> 5.3.18-lp152.98-default (recovery mode) (on /dev/sda1)", "openSUSE Leap 15.2,
> with Linux 5.3.18-lp152.75-default (on /dev/sda1)", "openSUSE Leap 15.2, with
> Linux 5.3.18-lp152.75-default (recovery mode) (on /dev/sda1)", "UEFI Firmware
> Settings"]

[2] 
> 2024-04-28 14:34:45 <1> moon(31190) [Ruby] lib/cheetah.rb(record_commands):160
> Executing "/usr/sbin/grub2-install --target\=i386-pc --force --skip-fs-probe
> /dev/disk/by-id/ata-Samsung_SSD_850_PRO_256GB_S251NX0H626286K".
Comment 19 Michael Chang 2024-04-29 13:04:35 UTC
Your system is also mixed with uefi boot disk (/dev/nvme0n1) and legacy boot disks. (/dev/sda and /dev/sdb) without esp partition. 

> 2024-04-28 14:25:34 <1> moon(4395) [libstorage] CmdBlkid.cc(parse):175 data[/dev/nvme0n1p1] -> is-fs:true fs-type:vfat fs-uuid:437E-C2C8
> 2024-04-28 14:25:34 <1> moon(4395) [libstorage] CmdBlkid.cc(parse):175 data[/dev/nvme0n1p3] -> is-fs:true fs-type:btrfs fs-uuid:099a8f4e-5c48-449c-8f6c-49c5ba70473b fs-sub-uuid:8acc3057-55a0-48c5-9e7d-b9428a98321a
> 2024-04-28 14:25:34 <1> moon(4395) [libstorage] CmdBlkid.cc(parse):175 data[/dev/nvme0n1p4] -> is-fs:true fs-type:xfs fs-uuid:634ceb43-9da7-46c6-9edc-3b84a94164c1
> 2024-04-28 14:25:34 <1> moon(4395) [libstorage] CmdBlkid.cc(parse):175 data[/dev/sda1] -> is-fs:true fs-type:xfs fs-uuid:cda30580-78d0-40bd-9d02-f8e39621fc0a
> 2024-04-28 14:25:34 <1> moon(4395) [libstorage] CmdBlkid.cc(parse):175 data[/dev/sda3] -> is-fs:true fs-type:xfs fs-uuid:8ece71d2-8e21-4303-bea5-497d3116e3c2
> 2024-04-28 14:25:34 <1> moon(4395) [libstorage] CmdBlkid.cc(parse):175 data[/dev/sdb1] -> is-fs:true fs-type:xfs fs-uuid:b938619e-d5e6-4323-b7c8-cc28ec2e76e9
> 2024-04-28 14:25:34 <1> moon(4395) [libstorage] CmdBlkid.cc(parse):175 data[/dev/sdb2] -> is-fs:true fs-type:xfs fs-uuid:3a38bfb2-6829-40d0-8cbe-5df026548ae3

In the log there also a shim-install, which is targeting a "uefi" platform :

> 2024-04-28 15:03:25 <1> moon(6809) [Ruby] lib/cheetah.rb(record_commands):160 Executing "/usr/sbin/shim-install --config-file\=/boot/grub2/grub.cfg --removable".
> 2024-04-28 15:03:25 <1> moon(6809) [Ruby] lib/cheetah.rb(log_stream_line):208 Standard output: copying /usr/share/efi/x86_64/grub.efi to /boot/efi/EFI/boot/grub.efi
> 2024-04-28 15:03:25 <3> moon(6809) [Ruby] lib/cheetah.rb(log_stream_line):208 Error output: Installing for x86_64-efi platform.
> 2024-04-28 15:03:28 <3> moon(6809) [Ruby] lib/cheetah.rb(log_stream_line):208 Error output: Installation finished. No error reported.

It is kind of weird to me, is there a explanation why the setup is used ?
Thanks.
Comment 20 Mark Overmeer 2024-04-29 17:04:12 UTC
(In reply to Michael Chang from comment #19)
> Your system is also mixed with uefi boot disk (/dev/nvme0n1) and legacy boot
> disks. (/dev/sda and /dev/sdb) without esp partition.
> It is kind of weird to me, is there a explanation why the setup is used?

I first tried to install Leap 15.6 with the defaults, on UEFI, but failed.
Then, I tried to do it twice in different ways "the old way", which also
failed with these grub problems.  "Old" disks from older installations.

This afternoon, I installed Leap15.6 again, in an attempt to reproduce the error messages I got.  With all the defaults.  And it simply worked :-((((  I have no idea why it failed me three times yesterday.

I use UNIX since 1985, and Linux since 1993... OpenSuSE since it exists... so: I am not inexperienced. But I have f*cked up something yesterday, which let me to this bug-report.  Sorry for consuming your time.  I am impressed by your team!
Please close this report.
Comment 21 Michael Chang 2024-04-30 00:40:31 UTC
Thanks for the update. Let's close this ticket and hope you enjoy the new leap 15.6 installation.