@Michael, since you did the backporting I'm assigning this one over to you, but let me know if you need any assistance from the xfs side of things.

Just to be more clear, the troubling commit is not one backported in openSUSE, it is one present upstream in grub 1.12:
  commit: 07318ee7e11a00b9c1dea4c6b4edf62af35a511a
  fs/xfs: Fix XFS directory extent parsing

(In reply to Giacomo Comes from comment #2)
> Just to be more clear, the troubling commit is not one backported in
> openSUSE, it is one present upstream in grub 1.12:
>   commit: 07318ee7e11a00b9c1dea4c6b4edf62af35a511a
>   fs/xfs: Fix XFS directory extent parsing

Understood, but Michael is in any case the grub2 package maintainer and handling all that, especially if we need to (temporarily?) revert the offending commit.

BTW I haven't seen any upstream discussion on grub-devel yet, but I assume it is forthcoming.

Hi Anthony and Giacomo,

Thanks for the heads up. I have created SR to factory to temporarily revert the commit :  07318ee7e fs/xfs: Fix XFS directory extent parsing:

https://build.opensuse.org/request/show/1139339

It is a bit strange that we didn't  run into the issue earlier given we had it backported to 2.12~rc1 a few months ago.

I'll monitor the mailing list for any relevant discussions in the coming days. If there's none, I'll initiate one.

Thanks.

If this is indeed easily reproducible, and you can provide a metadump of the affected partition, I can have a look and figure out in more detail what the exact issue is (and handle it also upstream).

Created attachment 871939 [details]
metadump1

metadump of the XFS partition after running one time update-bootloader --reinit
non error when booting

Created attachment 871940 [details]
metadump2

metadump of the XFS partition after running a second time 
  update-bootloader --reinit
one minute delay error when booting

Created attachment 871941 [details]
metadump3

metadump of the XFS partition after running a third time 
  update-bootloader --reinit
two minutes delay error when booting

Giacomo, how do you obtain those metadumps?

I'm asking because all of them seem to have a clean xfs log, but at the same time they appear to be quite corrupted (you can see that also with xfs_repair). 

All three metadumps appear to have an equally corrupted fs.

Are you forcing metadump (-f) on a live (mounted) filesystem? Because in this case, the metadata would be inconsistent and it is impossible to tell what the real problem is.

Ideally the second time you reboot (when you expect the errors), it would be possible to drop to initrd shell before the rootfs gets mounted (passing rd.break=pre-mount to grub shell as kernel param should do the trick).

From there you could obtain the metadump of the rootfs before it gets mounted.

I did the metadump after booting in Leap 15.5 and accessing the unmounted
tumbleweed disk image. There was a warning message on stdout while using xfs_metadump about some unsupported feature of the disk image that were not compatible with the Leap 15.5 kernel. But I did not mount the disk image.
Let me regenerate the metadump without using Leap 15.5 but only tumbleweed to access the disk image, this time I'll also add a metadump of the freshly installed tumbleweed disk image before running any update-bootloader --reinit.

(In reply to Giacomo Comes from comment #10)
> There was a warning message on stdout while using
> xfs_metadump about some unsupported feature of the disk image that were not
> compatible with the Leap 15.5 kernel.

Yeap, that's because of the nrext64 (large extent counters) feature that is by default enabled on upstream xfsprogs (and thus tumbleweed), which is incompatible with older releases that don't support it.

I just tested capturing a metadump of a filesystem with nrext64=1 with an older xfsprogs release xfs_metadump binary from 15.5 (v5.13.0), and then restoring it, and indeed it reproduces the exact corruption patterns that I saw in your metadumps.

That's because indeed the older metadump doesn't understand the large extent counters format and as such it records a corrupted view of the filesystem.

But yeah your suggestion makes sense, just capture the metadumps with the latest xfsprogs release (the kernel doesn't matter, this is all in userspace, so you could use either latest xfsprogs from source or from tumbleweed repos on any kernel).

Created attachment 871943 [details]
metadump0-try2

metadump of the XFS partition after fresh installation without running
 update-bootloader --reinit

Created attachment 871944 [details]
metadump1-try2

metadump of the XFS partition after running one time update-bootloader --reinit
non error when booting

Created attachment 871945 [details]
metadump2-try2

metadump of the XFS partition after running a second time 
  update-bootloader --reinit
one minute delay error when booting

Created attachment 871946 [details]
metadump3-try2

metadump of the XFS partition after running a third time 
  update-bootloader --reinit
two minutes delay error when booting

So all of the new metadumps contain completely healthy filesystems: the log is clean, the xfs_repair returns with no issues.

I wasn't able to reproduce the grub problem with them, at least grub-mount is not complaining and every file appears to be accessible.

Between metadump0-try2 and metadump1-try2, it seems that /boot/grub2/grub.cfg has been changed.

Between metadump1-try2 and metadump2-try2, every file under /boot/grub2/i386-pc/ has been updated (presumably due to --reinit).

Between metadump2-try2 and metadump3-try2, every file under /boot/grub2/locale/ has been updated.

But all of them are fully accessible, with no sign of problems.

Not sure what to make out of this, and why this would cause any issue, or how 07318ee7e is related here.

Perhaps Michael can reproduce this issue in some other way.

(In reply to Anthony Iliopoulos from comment #17)

> Perhaps Michael can reproduce this issue in some other way.

Not yet. The openSUSE-Tumbleweed-NET-x86_64-Snapshot20240117-Media.iso keeps throwing "repository not found" at me, despite vm network is working on my other instance. I gave up and now trying to download offline image but it takes me 4 hours from now to go.

@Giacomo,

One thing I notice from the fedora bugzilla is that the issue seemed to be restricted to legacy pc bios platform, could you please help to confirm that ? The i386-pc target  is built with `gcc -m32 ..' so that 64 bits host applications like grub2-mount may not be fully identical.

I did a test using UEFI bios and I didn't see the grub error message.
The issue appears to be related to legacy bios.

More info.
The "corrupted partition", that is the one that causes the message
"not a correct XFS inode" to be generated when it is booted from,
can be mounted with groub2-mount without issue.
I manage to enable debug with the command:
  grub2-editenv - set debug=xfs
then booted and recorded the boot process.
I have a 2:15 22MB video of the boot process. If interested, please let me know
how to make it available to you.
In such video it is possible to see:
  fs/xfs.c:1010:xfs: Reading sb
  fs/xfs.c:288:xfs: Validating superblock
  fs/xfs.c:300:xfs: XFS v5 superblock detected
  fs/xfs.c:1042:xfs: Reading root ino 128
  fs/xfs.c:533:xfs: reading inode (128) - 128, 0
and then more inodes are read until you see the following:
  fs/xfs.c:533:xfs: Reading inode (0) - 0, 0
  error: ../../grub-core/fs/xfs.c:541:not a correct XFS inode.

Later in the boot process you can also see:
  fs/xfs.c:533:xfs: Reading inode (25942926) - 16491888, 3072
  fs/xfs.c:533:xfs: Reading inode (25942927) - 16491888, 3584
  fs/xfs.c:533:xfs: Reading inode (197568495624) - 123371593736, 0
  error: ../../grub-core/kern/disk_common.c:26:attempt to read or write outside of partition.

So it looks like the code that decides which inode to read next have some glitches.

(In reply to Giacomo Comes from comment #20)
> More info.
> The "corrupted partition", that is the one that causes the message
> "not a correct XFS inode" to be generated when it is booted from,
> can be mounted with groub2-mount without issue.
> I manage to enable debug with the command:
>   grub2-editenv - set debug=xfs
> then booted and recorded the boot process.
> I have a 2:15 22MB video of the boot process. If interested, please let me
> know
> how to make it available to you.
> In such video it is possible to see:
>   fs/xfs.c:1010:xfs: Reading sb
>   fs/xfs.c:288:xfs: Validating superblock
>   fs/xfs.c:300:xfs: XFS v5 superblock detected
>   fs/xfs.c:1042:xfs: Reading root ino 128
>   fs/xfs.c:533:xfs: reading inode (128) - 128, 0
> and then more inodes are read until you see the following:
>   fs/xfs.c:533:xfs: Reading inode (0) - 0, 0
>   error: ../../grub-core/fs/xfs.c:541:not a correct XFS inode.

Yes this seems wrong, there should be no inode with inode number zero.

> Later in the boot process you can also see:
>   fs/xfs.c:533:xfs: Reading inode (25942926) - 16491888, 3072
>   fs/xfs.c:533:xfs: Reading inode (25942927) - 16491888, 3584
>   fs/xfs.c:533:xfs: Reading inode (197568495624) - 123371593736, 0
>   error: ../../grub-core/kern/disk_common.c:26:attempt to read or write
> outside of partition.
> 
> So it looks like the code that decides which inode to read next have some
> glitches.

Is this with 07318ee7e reverted? Does grub actually manage to boot the kernel after those errors?

(In reply to Michael Chang from comment #18)
> One thing I notice from the fedora bugzilla is that the issue seemed to be
> restricted to legacy pc bios platform, could you please help to confirm that
> ? The i386-pc target  is built with `gcc -m32 ..' so that 64 bits host
> applications like grub2-mount may not be fully identical.

This is interesting, but in my tests I am building with i386-pc (by default) and the target is x86_64.

That said:

- I have compiled grub-mount with -m32, and it mounts the metadump filesystems without presenting the error. The only issue I see is that due to the 32-bit magnitude, the metadata are not presented correctly, e.g.:

  File: /mnt/test/boot/vmlinuz
  Size: 2199023255552   Blocks: 7325092458772037632 IO Block: 16384  regular file
Device: 0,73    Inode: 1640        Links: 0
Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 1970-01-01 01:00:00.000000000 +0100
Modify: 1970-01-01 01:00:00.999999999 +0100
Change: 1970-01-01 01:00:00.000000000 +0100
 Birth: -

- In the x86_64 version, due to the 07318ee7e revert, not all of the filesystem contents are visible (a bug I noticed before, but it was indeed fixed via 07318ee7e), e.g.:

# grub2-mount (with 07318ee7e reverted)
# ls -al /mnt/test/bin/|wc -l
161

# grub2-mount (v2.12)
# ls -al /mnt/test/bin/|wc -l
1617

So reverting this patch introduces another problem.

(In reply to Anthony Iliopoulos from comment #21)
> Is this with 07318ee7e reverted? Does grub actually manage to boot the
> kernel after those errors?

No this is with 07318ee7e. If reverted, there is no error message. For the time being I'm keeping around the faulty grub package for testing purpose.

And yes, at the end grub boots the kernel.
I only saw the message "attempt to read or write outside of partition" in the recorded video with debug enabled, but it may well be that it is present also without enabling debug and I just missed it because of the fast scrolling of the error messages.

(In reply to Giacomo Comes from comment #15)
> Created attachment 871946 [details]
> metadump3-try2
> 
> metadump of the XFS partition after running a third time 
>   update-bootloader --reinit
> two minutes delay error when booting

I reproduced xfs error by this metadump with running grub-mount (or grub2-mount the utility name in the grub2 package) in foreground.

  sudo ./grub-mount -vvv -d xfs -- -f /dev/vdb /mnt

where /dev/vdb is device with xfs filesysem dump restored by xfs_mdrestore. The -f is the command line argument to fuse to start it in foreground that we can see xfs debug log printed.

Open another terminal, try to list file in /mnt/boot/grub2/i386-pc, eg
 
  sudo ls /mnt/boot/grub2/i386-pc/normal.mod

It doesn't matter if the xfs module is 32 or 64 bit build as I can reproduce the error in x86_64-efi environment by accessing the files in /boot/grub2/x86_64-efi. The reason why efi avoids the error is that grub.efi has all the modules builtin so it won't perform file access to that directory.

Created attachment 872050 [details]
grub-mount.log

Created attachment 872053 [details]
check-expected-entry.diff

It appears to me the error can be avoided if we check expected number of entries in the dir block regardless number of extend equals 1 or not. In other words, always do the check before iterating entries ..   

>        entries = grub_be_to_cpu32 (tail->leaf_count) - grub_be_to_cpu32 (tail->leaf_stale);
>
>        if (!entries)
>            continue;

.. and that seem to conflict with this block of comment ...

		/*
		 * The expected number of directory entries is only tracked for the
		 * single extent case.
		 */
		if (dir->inode.nextents == grub_cpu_to_be32_compile_time (1))
		  {
		    /* Check if last direntry in this block is reached. */
		    entries--;
		    if (!entries)
		      break;
		  }

Hi Anthony:

I apologize for my limited understanding of the XFS layout, which hinders my ability to delve deeper into the issue. The regression seems to be linked to the entry number check, where a recent commit has eliminated cases other than that of a single extent. I would greatly appreciate it if you could assess whether this change is sound and valid.

Thanks.

The original author of the patch causing the current problem has made some
comment on the fedora bugzilla entry. See comment 57,59,61.
I think what he says is consistent with that Michael found.

To chime in as a third party: On my tumbleweed installation with the latest change, I can no longer boot kernels from my root XFS. I had problems before that were fixed in

  https://build.opensuse.org/request/show/1138021

already before I could report them :)

But with the newest

  https://build.opensuse.org/request/show/1138021

it no longer works, again.

(A compounding factor seems to be the move of the kernels from /boot to /usr/lib/modules/$kernelversion, as the rather small /boot directory can always be parsed by grub in my tests so far, whereas the rather large /usr metadata is probably stored somehow differently by XFS, triggering this issue. This could blur the timeline of events, somewhat, not sure what came when)

Here are details about the currently installed package (which is not able to boot):

root@~> rpm -q grub2 --changelog | head
* Mi Jan 17 2024 Michael Chang <mchang@suse.com>
- Resolved XFS regression leading to the "not a correct XFS inode" error by
  temporarily reverting the problematic commit (bsc#1218864)
  * 0001-Revert-fs-xfs-Fix-XFS-directory-extent-parsing.patch
[...]
root@~> rpm -qi grub2
Name        : grub2
Version     : 2.12
Release     : 2.1
Architecture: x86_64
Install Date: Sa 20 Jan 2024 09:23:11 CET
Group       : System/Boot
Size        : 27941722
License     : GPL-3.0-or-later
Signature   : RSA/SHA512, Mi 17 Jan 2024 23:54:16 CET, Key ID 35a2f86e29b700a4
Source RPM  : grub2-2.12-2.1.src.rpm
Build Date  : Mi 17 Jan 2024 23:53:48 CET
Build Host  : i04-ch1d
Packager    : https://bugs.opensuse.org
Vendor      : openSUSE
URL         : http://www.gnu.org/software/grub/
Summary     : Bootloader with support for Linux, Multiboot and more
Description :
This is the second version of the GRUB (Grand Unified Bootloader), a
highly configurable and customizable bootloader with modular
architecture.  It support rich scale of kernel formats, file systems,
computer architectures and hardware devices.

This package includes user space utlities to manage GRUB on your system.
Distribution: openSUSE Tumbleweed

(In reply to Giacomo Comes from comment #28)
> The original author of the patch causing the current problem has made some
> comment on the fedora bugzilla entry. See comment 57,59,61.
> I think what he says is consistent with that Michael found.

Thanks. And yes that seemed to coincide with my finding.

(In reply to Lorenz Hüdepohl from comment #29)
> To chime in as a third party: On my tumbleweed installation with the latest
> change, I can no longer boot kernels from my root XFS. I had problems before
> that were fixed in
> 
>   https://build.opensuse.org/request/show/1138021
> 
> already before I could report them :)
> 
> But with the newest
> 
>   https://build.opensuse.org/request/show/1138021
> 
> it no longer works, again.
> 
> (A compounding factor seems to be the move of the kernels from /boot to
> /usr/lib/modules/$kernelversion, as the rather small /boot directory can
> always be parsed by grub in my tests so far, whereas the rather large /usr
> metadata is probably stored somehow differently by XFS, triggering this
> issue. This could blur the timeline of events, somewhat, not sure what came
> when)
> 
> Here are details about the currently installed package (which is not able to
> boot):
> 
> root@~> rpm -q grub2 --changelog | head
> * Mi Jan 17 2024 Michael Chang <mchang@suse.com>
> - Resolved XFS regression leading to the "not a correct XFS inode" error by
>   temporarily reverting the problematic commit (bsc#1218864)
>   * 0001-Revert-fs-xfs-Fix-XFS-directory-extent-parsing.patch
> [...]
> root@~> rpm -qi grub2
> Name        : grub2
> Version     : 2.12
> Release     : 2.1
> Architecture: x86_64
> Install Date: Sa 20 Jan 2024 09:23:11 CET
> Group       : System/Boot
> Size        : 27941722
> License     : GPL-3.0-or-later
> Signature   : RSA/SHA512, Mi 17 Jan 2024 23:54:16 CET, Key ID
> 35a2f86e29b700a4
> Source RPM  : grub2-2.12-2.1.src.rpm
> Build Date  : Mi 17 Jan 2024 23:53:48 CET
> Build Host  : i04-ch1d
> Packager    : https://bugs.opensuse.org
> Vendor      : openSUSE
> URL         : http://www.gnu.org/software/grub/
> Summary     : Bootloader with support for Linux, Multiboot and more
> Description :
> This is the second version of the GRUB (Grand Unified Bootloader), a
> highly configurable and customizable bootloader with modular
> architecture.  It support rich scale of kernel formats, file systems,
> computer architectures and hardware devices.
> 
> This package includes user space utlities to manage GRUB on your system.
> Distribution: openSUSE Tumbleweed

This can the consequence of reverting  "fs/xfs: Fix XFS directory extent parsing" that caused the xfs error here ...

Before a proper fix landing upstream, I think we may provide a band-aid fix by adding back the number of entries check to the "Fix XFS directory extent parsing" patch whereas the node is not single extent and hence filtering out those "unmapped blocks" globally. I will try to provide test package so that you can help to verify it on two different potential layouts.

The obs repository containing  the test package:

  https://download.opensuse.org/repositories/home:/michael-chang:/bsc:/1218864/standard/

For openSUSE Tumbleweed, you may follow the procedure to install the test package:

> zypper ar --repo https://download.opensuse.org/repositories/home:/michael-chang:/bsc:/1218864/standard/home:michael-chang:bsc:1218864.repo
> zypper ref
> zypper dup --from home_michael-chang_bsc_1218864 --allow-vendor-change

And this procedure to remove the test package:

> zypper rr home_michael-chang_bsc_1218864
> zypper dup --allow-vendor-change

For uefi, you may have to disable secure boot because the test grub.efi is not signed by SUSE. Optionally it is possible to keep secure boot on, but you have entrust the key of my home project by enrolling it into "Machine Owner Key" (MOK) keyring.

> mokutil --import /usr/share/efi/x86_64/grub.der

Thanks.

(In reply to Michael Chang from comment #32)
> The obs repository containing  the test package:
> 
>  
> https://download.opensuse.org/repositories/home:/michael-chang:/bsc:/1218864/
> standard/
> 
> For openSUSE Tumbleweed, you may follow the procedure to install the test
> package:
> 
> > zypper ar --repo https://download.opensuse.org/repositories/home:/michael-chang:/bsc:/1218864/standard/home:michael-chang:bsc:1218864.repo
> > zypper ref
> > zypper dup --from home_michael-chang_bsc_1218864 --allow-vendor-change
> 
> And this procedure to remove the test package:
> 
> > zypper rr home_michael-chang_bsc_1218864
> > zypper dup --allow-vendor-change
> 
> For uefi, you may have to disable secure boot because the test grub.efi is
> not signed by SUSE. Optionally it is possible to keep secure boot on, but
> you have entrust the key of my home project by enrolling it into "Machine
> Owner Key" (MOK) keyring.
> 
> > mokutil --import /usr/share/efi/x86_64/grub.der
> 
> Thanks.

Thank you for the detailed and easy to follow instructions: With the package in your branch the booting from XFS works, again!

Kind regards,
  Lorenz

For me too the patch fix the problem. I don't see anymore the 
error message at boot.

Many thanks for your help & support. The SR with the band-aid fix has been created to the openSUSE:Factory.

https://build.opensuse.org/request/show/1141140

The patch for the fix, provided by the original author, has been posted on the grub-devel mailing list.

https://lore.kernel.org/all/20240211153457.31786-2-nuxi@vault24.org/T/#m3ebe256000adc9fca8370c9d00286a5e858f92cb

I tested the patch. It works for me.