Bug 1178247 - [Build 20201028] openQA test fails in yast2_bootloader on Raspberry pi 2, 3 and 4
[Build 20201028] openQA test fails in yast2_bootloader on Raspberry pi 2, 3 a...
Status: RESOLVED FIXED
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Kernel
Current
aarch64 openSUSE Tumbleweed
: P4 - Low : Normal (vote)
: ---
Assigned To: Michael Chang
E-mail List
https://openqa.opensuse.org/tests/145...
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2020-10-29 07:40 UTC by Guillaume GARDET
Modified: 2021-03-29 14:46 UTC (History)
15 users (show)

See Also:
Found By: openQA
Services Priority:
Business Priority:
Blocker: Yes
Marketing QA Status: ---
IT Deployment: ---
jreidinger: needinfo? (afaerber)
mchang: needinfo? (jlee)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Guillaume GARDET 2020-10-29 07:40:04 UTC
## Observation

openQA test in scenario opensuse-Tumbleweed-JeOS-for-RPi-aarch64-jeos@RPi4 fails in
[yast2_bootloader](https://openqa.opensuse.org/tests/1453720/modules/yast2_bootloader/steps/5)

The problem seems to be the attempt to write to the nvram:
https://build.opensuse.org/request/show/843452
Due to bsc#1157550


## Test suite description
Maintainer: fvogt, mnowak

Start JeOS from the HDD image, configure it using the firstboot wizard and then run basic tests. console=tty0 added as needed for aarch64.


## Reproducible

Fails since (at least) Build [20201014](https://openqa.opensuse.org/tests/1434949)


## Expected result

Last good: (unknown) (or more recent)


## Further details

Always latest result in this scenario: [latest](https://openqa.opensuse.org/tests/latest?arch=aarch64&distri=opensuse&flavor=JeOS-for-RPi&machine=RPi4&test=jeos&version=Tumbleweed)
Comment 1 Michal Suchanek 2020-10-29 09:31:56 UTC
It's more likely due to the patch that does removable *and* non-removable install.

The test results did not make sense and looked inverted, and here we have attempt to install without --removable on platform where that fails.
Comment 2 Josef Reidinger 2020-10-29 09:52:24 UTC
To be honest I do not get what is goal of the test. Update NVRAM is checked in this case and it then it failed as it is read only? Maybe issue is that such HDD image has set update nvram variable to true on arm? Do you have idea who is author of such images?
Comment 3 Michal Suchanek 2020-10-29 09:58:47 UTC
I suppose that rPi is one of the platforms that does not support variables and needs removable installation with --removable argument.

However, grub is installed without the --removable argument which fails.

When merging the --no-nvram support the application of --removable looked opposite of what it should be.

The --no-nvram and --removable arguments are separate in the yast logic.
Comment 4 Andreas Färber 2020-10-29 11:34:41 UTC
There was a bug #1167015 (?) that in 15 SP2 led to the double-installation as workaround for a vendor's interesting UEFI firmware. Since then, upstream U-Boot grew UEFI variables support, I believe (but haven't verified) not read-only but not persistent at runtime for Raspberry Pi either, so same problem as elsewhere.

I believe the intended plan for SP3 and TW was to always install the Shim in the removable media location and the regular GRUB in the non-removable location?
Comment 5 Guillaume GARDET 2020-10-29 17:32:57 UTC
(In reply to Josef Reidinger from comment #2)
> To be honest I do not get what is goal of the test. Update NVRAM is checked
> in this case and it then it failed as it is read only? Maybe issue is that
> such HDD image has set update nvram variable to true on arm? Do you have
> idea who is author of such images?

According to the test header:
# Summary: Basic test for yast2 bootloader
# - Install yast2-bootloader
# - Launch yast2 bootloader
# - Handle missing package screen
# - Wait to yast2 to finish (initrd regenerated)

So, nothing fancy, just a check that yast2-bootloader works.

The image is a JeOS image built for the RPi, from: 
https://build.opensuse.org/package/show/openSUSE:Factory:ARM/JeOS
Comment 6 Josef Reidinger 2020-10-30 08:40:39 UTC
(In reply to Andreas Färber from comment #4)
> There was a bug #1167015 (?) that in 15 SP2 led to the double-installation
> as workaround for a vendor's interesting UEFI firmware. Since then, upstream
> U-Boot grew UEFI variables support, I believe (but haven't verified) not
> read-only but not persistent at runtime for Raspberry Pi either, so same
> problem as elsewhere.
> 
> I believe the intended plan for SP3 and TW was to always install the Shim in
> the removable media location and the regular GRUB in the non-removable
> location?

Well, I see jira entry to really use that new shim, but I do not know state. I am not against removing that workaround which is ugly. I just need to know what is the final plan. And if you need different binary according to configuration, I will need adapt code to add that logic.

So Andreas if you know state and final plan, can you please write it here and I will adapt that code?
and yes, bug was https://github.com/yast/yast-bootloader/blob/master/src/lib/bootloader/grub_install.rb#L43

bug#1167015
Comment 7 Gary Ching-Pang Lin 2020-11-04 06:11:46 UTC
I've backported a few patches for AArch64 shim and done some basic tests. I'm planning to disable the signature check for AArch64 shim in the spec file until we get a real workable signature.

On the other hand, Tumbleweed currently uses the shim-leap package which is directly copied from Leap. Since AArch64 shim doesn't exist in the current Leap, it cannot provides AArch64 shim for Tumbleweed. Will try to figure out a solution for that.
Comment 8 Stefan Schubert 2020-11-13 08:35:54 UTC
I think we close that bug until we are getting a "plan" in which direction we have to go. If the requirements are clear feel free to reopen the bug.
But I would prefer to generate a new one if there are more than one requirements.
Comment 9 Guillaume GARDET 2020-11-16 13:14:30 UTC
(In reply to Stefan Schubert from comment #8)
> I think we close that bug until we are getting a "plan" in which direction
> we have to go. If the requirements are clear feel free to reopen the bug.
> But I would prefer to generate a new one if there are more than one
> requirements.

Shouldn't we keep it open to find a plan, instead?
Comment 11 Ang 2020-12-02 04:59:55 UTC
On my particular machines, Raspberry Pi 3B (not plus) and 4B, I get this error when I do a Zypper dup from Tumbleweed 20201119 to 20201129:

Output of grub2-arm64-efi-2.04-21.2.noarch.rpm %posttrans script:
    update-bootloader: 2020-12-01 23:57:52 <3> update-bootloader-3564 run_command.294: '/usr/lib/bootloader/grub2-efi/install' failed with exit code 1, output:
    <<<<<<<<<<<<<<<<
    target = arm64-efi
    + /usr/sbin/grub2-install --target=arm64-efi --suse-force-signed --no-nvram --removable
    Installing for arm64-efi platform.
    Use signed file in /usr/share/grub2/arm64-efi/grub.efi for installation.
    Installation finished. No error reported.
    + /usr/sbin/grub2-install --target=arm64-efi --suse-force-signed
    Installing for arm64-efi platform.
    Use signed file in /usr/share/grub2/arm64-efi/grub.efi for installation.
    Could not prepare Boot variable: Read-only file system
    /usr/sbin/grub2-install: error: efibootmgr failed to register the boot entry: Input/output error.
    >>>>>>>>>>>>>>>>

Regardless of this error, both of my machines still boot normally.
Comment 12 Ang 2020-12-02 05:07:45 UTC
Both of my systems had openSUSE Tumbleweed installed via the ISO written to a USB drive. I tested this on both systems with the OS installed on an SD card and on the Raspberry Pi 4B with the OS installed on a USB flash drive and no SD card.

My software selections are xfce graphical desktop on both of my systems.

You will also get an error similar to the one produced by yast2 when doing a fresh install from the ISO written to a USB flash drive during the install bootloader step at the end of the installation process. The system will still boot regardless of this error.

The problem does not appear to be isolated to yast2 or the installer since it also produces similar errors when doing a Zypper dup on an established system.
Comment 13 Michael Chang 2020-12-02 10:20:18 UTC
This may be related to a kernel change that efivarfs is set to read-only if kernel detected SetVariable runtime service is not available or present in the system. Prior to the change efivarfs could expose no variable in such case so our detection worked.

See:
https://bugzilla.suse.com/show_bug.cgi?id=1174111#c9

To workaround, we could extend the efivars check to cover the permission ...

https://bugzilla.suse.com/show_bug.cgi?id=1174111#c10

@ Joey

Please correct if I am wrong.

@ Josef and Steffen

What do you think about adding the extended check to workaround the problem ?

Thanks.
Comment 14 Steffen Winterfeldt 2020-12-02 10:37:56 UTC
> What do you think about adding the extended check to workaround the problem ?

Basically ok, but I think bug 1174111 comment 10 will not work as root can always
write regardless of permission bits. Maybe experiment with 'stat'.

I just wish the kernel people would come up with an EFI api that's a bit more
abstracted than an 1:1 with that weird EFI interface.
Comment 15 Michael Chang 2020-12-03 07:57:43 UTC
(In reply to Steffen Winterfeldt from comment #14)
> > What do you think about adding the extended check to workaround the problem ?
> 
> Basically ok, but I think bug 1174111 comment 10 will not work as root can
> always
> write regardless of permission bits. Maybe experiment with 'stat'.

Yes it is true that root can always write regardless permission bits, but here the filesystem efivarfs is readonly, so the `test -w ...` actually worked to test whether the file/filesystem is writable.

Or if we want to test efivarfs readwritable, this command can be helpful 

> findmnt -O rw -t efivarfs --mountpoint /sys/firmware/efi/efivars && ... || ...

Thanks.
Comment 16 Steffen Winterfeldt 2020-12-03 08:53:50 UTC
Ah yes, you are right, the whole fs is read-only.
Comment 17 Stefan Schubert 2020-12-03 12:06:08 UTC
(In reply to Ang from comment #12)
> The problem does not appear to be isolated to yast2 or the installer since
> it also produces similar errors when doing a Zypper dup on an established
> system.
So I think it is not currently not a YAST issue. Feel free to assign it back, if we can do something.
Comment 23 Oliver Kurz 2021-03-06 06:06:36 UTC
This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: jeos@RPi3
https://openqa.opensuse.org/tests/1651911

To prevent further reminder comments one of the following options should be followed:
1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed
Comment 24 Michal Suchanek 2021-03-24 21:46:17 UTC
This will eventually affect SLE/Leap so it should be fixed - probably before 15 SP4.
Comment 25 Josef Reidinger 2021-03-29 06:54:26 UTC
Sorry, I am a bit lost now what is question for me, but now in bootloader is test for read only efivars, so it should be fixed, not?
Comment 26 Michal Suchanek 2021-03-29 10:00:40 UTC
The problem was just reported last week on opensuse-arm last week an is repeatedly flagged by openQA. 

Also the bug reference is not flagged in any SR by the OBS bots.

Where and when has it been fixed?
Comment 27 Michael Chang 2021-03-29 10:10:26 UTC
This should be set as dup to bsc#1174111. Wouldn't it ?
Comment 28 Josef Reidinger 2021-03-29 12:38:03 UTC
(In reply to Michael Chang from comment #27)
> This should be set as dup to bsc#1174111. Wouldn't it ?

yes, but it is for different product.

(In reply to Michal Suchanek from comment #26)
> The problem was just reported last week on opensuse-arm last week an is
> repeatedly flagged by openQA. 
> 
> Also the bug reference is not flagged in any SR by the OBS bots.
> 
> Where and when has it been fixed?

It should be fixed by this fix - https://github.com/yast/yast-bootloader/blob/master/package/yast2-bootloader.changes#L17
Comment 29 Michal Suchanek 2021-03-29 14:13:04 UTC
So it should be fixed since yast2-bootloader 4.3.23 which is pretty fresh and may not have made it to ARM Tumbleweed yet.
Comment 30 Guillaume GARDET 2021-03-29 14:46:17 UTC
openQA on RPi3/4 is green again since snapshot 20210315 and current released Tumbleweed is fixed.

See: https://openqa.opensuse.org/tests/1682147#step/yast2_bootloader/8