Bug 1182749

Summary: [Build 154.1] secure boot is enabled on aarch64 by installer even when disabled on test machine
Product: [SUSE Linux Enterprise Server] PUBLIC SUSE Linux Enterprise Server 15 SP3 Reporter: Alvaro Carvajal <acarvajal>
Component: InstallerAssignee: YaST Team <yast-internal>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P1 - Urgent CC: afaerber, chester.lin, fcrozat, glin, jlee, jreidinger, llzhao, ptesarik, riafarov, richard.fan
Version: Public Beta   
Target Milestone: unspecified   
Hardware: aarch64   
OS: SLES 15   
URL: https://trello.com/c/GSWoJy3g
See Also: https://bugzilla.suse.com/show_bug.cgi?id=1182640
https://bugzilla.suse.com/show_bug.cgi?id=1174111
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: Yes
Marketing QA Status: --- IT Deployment: ---
Attachments: YaST logs
Serial Console

Description Alvaro Carvajal 2021-02-25 12:58:17 UTC
* Platform and arch: aarch64, Virtual System

* OS Version: 15-SP3 Public Beta (Build 154.1)
  - Installed via: ISO
  - Using procedure: manual installation by openQA

* LOGS: y2logs and serial console output

* Results:
  - Expected: SLES is installed successfully
  - Real: secure boot is enabled by installer, and it hits bsc#1182640 which prevents automatic installation from continuing.

* Reproducible: yes

* openQA:
  * Link to failed test:
https://openqa.suse.de/tests/5523884#step/start_install/3
https://openqa.suse.de/tests/5523966#step/start_install/3
  * Link to last successful run: https://openqa.suse.de/tests/5475412 (Build 150.1/Snapshot 10)

* Description: installation of SLES in textmode on aarch64 and of SLES+HA in the same scenario is currently failing due to bsc#1182640. As described in that bug, issue is present when attempting to install SLES with secure boot on UEFI, which causes the installer to report an un-resolvable dependency issue while looking for the mokutil package. By explicitly disabling secure boot in the Bootloader Setting Screen, installation is possible (http://mango.qa.suse.de/tests/3553), however, per the test results it seems that secure boot is disabled on the VM at the start of the test and then it gets enabled by the installer automatically.

In the serial log from test https://openqa.suse.de/tests/5523884, the following can be seen in the serial console:

# grep -i secure serial0.txt 
Variable SecureBoot is 0
Variable SecureBootEnable is 0
add-symbol-file /home/abuild/rpmbuild/BUILD/edk2-edk2-stable201911/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/SecurityPkg/VariableAuthenticated/SecureBootConfigDxe/SecureBootConfigDxe/DEBUG/SecureBootConfigDxe.dll 0x7BA3C000
Loading driver at 0x0007BA3B000 EntryPoint=0x0007BA4738C SecureBootConfigDxe.efi
[    1.318400] ima: secureboot mode disabled
[    8.325810] Loaded X.509 cert 'SUSE Linux Enterprise Secure Boot Signkey: <snip>

It seems that during boot, VM is correctly configured with secure boot disabled, judging by 'Variable SecureBoot is 0', 'Variable SecureBootEnable is 0' and by 'ima: secureboot mode disabled' in the logs. However installer is enabling this setting, as can be seen on https://openqa.suse.de/tests/5523884#step/disable_grub_timeout/4, leading the test to subsequently fail with bsc#1182640.
Comment 1 Alvaro Carvajal 2021-02-25 12:58:58 UTC
Created attachment 846514 [details]
YaST logs
Comment 2 Alvaro Carvajal 2021-02-25 12:59:19 UTC
Created attachment 846515 [details]
Serial Console
Comment 4 Josef Reidinger 2021-02-25 15:58:17 UTC
Well, reason is simple. When we start with secure boot on x86_64 answer when to propose it is that we should propose it by default always as it is safe option and shim can handle it. And it is same for other architectures with EFI and only exception is s390 secure boot where it is checked if it is available at all https://github.com/yast/yast-bootloader/blob/master/src/lib/bootloader/systeminfo.rb#L105
Comment 5 Lukas Ocilka 2021-02-25 16:07:06 UTC
Andreas, it seems we are a bit stuck here. According to our knowledge, this is expected behavior, but it does not seem to work. We can'd decide what should happen now. Maybe asking shim maintainer? Please, decide.
Comment 6 Andreas Färber 2021-02-25 16:50:58 UTC
(In reply to Lukas Ocilka from comment #5)
> Andreas, it seems we are a bit stuck here. According to our knowledge, this
> is expected behavior, but it does not seem to work. We can'd decide what
> should happen now. Maybe asking shim maintainer? Please, decide.

Shim experts are already in CC.

Bug #1182640 (an extension of bug #1182210) simply needs to be fixed by making mokutil available on aarch64 (000package-groups). That'll solve the dependency.

I don't see another issue here right now - although we should keep an eye out for any Tumbleweed-style issues with systems that don't have read/writeable efivarfs.

If our shim gets installed as bootaa64.efi then I would expect (subject to tests) that all systems should be able to boot it, also without Secure Boot enabled and also with read-only or no UEFI variables.
If that assumption holds true, that leaves us with GRUB, mokutil, efibootmgr and related tools potentially being unable to add/modify UEFI variables for boot entries and their callers needing to handle that gracefully one way or another.

If we're unable/unwilling to handle it that way, we may need to detect read-only UEFI variables (or lack of UEFI variables, thinking of Automotive or recovery) and disable the Secure Boot default at runtime accordingly.
Comment 9 Josef Reidinger 2021-02-26 07:33:41 UTC
(In reply to Andreas Färber from comment #6)
> (In reply to Lukas Ocilka from comment #5)
> > Andreas, it seems we are a bit stuck here. According to our knowledge, this
> > is expected behavior, but it does not seem to work. We can'd decide what
> > should happen now. Maybe asking shim maintainer? Please, decide.
> 
> Shim experts are already in CC.
> 
> Bug #1182640 (an extension of bug #1182210) simply needs to be fixed by
> making mokutil available on aarch64 (000package-groups). That'll solve the
> dependency.
> 
> I don't see another issue here right now - although we should keep an eye
> out for any Tumbleweed-style issues with systems that don't have
> read/writeable efivarfs.
> 
> If our shim gets installed as bootaa64.efi then I would expect (subject to
> tests) that all systems should be able to boot it, also without Secure Boot
> enabled and also with read-only or no UEFI variables.
> If that assumption holds true, that leaves us with GRUB, mokutil, efibootmgr
> and related tools potentially being unable to add/modify UEFI variables for
> boot entries and their callers needing to handle that gracefully one way or
> another.
> 
> If we're unable/unwilling to handle it that way, we may need to detect
> read-only UEFI variables (or lack of UEFI variables, thinking of Automotive
> or recovery) and disable the Secure Boot default at runtime accordingly.

Well, in installer we have that no_nvram and also removable if we detect something like ro EFI and we apply it also when calling shim
https://github.com/yast/yast-bootloader/blob/master/src/lib/bootloader/grub_install.rb#L102
I am not sure if it is enough? In fact for aarch64 we run always removable and when we found efivars we run also without removable https://github.com/yast/yast-bootloader/blob/master/src/lib/bootloader/grub_install.rb#L102
so booting on aarch is currently probably the most complex one in installer :)
Comment 10 Joey Lee 2021-02-26 07:51:30 UTC
Hi Josef, 

(In reply to Josef Reidinger from comment #9)
> (In reply to Andreas Färber from comment #6)
> > (In reply to Lukas Ocilka from comment #5)
> > > Andreas, it seems we are a bit stuck here. According to our knowledge, this
> > > is expected behavior, but it does not seem to work. We can'd decide what
> > > should happen now. Maybe asking shim maintainer? Please, decide.
> > 
> > Shim experts are already in CC.
> > 
> > Bug #1182640 (an extension of bug #1182210) simply needs to be fixed by
> > making mokutil available on aarch64 (000package-groups). That'll solve the
> > dependency.
> > 
> > I don't see another issue here right now - although we should keep an eye
> > out for any Tumbleweed-style issues with systems that don't have
> > read/writeable efivarfs.
> > 
> > If our shim gets installed as bootaa64.efi then I would expect (subject to
> > tests) that all systems should be able to boot it, also without Secure Boot
> > enabled and also with read-only or no UEFI variables.
> > If that assumption holds true, that leaves us with GRUB, mokutil, efibootmgr
> > and related tools potentially being unable to add/modify UEFI variables for
> > boot entries and their callers needing to handle that gracefully one way or
> > another.
> > 
> > If we're unable/unwilling to handle it that way, we may need to detect
> > read-only UEFI variables (or lack of UEFI variables, thinking of Automotive
> > or recovery) and disable the Secure Boot default at runtime accordingly.
> 
> Well, in installer we have that no_nvram and also removable if we detect
> something like ro EFI and we apply it also when calling shim
> https://github.com/yast/yast-bootloader/blob/master/src/lib/bootloader/
> grub_install.rb#L102
> I am not sure if it is enough? In fact for aarch64 we run always removable
> and when we found efivars we run also without removable
> https://github.com/yast/yast-bootloader/blob/master/src/lib/bootloader/
> grub_install.rb#L102
> so booting on aarch is currently probably the most complex one in installer
> :)

Could you please also add the detection of the Read_Only attribute on efivarfs? Kernel will mount efivarfs with RO when firmware doesn't support SetVariable EFI protocol, which means the EFI variables can only be read, so boot entry can not be created success.

e.g. For detection, Using mount command or cat /proc/mounts 

linux-npj3:~ # mount | grep efivarfs
efivarfs on /sys/firmware/efi/efivars type efivarfs (ro,nosuid,nodev,noexec,relatime)

linux-npj3:~ # cat /proc/mounts | grep efivarfs
efivarfs /sys/firmware/efi/efivars efivarfs ro,nosuid,nodev,noexec,relatime 0 0

Please also reference comment#37-39 on bsc#1174111.

Thanks a lot!
Comment 11 Josef Reidinger 2021-02-26 08:18:28 UTC
(In reply to Joey Lee from comment #10)
> Hi Josef, 
> 
> (In reply to Josef Reidinger from comment #9)
> > (In reply to Andreas Färber from comment #6)
> > > (In reply to Lukas Ocilka from comment #5)
> > > > Andreas, it seems we are a bit stuck here. According to our knowledge, this
> > > > is expected behavior, but it does not seem to work. We can'd decide what
> > > > should happen now. Maybe asking shim maintainer? Please, decide.
> > > 
> > > Shim experts are already in CC.
> > > 
> > > Bug #1182640 (an extension of bug #1182210) simply needs to be fixed by
> > > making mokutil available on aarch64 (000package-groups). That'll solve the
> > > dependency.
> > > 
> > > I don't see another issue here right now - although we should keep an eye
> > > out for any Tumbleweed-style issues with systems that don't have
> > > read/writeable efivarfs.
> > > 
> > > If our shim gets installed as bootaa64.efi then I would expect (subject to
> > > tests) that all systems should be able to boot it, also without Secure Boot
> > > enabled and also with read-only or no UEFI variables.
> > > If that assumption holds true, that leaves us with GRUB, mokutil, efibootmgr
> > > and related tools potentially being unable to add/modify UEFI variables for
> > > boot entries and their callers needing to handle that gracefully one way or
> > > another.
> > > 
> > > If we're unable/unwilling to handle it that way, we may need to detect
> > > read-only UEFI variables (or lack of UEFI variables, thinking of Automotive
> > > or recovery) and disable the Secure Boot default at runtime accordingly.
> > 
> > Well, in installer we have that no_nvram and also removable if we detect
> > something like ro EFI and we apply it also when calling shim
> > https://github.com/yast/yast-bootloader/blob/master/src/lib/bootloader/
> > grub_install.rb#L102
> > I am not sure if it is enough? In fact for aarch64 we run always removable
> > and when we found efivars we run also without removable
> > https://github.com/yast/yast-bootloader/blob/master/src/lib/bootloader/
> > grub_install.rb#L102
> > so booting on aarch is currently probably the most complex one in installer
> > :)
> 
> Could you please also add the detection of the Read_Only attribute on
> efivarfs? Kernel will mount efivarfs with RO when firmware doesn't support
> SetVariable EFI protocol, which means the EFI variables can only be read, so
> boot entry can not be created success.
> 
> e.g. For detection, Using mount command or cat /proc/mounts 
> 
> linux-npj3:~ # mount | grep efivarfs
> efivarfs on /sys/firmware/efi/efivars type efivarfs
> (ro,nosuid,nodev,noexec,relatime)
> 
> linux-npj3:~ # cat /proc/mounts | grep efivarfs
> efivarfs /sys/firmware/efi/efivars efivarfs ro,nosuid,nodev,noexec,relatime
> 0 0
> 
> Please also reference comment#37-39 on bsc#1174111.
> 
> Thanks a lot!

Hi Joey,
I can do it, but need more info. At first does it effect all architectures or is it ARM specific?
And as ARM booting getting more and more complex, is there some good source of documentation that we can link from our code? As currently we are already referencing bunch of bug reports and features and it is getting more and more hard to keep overall picture.

Thanks
Josef
Comment 12 Lukas Ocilka 2021-02-26 09:36:35 UTC
(In reply to Josef Reidinger from comment #11)
> And as ARM booting getting more and more complex, is there some good source
> of documentation that we can link from our code? As currently we are already
> referencing bunch of bug reports and features and it is getting more and
> more hard to keep overall picture.


In other words: We'd ideally need some document, description, workflow, diagram, ... how it should ALL work "tomorrow" maybe in if-this-than-that form. Maybe this part even deserves rewriting from scratch, we don't know yet.
Comment 13 Joey Lee 2021-02-26 11:06:56 UTC
Hi Josef,

(In reply to Josef Reidinger from comment #11)
> (In reply to Joey Lee from comment #10)
> > Hi Josef, 
> > 
> > (In reply to Josef Reidinger from comment #9)
> > > (In reply to Andreas Färber from comment #6)
> > > > (In reply to Lukas Ocilka from comment #5)
> > > > > Andreas, it seems we are a bit stuck here. According to our knowledge, this
> > > > > is expected behavior, but it does not seem to work. We can'd decide what
> > > > > should happen now. Maybe asking shim maintainer? Please, decide.
> > > > 
> > > > Shim experts are already in CC.
> > > > 
> > > > Bug #1182640 (an extension of bug #1182210) simply needs to be fixed by
> > > > making mokutil available on aarch64 (000package-groups). That'll solve the
> > > > dependency.
> > > > 
> > > > I don't see another issue here right now - although we should keep an e

[...ship]

> > > 
> > > Well, in installer we have that no_nvram and also removable if we detect
> > > something like ro EFI and we apply it also when calling shim
> > > https://github.com/yast/yast-bootloader/blob/master/src/lib/bootloader/
> > > grub_install.rb#L102
> > > I am not sure if it is enough? In fact for aarch64 we run always removable
> > > and when we found efivars we run also without removable
> > > https://github.com/yast/yast-bootloader/blob/master/src/lib/bootloader/
> > > grub_install.rb#L102
> > > so booting on aarch is currently probably the most complex one in installer
> > > :)
> > 
> > Could you please also add the detection of the Read_Only attribute on
> > efivarfs? Kernel will mount efivarfs with RO when firmware doesn't support
> > SetVariable EFI protocol, which means the EFI variables can only be read, so
> > boot entry can not be created success.
> > 
> > e.g. For detection, Using mount command or cat /proc/mounts 
> > 
> > linux-npj3:~ # mount | grep efivarfs
> > efivarfs on /sys/firmware/efi/efivars type efivarfs
> > (ro,nosuid,nodev,noexec,relatime)
> > 
> > linux-npj3:~ # cat /proc/mounts | grep efivarfs
> > efivarfs /sys/firmware/efi/efivars efivarfs ro,nosuid,nodev,noexec,relatime
> > 0 0
> > 
> > Please also reference comment#37-39 on bsc#1174111.
> > 
> > Thanks a lot!
> 
> Hi Joey,
> I can do it, but need more info. At first does it effect all architectures
> or is it ARM specific?
> And as ARM booting getting more and more complex, is there some good source
> of documentation that we can link from our code? As currently we are already
> referencing bunch of bug reports and features and it is getting more and
> more hard to keep overall picture.
> 

It is architecture independent. It is base on EFI_RT_PROPERTIES table which works both with x86_64 and aarch64.

Thanks
Comment 15 Stefan Weiberg 2021-03-01 13:30:03 UTC
Also see Rodion's comment on bsc#1182776: https://bugzilla.suse.com/show_bug.cgi?id=1182776#c12

The other two aarch64 related bugs blocking installation of aarch64 are meanwhile resolved in build156.3 (bsc#1182640 and bsc#1182663), with this issue still occurring and also bsc#1182776 having an impact on aarch64 and secureboot.
Comment 16 Josef Reidinger 2021-03-03 12:31:52 UTC
fix for read only efivars is in place https://github.com/yast/yast-bootloader/pull/635
Comment 24 Swamp Workflow Management 2021-03-29 16:18:38 UTC
SUSE-RU-2021:0968-1: An update that has two recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1174111,1182749
CVE References: 
JIRA References: 
Sources used:
SUSE Linux Enterprise Module for Basesystem 15-SP2 (src):    yast2-bootloader-4.2.28-3.9.1
SUSE Linux Enterprise Installer 15-SP2 (src):    yast2-bootloader-4.2.28-3.9.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 25 Swamp Workflow Management 2021-04-02 13:17:37 UTC
openSUSE-RU-2021:0497-1: An update that has two recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1174111,1182749
CVE References: 
JIRA References: 
Sources used:
openSUSE Leap 15.2 (src):    yast2-bootloader-4.2.28-lp152.2.9.1
Comment 31 Maintenance Automation 2023-08-02 08:44:47 UTC
SUSE-RU-2023:3149-1: An update that contains four features and has 15 recommended fixes can now be installed.

Category: recommended (important)
Bug References: 1136601, 1157550, 1167015, 1172293, 1174111, 1174320, 1182749, 1184160, 1188768, 1192764, 1198197, 1198828, 1201399, 1208003, 1210799
Jira References: SLE-11500, SLE-15020, SLE-15823, SLE-18271
Sources used:
SUSE OpenStack Cloud 9 (src): perl-Bootloader-0.944-3.3.1
SUSE OpenStack Cloud Crowbar 9 (src): perl-Bootloader-0.944-3.3.1
SUSE Linux Enterprise Server for SAP Applications 12 SP4 (src): perl-Bootloader-0.944-3.3.1
SUSE Linux Enterprise Server 12 SP4 ESPOS 12-SP4 (src): perl-Bootloader-0.944-3.3.1
SUSE Linux Enterprise Server 12 SP4 LTSS 12-SP4 (src): perl-Bootloader-0.944-3.3.1
SUSE Linux Enterprise High Performance Computing 12 SP5 (src): perl-Bootloader-0.944-3.3.1
SUSE Linux Enterprise Server 12 SP5 (src): perl-Bootloader-0.944-3.3.1
SUSE Linux Enterprise Server for SAP Applications 12 SP5 (src): perl-Bootloader-0.944-3.3.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.