Bug 1191480 - Kernel:stable kernel 5.14.10-2.1.g2878fd1 cannot boot due to "bad shim signature"
Kernel:stable kernel 5.14.10-2.1.g2878fd1 cannot boot due to "bad shim signat...
Status: RESOLVED FIXED
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Kernel
Current
x86-64 openSUSE Tumbleweed
: P5 - None : Normal (vote)
: ---
Assigned To: openSUSE Kernel Bugs
E-mail List
https://github.com/openSUSE/suse-modu...
:
Depends on:
Blocks: 1189841
  Show dependency treegraph
 
Reported: 2021-10-08 10:24 UTC by Yan Huang
Modified: 2021-12-07 20:31 UTC (History)
13 users (show)

See Also:
Found By: Field Engineer
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
mokutil --list-enrolled (10.11 KB, text/plain)
2021-10-12 16:26 UTC, Yan Huang
Details
mokutil --list-enrolled (13.36 KB, text/plain)
2021-10-12 16:54 UTC, Yan Huang
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Yan Huang 2021-10-08 10:24:11 UTC
On openSUSE Tumbleweed 20211005 (with shim 15.4-4.2 and Secure Boot enabled), booting the newest Kernel:stable kernel 5.14.10-2.1.g2878fd1 leads to:

> Loading Linux kernel-default-5.14.10-2.g2878fd1-default ...
> error: ../../grub-core/kern/efi/sb.c:150:bad shim signature.
> Loading initial ramdisk ...
> error: ../../grub-core/loader/i386/efi/linux.c.98:you need to load the kernel first.
> 
> Press any key to continue..._

The previous Kernel:stable kernel 5.14.9-2.1.gd0ace7f did not have this issue.

The issue seems to be similar to the previous boo#1188142 (Kernel 5.13.1 does not boot due to bad shim signature).
Comment 1 Frank Krüger 2021-10-08 18:13:19 UTC
(In reply to Yan Huang from comment #0)
> On openSUSE Tumbleweed 20211005 (with shim 15.4-4.2 and Secure Boot
> enabled), booting the newest Kernel:stable kernel 5.14.10-2.1.g2878fd1 leads
> to:
> 
> > Loading Linux kernel-default-5.14.10-2.g2878fd1-default ...
> > error: ../../grub-core/kern/efi/sb.c:150:bad shim signature.
> > Loading initial ramdisk ...
> > error: ../../grub-core/loader/i386/efi/linux.c.98:you need to load the kernel first.
> > 
> > Press any key to continue..._
> 
> The previous Kernel:stable kernel 5.14.9-2.1.gd0ace7f did not have this
> issue.
> 
> The issue seems to be similar to the previous boo#1188142 (Kernel 5.13.1
> does not boot due to bad shim signature).

Confirmed with TW20201005, shim-15.4-4.2.x86_64 and kernel-default-5.14.10-2.1.g2878fd1.x86_64 from kernel:stable.
Comment 2 Michal Suchanek 2021-10-08 20:00:37 UTC
The kernel is signed but pesign signatures are apprently write-only there is no tool to tell you with which key specifically.

sbverify --list kernel-default-5.14.10-2.1.g2878fd1.x86_64/usr/lib/modules/5.14.10-2.g2878fd1-default/vmlinuz 
signature 1
image signature issuers:
 - /CN=Kernel OBS Project/emailAddress=Kernel@build.opensuse.org
image signature certificates:
 - subject: /CN=Kernel OBS Project/emailAddress=Kernel@build.opensuse.org
   issuer:  /CN=Kernel OBS Project/emailAddress=Kernel@build.opensuse.org

It verifies with the project certificate here: https://build.opensuse.org/projects/Kernel:stable/ssl_certificate

sbverify --cert ~/Downloads/ssl_certificate.txt ~/Downloads/kernel-default-5.14.10-2.1.g2878fd1.x86_64/usr/lib/modules/5.14.10-2.g2878fd1-default/vmlinuz 
Signature verification OK

The kernel package contains etc/uefi/certs/6A4E915C.crt so you can check that

mokutil --list contains a certificate with hash starting with 6A4E915C and enroll it if not.

Verifies with this certificate as well:

openssl x509 --inform DER --outform PEM --in ~/Downloads/kernel-default-5.14.10-2.1.g2878fd1.x86_64/etc/uefi/certs/6A4E915C.crt > /tmp/cert
sbverify --cert /tmp/cert ~/Downloads/kernel-default-5.14.10-2.1.g2878fd1.x86_64/usr/lib/modules/5.14.10-2.g2878fd1-default/vmlinuz 
Signature verification OK
Comment 3 Yan Huang 2021-10-11 12:48:42 UTC
The issue persists with the Kernel:stable kernel 5.14.11-1.1.g834dddd.
Comment 4 Michal Suchanek 2021-10-11 18:39:32 UTC
Please verify output of 'mokutil --list' contains a certificate with fingerprint starting with 6A4E915C and enroll it if not.
Comment 5 Jiri Slaby 2021-10-12 04:24:02 UTC
Works for me:
# mokutil --sb-state
SecureBoot enabled
# uname -r
5.14.11-20.g834dddd-default
# dmesg|grep -i Secure
Secure boot enabled
integrity: Loaded X.509 cert 'openSUSE Secure Boot CA: 6842600de22c4c477e95be23dfea9513e5971762'
integrity: Loaded X.509 cert 'openSUSE Secure Boot CA: 6842600de22c4c477e95be23dfea9513e5971762'
integrity: Loaded X.509 cert 'openSUSE Secure Boot Signkey: c8bdc7ac1a1d85966217fd93ebfc14f4a200b814'
Comment 6 Yan Huang 2021-10-12 16:26:29 UTC
Created attachment 853088 [details]
mokutil --list-enrolled

The current state of my system:
> # mokutil --sb-state
> SecureBoot enabled
> # uname -r
> 5.14.9-2.gd0ace7f-default
> # dmesg | grep -i secure
> [    0.009083] Secure boot enabled
> [    1.461959] integrity: Loaded X.509 cert 'openSUSE Secure Boot CA: 6842600de22c4c477e95be23dfea9513e5971762'
> [    1.463127] integrity: Loaded X.509 cert 'openSUSE Secure Boot Signkey: 0332fa9cbf0d88bf21924b0de82a09a54d5defc8'
> [    6.485674] Bluetooth: hci0: Secure boot is enabled

~~~~~~~~~

The mentioned certificate 6A4E915C.crt has been available only since the kernel 5.14.10-2.1.g2878fd1:
> # rpm -q --whatprovides /etc/uefi/certs/6A4E915C.crt
> kernel-default-5.14.10-2.1.g2878fd1.x86_64
> kernel-default-5.14.11-1.1.g834dddd.x86_64

More information about 6A4E915C.crt:
> # openssl x509 --inform DER --outform PEM --in /etc/uefi/certs/6A4E915C.crt > /tmp/6A4E915C.crt-pem
> # openssl x509 -in /tmp/6A4E915C.crt-pem -text | grep -e Before -e After
>             Not Before: Oct  5 16:48:55 2021 GMT
>             Not After : Dec 14 16:48:55 2023 GMT

~~~~~~~~~

The previous, known-to-be-working kernel 5.14.9-2.1.gd0ace7f provided a different certificate 1AA60533.crt:
# rpm -q --whatprovides /etc/uefi/certs/1AA60533.crt
kernel-default-5.14.9-2.1.gd0ace7f.x86_64

More information about 1AA60533.crt:
> # openssl x509 --inform DER --outform PEM --in /etc/uefi/certs/1AA60533.crt > /tmp/1AA60533.crt-pem
> # openssl x509 -in /tmp/1AA60533.crt-pem -text | grep -e Before -e After
>             Not Before: Aug 11 16:46:49 2019 GMT
>             Not After : Oct 19 16:46:49 2021 GMT

~~~~~~~~~

I tried to enroll the new certificate 6A4E915C.crt:
> # mokutil --import /etc/uefi/certs/6A4E915C.crt
> Already in kernel trusted keyring. Skip /etc/uefi/certs/6A4E915C.crt

However, 6A4E915C.crt is still not seen in "mokutil --list-enrolled" (judging by the certificates' validity) - I attached the output.
Comment 7 Michal Suchanek 2021-10-12 16:37:23 UTC
It will be enrolled only after reboot.

Thanks for verification.

There is a problem with enrolling certificates ATM which should be resolved in some weeks.

The released tumbleweed kernels should not be affected but the situation is different for the development snapshots.
Comment 8 Michal Suchanek 2021-10-12 16:41:19 UTC
Also you might need        --ignore-keyring option
Comment 9 Yan Huang 2021-10-12 16:54:39 UTC
Created attachment 853091 [details]
mokutil --list-enrolled

(In reply to Michal Suchanek from comment #8)
> Also you might need        --ignore-keyring option

Thanks a lot, Michal.

The "--ignore-keyring" option worked:
> # mokutil --import /etc/uefi/certs/6A4E915C.crt --ignore-keyring
> input password: 
> input password again: 

After reboot, the new certificate 6A4E915C.crt is enrolled (also seen in the attached "mokutil --list-enrolled" output) and the newest Kernel:stable kernel 5.14.11-1.1.g834dddd successfully booted with Secure Boot enabled.
Comment 10 Frank Krüger 2021-10-12 17:37:49 UTC
(In reply to Michal Suchanek from comment #7)
> It will be enrolled only after reboot.
> 
> Thanks for verification.
> 
> There is a problem with enrolling certificates ATM which should be resolved
> in some weeks.
> 
> The released tumbleweed kernels should not be affected but the situation is
> different for the development snapshots.

JFYI: "sudo mokutil --import /etc/uefi/certs/6A4E915C.crt --ignore-keyring" works also for me after a reboot. Apart from this workaround, which kind of solution is in sight?
Comment 11 Stephan Hemeier 2021-10-12 17:40:53 UTC
Same here, with --ignore-keyring I got my key to mok.

Before it was always:
# mokutil --import /etc/uefi/certs/F2B7BCC9.crt
> Already in kernel trusted keyring. Skip /etc/uefi/certs/F2B7BCC9.crt

And not shown in mokutil......

Now after restart, I could enroll the key and all is working.

PS:
I branch the kernel:stable:backport in my Repo so I have another key.
Comment 12 Michal Suchanek 2021-10-12 17:52:49 UTC
The key should be enrolled automagically but the --ignore-keyring option is not used.

If it's now needed to successfully enroll the key it needs to be adde in the scripts.
Comment 13 Joey Lee 2021-10-13 09:45:43 UTC
(In reply to Michal Suchanek from comment #12)
> The key should be enrolled automagically but the --ignore-keyring option is
> not used.
> 
> If it's now needed to successfully enroll the key it needs to be adde in the
> scripts.

I prefer to keep the logic for checking keyring (--ignore-keyring option can disable it) but not add it to scripts. 

This mokutil function be added to prevent that the nvram space be wasted. When a shim and kernel be produced by the same project. The shim should be embedded a openSUSE CA that it can verify the kernel that be signed by openSUSE signkey. And, the kernel is emabedded a openSUSE signkey. So we don't need enroll openSUSE signkey to MOK. It can save limited nvraom space of firmware.

About this issue, user installed a kernel be signed by another project (Kernel OBS Project/emailAddress=Kernel@build.opensuse.org, in this case). So shim's embedded CA can not verify the non-openSUSE signed kernel. And, mokutil checks the signkey is in kernel keyring because it be embedded by kernel. So the key can not be auto-enrolled.
Comment 14 Michal Suchanek 2021-10-13 13:09:49 UTC
But the key must be auto-enrolled, otherwise the system cannot boot.
Comment 15 Han Hui Teoh 2021-10-18 15:58:26 UTC
On MicroOS Desktop:

```
(15/23) Installing: kernel-default-5.14.11-1.2.x86_64 ...................[error]
Installation of kernel-default-5.14.11-1.2.x86_64 failed:
Error: Subprocess failed. Error: RPM failed: SecureBoot enabled
```

Is it related to this bug?
Comment 16 Michal Suchanek 2021-10-18 16:08:04 UTC
If you have 

/usr/lib/module-init-tools/kernel-scriptlets/cert-script
/usr/lib/module-init-tools/kernel-scriptlets/rpm-script

you can add 

set -x

at the start of these scripts and retry the kernel installation for _very_ verbose output.

I expect that the 'SecureBoot enabled' part comes from one of these but the rpm error does not say that a script failed so not sure.
Comment 17 Michal Suchanek 2021-10-19 09:30:53 UTC
Proposed fix

https://github.com/openSUSE/suse-module-tools/pull/48/commits/1bbbe4c04286ba917c9f5dc5817927d017447d95

(In reply to Joey Lee from comment #13)
> (In reply to Michal Suchanek from comment #12)
> > The key should be enrolled automagically but the --ignore-keyring option is
> > not used.
> > 
> > If it's now needed to successfully enroll the key it needs to be adde in the
> > scripts.
> 
> I prefer to keep the logic for checking keyring (--ignore-keyring option can
> disable it) but not add it to scripts. 
> 
> This mokutil function be added to prevent that the nvram space be wasted.
> When a shim and kernel be produced by the same project. The shim should be
> embedded a openSUSE CA that it can verify the kernel that be signed by
> openSUSE signkey. And, the kernel is emabedded a openSUSE signkey. So we
> don't need enroll openSUSE signkey to MOK. It can save limited nvraom space
> of firmware.
> 
> About this issue, user installed a kernel be signed by another project
> (Kernel OBS Project/emailAddress=Kernel@build.opensuse.org, in this case).
> So shim's embedded CA can not verify the non-openSUSE signed kernel. And,
> mokutil checks the signkey is in kernel keyring because it be embedded by
> kernel. So the key can not be auto-enrolled.

Then the check is wrong. The CA check should suffice to not enroll keys needlessly. On the other hand, kernel keys should be enrolled and they are expected to be in the kernel keyring so checking against the kernel keyring is pointless for kernel keys.
Comment 20 OBSbugzilla Bot 2021-10-22 14:40:06 UTC
This is an autogenerated message for OBS integration:
This bug (1191480) was mentioned in
https://build.opensuse.org/request/show/926988 Factory / suse-module-tools
Comment 22 Han Hui Teoh 2021-10-23 10:29:50 UTC
(In reply to Michal Suchanek from comment #16)
> If you have 
> 
> /usr/lib/module-init-tools/kernel-scriptlets/cert-script
> /usr/lib/module-init-tools/kernel-scriptlets/rpm-script
> 
> you can add 
> 
> set -x
> 
> at the start of these scripts and retry the kernel installation for _very_
> verbose output.
> 
> I expect that the 'SecureBoot enabled' part comes from one of these but the
> rpm error does not say that a script failed so not sure.

If I try removing `kernel-default-5.14.9-1.2.x86_64` it gives:


```
(1/6) Removing kernel-default-5.14.9-1.2.x86_64 .........................[error]
Removal of (68110)kernel-default-5.14.9-1.2.x86_64(@System) failed:
Error: Subprocess failed. Error: RPM failed: SecureBoot enabled
ln: failed to create hard link '/etc/uefi/certs/BDD31A9E.crt.delete': File exists
error: %preun(kernel-default-5.14.9-1.2.x86_64) scriptlet failed, exit status 1
error: kernel-default-5.14.9-1.2.x86_64: erase failed
```

So applying the same I ran `rm /etc/uefi/certs/BDD31A9E.crt.delete` before `zypper dup` and the upgrade worked.
Comment 23 Han Hui Teoh 2021-10-23 10:34:11 UTC
And looks like that was already fixed in https://github.com/openSUSE/suse-module-tools/commit/db74b3d9829ba69862bf0fe9ff3487d07ae00427
Comment 24 Michal Suchanek 2021-10-25 07:58:57 UTC
Should be fixed with suse-module-tools from Base:System
Comment 25 Swamp Workflow Management 2021-10-26 13:27:53 UTC
openSUSE-RU-2021:3509-1: An update that has 5 recommended fixes can now be installed.

Category: recommended (important)
Bug References: 1191200,1191260,1191480,1191804,1191922
CVE References: 
JIRA References: 
Sources used:
openSUSE Leap 15.3 (src):    suse-module-tools-15.3.13-3.11.1
Comment 26 Swamp Workflow Management 2021-10-26 13:29:43 UTC
SUSE-RU-2021:3509-1: An update that has 5 recommended fixes can now be installed.

Category: recommended (important)
Bug References: 1191200,1191260,1191480,1191804,1191922
CVE References: 
JIRA References: 
Sources used:
SUSE MicroOS 5.1 (src):    suse-module-tools-15.3.13-3.11.1
SUSE Linux Enterprise Module for Basesystem 15-SP3 (src):    suse-module-tools-15.3.13-3.11.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 27 Swamp Workflow Management 2021-10-26 16:25:49 UTC
SUSE-RU-2021:3515-1: An update that has 5 recommended fixes can now be installed.

Category: recommended (important)
Bug References: 1191200,1191260,1191480,1191804,1191922
CVE References: 
JIRA References: 
Sources used:
SUSE MicroOS 5.0 (src):    suse-module-tools-15.2.15-4.9.1
SUSE Linux Enterprise Module for Basesystem 15-SP2 (src):    suse-module-tools-15.2.15-4.9.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 28 Swamp Workflow Management 2021-10-31 20:54:27 UTC
openSUSE-RU-2021:1406-1: An update that has 5 recommended fixes can now be installed.

Category: recommended (important)
Bug References: 1191200,1191260,1191480,1191804,1191922
CVE References: 
JIRA References: 
Sources used:
openSUSE Leap 15.2 (src):    suse-module-tools-15.2.15-lp152.5.9.1
Comment 32 Michal Suchanek 2021-11-16 17:05:27 UTC
Update released.
Comment 35 Swamp Workflow Management 2021-12-01 17:26:45 UTC
SUSE-RU-2021:3820-1: An update that has 9 recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1158817,1189841,1189879,1190598,1191200,1191260,1191480,1191804,1191922
CVE References: 
JIRA References: 
Sources used:
SUSE Linux Enterprise Server for SAP 15 (src):    suse-module-tools-15.0.10-3.12.1
SUSE Linux Enterprise Server 15-LTSS (src):    suse-module-tools-15.0.10-3.12.1
SUSE Linux Enterprise High Performance Computing 15-LTSS (src):    suse-module-tools-15.0.10-3.12.1
SUSE Linux Enterprise High Performance Computing 15-ESPOS (src):    suse-module-tools-15.0.10-3.12.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 36 Swamp Workflow Management 2021-12-02 11:50:06 UTC
SUSE-RU-2021:3869-1: An update that has 8 recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1189841,1189879,1190598,1191200,1191260,1191480,1191804,1191922
CVE References: 
JIRA References: 
Sources used:
SUSE Linux Enterprise Server for SAP 15-SP1 (src):    suse-module-tools-15.1.23-3.19.1
SUSE Linux Enterprise Server 15-SP1-LTSS (src):    suse-module-tools-15.1.23-3.19.1
SUSE Linux Enterprise Server 15-SP1-BCL (src):    suse-module-tools-15.1.23-3.19.1
SUSE Linux Enterprise High Performance Computing 15-SP1-LTSS (src):    suse-module-tools-15.1.23-3.19.1
SUSE Linux Enterprise High Performance Computing 15-SP1-ESPOS (src):    suse-module-tools-15.1.23-3.19.1
SUSE Enterprise Storage 6 (src):    suse-module-tools-15.1.23-3.19.1
SUSE CaaS Platform 4.0 (src):    suse-module-tools-15.1.23-3.19.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 37 Swamp Workflow Management 2021-12-07 17:17:28 UTC
SUSE-RU-2021:3966-1: An update that has 8 recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1189841,1189879,1190598,1191200,1191260,1191480,1191804,1191922
CVE References: 
JIRA References: 
Sources used:
SUSE Linux Enterprise Server 12-SP5 (src):    suse-module-tools-12.11-3.8.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 38 Swamp Workflow Management 2021-12-07 20:31:57 UTC
SUSE-RU-2021:3970-1: An update that has 8 recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1189841,1189879,1190598,1191200,1191260,1191480,1191804,1191922
CVE References: 
JIRA References: 
Sources used:
SUSE OpenStack Cloud Crowbar 9 (src):    suse-module-tools-12.6.1-27.6.1
SUSE OpenStack Cloud Crowbar 8 (src):    suse-module-tools-12.6.1-27.6.1
SUSE OpenStack Cloud 9 (src):    suse-module-tools-12.6.1-27.6.1
SUSE OpenStack Cloud 8 (src):    suse-module-tools-12.6.1-27.6.1
SUSE Linux Enterprise Server for SAP 12-SP4 (src):    suse-module-tools-12.6.1-27.6.1
SUSE Linux Enterprise Server for SAP 12-SP3 (src):    suse-module-tools-12.6.1-27.6.1
SUSE Linux Enterprise Server 12-SP4-LTSS (src):    suse-module-tools-12.6.1-27.6.1
SUSE Linux Enterprise Server 12-SP3-LTSS (src):    suse-module-tools-12.6.1-27.6.1
SUSE Linux Enterprise Server 12-SP3-BCL (src):    suse-module-tools-12.6.1-27.6.1
SUSE Linux Enterprise Server 12-SP2-BCL (src):    suse-module-tools-12.6.1-27.6.1
HPE Helion Openstack 8 (src):    suse-module-tools-12.6.1-27.6.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.