Kernel recommended that dracut folks would look into it first

Happens also on a brand new installation of 5.4. I did use a preconfigured image.

(In reply to Lubos Kocman from comment #0)
> dracut: *** Creating image file '/boot/initrd-5.14.21-150400.24.69-default'
> ***
> dracut: *** Creating initramfs image file
> '/boot/initrd-5.14.21-150400.24.69-default' done ***

This is the last line printed by dracut, all good.

> Failed to enroll new keys
> Failed to import /etc/uefi/certs/76B6A6A0.crt

Maybe I'm wrong, but this looks like suse-module-tools [1], Martin?

[1] https://github.com/openSUSE/suse-module-tools/blob/eecfaab275ec7a166a6fefebadd884a0bec0b8e3/kernel-scriptlets/cert-script#L79

I did manually following on a new install

# transactional-update shell
# zypper up
...
# echo $?
107
# mokutil --import /etc/uefi/certs/76B6A6A0.crt
Already in kernel trusted keyring. Skip /etc/uefi/certs/76B6A6A0.crt
# echo $?
0

Oky, just tested to reboot into this "manually updated" snapshot, and the system does boot.

Any update dracut-maintainers? This is bit important as it blocks automated updates. And mistifies people who rely on this functionality simply working. (Increasing severity).

(In reply to Lubos Kocman from comment #6)
> Any update dracut-maintainers? This is bit important as it blocks automated
> updates. And mistifies people who rely on this functionality simply working.
> (Increasing severity).

This issue belongs to another package. See comment #3

(In reply to Lubos Kocman from comment #4)

> # mokutil --import /etc/uefi/certs/76B6A6A0.crt
> Already in kernel trusted keyring. Skip /etc/uefi/certs/76B6A6A0.crt

the script would run 
  "mokutil --import /etc/uefi/certs/76B6A6A0.crt --root-pw --ca-check"

Can you try if that manual command succeeds as well?

The script error handling was just imported into the s-m-t kernel scriptlets from the kernel package. It's generally discouraged to use non-zero exit codes for rpm scriptlets. (https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/: "All scriptlets MUST exit with the zero exit status").

So if something fails, we should probably just print an error message and return 0 nonetheless, unless we *know* that the error is *really* fatal, and the rpm transaction must be aborted.

Michal, thoughts?

Does the transaction fail when the scriptlets fail, though?

As far as I know the return value is only advisory, and scritlet failure causes zypper to return a specific return value that the caller can check or ignore.

There has been the opposite bug complaining that kernel scriptlets do not fail when something fails.

> Does the transaction fail when the scriptlets fail, though?

Yes. IMO that's a general property of rpm. See dedora docs link above.

That's Fedora dos, not a general rpm doc.

Adding libzypp maintainers for intended semantics when using zypper.

IIRC failing scritplets are strongly discouraged on openSUSE, too. But I couldn't find the docs openSUSE, only Fedora.

Yes the 'transactional-update' will simply fail and automatic-systems are basically paralelized, until somebody runs transactional-update shell and will run zypper in instide, then the snapshot will be created. And you can try your luck until the next kernel update.

I don't have a Leap Micro test system here... while the general question about the scriptlets needs to be clarified, can you try to answer comment 8?

Yes, if the key fails to enroll it's completely fine that transactional update fails, and it's in fact correct.

With the key not enrolled you would not be able to boot the system.

Ah sorry for missing that part, it all seems to succeed. Can you clarify on the selinux ownership? Does this look okay. There was a one more update of kernel since the bug report and it failed too. No matter that the manual import did't exit with error.

localhost:~ # cat /etc/os-release 
NAME="openSUSE Leap Micro"
VERSION="5.4"
ID="opensuse-leap-micro"
ID_LIKE="suse opensuse opensuse-leap suse-microos"
VERSION_ID="5.4"
PRETTY_NAME="openSUSE Leap Micro 5.4"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap-micro:5.4"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:LeapMicro"
LOGO="distributor-logo-LeapMicro"
localhost:~ # mokutil --import /etc/uefi/certs/76B6A6A0.crt --root-pw --ca-check
Already in kernel trusted keyring. Skip /etc/uefi/certs/76B6A6A0.crt
localhost:~ # echo $?
0
localhost:~ # ls -laZ /etc/uefi/certs/76B6A6A0.crt
-rw-r--r--. 1 root root system_u:object_r:etc_t:s0 1288 Aug  9 12:53 /etc/uefi/certs/76B6A6A0.crt

Not sure if the delete part helps

transactional update # mokutil --import /etc/uefi/certs/76B6A6A0.crt --root-pw --ca-check       
Already in kernel trusted keyring. Skip /etc/uefi/certs/76B6A6A0.crt
transactional update # mokutil --delete /etc/uefi/certs/76B6A6A0.crt --root-pw --ca-check
SKIP: /etc/uefi/certs/76B6A6A0.crt is not in MokList
transactional update # mokutil --delete /etc/uefi/certs/76B6A6A0.crt                     
SKIP: /etc/uefi/certs/76B6A6A0.crt is not in MokList

Perhaps we're experiencing the same in %post as with --ignore-keyring

transactional update # mokutil --delete /etc/uefi/certs/76B6A6A0.crt                     
SKIP: /etc/uefi/certs/76B6A6A0.crt is not in MokList
transactional update # mokutil --db                                                             
transactional update # mokutil --dbx
transactional update # mokutil --pk 
transactional update # mokutil --kek
transactional update # mokutil --import /etc/uefi/certs/76B6A6A0.crt --root-pw --ca-check 
Already in kernel trusted keyring. Skip /etc/uefi/certs/76B6A6A0.crt
transactional update # mokutil --list-enrolled                                            
transactional update # mokutil --list-new     
transactional update # mokutil --import /etc/uefi/certs/76B6A6A0.crt --root-pw --ca-check --ignore-keyring
Failed to enroll new keys

(In reply to Lubos Kocman from comment #19)

> Perhaps we're experiencing the same in %post as with --ignore-keyring

yes, the script would use --ignore-keyring if supported by mokutil.

> transactional update # mokutil --import /etc/uefi/certs/76B6A6A0.crt
> --root-pw --ca-check --ignore-keyring
> Failed to enroll new keys

... and does this atually exit with non-0? On my (non-micro) system, it get 

> # mokutil --import /etc/uefi/certs/1F673297.crt --root-pw  --ca-check --ignore-keyring
> CA enrolled. Skip /etc/uefi/certs/1F673297.crt
> # echo $?
> 0

It's not that easy to make mokutil --import exit with non-0 ...

(In reply to Lubos Kocman from comment #19)

> transactional update # mokutil --import /etc/uefi/certs/76B6A6A0.crt
> --root-pw --ca-check --ignore-keyring
> Failed to enroll new keys

This is the same error as in comment 0. Could you run this through strace in order to figure out what's going wrong?

I've been testing this in a VM now. No luck reproducing the issue, not even with 
/sys/firmware/efi/efivars mounted read-only.

Could you try with some debugging enabled?

Run transactional-update shell and edit /usr/lib/module-init-tools/kernel-scriplets/cert-script as follows:

At the top, add:

exec 2>>/var/log/cert-script.log
echo "$0 $@" >&2
set -x

And change the run_mokutil function as follows:

run_mokutil () {
    [ -z "$KERNEL_PACKAGE_SCRIPT_DEBUG" ] || echo mokutil "$@" >&2
    strace -f -ttt -o /var/tmp/cert-$$.log mokutil "$@"
}

You must also install strace.

Reboot, run the update, and provide the /var/log/cert* ouptput files.

(In reply to Martin Wilck from comment #9)
> The script error handling was just imported into the s-m-t kernel scriptlets
> from the kernel package. It's generally discouraged to use non-zero exit
> codes for rpm scriptlets.
> (https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/: "All
> scriptlets MUST exit with the zero exit status").
> 
> So if something fails, we should probably just print an error message and
> return 0 nonetheless, unless we *know* that the error is *really* fatal, and
> the rpm transaction must be aborted.
> 
> Michal, thoughts?

This is completely up to rpm. Basically the point in time where the package is  unpacked to disk and registered in the rpm database is the point where rpm calls the package 'installed'.

This is why an error in a %pre script leads to a non-zero rpm exit code and the package is still considered to be not installed (or not updated). While an error in a %post script is ignored, rpm returns 0 and the package is considered to be installed (or updated).

Regarding the individual package this is an easy rule. If scripts before unpacking fail (new packages %pre, old packages %preun) the old one stays installed. If scripts after unpacking fail, the new one is installed. 


Regarding the whole transaction it depends on the backend. Zypp's current default rpm backend, which issues a single rpm call for every single package, is able to abort/retry/ignore after each failing package. (But we have other issues with this mode).

For rpm transactions processing multiple packages (rpm CLI or the new ZYPP_SINGLE_RPMTRANS backend) it's completely up to librpm to decide which error causes a package to be skipped or the whole transaction to be aborted. 

But the rule regarding the individual package is the same: 
  unpacked to disk == new version installed


Regarding zypper's exit code: 
If rpm returns not-zero, zypper also returns not-zero. 
If rpm returns zero, it depends.
We monitor rpm's debug output to detect failing scripts. The errors are highlighted in the output (and should be visible in the zypp/history) and if rpm returns zero, zypper returns 107 - ZYPPER_EXIT_INF_RPM_SCRIPT_FAILED in case script errors were seen.

Thanks for the detailed response. So it seems that failed %post scripts aren't as fatal as I thought they were. Maybe transactional-update's policy for dealing with errors needs to be reviewed?

Anyway, we won't make progress wrt the mokutil issue if it's not reproducible (see comment 22). Any ideas how to proceed?

Created attachment 869208 [details]
stra

strace mokutil --import /etc/uefi/certs/76B6A6A0.crt     --root-pw  --ca-check --ignore-keyring  2> /tmp/strace-mokutil.txt

Seems like this is the issue

openat(AT_FDCWD, "/sys/firmware/efi/efivars/MokNew-605dab50-e046-4300-abb6-3dd810dd8b23", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/sys/firmware/efi/efivars/MokNew-605dab50-e046-4300-abb6-3dd810dd8b23", O_WRONLY|O_CREAT|O_EXCL, 0600) = -1 EROFS (Read-only file system)

inside tu 

transactional update # ls -la /sys/firmware/efi/efivars/    
total 0
drwxr-xr-x. 2 root root  0 Aug 16 14:00 .
drwxr-xr-x. 3 root root  0 Aug 16 14:00 ..
-rw-r--r--. 1 root root  5 Aug 16 14:00 AuditMode-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r--. 1 root root  5 Aug 16 14:00 DeployedMode-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r--. 1 root root 12 Aug 16 14:00 OsIndications-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r--. 1 root root 12 Aug 16 14:00 OsIndicationsSupported-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r--. 1 root root 10 Aug 16 14:00 PlatformLang-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r--. 1 root root 10 Aug 16 14:00 PlatformLangCodes-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r--. 1 root root  5 Aug 16 14:00 SecureBoot-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r--. 1 root root  5 Aug 16 14:00 SetupMode-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r--. 1 root root  5 Aug 16 14:00 VendorKeys-8be4df61-93ca-11d2-aa0d-00e098032b8c

My strace starts out similar, but then diverges. It reads MokListRT and seems to be fine with that. None of the Mok variables is present on your Raspi, though. Perhaps that's the reason?

1693577033.432890 openat(AT_FDCWD, "/sys/firmware/efi/efivars/MokNew-605dab50-e046-4300-abb6-3dd810dd8b23", O_RDONLY) = -1 ENOENT (No such file or directory)
1693577033.433036 openat(AT_FDCWD, "/etc/uefi/certs/76B6A6A0.crt", O_RDONLY) = 6
1693577033.433139 read(6, "0\202\5\0040\202\3\354\240\3\2\1\2\2\t\0\312\374\265\327^\305\211\2020\r\6\t*\206H\206"..., 1288) = 1288
1693577033.436275 openat(AT_FDCWD, "/sys/firmware/efi/efivars/MokListRT-605dab50-e046-4300-abb6-3dd810dd8b23", O_RDONLY) = 7

On my VM, I see:

# mokutil --test-key /etc/uefi/certs/76B6A6A0.crt 
/etc/uefi/certs/76B6A6A0.crt is already in the built-in trusted keyring

it's hard-coded into the kernel, so I wonder why it would ever need to be written to the MoK database.

Btw I'd re-mounted efivarfs read-only for the strace I took in comment 28.

I was finally able to reproduce the issue on my VM by copying another cert (1F673297.crt) from my TW system to the Leap micro VM.

I now see the same behavior Lubos did, if efivarfs is mounted read-only.

> 3513  1693577836.497635 openat(AT_FDCWD, "/sys/firmware/efi/efivars/MokNew-605dab50-e046-4300-abb6-3dd810dd8b23", O_RDONLY) = -1 ENOENT (No such file or directory)
> 3513  1693577836.497808 openat(AT_FDCWD, "/sys/firmware/efi/efivars/MokNew-605dab50-e046-4300-abb6-3dd810dd8b23", O_WRONLY|O_CREAT|O_EXCL, 0600) = -1 EROFS (Read-only file system)

If efivarfs is mounted read-write, I see 

> 3541  1693578005.505298 openat(AT_FDCWD, "/sys/firmware/efi/efivars/MokNew-605dab50-e046-4300-abb6-3dd810dd8b23", O_RDONLY) = -1 ENOENT (No such file or directory)
> 3541  1693578005.505392 openat(AT_FDCWD, "/sys/firmware/efi/efivars/MokNew-605dab50-e046-4300-abb6-3dd810dd8b23", O_WRONLY|O_CREAT|O_EXCL, 0600) = 6
> 3541  1693578005.630770 openat(AT_FDCWD, "/sys/firmware/efi/efivars/MokAuth-605dab50-e046-4300-abb6-3dd810dd8b23", O_RDONLY) = -1 ENOENT (No such file or directory)
> 3541  1693578005.630976 openat(AT_FDCWD, "/sys/firmware/efi/efivars/MokAuth-605dab50-e046-4300-abb6-3dd810dd8b23", O_WRONLY|O_CREAT|O_EXCL, 0600) = 6


On my VM, enrolling 76B6A6A0.crt would never do anything, because this cert was already in the kernel's trusted keyring. Was that not the case on Lubos' Raspi, and if yes, why?

ARGH, this is the same problem as bug 1201066, and I just didn't backport the fix to SLE/Leap so far. 

https://github.com/openSUSE/suse-module-tools/pull/66

I've submitted updates to SLE15-SP5 and SP4. SP6 had the fix already, TW anyway.

Be aware though that the "fix" just avoids an error exit status from mokutil.

If you really ran secure boot on a system with emulated UEFI variables like the Raspi, booting would fail unless you find some other way to import the certificate. See bug 1201066, comment 17.

SUSE-RU-2023:3986-1: An update that has two fixes can now be installed.

Category: recommended (important)
Bug References: 1201066, 1213428
Sources used:
openSUSE Leap 15.4 (src): suse-module-tools-15.4.17-150400.3.11.1
SUSE Linux Enterprise Micro for Rancher 5.3 (src): suse-module-tools-15.4.17-150400.3.11.1
SUSE Linux Enterprise Micro 5.3 (src): suse-module-tools-15.4.17-150400.3.11.1
SUSE Linux Enterprise Micro for Rancher 5.4 (src): suse-module-tools-15.4.17-150400.3.11.1
SUSE Linux Enterprise Micro 5.4 (src): suse-module-tools-15.4.17-150400.3.11.1
Basesystem Module 15-SP4 (src): suse-module-tools-15.4.17-150400.3.11.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

SUSE-RU-2023:3985-1: An update that has four fixes can now be installed.

Category: recommended (important)
Bug References: 1201066, 1212957, 1213428, 1213822
Sources used:
openSUSE Leap 15.5 (src): suse-module-tools-15.5.2-150500.3.3.1
Basesystem Module 15-SP5 (src): suse-module-tools-15.5.2-150500.3.3.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

Marking as resolved