Bug 1195118 - some kernel modules cannot be loaded if secure boot is enabled
some kernel modules cannot be loaded if secure boot is enabled
Status: IN_PROGRESS
: 1196563 (view as bug list)
Classification: openSUSE
Product: openSUSE Distribution
Classification: openSUSE
Component: Installation
Leap 15.3
Other Other
: P5 - None : Enhancement (vote)
: ---
Assigned To: YaST Team
Jiri Srain
https://trello.com/c/I8w56O5E
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2022-01-26 07:09 UTC by Giacomo Comes
Modified: 2022-07-05 15:19 UTC (History)
14 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---
ancor: needinfo? (mchang)
ancor: needinfo? (jlee)
meissner: needinfo? (meissner)


Attachments
yast2 logs (2.19 MB, application/x-compressed-tar)
2022-02-09 13:29 UTC, Takashi Iwai
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Giacomo Comes 2022-01-26 07:09:12 UTC
If secure boot is enabled some modules provided by the -kmp- packages cannot be loaded. Example:
  modprobe vhba
modprobe: ERROR: could not insert 'vhba': Key was rejected by service
  modprobe vboxdrv
modprobe: ERROR: could not insert 'vboxdrv': Key was rejected by service
  modprobe pcfclock
modprobe: ERROR: could not insert 'pcfclock': Key was rejected by service
These are modules available since the release of 15.3 and for which no updates exist.

Instead modules for which there exist updates can be loaded without error. Example:
  modprobe dlm
  modprobe ocfs2
  modprobe gfs2
Comment 1 Takashi Iwai 2022-01-26 07:29:03 UTC
As default, the kernel contains only its static built (SLE) cert and the ones passed from GRUB shim, and the error implies that the cert corresponding to your KMPs aren't included there.

There are different builds of KMPs, a few of them are built with SLE, hence they work as-is.  Others built from other OBS projects, including the official openSUSE:Leap:*, they won't work unless you install the corresponding cert on your system and enroll it via MOK manually.

There is a brief description in the release notes.  For Leap official KMPs, install openSUSE-signkey-cert package, if you didn't install it yet, reboot and enroll it at GRUB.  Note that the trigger happens only once after installing this package.  You can uninstall it, reboot, and re-install it, and reboot.
Comment 2 Giacomo Comes 2022-01-26 16:48:51 UTC
The package openSUSE-signkey-cert is already installed however
  mokutil --test-key /etc/uefi/certs/BDD31A9E-kmp.crt
was telling that the key was not enrolled
After running:
  mokutil --import /etc/uefi/certs/BDD31A9E-kmp.crt --root-pw
I rebooted and enrolled the key. Now the modules can be imported.

I assume that openSUSE-signkey-cert was installed during the OS installation,
but when the machine rebooted, no one was in front of the console to enroll the new key.

In my opinion this is a flaw of the openSUSE installation process.
Once the installation process completes, the machine reboots and if no one is in front of the console the enroll process is missed and/or boot error messages as well.
Fedora instead, after the installation process is completed, ask you to press a
reboot button before continuing. I this way there is a person in front of the console when the machine reboots and he can take appropriate action if required.
Comment 3 Takashi Iwai 2022-01-26 16:52:25 UTC
On openSUSE, after the installation, the installer also asks for reboot in a dialog, too, but with a certain timeout.  It has both merits and demerits.

In anyway, this is more about the installation and enhancements.  Reassigned.
Comment 4 Stefan Hundhammer 2022-01-31 10:19:25 UTC
I don't think we can or want to hard-code that kind of information in the installer; the possible permutations with modules, security modes and released products would explode in a very short time. There must be a better way.
Comment 5 Giacomo Comes 2022-02-09 07:00:48 UTC
I did the following test installation on an UEFI machine and a virtual machine with secure boot enabled. The results are identical:
I installed a KDE desktop. The package openSUSE-signkey-cert is installed automatically. After the installation completes and the machine reboots, there is no request to enroll the key present in openSUSE-signkey-cert and the command:
  mokutil --test-key /etc/uefi/certs/BDD31A9E-kmp.crt
returns:
  /etc/uefi/certs/BDD31A9E-kmp.crt is not enrolled
At this point you have to run:
  mokutil --import /etc/uefi/certs/BDD31A9E-kmp.crt --root-pw
or
  zypper in --force openSUSE-signkey-cert
then reboot and enroll the key.

The release notes (section 4.1) are somehow incorrect:
they assume that you have to manually install openSUSE-signkey-cert and after the reboot enroll the key. 
If I see that openSUSE-signkey-cert is already installed, I assume (incorrectly) that there is nothing more to do.
If nothing else changes, at least the release notes should be updated.
Comment 6 Takashi Iwai 2022-02-09 09:56:51 UTC
Hrm, then indeed there must be some flaw in the installation process or cert packaging script.  I tried by myself and found the same problem.

In the zypper log, I found the lines:
2022-02-09 10:10:54|install|perl-libwww-perl|6.31-1.17|noarch||openSUSE-Leap-15.3-2|625baf3b98a428ca87c6a00193c31bf9e9fb63dfd6e74766fff1068a61144aa1|
# 2022-02-09 10:10:55 openSUSE-signkey-cert-20210302-lp153.1.1.x86_64.rpm installed ok
# Additional rpm output:
# Failed to get root password hash
# Failed to import /etc/uefi/certs/BDD31A9E-kmp.crt

I guess that's the culprit?  The package %post invokes
   mokutil --import "$cert" --root-pw

and it seems accessing /etc/shadow and co.

(Reassigned back to category Installation)
Comment 8 Lukas Ocilka 2022-02-09 10:59:56 UTC
From my POV, this is a feature request but with quite some pieces of information missing. Such as: how is this expected to actually work? And moreover what about AutoYaST? There's nobody "in front of the computer".

Honestly, it's not very likely that the Installer will contain anything that would stop at the end of installation and ask the user to do something. Most of people want that to "just work" whatever it takes. Maybe the installation proposal could possibly contain something new, but that's in a huge contrast with request to make the installer simpler. Other similar cases, e.g., SSL certificates are solved just by installing the right package, nothing else.

Side note: Packages to install are not usually selected by the installer. They depend on the product to install, selected role, requirements coming from user-decisions, etc.

So, I believe this should be defined and driven by some Kernel PjM / PO. Reassigning to Kernel for that reason.
Comment 9 Takashi Iwai 2022-02-09 11:35:53 UTC
Lukas, the comment 8 looks like a wrong analysis.

This bug is in essentially two parts: the real bug that hinders the MOK cert deployment at installation, and the lack of intuitive usage.  The package openSUSE-signkey-cert *is* installed as default, so the patterns or whatever choice of package itself works.  The actual problem is that the end result is not what we expected due to the bug.

The first item, the bug in the installation step, is described in comment 6, and it has to be evaluated by your team, I suppose.  This isn't directly related with the kernel itself, but rather how mokutil gets executed at which state.  The error indicates some insufficient installation at the mokutil execution time.

The rest is rather about the documentation, or some instruction that could be shown in installer or something else.  It's an enhancement, and you can take it as a sort of feature request in future products.  Again, the kernel plays little role in this regard.

So, please take a look at the first problem (comment 6).  Once when this gets fixed, we can consider the rest (either more documentation or UI enhancement if really appropriate -- maybe it's better to continue in another bug, though).
Comment 10 Lukas Ocilka 2022-02-09 12:57:59 UTC
I'm a bit confused by comment #2, but OK, let's take another view.

The message in comment #6 seems to come from mokutil, somewhere like here:
https://github.com/lcp/mokutil/blob/master/src/mokutil.c#L460-L463

YaST / Installer does not do anything about mokutil besides installing it in some cases, see https://github.com/yast/yast-bootloader/blob/d7e49bd5f8c5d2bbe572c1e200167e9e1b7754f2/SUPPORTED_SCENARIOS.md#grub2-efi

Code is here https://github.com/yast/yast-bootloader/blob/6b8b1680fa340aa5883ed3f1db813ca83b4c7126/src/lib/bootloader/grub2efi.rb#L83-L104

So what comes to my mind that maybe root's hash is not yet written into /etc/shadow when the post-install script is executed. I'm not an expert for mokutil though so we might need some help here.

Additionally, if you believe that this is an installer issue, please, someone attach YaST logs from the installation attempt. Ideally including those zypp logs (/var/log/zypp)
https://en.opensuse.org/openSUSE:Report_a_YaST_bug
Comment 11 Takashi Iwai 2022-02-09 13:29:32 UTC
I attach logs I gathered from a VM I installed for testing shortly ago.

And, yes, that's the place where mokutil tries to retrieve the root password.
Comment 12 Takashi Iwai 2022-02-09 13:29:57 UTC
Created attachment 856010 [details]
yast2 logs
Comment 13 Ancor Gonzalez Sosa 2022-02-09 16:34:25 UTC
(In reply to Lukas Ocilka from comment #10)
> 
> So what comes to my mind that maybe root's hash is not yet written into
> /etc/shadow when the post-install script is executed. I'm not an expert for
> mokutil though so we might need some help here.

I can confirm that in the moment in which RPM packages are installed into the /mnt chroot during the installation process. The content of /mnt/etc/shadow is something like this:

> root:*:19032::::::

So there is indeed no root hash
Comment 14 Ancor Gonzalez Sosa 2022-02-10 09:14:57 UTC
(In reply to Ancor Gonzalez Sosa from comment #13)
> 
> I can confirm that in the moment in which RPM packages are installed into
> the /mnt chroot during the installation process. The content of
> /mnt/etc/shadow is something like this:
> 
> > root:*:19032::::::
> 
> So there is indeed no root hash

Can somebody confirm this is indeed the culprit of the whole problem? Or it's maybe just a red herring?

The installation process has always been like that - the root password of the target system (and, thus, the root hash) is written after installing the packages. If that's a problem we may reconsider adjusting the post-install script of that RPM if possible. Altering the order of the installation steps sounds like a more aggressive change, a bit too much just to please a concrete package.
Comment 15 Ancor Gonzalez Sosa 2022-02-10 09:21:07 UTC
(In reply to Ancor Gonzalez Sosa from comment #14)
> 
> Can somebody confirm this is indeed the culprit of the whole problem?

Adding needinfo on the maintainer of the mokutil package with the hope he can answer this.
Comment 16 Lukas Ocilka 2022-02-10 10:15:32 UTC
BTW, we were changing YaST Users (incl. root password handling during installation) in 15.4, but this is 15.3.
Comment 18 Takashi Iwai 2022-02-10 12:17:00 UTC
(In reply to Ancor Gonzalez Sosa from comment #14)
> (In reply to Ancor Gonzalez Sosa from comment #13)
> > 
> > I can confirm that in the moment in which RPM packages are installed into
> > the /mnt chroot during the installation process. The content of
> > /mnt/etc/shadow is something like this:
> > 
> > > root:*:19032::::::
> > 
> > So there is indeed no root hash
> 
> Can somebody confirm this is indeed the culprit of the whole problem? Or
> it's maybe just a red herring?

This is the very likely cause.  I tested on VM and mokutil execution with --root-pw option failed with the same error message when I edited /etc/shadow like the above.

> The installation process has always been like that - the root password of
> the target system (and, thus, the root hash) is written after installing the
> packages. If that's a problem we may reconsider adjusting the post-install
> script of that RPM if possible. Altering the order of the installation steps
> sounds like a more aggressive change, a bit too much just to please a
> concrete package.

Currently it's a simple %post script triggering from openSUSE-signkey-cert package.  If we need an extra procedure after the root passwd setup in the installation step, it has to be executed manually?

The %post consists of:

if command -v mokutil >/dev/null; then
        # enroll all *-kmp.crt certificates
        # the mokutil import command will auto-skip enrolled certificates
        certs=$(ls %{_sysconfdir}/uefi/certs/*-kmp.crt 2>/dev/null || true)
        for cert in $certs
        do
                if ! mokutil --import "$cert" --root-pw; then
                        echo "Failed to import $cert"
                fi
        done
fi
Comment 19 Takashi Iwai 2022-02-10 12:17:46 UTC
(In reply to Lukas Ocilka from comment #16)
> BTW, we were changing YaST Users (incl. root password handling during
> installation) in 15.4, but this is 15.3.

I didn't check Leap 15.4, but unless the root passwd is set up before package installation, the same problem must remain.
Comment 20 Joey Lee 2022-02-10 14:57:40 UTC
(In reply to Takashi Iwai from comment #19)
> (In reply to Lukas Ocilka from comment #16)
> > BTW, we were changing YaST Users (incl. root password handling during
> > installation) in 15.4, but this is 15.3.
> 
> I didn't check Leap 15.4, but unless the root passwd is set up before
> package installation, the same problem must remain.

Thanks for everyone's help!

I have tested openSUSE-signkey-cert package manually but didn't check it with ISO installation. I will think another way for enrolling key after the root password be added.
Comment 21 Lukas Ocilka 2022-02-10 15:50:03 UTC
(In reply to Takashi Iwai from comment #18)
> Currently it's a simple %post script triggering from openSUSE-signkey-cert
> package.  If we need an extra procedure after the root passwd setup in the
> installation step, it has to be executed manually?

Has this actually ever worked during installation? Has the feature been tested only on a running system where root obviously already has their hash in shadow? I'm trying to track it down by finding a working state and this one (15.3) to find the difference.

The reason why I mentioned 15.4 is that we were changing some internal logic recently, but that would not change already release products.

As far as I can see, this is actually a new thing in 15.3: https://build.opensuse.org/package/view_file/openSUSE:Leap:15.3/openSUSE-signkey-cert/openSUSE-signkey-cert.changes?expand=1
Comment 22 Takashi Iwai 2022-02-10 16:02:48 UTC
(In reply to Lukas Ocilka from comment #21)
> (In reply to Takashi Iwai from comment #18)
> > Currently it's a simple %post script triggering from openSUSE-signkey-cert
> > package.  If we need an extra procedure after the root passwd setup in the
> > installation step, it has to be executed manually?
> 
> Has this actually ever worked during installation?

I guess not, it seems that we haven't tested it full scenarios :-<

> Has the feature been
> tested only on a running system where root obviously already has their hash
> in shadow? I'm trying to track it down by finding a working state and this
> one (15.3) to find the difference.

This must be broken from the beginning.  Basically the feature is present only since Leap 15.3, and it's confirmed to broken, so...

> The reason why I mentioned 15.4 is that we were changing some internal logic
> recently, but that would not change already release products.

Would 15.4 set up the root passwd before package installation?  If so, the fix would be needed only for Leap 15.3 (for next possible quarterly update).

> As far as I can see, this is actually a new thing in 15.3:
> https://build.opensuse.org/package/view_file/openSUSE:Leap:15.3/openSUSE-
> signkey-cert/openSUSE-signkey-cert.changes?expand=1

Right.
Comment 23 Lukas Ocilka 2022-02-10 16:26:57 UTC
Root is created / password set after packages are installed and after post-install scripts are called (IIRC, done by libzypp in one transaction). This is because the filesystem needs to be ready and progs and libraries available (e.g., useradd to work correctly).

I agree that root might deserve different handling as it might be needed for special cases like this one. Except that this is the first time we actually see it.

Looking at https://github.com/lcp/mokutil/blob/master/src/mokutil.c#L454-L475 we might actually put the root-hash into a special file, correct? I'm not sure if that helps, but it might be one of the possible solutions - to write the hash temporarily somewhere. Well, which sounds like an ugly hack, but ...

We'll need to discuss this with my team.
Comment 24 Takashi Iwai 2022-02-10 16:43:35 UTC
(In reply to Lukas Ocilka from comment #23)
> Root is created / password set after packages are installed and after
> post-install scripts are called (IIRC, done by libzypp in one transaction).
> This is because the filesystem needs to be ready and progs and libraries
> available (e.g., useradd to work correctly).

Would the root passwd appearing if it's handled in %posttrans instead?
(Even if so, there might be ordering problem...)

> I agree that root might deserve different handling as it might be needed for
> special cases like this one. Except that this is the first time we actually
> see it.
> 
> Looking at
> https://github.com/lcp/mokutil/blob/master/src/mokutil.c#L454-L475 we might
> actually put the root-hash into a special file, correct? I'm not sure if
> that helps, but it might be one of the possible solutions - to write the
> hash temporarily somewhere. Well, which sounds like an ugly hack, but ...

We invoke mokutil with --root-pw option (the second case, root_pw == true) so that the same root password is used for MOK deployment.  It's for allowing installing package without interaction.

In theory it should be possible to pass via some temporary file, but how can the package know such a file?  A fixed path might be not good from security POV, IMO.
Comment 25 Joey Lee 2022-02-11 06:15:15 UTC
(In reply to Takashi Iwai from comment #24)
[...snip]
> 
> In theory it should be possible to pass via some temporary file, but how can
> the package know such a file?  A fixed path might be not good from security
> POV, IMO.

hm... Yes, the fixed path is not good for security. 

I am thinking that has any approach in post-install script for waiting the root password shows up?
Comment 26 Lukas Ocilka 2022-02-11 08:38:40 UTC
I somehow fear that when a script waits for root has to appear, you'd simply put the process in a deadlock.

Just an idea: There is one very old feature that indicates whether YaST is the current handler of things: YAST_IS_RUNNING

See https://github.com/yast/yast.github.io/blob/92e38f2ab85d7ad33f762a3394e039610db1d644/doc/environment-variables.md#variables-set-by-yast

This can (and used to be) be used in RPM scripts. It's not a complete solution, just a hint.

Then maybe the post-* script could create some kind of hook. Installer provides several possibilities, but as this is already in chroot, it's a bit more complicated (I need someone else's brain ;) TBDiscussed).

I was also thinking about using systemd-firstboot https://www.freedesktop.org/software/systemd/man/systemd-firstboot.html but that is limited to predefined set of actions.

Do you have any guidance when exactly the mokutil needs to be called, i.e., when is the latest time a certificate needs to be imported? The earliest is simple: when root password hash is available and when those packages are installed.
Comment 27 Takashi Iwai 2022-02-11 08:59:01 UTC
I'm afraid that firstboot won't work either, because the MOK deployment is required before rebooting.  It's a stuff for GRUB/shim, after all.

Is there any hook the installer itself runs after packaging installation and root passwd setup?
Comment 28 Ancor Gonzalez Sosa 2022-02-11 09:18:55 UTC
(In reply to Takashi Iwai from comment #27)
> I'm afraid that firstboot won't work either, because the MOK deployment is
> required before rebooting.  It's a stuff for GRUB/shim, after all.
> 
> Is there any hook the installer itself runs after packaging installation and
> root passwd setup?

We could hook after users_finish?

Info about installer hooks
https://github.com/yast/yast-yast2/blob/master/library/general/doc/Hooks.md
Comment 29 Ancor Gonzalez Sosa 2022-02-11 09:28:53 UTC
And there is also the possibility of embedding the hook itself in the control file:

https://github.com/yast/yast-installation/blob/b090564195597f6db4688e0d45a93be9bc2bb6f6/doc/control-file.md#hooks
Comment 30 Lukas Ocilka 2022-02-11 10:06:30 UTC
I'm afraid that at the end we might need to add something to YaST Bootlolader, e.g., somewhere into https://github.com/yast/yast-bootloader/blob/master/src/lib/bootloader/grub_install.rb
Comment 31 Ancor Gonzalez Sosa 2022-02-11 11:11:11 UTC
(In reply to Takashi Iwai from comment #27)
> I'm afraid that firstboot won't work either, because the MOK deployment is
> required before rebooting.  It's a stuff for GRUB/shim, after all.

Which makes me think we may consider to fix it in the Grub2 side. Like making sure Grub always imports the missing keys during grub_install or any other of its steps/scripts.
 
> Is there any hook the installer itself runs after packaging installation and
> root passwd setup?

I keep working on finding such a hook that could be used by the openSUSE-signjey-cert package (no clear candidate so far). I will update the bug at the end of today with my findings. But meanwhile I would like to add Michal Chang to NEEDINFO for him to check how feasible would be to implement a fix in the Grub2 side. Grub2 is installed from the chrooted (target) system at the very end of the installation process when the root password and everything else is already available.
Comment 32 Ancor Gonzalez Sosa 2022-02-11 13:33:45 UTC
(In reply to Takashi Iwai from comment #27)
> 
> Is there any hook the installer itself runs after packaging installation and
> root passwd setup?

Short answer: no.

Longer answer:

In fact there are different kinds of hooks in which the installer could run "third party" code after setting the root password and before configuring bootloader. But none of them could be used as-is by the openSUSE-signkey-cert package to run its custom code.

- Driver Update Disks can provide scripts that would be run inside the chroot (target) system. But that mechanism is only activated if a DUD is indeed detected, cannot be triggered from a regular package in a regular repository.

- System roles can provide their own scripts that would be executed also inside the chroot. But they are run only if the given role was selected, so they are not the solution for openSUSE-signkey-cert either.

- YaST Hooks (see links in a previous comment) can also be used to run scripts at any point of the installation. But the whole mechanism takes place in the installation system, not in the target one (chroot). That means openSUSE-signkey-cert simply cannot place its scripts in the directory used by the hooks mechanisms (since they are outside the chroot).
Comment 33 Ancor Gonzalez Sosa 2022-02-11 13:46:13 UTC
As far as I see it (I may be overlooking alternatives), we have now the following options:

A) Somehow make mokutils work without needing that root hash.

B) Modify grub2 to incorporate an extra step during its installation for importing the keys in case they are not all imported.

C) Similar to (B) but in yast2-bootloader instead of in Grub2.

D) Implement a new generic mechanism similar to the YaST Hooks but that runs inside the chroot. That would allow the packages being installed to take advantage of it. Unfortunately the current hooks mechanism is not exactly flexible (for example it reads the hooks from only one directory, you cannot have a list of alternative directories to fallback). So implementing that mechanism in a secure and robust way may not be as easy as adding a couple of lines of code to the existing mechanism. :-(
Comment 34 Giacomo Comes 2022-02-11 19:01:11 UTC
I would like to add few comments about the installation hooks.
The hooks are described in the link posted in comment 28. They are executed in the target system (chroot) but currently they are broken (if I'm not wrong, see boo#1195850).
The link in comment 29 about embedding the hook itself in the control file refers to the DebugHooks library. Such hook is executed too in the target system (chroot), but the DebugHooks library has been removed since Leap 15.1.
See boo#1132137 for more details.
Comment 35 Michael Chang 2022-02-14 05:59:22 UTC
(In reply to Ancor Gonzalez Sosa from comment #33)

[snip]

> B) Modify grub2 to incorporate an extra step during its installation for
> importing the keys in case they are not all imported.

Does it have to always happen or an one-off thing during the last stage of installation process ? If it is on-off then I think yast hook is better.

> 
> C) Similar to (B) but in yast2-bootloader instead of in Grub2.

Maybe in shim-install ? As it is taking care MokManager installation.
Comment 36 Ancor Gonzalez Sosa 2022-02-14 09:25:43 UTC
(In reply to Michael Chang from comment #35)
> (In reply to Ancor Gonzalez Sosa from comment #33)
> > 
> > C) Similar to (B) but in yast2-bootloader instead of in Grub2.
> 
> Maybe in shim-install ? As it is taking care MokManager installation.

It sounds to me like that could be indeed the right place.

Since we don't have Gary (shim-install maintainer) around anymore, I'm putting the needinfo on Joey Lee to check if it would be possible to modify shim-install so it makes sure the keys are imported as part of its process.
Comment 37 Lukas Ocilka 2022-02-14 11:26:02 UTC
(In reply to Michael Chang from comment #35)
> Does it have to always happen or an one-off thing during the last stage of
> installation process ? If it is on-off then I think yast hook is better.

Currently all is in post-install script of openSUSE-signkey-cert. YaST/Installer does not need to be involved at all. It works on the running system, but not during installation. Ideally Installer (YaST Bootloader) and the post-install script use the same method, e.g, calling a (some) script or shim.
Comment 38 Stefan Hundhammer 2022-02-14 13:43:41 UTC
Please correct me if I am wrong, but how about this general approach:

- Don't rely on that post-install mokutil call being successful.

- Use mokutil's introspection features when the certificates are actually needed; i.e. during the dracut run.

- If the certs are not already imported at that time, do it then. At that point in time, there are better chances of the crucial infrastructure (/etc/shadow) being in place.


I guess right now it's not only during a YaST installation that this doesn't work; it's probably the same for a kiwi or an image installation, or when doing zypper calls manually.

Moving that part to the dracut run should fix all those scenarios.
Comment 39 Stefan Hundhammer 2022-02-14 13:45:40 UTC
man mokutil:


MOKUTIL(1)               General Commands Manual               MOKUTIL(1)

NAME
       mokutil - utility to manipulate machine owner keys

SYNOPSIS
       mokutil [--list-enrolled | -l]
               ([--mokx | -X])
       mokutil [--list-new | -N]
               ([--mokx | -X])
       ...
       mokutil [--import keylist| -i keylist]
               ([--hash-file hashfile | -f hashfile] | [--root-pw | -P] |
                [--mokx | -X] | [--ca-check] | [--ignore-keyring])

...
...
OPTIONS
       -l, --list-enrolled
              List the keys the already stored in the database

       -N, --list-new
              List the keys to be enrolled

       ...
       ...

       -i, --import
              Collect the following files and form an  enrolling  request
              to shim. The files must be in DER format.
Comment 40 Lukas Ocilka 2022-02-14 14:00:23 UTC
(In reply to Lukas Ocilka from comment #8)
> So, I believe this should be defined and driven by some Kernel PjM / PO.
> Reassigning to Kernel for that reason.

And now I'm back to my comment #8. This is exactly why I reassigned that to Kernel and asked for having some PjM to drive it. We don't know the supported use-cases and we can't really decide which solution to choose to fulfill them. Moreover, I can only decide about YaST / Installer, but if the right place is actually elsewhere then what?
Comment 41 Takashi Iwai 2022-02-14 15:35:26 UTC
Note that dracut itself merely creates an initrd from the given kernel and modules.  Deploying an initrd is done rather by other tool like scripts in suse-module-tools (or kernel post/posttrans).

So, if any, this could be in suse-module-tools, but it really depends on how it's called from YaST side at which timing...
Comment 42 Lukas Ocilka 2022-02-15 07:37:31 UTC
Please, can anyone point me to the original feature request for the current implementation? It seems we are somehow stuck now.
Comment 43 Joey Lee 2022-02-15 12:40:41 UTC
(In reply to Lukas Ocilka from comment #42)
> Please, can anyone point me to the original feature request for the current
> implementation? It seems we are somehow stuck now.

You can reference bsc#1182641.

The original feature request is after the SLE-Leap closing gap, some KMPs be signed by openSUSE signkey because they are in Leap repo but not in SLE. Those kernel modules can not be loaded by SLE-Leap kernel (closing gap) when secure boot is enabled.

So, we need a package to help user to enroll openSUSE signkey to MOK when they want to use those Leap-only KMPs. But user doesn't know they need to install the signkey RPM. So I ask Max Lin's help to add it in installation pattern.
Comment 44 Giacomo Comes 2022-02-15 13:08:42 UTC
(In reply to Joey Lee from comment #43)
> (In reply to Lukas Ocilka from comment #42)
> > Please, can anyone point me to the original feature request for the current
> > implementation? It seems we are somehow stuck now.
> 
> You can reference bsc#1182641.

I see something important in bsc#1182641 that's not there in the current implementation:

| On the other hand, in the latest step of YaST installation. YaST should guide
| user for what will user see after system first reboot. It should tell user that 
| a mok-manager UI will show up after reboot, and user can follow the mok-manager 
| UI to enroll openSUSE key to MOK.

Right now, the user does not know that a enroll step is necessary after the
first reboot. He can only guess it (if by chance he is in front of the monitor
when the machine reboots).
Comment 45 Lukas Ocilka 2022-02-15 13:26:35 UTC
(In reply to Giacomo Comes from comment #44)
> Right now, the user does not know that a enroll step is necessary after the
> first reboot. He can only guess it (if by chance he is in front of the
> monitor
> when the machine reboots).

Let's call this a wish that did not come true. It wasn't a feature request wanted and blessed by a project manager (Lubos in this case). Frankly, not everything that is requested gets implemented or at least not the way how it's been requested.

In fact, if the machine was installed from an Image or Yomi it would not show such message either. We usually try to implement Installer features in a generic way. We've learned that hard-coding messages, package names, etc., works for a short period of time and then it breaks and often goes unnoticed till it causes a (severe) bug and the Installer is to blame.

That's why we request features to be filed, discussed and the implementation decided based on that process and its output. QA and Docu needs to be involved too. Which is often not the case in case of a bugreport (enhancement).
Comment 46 Takashi Iwai 2022-02-15 14:00:43 UTC
Yes, admittedly, the enhancement part should be handled better via JIRA.

Meanwhile, we can concentrate on the real bug in this bugzilla entry, about the missing MOK cert enrollment after the fresh installation.
Comment 47 Stefan Hundhammer 2022-02-15 14:40:19 UTC
The very idea to do that in some post-install script is broken by design. That's the root of the problem. It makes assumptions, and those assumptions are sometimes simply not true.
Comment 48 Takashi Iwai 2022-02-16 11:32:57 UTC
(In reply to Stefan Hundhammer from comment #47)
> The very idea to do that in some post-install script is broken by design.
> That's the root of the problem. It makes assumptions, and those assumptions
> are sometimes simply not true.

OTOH, a cert package installation doesn't happen only at a fresh installation or the system upgrade.  It can be installed at any time on the running system additionally.  And in that case, we don't want / need to touch the kernel, initrd nor GRUB setup itself.  It's merely a call of mokutil.  So, for that purpose, the post or posttrans script is the best place to execute.

That said, handling it in the post script itself isn't wrong.  The problem is that we rely only on that mechanism.
Comment 49 Lukas Ocilka 2022-02-16 12:01:20 UTC
Let me rephrase my proposal in comment #37 then.

There's currently just a post-install script that should do the job. To make it working during installation, we could:

- Move the post-install script content to a script file owned by that package
- Post-install would call this script
- ... and fail during installation or even skip in case 
  YAST_IS_RUNNING=="instsys"
- YaST Bootloader would call the same script at the end of installation and 
  succeed

IMO does not sound like a hack, no big changes to the code.

If then something more is actually needed during the first boot, this would be out of scope for the Installer for now.
Comment 50 Takashi Iwai 2022-02-16 13:19:15 UTC
(In reply to Lukas Ocilka from comment #49)
> Let me rephrase my proposal in comment #37 then.
> 
> There's currently just a post-install script that should do the job. To make
> it working during installation, we could:
> 
> - Move the post-install script content to a script file owned by that package
> - Post-install would call this script
> - ... and fail during installation or even skip in case 
>   YAST_IS_RUNNING=="instsys"
> - YaST Bootloader would call the same script at the end of installation and 
>   succeed
> 
> IMO does not sound like a hack, no big changes to the code.
> 
> If then something more is actually needed during the first boot, this would
> be out of scope for the Installer for now.

I guess this will work.  The remaining points are:
- Which package does the script belong to?  suse-module-tools, shim or anything else?
- A cert package installation is optional, so yast bootloader should handle it conditionally
- Similar fixes are needed for other cert packages
Comment 51 Michael Chang 2022-02-16 13:42:18 UTC
(In reply to Takashi Iwai from comment #50)
> (In reply to Lukas Ocilka from comment #49)

[snip]

> I guess this will work.  The remaining points are:
> - Which package does the script belong to?  suse-module-tools, shim or
> anything else?

Why not cert package itself (openSUSE-signkey-cert) ?

I'd suggest to avoid shim package here, as it at times may have to wait shim binary update from Microsoft.

> - A cert package installation is optional, so yast bootloader should handle
> it conditionally

> - Similar fixes are needed for other cert packages

These all sound good reason to use cert package to provide the scripts. :)
Comment 52 Lukas Ocilka 2022-02-16 15:06:13 UTC
(In reply to Michael Chang from comment #51)
> Why not cert package itself (openSUSE-signkey-cert) ?

Comment #49: Move the post-install script content to a script file owned by that package (== openSUSE-signkey-cert)
Comment 53 Lukas Ocilka 2022-02-16 15:38:44 UTC
Plus I believe that we should get the Security Team involved (Adding them to CC now).
Comment 54 Lukas Ocilka 2022-03-01 08:18:21 UTC
See also bug #1196563
Comment 55 Lukas Ocilka 2022-03-01 08:20:45 UTC
... which points to https://bugzilla.suse.com/show_bug.cgi?id=1186784#c14
Comment 56 Josef Reidinger 2022-03-01 20:19:54 UTC
*** Bug 1196563 has been marked as a duplicate of this bug. ***
Comment 57 John 2022-07-05 14:21:18 UTC
This is still an issue, after doing an upgrade it seems the openSUSE Secure Boot cert has changed. On reboot after the upgrade you get an enrollment prompt and it requests you to provide a password, which you can only guess. After failing several times I decided to boot without enrolling it and run mokutil -i /etc/uefi/certs/1F673297-kmp.crt set a password and reboot again to enroll it, this time knowing which password I have to provide.

I don't understand why end users are required to do this instead of either having only one cert and everything signed with the same key or having both certs installed by default. This is not user friendly at all, it should "just work" without having to search on the internet why virtualbox fails to load a VM, possible causes, find out forum posts or bugs like https://bugzilla.opensuse.org/show_bug.cgi?id=1186784 then learn how to make enrollment requests on command line etc etc. It's really a pain and doesn't seem to be good practice..

This time I knew what to do, but many other users will probably have to spend a lot time to figure out why it doesn't work and what they have to manually do after a fresh install or upgrade... :(
Comment 58 Marcus Meissner 2022-07-05 14:23:31 UTC
the MOK dialog password is your systems root password.
Comment 59 John 2022-07-05 14:38:50 UTC
(In reply to Marcus Meissner from comment #58)
> the MOK dialog password is your systems root password.

Thanks. I tried the root password too and it didn't work. A possible reason are the special characters in my password, I have no idea if the unexpected blue enrollment screen demanding an unspecified password expects a US keyboard, I can't see what I type either since it only shows asterisks and there is no shell to type in and see if special characters are correct or what the equivalent key is. 

Also it isn't specified anywhere that it's the root password, are end users really expected to guess this? Is this really the desired design for openSUSE? I ask this after spending many hours months ago until I figured out why Virtualbox did no longer work after a fresh openSUSE 15.3 install, and after searching for the cause I see many other users had the same problem..
Comment 60 Marcus Meissner 2022-07-05 15:19:42 UTC
> Also it isn't specified anywhere that it's the root password, are end users
> really expected to guess this? Is this really the desired design for openSUSE?

It is a problem with the kind of strict  secure boot requirements and split-brain SUSE Linux Enterprise / openSUSE development in two entirely different environments.

But it is something to review and think about to improve.