I agree, this is a valid concern. The kernel command line should be included in the state that is used to seal the key to prevent the cmd line from being tampered.

Additionally users with higher security needs should have a way to prevent the booting of older kernels/snapshots or be advised to remove them/reduce the count.

Apparently modified commandline fails to unlock the key. Is this possible to lock the volume if older snapshot or kernel is being selected?

It's expected behavior. The kernel and initrd are not covered by design because those files are in the encrypted and trusted partition. Besides, TPM key sealing with authorized policy won't stop downgrading attack even if you extend the coverage to kernel and initrd, because the attacker may perform the downgrading attack with an older and valid sealed key.

Actually, I'd suggest the laptop users with security concerns don't enable unattended boot because the main purpose is to make the stolen disk unusable in other machine, but it's very easy to take the whole laptop away...

> The kernel and initrd are not covered by design because those files are in the encrypted and trusted partition.

I get it they're not measured.

Let me explain how I see the perfect trusted boot chain (and, much thanks, openSUSE almost provides that to our systems):

1. BootGuard or similar safe firmware start, trustworthy firmware populates TPM2 PCRs with it configuration (mostly NVRAM contents)

2. GRUB2 is measured, extending PCRs. It boots as a trusted environment given power to protect following system integrity.

3. GRUB2 measures unencrypted parts of its configs (stored in the ESP). After that it passes sealed.tpm file data to the TPM. TPM matches PCRs with PCR policy and only if it's assured that GRUB is good and has good config, continuing boot into good state, TPM will give LUKS key to the GRUB.

From now we rely on GRUB and its config to protect system integrity as the key is now unlocked (and it couldn't be safely unlocked later. This is a good and rarely found decision to encrypt kernel and initrd, thanks for that). Thus it should only allow latest kernel with cmdline and initrd approved by the user (the default config as stored on the encrypted boot fs is believed to be set by user). If the user decides to modify cmdline (this is already protected in the current state), go into recovery, use older kernels or snapshots, it's GRUB's responsibility to neutralize the secret after loading kernel and initrd into memory from the encrypted boot and request password in initrd. GRUB config scripts should ensure next participants are to be trusted to handle the key.

Another variant would be locking and unlocking the volume in GRUB using its own LUKS infrastructure, but using a password, not TPM-backed secret.

4. System being booted is trusted to keep user's data secure. We are sure in it as it's recent and has booted from the encrypted partition thus shouldn't have been tampered with. We believe it won't leak the data and cannot be broken into without proper auth in runtime.

5. We get remotely-accessible safe multi-user system without any shared secret, just as we want.

I don't think there're any drawbacks in the solution I proposed, however, I'm not sure I thought about all use cases. If there're any issues in my proposal, please let me know.

If it's still to be considered an expected behavior, there should be an explicit note in the docs that it's not resistant to situations when both mainboard and drive are in the unsafe environment (or needs extra hardening to handle such) and only protects from theft of drive alone. Current state of the docs doesn't suggest any security concerns except this being an experimental feature.

Thank for your answers!

(In reply to Dmitry Sharshakov from comment #4)
> > The kernel and initrd are not covered by design because those files are in the encrypted and trusted partition.
> 
> I get it they're not measured.
> 
> Let me explain how I see the perfect trusted boot chain (and, much thanks,
> openSUSE almost provides that to our systems):
> 
> 1. BootGuard or similar safe firmware start, trustworthy firmware populates
> TPM2 PCRs with it configuration (mostly NVRAM contents)
> 
> 2. GRUB2 is measured, extending PCRs. It boots as a trusted environment
> given power to protect following system integrity.
> 

> 3. GRUB2 measures unencrypted parts of its configs (stored in the ESP).
> After that it passes sealed.tpm file data to the TPM. TPM matches PCRs with
> PCR policy and only if it's assured that GRUB is good and has good config,
> continuing boot into good state, TPM will give LUKS key to the GRUB.
> 
> From now we rely on GRUB and its config to protect system integrity as the
> key is now unlocked (and it couldn't be safely unlocked later. This is a
> good and rarely found decision to encrypt kernel and initrd, thanks for
> that). Thus it should only allow latest kernel with cmdline and initrd
> approved by the user (the default config as stored on the encrypted boot fs
> is believed to be set by user). If the user decides to modify cmdline (this
> is already protected in the current state), go into recovery, use older
> kernels or snapshots, it's GRUB's responsibility to neutralize the secret
> after loading kernel and initrd into memory from the encrypted boot and
> request password in initrd. GRUB config scripts should ensure next
> participants are to be trusted to handle the key.
> 
> Another variant would be locking and unlocking the volume in GRUB using its
> own LUKS infrastructure, but using a password, not TPM-backed secret.
> 

If you want to harden the menu entries other than the default one, it's possible to lock down the specific entries with 'password_pbkdf2' or 'password' in /boot/grub2/grub.cfg.

https://www.gnu.org/software/grub/manual/grub/grub.html#Security

You can try to customize the config file by modifying /etc/grub.d/40_custom.

> 4. System being booted is trusted to keep user's data secure. We are sure in
> it as it's recent and has booted from the encrypted partition thus shouldn't
> have been tampered with. We believe it won't leak the data and cannot be
> broken into without proper auth in runtime.
> 
> 5. We get remotely-accessible safe multi-user system without any shared
> secret, just as we want.
> 
> I don't think there're any drawbacks in the solution I proposed, however,
> I'm not sure I thought about all use cases. If there're any issues in my
> proposal, please let me know.
> 
> If it's still to be considered an expected behavior, there should be an
> explicit note in the docs that it's not resistant to situations when both
> mainboard and drive are in the unsafe environment (or needs extra hardening
> to handle such) and only protects from theft of drive alone. Current state
> of the docs doesn't suggest any security concerns except this being an
> experimental feature.
> 
> Thank for your answers!

> If you want to harden the menu entries other than the default one, it's possible to lock down the specific entries with 'password_pbkdf2' or 'password' in /boot/grub2/grub.cfg.

Thank you, will try.

(In reply to Gary Ching-Pang Lin from comment #5)
> (In reply to Dmitry Sharshakov from comment #4)
> > > The kernel and initrd are not covered by design because those files are in the encrypted and trusted partition.
> > 
> > I get it they're not measured.
> > 
> > Let me explain how I see the perfect trusted boot chain (and, much thanks,
> > openSUSE almost provides that to our systems):
> > 
> > 1. BootGuard or similar safe firmware start, trustworthy firmware populates
> > TPM2 PCRs with it configuration (mostly NVRAM contents)
> > 
> > 2. GRUB2 is measured, extending PCRs. It boots as a trusted environment
> > given power to protect following system integrity.
> > 
> 
> > 3. GRUB2 measures unencrypted parts of its configs (stored in the ESP).
> > After that it passes sealed.tpm file data to the TPM. TPM matches PCRs with
> > PCR policy and only if it's assured that GRUB is good and has good config,
> > continuing boot into good state, TPM will give LUKS key to the GRUB.
> > 
> > From now we rely on GRUB and its config to protect system integrity as the
> > key is now unlocked (and it couldn't be safely unlocked later. This is a
> > good and rarely found decision to encrypt kernel and initrd, thanks for
> > that). Thus it should only allow latest kernel with cmdline and initrd
> > approved by the user (the default config as stored on the encrypted boot fs
> > is believed to be set by user). If the user decides to modify cmdline (this
> > is already protected in the current state), go into recovery, use older
> > kernels or snapshots, it's GRUB's responsibility to neutralize the secret
> > after loading kernel and initrd into memory from the encrypted boot and
> > request password in initrd. GRUB config scripts should ensure next
> > participants are to be trusted to handle the key.
> > 
> > Another variant would be locking and unlocking the volume in GRUB using its
> > own LUKS infrastructure, but using a password, not TPM-backed secret.
> > 
> 
> If you want to harden the menu entries other than the default one, it's
> possible to lock down the specific entries with 'password_pbkdf2' or
> 'password' in /boot/grub2/grub.cfg.
> 
> https://www.gnu.org/software/grub/manual/grub/grub.html#Security
> 
> You can try to customize the config file by modifying /etc/grub.d/40_custom.

I'd like to propose a new default for the FDE unattended boot by locking down grub shell and menu_entry editor mode, and to regain access to them, users are mandatory to setup password protection  from YaST or manually.

After reading the discussion and I think Dmitry provided very good argument to do that. We have to be very careful to ensure the boot flow is carried out in a controlled manner, that is it should not trust anything else other than following the instructions written in grub.cfg from the target of (unlocked) encrypted root (ie trusted domain). The config execution related commands like 'source' and 'configfiles' are easily accessible from grub shell to detour it to other target's configs. Besides, closing the command shell can greatly reduce potential attacking via abusing commands trying to use a flawed command to compromise system. 

The downside would be inconvenient user experience, but I thought it is understandable decision and still they can regain access by setting up password.

What do you think ?

> 
> > 4. System being booted is trusted to keep user's data secure. We are sure in
> > it as it's recent and has booted from the encrypted partition thus shouldn't
> > have been tampered with. We believe it won't leak the data and cannot be
> > broken into without proper auth in runtime.
> > 
> > 5. We get remotely-accessible safe multi-user system without any shared
> > secret, just as we want.
> > 
> > I don't think there're any drawbacks in the solution I proposed, however,
> > I'm not sure I thought about all use cases. If there're any issues in my
> > proposal, please let me know.

The old bootable snapshots may be set as automatic fallback if kernel updated in new snapshot fails to boot, in this case we may not want the password prompt gets in the way for remote managed server.

Thanks.

> > 
> > If it's still to be considered an expected behavior, there should be an
> > explicit note in the docs that it's not resistant to situations when both
> > mainboard and drive are in the unsafe environment (or needs extra hardening
> > to handle such) and only protects from theft of drive alone. Current state
> > of the docs doesn't suggest any security concerns except this being an
> > experimental feature.
> > 
> > Thank for your answers!

I would propose the following:

 - if we boot the default kernel without editing anything, we unseal the root partition the way we do today
 - in any other case, grub "forgets" the LUKS key and does not include it in the initrd

This does not require a separate password for grub, it just means that the user has to manually unlock the root partition when they fiddle around, by providing their recovery passphrase. Which, IMO, also makes sense in the context, because most of the time when you boot an older kernel, it's related to recovery.



One additional comment to Johannes' suggestion to measure the kernel command line: that is not possible. Unlocking the FS happens in the "early" grub.cfg file that resides in the EFI partition. The kernel entries are in the "late" grub.cfg file that resides inside the encrypted root partition.

So while we do seal the LUKS key against PCR9 (among others), at the point in time where we unseal it, the kernel command line has not been hashed into PCR9 yet.

The current design does not require re-signing or re-sealing when performing a kernel update, or when the user edits the late grub.cfg file. In this regard, it is fairly robust against accidental lock-outs, and I would be very careful about any changes that sacrifice this robustness.

(In reply to Michael Chang from comment #7)

> I'd like to propose a new default for the FDE unattended boot by locking down grub shell and menu_entry editor mode, and to regain access to them, users are mandatory to setup password protection  from YaST or manually.


Thanks Michael! You understood and explained the idea I wanted to communicate. Yes, this looks like another way of securing following boot flow.

> The downside would be inconvenient user experience, but I thought it is understandable decision and still they can regain access by setting up password

AIUI currently we can only setup TPM-backed encryption from the CLI. Let's just add a warning there that if user wants to retain those options they should proceed with the following instructions. When YaST would be able to setup this, boot menu password could be set up in the YaST flow.

> The old bootable snapshots may be set as automatic fallback if kernel updated in new snapshot fails to boot, in this case we may not want the password prompt gets in the way for remote managed server.

I know some setups feature a systemd service marking boot as successful. Fedora does such for an automated GRUB entry. This way auto-fallback can be locked after successful boot (locked in the config from the secure land of /boot on LUKS). If sysadmin notices misbehavior of the new snapshot and wants to rollback, they can unlock the rollback having access to the config and then reboot unattendedly.

(In reply to Olaf Kirch from comment #8)

> Unlocking the FS happens in the "early" grub.cfg file that resides in the EFI partition. The kernel entries are in the "late" grub.cfg file that resides inside the encrypted root partition.

And yes, we don't need to measure the kernel and initrd as it would require more pcr-oracle invocations and is not needed as the kernel comes from unlocked LUKS source which is trusted. We just need to make sure grub handles the boot safely.

Robustness is very important as this should be workable for servers and cloud VMs.

By the way, I wanted to know from you whether PCR sealing does write something into TPM's persistent memory? This isn't related, but I just wanted to understand that and didn't manage to perceive this from the code. Thanks for answers.

> - if we boot the default kernel without editing anything, we unseal the root partition the way we do today
> - in any other case, grub "forgets" the LUKS key and does not include it in the initrd

Actually this is even closer to what I meant than the previous comment. Forget the key in any case except the unmodified default from the trusted source. Else ask the password, we expect admin to be at the console anyway. If that's not admin, they probably shouldn't be able to do recovery actions. If many people need to do so, we can have multiple LUKS recovery passwords, unlike GRUB passwords which have to be separately accounted. Thanks for explaining my thoughts in a better way!

Also, is grub config part on the ESP measured into PCR registers? Can we be sure it's pristine and won't cause our encrypted boot scripts to be untrustworthy?

Also, will YaST support come into place for this feature? Probably it's difficult to enable this during installation because of the need to guess too many PCRs (firmware configuration data might be different during the USB boot), but should be good to enable from the running OS with a YaST GUI tool panel. Probably this is already planned, as I see this feature is experimental yet.

Even being a power user and admin I like openSUSE for YaST. It provides such a minimalist, universal and desktop-agnostic tool for basically everything. Good job!

(In reply to Dmitry Sharshakov from comment #10)
> (In reply to Olaf Kirch from comment #8)
> 
> > Unlocking the FS happens in the "early" grub.cfg file that resides in the EFI partition. The kernel entries are in the "late" grub.cfg file that resides inside the encrypted root partition.
> 
> And yes, we don't need to measure the kernel and initrd as it would require
> more pcr-oracle invocations and is not needed as the kernel comes from
> unlocked LUKS source which is trusted. We just need to make sure grub
> handles the boot safely.
> 
> Robustness is very important as this should be workable for servers and
> cloud VMs.
> 
> By the way, I wanted to know from you whether PCR sealing does write
> something into TPM's persistent memory? This isn't related, but I just
> wanted to understand that and didn't manage to perceive this from the code.
> Thanks for answers.
> 
We currently do not write anything into the persistent memory. The primary key is created on demand with a fixed template.

> > - if we boot the default kernel without editing anything, we unseal the root partition the way we do today
> > - in any other case, grub "forgets" the LUKS key and does not include it in the initrd
> 
> Actually this is even closer to what I meant than the previous comment.
> Forget the key in any case except the unmodified default from the trusted
> source. Else ask the password, we expect admin to be at the console anyway.
> If that's not admin, they probably shouldn't be able to do recovery actions.
> If many people need to do so, we can have multiple LUKS recovery passwords,
> unlike GRUB passwords which have to be separately accounted. Thanks for
> explaining my thoughts in a better way!
> 
> Also, is grub config part on the ESP measured into PCR registers? Can we be
> sure it's pristine and won't cause our encrypted boot scripts to be
> untrustworthy?

Yes, shim.efi, grub.efi, and grub.cfg in ESP are all measured.

(In reply to Dmitry Sharshakov from comment #11)
> Also, will YaST support come into place for this feature? Probably it's
> difficult to enable this during installation because of the need to guess
> too many PCRs (firmware configuration data might be different during the USB
> boot), but should be good to enable from the running OS with a YaST GUI tool
> panel. Probably this is already planned, as I see this feature is
> experimental yet.
> 
The integration with YaST installation is on the way. The PCR measurement will happen in the firstboot rather than installation because it's way too difficult to predict the PCR values during installation. For the firstboot, there will be an one-time password written in grub.cfg and the password will be dropped by fde-tpm-enroll.service later.

> Even being a power user and admin I like openSUSE for YaST. It provides such
> a minimalist, universal and desktop-agnostic tool for basically everything.
> Good job!

(In reply to Dmitry Sharshakov from comment #10)
> (In reply to Olaf Kirch from comment #8)
> 
> > Unlocking the FS happens in the "early" grub.cfg file that resides in the EFI partition. The kernel entries are in the "late" grub.cfg file that resides inside the encrypted root partition.
> 
> And yes, we don't need to measure the kernel and initrd as it would require
> more pcr-oracle invocations and is not needed as the kernel comes from
> unlocked LUKS source which is trusted. We just need to make sure grub
> handles the boot safely.
> 
> Robustness is very important as this should be workable for servers and
> cloud VMs.
> 
> By the way, I wanted to know from you whether PCR sealing does write
> something into TPM's persistent memory? This isn't related, but I just
> wanted to understand that and didn't manage to perceive this from the code.
> Thanks for answers.
> 
> > - if we boot the default kernel without editing anything, we unseal the root partition the way we do today
> > - in any other case, grub "forgets" the LUKS key and does not include it in the initrd
> 
> Actually this is even closer to what I meant than the previous comment.
> Forget the key in any case except the unmodified default from the trusted
> source. Else ask the password, we expect admin to be at the console anyway.

It sounds good to me as well. However to make the deal happen we have to agree on the definition/handling of  "default":

1. In the sense of downgrade attack prevention, default should always be highest version (In other words, the topmost entry).
2. However grub offers capability to set default entry to boot after timeout, the default can literately be any entry thus booting a downgraded kernel. This can be intentional and we can't really tell user's intention.

So what is our take ? If we go for #1, then unattended boot may not work with grub2-set-default. If we go for #2, unattended boot may always work despite not being the topmost.

My preference is #2 as we can leave the decision to user discretion. If they don't care, always topmost and safe one is the silent default. But I'm really sure if you think the situation differently and topmost must be enforced all the time.

Thanks.

> If that's not admin, they probably shouldn't be able to do recovery actions.
> If many people need to do so, we can have multiple LUKS recovery passwords,
> unlike GRUB passwords which have to be separately accounted. Thanks for
> explaining my thoughts in a better way!
> 
> Also, is grub config part on the ESP measured into PCR registers? Can we be
> sure it's pristine and won't cause our encrypted boot scripts to be
> untrustworthy?

> We currently do not write anything into the persistent memory. The primary key is created on demand with a fixed template.

Thanks. So sealed.tpm is like a wrapped container which contains cyphertext and TPM-signed PCR policy, we pass it to the TPM2 and it will unlock it if it trusts us? The only thing stored in the NV is TPMs initially generated global private key?

> Yes, shim.efi, grub.efi, and grub.cfg in ESP are all measured.

Amazing, a finally secure approach! No need to even have a custom secure boot policy enrolled for initrd checking.

> The integration with YaST installation is on the way. The PCR measurement will happen in the firstboot rather than installation because it's way too difficult to predict the PCR values during installation. For the firstboot, there will be an one-time password written in grub.cfg and the password will be dropped by fde-tpm-enroll.service later.

Also thought like this, making system auto-enroll the TPM key after first booting into the expected state.

> So what is our take ? If we go for #1, then unattended boot may not work with grub2-set-default. If we go for #2, unattended boot may always work despite not being the topmost.

> My preference is #2 as we can leave the decision to user discretion. If they don't care, always topmost and safe one is the silent default. But I'm really sure if you think the situation differently and topmost must be enforced all the time.

Well, unless root wants to change the default it's being the latest. Timeout boot should be allowed for the first time after update and then locked by a systemd service running on boot success. Some systems have such a service to detect boot failures and offer recovery options in such cases. So on systems with automatic rollback (is any available in Tumbleweed?) we can first boot with that timeout entry enabled and this saves us in case of a boot failure after update, but if our boot is fine and automatic recovery wasn't needed, a systemd unit will adjust /boot config files to restrict old snapshot to only booting with LUKS re-unlocking.

I assume we choose Olaf's approach and drop the key on booting options not believed to be currently secure, not using a GRUB password.

(In reply to Dmitry Sharshakov from comment #15)
> > We currently do not write anything into the persistent memory. The primary key is created on demand with a fixed template.
> 
> Thanks. So sealed.tpm is like a wrapped container which contains cyphertext
> and TPM-signed PCR policy, we pass it to the TPM2 and it will unlock it if
> it trusts us? 

Yes, grub2 sends the encrypted data to TPM and TPM returns the decrypted LUKS key if the PCR policy matches.

> The only thing stored in the NV is TPMs initially generated global private key?

TPM2 doesn't need to store any private key. As long as you provide the consistent template, TPM2 will always give your the same key pair derived from its internal seed. You still can save the key in the persistent memory in case you want to save the time for the key generation.

(In reply to Dmitry Sharshakov from comment #15)

[snip]

> 
> > So what is our take ? If we go for #1, then unattended boot may not work with grub2-set-default. If we go for #2, unattended boot may always work despite not being the topmost.
> 
> > My preference is #2 as we can leave the decision to user discretion. If they don't care, always topmost and safe one is the silent default. But I'm really sure if you think the situation differently and topmost must be enforced all the time.
> 
> Well, unless root wants to change the default it's being the latest. Timeout
> boot should be allowed for the first time after update and then locked by a
> systemd service running on boot success. Some systems have such a service to
> detect boot failures and offer recovery options in such cases. So on systems
> with automatic rollback (is any available in Tumbleweed?)

I'm not sure for Tumbleweed, but openSUSE MicroOS(Aeon) provides similar service called health-checker [1], and in it's grub integration setting 'default' variable is used.

[1] https://github.com/openSUSE/health-checker  
[2] https://github.com/openSUSE/health-checker/blob/master/grub/05_health_check#L45

> we can first boot
> with that timeout entry enabled and this saves us in case of a boot failure
> after update, but if our boot is fine and automatic recovery wasn't needed,
> a systemd unit will adjust /boot config files to restrict old snapshot to
> only booting with LUKS re-unlocking.

There's also usercase that YaST can be used to select boot entries as default entry or running grub2-set-default from console to set an entrty as permanent default unless it is changed again. In  addition there's also grub2-once which will set an temporary entry that boots only "once" and will restore to old default after. This is often used in situation like hibernation (s4) to make sure the menu entry containing the running kernel is used for resuming in case of the default is a different one.

I think we'll need take care of the best not breaking the feature we have shipped or in use.

> I assume we choose Olaf's approach and drop the key on booting options not
> believed to be currently secure, not using a GRUB password.

Yes. Password is optional like what it is now.
Thanks.

> There's also usercase that YaST can be used to select boot entries as default entry or running grub2-set-default from console to set an entrty as permanent default unless it is changed again. In  addition there's also grub2-once which will set an temporary entry that boots only "once" and will restore to old default after. This is often used in situation like hibernation (s4) to make sure the menu entry containing the running kernel is used for resuming in case of the default is a different one.

Well, all these should modify the encrypted boot directory, so it's okay. So the only parts needed are hardening GRUB runtime config (manual snapshot/kernel selection etc) and adding a service blocking rollback after a good boot, preventing automatic and manual rollback without the password entry after new snapshot has been proven booting

(In reply to Dmitry Sharshakov from comment #18)
> > There's also usercase that YaST can be used to select boot entries as default entry or running grub2-set-default from console to set an entrty as permanent default unless it is changed again. In  addition there's also grub2-once which will set an temporary entry that boots only "once" and will restore to old default after. This is often used in situation like hibernation (s4) to make sure the menu entry containing the running kernel is used for resuming in case of the default is a different one.
> 
> Well, all these should modify the encrypted boot directory, so it's okay. So
> the only parts needed are hardening GRUB runtime config (manual
> snapshot/kernel selection etc) and adding a service blocking rollback after
> a good boot, preventing automatic and manual rollback without the password
> entry after new snapshot has been proven booting

OK. So I think it is enough understanding to implement the proposed grub key handling. After that, it can be more clear how that works in some special case if the general hardening in grub is done.

By the way, next entire week will be SUSE's hackweek so it is expected to be less active during that time.
Thanks.

Hi Michael,

(In reply to Michael Chang from comment #14)
> It sounds good to me as well. However to make the deal happen we have to
> agree on the definition/handling of  "default":

For me, the definition of "default" would be "no interference from untrusted input". Our trigger to forget the LUKS key should be when untrusted input changes what we would boot.

For example, we could drop the LUKS key as soon as the user enters the boot menu, because this is the moment they can start to do potentially harmful stuff. Or we do it whenever we register any keyboard input (except for the ENTER key).

There may be other untrusted sources that control what we boot and how. I'm not sure about grub-set-default and grub-editenv, for instance. In the BIOS days, this used to be stored in some reserved block on disk, which means it wouldn't be protected. I have no idea where that stuff is kept in a UEFI environment, though.

[Side note: We have to check what MicroOS/transactional uses roll back to the previous snapshot. We don't want to break any of that by being overly restrictive]

(In reply to Olaf Kirch from comment #20)
> Hi Michael,
> 
> (In reply to Michael Chang from comment #14)
> > It sounds good to me as well. However to make the deal happen we have to
> > agree on the definition/handling of  "default":
> 
> For me, the definition of "default" would be "no interference from untrusted
> input". Our trigger to forget the LUKS key should be when untrusted input
> changes what we would boot.

Totally. Thanks to clear it up . 

The input mostly originated from boot menu being too helpful in providing many utilities to give out users freedom to modifying or amending a boot flow at runtime. I am thinking for unattended boot maybe it should be disabled with a countdown timer. The timer is used to give users a chance to press ESC key so they can bring up boot menu to work on stuff, and of course by the time it is brought up, the key is thrown away. This behavior seemed to fix in the situation better and more comprehensible.

The settings is easily accomplished in /etc/default/grub, like this:

> GRUB_HIDDEN_TIMEOUT=3
> GRUB_HIDDEN_TIMEOUT_QUIET=false

Of course, we still need to improve current key handing as it is totally legitimate case to enable boot menu. Here just thinking about what would be better alternative to adapt to tpm unattended boot.   

> 
> For example, we could drop the LUKS key as soon as the user enters the boot
> menu, because this is the moment they can start to do potentially harmful
> stuff. Or we do it whenever we register any keyboard input (except for the
> ENTER key).

Sounds good to me. And yes may only keep ENTER for the impatient user to cut in the timeout.

> 
> There may be other untrusted sources that control what we boot and how. I'm
> not sure about grub-set-default and grub-editenv, for instance. In the BIOS
> days, this used to be stored in some reserved block on disk, which means it
> wouldn't be protected. I have no idea where that stuff is kept in a UEFI
> environment, though.

It is kept in a file /boot/grub2/grubenv. For btrfs, additionally we use a portion of btrfs reserved area for bootloaders as raw blocks to extend grubenv to be writable by grub. Both are in the encrypted root partition and are therefore trusted. 

> [Side note: We have to check what MicroOS/transactional uses roll back to
> the previous snapshot. We don't want to break any of that by being overly
> restrictive]

OK. I will do to make sure it won't break existing feature.
Thanks.

> The timer is used to give users a chance to press ESC key so they can bring up boot menu to work on stuff

Maybe GRUB could check and interrupt normal startup if a key is being held during startup? See https://fedoraproject.org/wiki/Changes/HiddenGrubMenu . This way no extra timeout, as even 3 seconds might be a lot of unnecessary waiting time for some users while key hold should be perfectly fine.

(In reply to Dmitry Sharshakov from comment #22)
> > The timer is used to give users a chance to press ESC key so they can bring up boot menu to work on stuff
> 
> Maybe GRUB could check and interrupt normal startup if a key is being held
> during startup? See https://fedoraproject.org/wiki/Changes/HiddenGrubMenu .
> This way no extra timeout, as even 3 seconds might be a lot of unnecessary
> waiting time for some users while key hold should be perfectly fine.

Thanks for providing the information. In the course of working on the subject I just discovered that upstream provided similar feature by holding "shift" key to interrupt timeout:

https://git.savannah.gnu.org/cgit/grub.git/commit/?id=12341958d28f93d381c1a218b28ea1aaf80f6a9b

By the way the patch for better key handling support is mostly done, we are still testing it and will try to make it before end of this week.

I managed to test this and now whatever boot process modification occurs (readonly snapshot boot or older kernel selected) I get a password prompt in Linux. Thanks for your great job! Happy holidays!

I believe this issue is now fixed and can be made visible to everyone, may I do so? Also, what are the missing parts still making this an experimental feature? Is it safe enough for laptops and physical servers? I think it'd be great if https://en.opensuse.org/SDB:Encrypted_root_file_system#Unattended_boot_with_TPM_2.0 mentioned reasons for experimental status. Or does it only need some more testing?

Ideally, I would like to see the full LUKS2 support in grub2 so that the user won't have to choose the LUKS2 algorithm with extra care. Unfortunately, grub2 only supports PBKDF2 for now. Grub2 upstream haven't decided how to support Argon2, and the question has been raised again recently(*).

(*) https://lists.gnu.org/archive/html/grub-devel/2024-01/msg00001.html

(In reply to Dmitry Sharshakov from comment #27)
> > Only users in all of the selected groups can view this bug:
> > Unchecking all boxes makes this a more user accessible bug.
> > Only members of a group can change the visibility of a bug for that group.
> 
> If that's fine, could you please open this discussion?

Will do. But it may take some time because I need to check with people how to do it and many are still in vacation ...

That's alright.

Another idea: maybe by default fde-tools should add the keyslot with less algorithm complexity (aka number of rounds aka iter-time) because the keyfile from TPM is huge enough to be quite resilient to bruteforce even with smaller iteration counts? Or at least it could be useful to have a configuration option to make this faster for a tradeoff of a bit of reduction in bruteforce resistance

(In reply to Dmitry Sharshakov from comment #30)
> That's alright.
> 
> Another idea: maybe by default fde-tools should add the keyslot with less
> algorithm complexity (aka number of rounds aka iter-time) because the
> keyfile from TPM is huge enough to be quite resilient to bruteforce even
> with smaller iteration counts? Or at least it could be useful to have a
> configuration option to make this faster for a tradeoff of a bit of
> reduction in bruteforce resistance

fde-tools did set the smaller iteration number (1000) to the TPM2 keyslot:

https://github.com/openSUSE/fde-tools/blob/main/share/luks#L330-L332

There is also an improvement (already released) in grub2 to try the TPM2 keyslot first. Per my test, it reduces the boot time a lot.

(In reply to Gary Ching-Pang Lin from comment #31)
> (In reply to Dmitry Sharshakov from comment #30)
> > That's alright.
> > 
> > Another idea: maybe by default fde-tools should add the keyslot with less
> > algorithm complexity (aka number of rounds aka iter-time) because the
> > keyfile from TPM is huge enough to be quite resilient to bruteforce even
> > with smaller iteration counts? Or at least it could be useful to have a
> > configuration option to make this faster for a tradeoff of a bit of
> > reduction in bruteforce resistance
> 
> fde-tools did set the smaller iteration number (1000) to the TPM2 keyslot:
> 
> https://github.com/openSUSE/fde-tools/blob/main/share/luks#L330-L332
> 
> There is also an improvement (already released) in grub2 to try the TPM2
> keyslot first. Per my test, it reduces the boot time a lot.

Oh, great to know! Thanks for this work, it's truly amazing for many use cases

(In reply to Dmitry Sharshakov from comment #25)

> I believe this issue is now fixed and can be made visible to everyone, may I
> do so? 

We can set resolution to fixed and the issue is public now. Thanks a lot for everyone's help and support.