Bug 1214849 - Problem with full disk encryption and hibernation
Summary: Problem with full disk encryption and hibernation
Status: NEW
Alias: None
Product: openSUSE Distribution
Classification: openSUSE
Component: Bootloader (show other bugs)
Version: Leap 15.5
Hardware: Other Other
: P5 - None : Normal (vote)
Target Milestone: ---
Assignee: Bootloader Maintainers
QA Contact: E-mail List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-08-31 21:55 UTC by Carlos Robinson
Modified: 2023-09-06 07:47 UTC (History)
2 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
dmesg output on resume (85.29 KB, text/plain)
2023-08-31 22:00 UTC, Carlos Robinson
Details
journal of "before" session (601.62 KB, text/plain)
2023-08-31 22:02 UTC, Carlos Robinson
Details
journal of "after" session (144.28 KB, text/plain)
2023-08-31 22:03 UTC, Carlos Robinson
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Carlos Robinson 2023-08-31 21:55:25 UTC
I have a Lenovo L14 Gen 3 AMD laptop, on which originally I installed Leap 15.4 with full disk encryption with YaST, then upgraded to 15.5.

It has separate partitions (no LVM). Each partition is encrypted separately but with the same password.

The partition layout is:

/          ext4
/home      xfs
swap
/boot/efi  vfat


When the system boots, grub asks for the password. After a longish while, the grub menu appears, the appropriate entry selected, loads, and then plymouth asks for the password again, and applies it to the three encrypted partitions.

So far, so good.

Then I found, with help from the mail list, an openSUSE wiki page with instructions for having only grub prompting for the encryption password:

<https://en.opensuse.org/SDB:Encrypted_root_file_system#Avoiding_to_type_the_passphrase_twice_in_Leap_and_Tumbleweed>

(mail thread Archived-At: 
 <https://lists.opensuse.org/archives/list/users@lists.opensuse.org/message/WE4G3MK6QDUNLVHWROWNS6R6IBSM7QXY/>)

touch /.root.key
chmod 600 /.root.key
dd if=/dev/urandom of=/.root.key bs=1024 count=1

cryptsetup luksAddKey /dev/nvme0n1p3 /.root.key
cryptsetup luksAddKey /dev/nvme0n1p4 /.root.key
cryptsetup luksAddKey /dev/nvme0n1p2 /.root.key

Laicolasse:~ # cat /etc/crypttab
cr-auto-3  UUID=253d3fd9-7f53-465a-...    /.root.key
cr-auto-2  UUID=d153e878-b32c-4a14-...    /.root.key
cr-auto-1  UUID=43662ac8-d98d-4b1a-...    /.root.key  x-initrd.attach
Laicolasse:~ #

The "x-initrd.attach" keyword was already there.

(uuid trimmed for clarity)

Laicolasse:~ # cat /etc/dracut.conf.d/99-root-key.conf
install_items+=" /.root.key "
Laicolasse:~ #


Laicolasse:~ # tail /etc/permissions.local

dracut -f
mkinitrd.



This did not work correctly, I was still being asked for the encryption password twice. Turned out I had modify crypttab:

Laicolasse:~ # lsinitrd /boot/initrd-5.14.21-150500.55.12-default /etc/crypttab
cr-auto-3 /dev/disk/by-uuid/253d3fd9-7f53... /.root.key x-initrd.attach
cr-auto-2 /dev/disk/by-uuid/d153e878-b32c-... /.root.key x-initrd.attach
cr-auto-1 /dev/disk/by-uuid/43662ac8-d98d-... /.root.key x-initrd.attach
Laicolasse:~ # 

Laicolasse:~ # cat /etc/crypttab
cr-auto-3  UUID=253d3fd9-7f53-465a-85f9-1900b6b87a3c	/.root.key  x-initrd.attach
cr-auto-2  UUID=d153e878-b32c-4a14-856e-cbc8c6101280	/.root.key  x-initrd.attach
#cr-auto-1  UUID=43662ac8-d98d-4b1a-a483-0f16e06b419c	/.root.key  x-initrd.attach


/etc/fstab:
LABEL=Main  /          ext4  defaults            0  1
LABEL=Home  /home      xfs   defaults,x-initrd.mount    0  0
LABEL=ESP   /boot/efi  vfat  utf8            0  2
LABEL=Swap  swap       swap  defaults,x-initrd.mount    0  0
LABEL=Beta  /Other     ext4  data=ordered        0  2
Laicolasse:~ # 



This worked, I was prompted for the password once. However, hibernation broke! The machine hibernated, but would not restore, it booted instead. 

Then I reverted the changed to /etc/crypttab, and I was back to the previous situation: resuming worked again, but I am prompted for the encryption password twice.


The diagnosis from Andrei Borzenkov in the mail list is:

+++···································
dracut explicitly skips /etc/crypttab entries with valid third field (which refers to the existing key file on disk). There is no explanation, why. So, dracut does not find any swap device and skips "resume" module.

Why it does it for swaps only I do not know. Otherwise it is happy to actually unlock this device in initrd.

So yes, it does sound like a bug. It should either completely skip this device or fully configure it. 

--------------

dracut ignores swap on LUKS device with key file as resume device but
has no problems adding the same LUKS device if it has x-initrd.mount
option. This is inconsistent.
···································++-


I will later attach the logs.
Comment 1 Carlos Robinson 2023-08-31 22:00:41 UTC
Created attachment 869189 [details]
dmesg output on resume

dmesg output on resume, which is actually a boot.
Comment 2 Carlos Robinson 2023-08-31 22:02:16 UTC
Created attachment 869190 [details]
journal of "before" session

journal of "before" session

It is the journal of the session where I issue the "systemctl hibernate" command as root.
Comment 3 Carlos Robinson 2023-08-31 22:03:40 UTC
Created attachment 869191 [details]
journal of "after" session

journal of "after" session

It is the journal of the session where it should restore from hibernation, but instead it boots fresh.
Comment 4 Antonio Feijoo 2023-09-01 06:46:03 UTC
I think you are experiencing this issue: https://github.com/dracutdevs/dracut/issues/2128

dracut ignores the `x-initrd.attach` option in /etc/crypttab, but it understands it's own `force` option.
Comment 5 Antonio Feijoo 2023-09-01 07:27:02 UTC
(In reply to Carlos Robinson from comment #0)
> +++···································
> dracut explicitly skips /etc/crypttab entries with valid third field (which
> refers to the existing key file on disk). There is no explanation, why. So,
> dracut does not find any swap device and skips "resume" module.
> 
> Why it does it for swaps only I do not know. Otherwise it is happy to
> actually unlock this device in initrd.
> 
> So yes, it does sound like a bug. It should either completely skip this
> device or fully configure it. 
> 
> --------------
> 
> dracut ignores swap on LUKS device with key file as resume device but
> has no problems adding the same LUKS device if it has x-initrd.mount
> option. This is inconsistent.
> ···································++-

The other specific issue of dracut skipping encrypted swaps with password files: 

https://github.com/dracutdevs/dracut/issues/2425

I'd suggest to force the addition of the `resume` module in this case via `add_dracutmodules+=" resume "`.
Comment 6 Carlos Robinson 2023-09-01 20:13:06 UTC
It worked! Thanks.

I'll add that information to the wiki.

[...]

Well, no, now there is a new problem:

I get the prompt from grub for the password, goes through the motions to restore, but at some point, it powers off instead!

I have to press the power button, enter the password a second time, and now it fully restores.

I'm not sure it happens every time, but for sure happened twice.

Nothing is displayed on the screen, and nothing can be in the logs, system is not "running" while this is going on.
Comment 7 Antonio Feijoo 2023-09-04 06:35:36 UTC
(In reply to Carlos Robinson from comment #6)
> It worked! Thanks.
> 
> I'll add that information to the wiki.
> 
> [...]
> 
> Well, no, now there is a new problem:
> 
> I get the prompt from grub for the password, goes through the motions to
> restore, but at some point, it powers off instead!
> 
> I have to press the power button, enter the password a second time, and now
> it fully restores.
> 
> I'm not sure it happens every time, but for sure happened twice.
> 
> Nothing is displayed on the screen, and nothing can be in the logs, system
> is not "running" while this is going on.

Does the system power off before or after displaying the GRUB menu? If the answer is after, check if your the kernel command line of the boot option contains `quiet` or `plymouth=whatever` and remove them.

BTW, I don't think you need to mark with `x-initrd.mount` the encrypted partitions in /etc/fstab only to make resume from hibernation work (I don't know if you have other use case that needs these partitions mounted in the initrd).
Comment 8 Carlos Robinson 2023-09-05 14:02:51 UTC
(In reply to Antonio Feijoo from comment #7)
> (In reply to Carlos Robinson from comment #6)

It happens, apparently, before the grub menu. 
However, it has only happened those two times, so I haven't been able to investigate more. Yes, I already did those things and I'm booting in verbose mode since I wrote the previous comment.

So I'm happy that it works, but no idea why that error happened or if will repeat.


> BTW, I don't think you need to mark with `x-initrd.mount` the encrypted
> partitions in /etc/fstab only to make resume from hibernation work (I don't
> know if you have other use case that needs these partitions mounted in the
> initrd).

Unless I write those options to fstab and crypttab, all the lines in crypttab and not written to the initrd archive, and thus, they are not decrypted using the key file, but instead the system asks me to type the password _again_. Once by grub, once by systemd.

That's the problem, that some logic in dracut decides what lines to copy and which not, and does the decision wrongly.

You can follow the problem tracking and why each option was added and the testing in the mail list thread I linked.
Comment 9 Antonio Feijoo 2023-09-06 06:21:10 UTC
(In reply to Carlos Robinson from comment #8)
> (In reply to Antonio Feijoo from comment #7)
> > (In reply to Carlos Robinson from comment #6)
> 
> It happens, apparently, before the grub menu. 
> However, it has only happened those two times, so I haven't been able to
> investigate more. Yes, I already did those things and I'm booting in verbose
> mode since I wrote the previous comment.
> 
> So I'm happy that it works, but no idea why that error happened or if will
> repeat.

Based on this, re-assigning to @bootloader-maintainers to check this remaining issue. See comment #6.

> > BTW, I don't think you need to mark with `x-initrd.mount` the encrypted
> > partitions in /etc/fstab only to make resume from hibernation work (I don't
> > know if you have other use case that needs these partitions mounted in the
> > initrd).
> 
> Unless I write those options to fstab and crypttab, all the lines in
> crypttab and not written to the initrd archive, and thus, they are not
> decrypted using the key file, but instead the system asks me to type the
> password _again_. Once by grub, once by systemd.
> 
> That's the problem, that some logic in dracut decides what lines to copy and
> which not, and does the decision wrongly.
> 
> You can follow the problem tracking and why each option was added and the
> testing in the mail list thread I linked.

Yes, I followed that. If you want to ensure that a crypttab entry is added to the initrd crypttab, you just need to add the `force` option in the 4th column.
Comment 10 Michael Chang 2023-09-06 07:47:38 UTC
(In reply to Carlos Robinson from comment #8)
> (In reply to Antonio Feijoo from comment #7)
> > (In reply to Carlos Robinson from comment #6)
> 
> It happens, apparently, before the grub menu. 
> However, it has only happened those two times, so I haven't been able to
> investigate more. Yes, I already did those things and I'm booting in verbose
> mode since I wrote the previous comment.
> 
> So I'm happy that it works, but no idea why that error happened or if will
> repeat.

Could the CPU overheats in the course of running PBKDF iteration ? The process costs a lot to grub in terms of CPU resource consumption because no CPU hardware acceleration (SSE/AVX and so on) enabled in the pre-boot environment to make the cryoto related calculations efficient, especially the hash algorithms that is heavily used by the KDF. If the cooling system is also somehow not in a good condition to process the heat, as a safe measure the CPU may power itself off.

If it is the case, either to improve the cooling system by cleaning it's fan and outlet, or you can try to reduce the PBKDF iteration count which would also shorten the time in grub to unlock the volume, as an exchange that your password might be easier to crack. 

You can follow the link to do that:

[1] https://wiki.archlinux.org/title/GRUB/Tips_and_tricks#Speeding_up_LUKS_decryption_in_GRUB

Thanks.