Bug 1047598 - udev: stuck boot using LUKS encrypted LVM on NVMe
Summary: udev: stuck boot using LUKS encrypted LVM on NVMe
Status: RESOLVED FIXED
Alias: None
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Bootloader (show other bugs)
Version: Current
Hardware: x86-64 Other
: P2 - High : Major (vote)
Target Milestone: ---
Assignee: Steffen Winterfeldt
QA Contact: Jiri Srain
URL:
Whiteboard: maint:planned:update
Keywords:
Depends on:
Blocks: 1048679
  Show dependency treegraph
 
Reported: 2017-07-06 22:05 UTC by Carmen Bianca Bakker
Modified: 2023-07-14 08:30 UTC (History)
9 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
Boot log (86.21 KB, text/plain)
2017-07-06 22:05 UTC, Carmen Bianca Bakker
Details
Boot log after upgrading Leap 42.2 to Tumbleweed (82.64 KB, text/plain)
2017-07-07 07:50 UTC, Carmen Bianca Bakker
Details
journalctl -b on a working Leap 42.2 install (99.24 KB, text/plain)
2017-07-07 13:20 UTC, Carmen Bianca Bakker
Details
Output from save_y2logs (2.15 MB, application/x-xz)
2017-07-14 11:26 UTC, Carmen Bianca Bakker
Details
udevadm info -e (161.37 KB, text/plain)
2017-07-18 14:06 UTC, Carmen Bianca Bakker
Details
udevadm test --action=add /block/nvme3 (7.21 KB, text/plain)
2017-07-18 14:06 UTC, Carmen Bianca Bakker
Details
edited rules (3.11 KB, text/plain)
2017-07-18 14:08 UTC, Carmen Bianca Bakker
Details
udevadm test --action=add /block/nvme0n1 (854 bytes, text/plain)
2017-07-18 20:29 UTC, Carmen Bianca Bakker
Details
udevadm test --action=add /sys/class/block/nvme0n1p3 (1.48 KB, text/plain)
2017-07-19 15:11 UTC, Carmen Bianca Bakker
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Carmen Bianca Bakker 2017-07-06 22:05:38 UTC
Created attachment 731483 [details]
Boot log

The summary title probably isn't too accurate because there are a lot of variables at play here.

As an introduction then: I have a Thinkpad T470 that I want to install openSUSE Tumbleweed on. The laptop has an NVMe SSD. For some reason, the laptop would not boot from a usual Tumbleweed installation ISO (i.e., `dd if=tumbleweed.iso of=/dev/sdb`).

To get around that issue, I consulted this bug report: https://bugzilla.opensuse.org/show_bug.cgi?id=939456

The gist of it:

    sudo mount tumbleweed.iso /mnt  # Not sure if this intermediary step is necessary, but it works
    sudo mksusecd -c suse.iso /mnt
    sudo dd if=suse.iso of=/dev/sdb bs=4M && sync

And then it boots from USB and installs just fine.

During installation, I selected LUKS encrypted LVM. This creates three partitions:

- /dev/nvme0n1p1 - /boot/efi, FAT32
- /dev/nvme0n1p2 - This partition is empty. Not sure why, but it is.
- /dev/nvme0n1p3 - LVM/LUKS

Installation finishes without issues.

When booting, grub2-efi prompts for the LUKS password. It successfully decrypts and reaches the selection screen.

When selecting Tumbleweed, it begins to boot and reaches Plymouth. Here, it gets stuck. As a side note, passing `plymouth.enable=0` does _not_ fix this issue.

When you hit a key to hide Plymouth and show (some of the) boot logs, it shows these in particular:

    [    5.251779] localhost systemd[1]: Started Show Plymouth Boot Screen.
    [    5.251947] localhost systemd[1]: Reached target Paths.
    [    5.252069] localhost systemd[1]: Started Forward Password Requests to Plymouth Directory Watch.
    [    5.529328] localhost systemd[1]: Found device /dev/disk/by-uuid/0CC3-B108.

/dev/disk/by-uuid/0CC3-B108 is equal to /dev/nvme0n1p1 (AKA /boot/efi).

And that's where I personally get stuck.  I'm not quite sure what to do from here on.  There's a similar-ish issue on AskUbuntu:

https://askubuntu.com/questions/772494/stuck-on-ubuntu-installation-trial-started-dispatch-password-requests

But the offered solution does not fix my problem.

I have attached the full boot logs as rdsosreport.txt.

Thank you for helping and for making openSUSE!
Comment 2 Carmen Bianca Bakker 2017-07-07 07:49:31 UTC
Okay I've learnt a few things since yesterday.

1. Selecting /dev/nvme0n1p2 (the unused partition) as unencrypted /boot (ext4) during installation does not change anything. Short of not having to type in a password to access grub, anyway.

2. openSUSE Leap 42.2 installs and boots correctly when using encrypted LVM. (I used the same `mksusecd` workaround, though I didn't test if this was necessary)

3. After upgrading openSUSE Leap 42.2 to Tumbleweed, the problem re-appears and you get stuck during boot.  The boot log looks a little different though (didn't reach target Paths), but it gets stuck on the same thing (found device /boot/efi). I'll attach this boot log.

4. openSUSE Tumbleweed installs and boots correctly when selecting the default, unencrypted partition-based layout.

I want to try and see whether unencrypted LVM installs and boots.
Comment 3 Carmen Bianca Bakker 2017-07-07 07:50:21 UTC
Created attachment 731520 [details]
Boot log after upgrading Leap 42.2 to Tumbleweed
Comment 4 Carmen Bianca Bakker 2017-07-07 10:28:02 UTC
I tried unencrypted LVM: It works just fine.

I did also try to install encrypted LVM via legacy BIOS instead of EFI: Doesn't work.

I think I've narrowed the problem down quite enough now (Tumbleweed + LVM/LUKS + my hardware).  Just not quite sure what I can try now.
Comment 5 Carmen Bianca Bakker 2017-07-07 13:11:19 UTC
I may've narrowed it down just a bit further. When upgrading a Leap 42.2 installation to Tumbleweed, it no longer boots. But it's also noteworthy that the old Leap snapshots stop booting as well. So it's probably not a kernel issue, but more of a Grub2 issue.

Moreover, there's this line in the logs (copied from the attachment in #1):

    [    5.067028] localhost systemd[1]: Unnecessary job for dev-mapper-cr_\x2dINTEL_SSDPEKKF256G7L_BTPY7136185T256D\x2dpart3.device was removed.


I guess it's not mounting LVM at all?
Comment 6 Carmen Bianca Bakker 2017-07-07 13:20:58 UTC
Created attachment 731587 [details]
journalctl -b on a working Leap 42.2 install
Comment 7 Carmen Bianca Bakker 2017-07-07 15:05:52 UTC
Lovely, I got the system booting.

Steps for workaround:

- Boot into live system (I used Kubuntu, but use whatever really)

- Mount the partitions for chrooting (execute as root):

    crypsetup luksOpen /dev/nvme0n1p3 system
    vgchange -a y
    mount /dev/mapper/system-root /mnt
    mount /dev/nvme0n1p1 /mnt/boot/efi
    mount --bind /dev /mnt/dev
    mount --bind /sys /mnt/sys
    mount --bind /proc /mnt/proc
    chroot /mnt

- Edit /etc/crypttab. Change `/dev/disk/by-id/-INTEL-gibberish-part3` to `/dev/disk/by-id/nvme-INTEL-gibberish-part3`.

- Run `mkinitrd`

And that should just about do it.

See https://bugzilla.opensuse.org/show_bug.cgi?id=1030299 for more information. This bug and that bug are not duplicates; I'm letting the installer take care of partitioning. But the end result is the same: A broken /etc/crypttab.

I don't know why Leap 42.2 worked but Tumbleweed did not. But at this point after so many reinstalls, I'm honestly lethargic to figure it out.
Comment 8 Daniel Molkentin 2017-07-13 14:29:46 UTC
Quote from #7:

> - Edit /etc/crypttab. Change `/dev/disk/by-id/-INTEL-gibberish-part3` to `/dev/disk/by-id/nvme-INTEL-gibberish-part3`.

This looks like a YaST problem to me. I'm re-assigning this to the YaST team.
Comment 9 Steffen Winterfeldt 2017-07-14 11:09:53 UTC
This looks like udev not generating the correct device link.

Carmen, could you attach the yast logs, please.

Run 'save_y2logs' and attach the log file archive this command produces.

If you're unsure what to do, please have a look at

https://en.opensuse.org/openSUSE:Report_a_YaST_bug#logfiles
Comment 10 Carmen Bianca Bakker 2017-07-14 11:26:02 UTC
Created attachment 732437 [details]
Output from save_y2logs

This should do it, hopefully.
Comment 11 Steffen Winterfeldt 2017-07-14 12:24:57 UTC
Thanks!

The logs indeed show that the 'nvme' string is sometimes missing. Like

/dev/disk/by-id/-INTEL_SSDPEKKF256G7L_BTPY7136185T256D
vs.
/dev/disk/by-id/nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D

or

/dev/mapper/cr_-INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part3
vs.
/dev/mapper/cr_nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part3

Looks like a udev thing.
Comment 12 Sebastian Parschauer 2017-07-14 14:32:31 UTC
Definitely udev. With the udev update also SLES12-SP2 and Leap 42.2 are affected.
Comment 13 Carmen Bianca Bakker 2017-07-14 14:47:07 UTC
A bit separate from the issue at hand, but wouldn't it make sense to use the UUID in /etc/crypttab instead of /dev/disk/by-id?  After all, UUID is used in /etc/fstab as well.
Comment 14 Aaron Digulla 2017-07-15 11:44:57 UTC
I'm on Leap.

When I try to mount the EFI partition, I get this error:

[...] FAT-fs (nvme0n1p1): IO charset iso8859-1 not found
mount: wrong fs type, bad option, bad superblock.

I'm sure I have the right partition (blkid lists it as TYPE="vfat" PARTLABEL="EFI System Partition".

After loading the modules nls_cp437.ko and nls_iso8859-1.ko, I can mount.
Comment 15 Andreas Stieger 2017-07-16 08:43:49 UTC
*** Bug 1048679 has been marked as a duplicate of this bug. ***
Comment 16 Mischa Salle 2017-07-16 13:26:20 UTC
(In reply to Andreas Stieger from comment #15)
> *** Bug 1048679 has been marked as a duplicate of this bug. ***

although Bug 1048679 is indeed the same bug, the IDs are slightly different due to Tumbleweed having a different systemd: compare the old ID in https://bugzilla.suse.com/show_bug.cgi?id=1048679#c4 with the ID in https://bugzilla.suse.com/show_bug.cgi?id=1047598#c8

Concerning a solution, it would be very helpful if dracut / mkinitrd would backup the initrd which was used for booting the currently running system before replacing all the existing initrds in /boot. That would make a roll-back much easier. For a kernel update, replacing them is needed, but for e.g. a systemd update, it can break booting.
Comment 17 Thomas Blume 2017-07-18 12:57:33 UTC
(In reply to Carmen Bianca Bakker from comment #13)
> A bit separate from the issue at hand, but wouldn't it make sense to use the
> UUID in /etc/crypttab instead of /dev/disk/by-id?  After all, UUID is used
> in /etc/fstab as well.

The UUID is a file system property.
Using the UUID in crypttab would require that the system in able to read it from the encrypted device.
Not sure whether this is possible.

Please attach the output of:

udevadm info -e

and:

udevadm test --action=add /block/nvme3

In addition, could you please do the following test?
If your systemd has a file:

/lib/udev/rules.d/60-persistent-storage-compat.rules

edit it and comment out all rules referring to nvme devices.
You will need to rebuild the initrd in order to sync the changed rules there.
Does that change the behavior?
Comment 18 Carmen Bianca Bakker 2017-07-18 14:04:53 UTC
(In reply to Thomas Blume from comment #17)
> (In reply to Carmen Bianca Bakker from comment #13)
> > A bit separate from the issue at hand, but wouldn't it make sense to use the
> > UUID in /etc/crypttab instead of /dev/disk/by-id?  After all, UUID is used
> > in /etc/fstab as well.
> 
> The UUID is a file system property.
> Using the UUID in crypttab would require that the system in able to read it
> from the encrypted device.
> Not sure whether this is possible.

It appears possible.  If I change my crypttab to this, the system boots:

    cr_nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part3 UUID=4847d296-1a27-458d-8988-45d5b345ddc8 none none

Output of `ls -la /dev/disk/by-uuid`:

    carmen@carmen-thinkpad:~> ls -la /dev/disk/by-uuid
    total 0
    drwxr-xr-x 2 root root 120 Jul 17 14:45 .
    drwxr-xr-x 7 root root 140 Jul 17 14:45 ..
    lrwxrwxrwx 1 root root  10 Jul 17 14:45 016eb514-736a-4a21-95d0-894f83e69e70 -> ../../dm-1
    lrwxrwxrwx 1 root root  15 Jul 17 14:45 4847d296-1a27-458d-8988-45d5b345ddc8 -> ../../nvme0n1p3
    lrwxrwxrwx 1 root root  15 Jul 17 14:45 EB08-0D77 -> ../../nvme0n1p1
    lrwxrwxrwx 1 root root  10 Jul 17 14:45 ec42e85c-7406-4d4b-af75-14d09a0b8b9a -> ../../dm-2


Output of `ls -la /dev/disk/by-id`:

    carmen@carmen-thinkpad:~> ls -la /dev/disk/by-id/
    total 0
    drwxr-xr-x 3 root root 440 Jul 18 13:52 .
    drwxr-xr-x 7 root root 140 Jul 17 14:45 ..
    lrwxrwxrwx 1 root root  10 Jul 17 14:45 dm-name-cr_nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part3 -> ../../dm-0
    lrwxrwxrwx 1 root root  10 Jul 17 14:45 dm-name-system-root -> ../../dm-1
    lrwxrwxrwx 1 root root  10 Jul 17 14:45 dm-name-system-swap -> ../../dm-2
    lrwxrwxrwx 1 root root  10 Jul 17 14:45 dm-uuid-CRYPT-LUKS1-4847d2961a27458d898845d5b345ddc8-cr_nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part3 -> ../../dm-0
    lrwxrwxrwx 1 root root  10 Jul 17 14:45 dm-uuid-LVM-Zbp31neIR3fn7AceHr9UrnkSY7p0Si3S24tPKJqfhD8KMIt83NoEbg5hXdNUJkQz -> ../../dm-1
    lrwxrwxrwx 1 root root  10 Jul 17 14:45 dm-uuid-LVM-Zbp31neIR3fn7AceHr9UrnkSY7p0Si3StKPx9IEAikEb1ApNlTFg99TRYm61dDCs -> ../../dm-2
    lrwxrwxrwx 1 root root  10 Jul 17 14:45 lvm-pv-uuid-ArrrE3-lGOe-KZV4-r2WW-AerJ-Hoxr-GQQnyq -> ../../dm-0
    lrwxrwxrwx 1 root root  13 Jul 17 14:45 nvme-eui.5cd2e41471a0a605 -> ../../nvme0n1
    lrwxrwxrwx 1 root root  15 Jul 17 14:45 nvme-eui.5cd2e41471a0a605-part1 -> ../../nvme0n1p1
    lrwxrwxrwx 1 root root  15 Jul 17 14:45 nvme-eui.5cd2e41471a0a605-part2 -> ../../nvme0n1p2
    lrwxrwxrwx 1 root root  15 Jul 17 14:45 nvme-eui.5cd2e41471a0a605-part3 -> ../../nvme0n1p3
    lrwxrwxrwx 1 root root  13 Jul 17 14:45 nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D -> ../../nvme0n1
    lrwxrwxrwx 1 root root  15 Jul 17 14:45 nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part1 -> ../../nvme0n1p1
    lrwxrwxrwx 1 root root  15 Jul 17 14:45 nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part2 -> ../../nvme0n1p2
    lrwxrwxrwx 1 root root  15 Jul 17 14:45 nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part3 -> ../../nvme0n1p3
    lrwxrwxrwx 1 root root  10 Jul 17 14:45 raid-cr_nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part3 -> ../../dm-0
    lrwxrwxrwx 1 root root  10 Jul 17 14:45 raid-system-root -> ../../dm-1
    lrwxrwxrwx 1 root root  10 Jul 17 14:45 raid-system-swap -> ../../dm-2
    lrwxrwxrwx 1 root root   9 Jul 18 13:52 scsi-200e04c2020202000 -> ../../sda
    drwxr-xr-x 2 root root  60 Jul 18 13:52 scsi-SGeneric-_SD

> Please attach the output of:
> 
> udevadm info -e
> 
> and:
> 
> udevadm test --action=add /block/nvme3

The second command barely writes anything to stdout, writes a lot to stderr.  Giving you both in a single file.

> In addition, could you please do the following test?
> If your systemd has a file:
> 
> /lib/udev/rules.d/60-persistent-storage-compat.rules
> 
> edit it and comment out all rules referring to nvme devices.
> You will need to rebuild the initrd in order to sync the changed rules there.
> Does that change the behavior?

This boots (where crypttab is /dev/disk/by-id/[...]).  `ls -la /dev/disk/by-id` is identical.  I'll attach the edited /lib/udev/rules.d/61-persistent-storage-compat.rules.  Lines with '##' are edited by me.
Comment 19 Carmen Bianca Bakker 2017-07-18 14:06:21 UTC
Created attachment 732770 [details]
udevadm info -e
Comment 20 Carmen Bianca Bakker 2017-07-18 14:06:53 UTC
Created attachment 732771 [details]
udevadm test --action=add /block/nvme3
Comment 21 Carmen Bianca Bakker 2017-07-18 14:08:09 UTC
Created attachment 732772 [details]
edited rules

Lines with '##' were edited by me.
Comment 22 Thomas Blume 2017-07-18 14:13:42 UTC
(In reply to Carmen Bianca Bakker from comment #20)
> Created attachment 732771 [details]
> udevadm test --action=add /block/nvme3

Sorry, I mistyped the device name.
The correct command would be:

udevadm test --action=add /block/nvme0n1p3
Comment 23 Carmen Bianca Bakker 2017-07-18 20:29:22 UTC
Created attachment 732836 [details]
udevadm test --action=add /block/nvme0n1

/block/nvme0n1, not /block/nvme0n1p3
Comment 24 Franck Bui 2017-07-19 11:32:27 UTC
(In reply to Carmen Bianca Bakker from comment #3)
> Created attachment 731520 [details]
> Boot log after upgrading Leap 42.2 to Tumbleweed

Hi Carmen, could you please tell us how did you generate this logs exactly ?

Did you generate it from the dracut emergency shell after upgrading and rebooting from 42.2 to TW ?

Also which version of systemd/udev was 42.2 using before upgrading to TW ?

Thanks
Comment 25 Thomas Blume 2017-07-19 13:13:36 UTC
(In reply to Carmen Bianca Bakker from comment #23)
> Created attachment 732836 [details]
> udevadm test --action=add /block/nvme0n1
> 
> /block/nvme0n1, not /block/nvme0n1p3

Hm, but your encrypted device is partition 3, e.g. /dev/nvme0n1p3 on this disk, no?
Comment 26 Carmen Bianca Bakker 2017-07-19 13:21:53 UTC
(In reply to Thomas Blume from comment #25)
> (In reply to Carmen Bianca Bakker from comment #23)
> > Created attachment 732836 [details]
> > udevadm test --action=add /block/nvme0n1
> > 
> > /block/nvme0n1, not /block/nvme0n1p3
> 
> Hm, but your encrypted device is partition 3, e.g. /dev/nvme0n1p3 on this
> disk, no?

Correct, but there's no such thing as /block/nvme0n1p3.

    carmen@carmen-thinkpad:~> ls -la /sys/block
    total 0
    drwxr-xr-x  2 root root 0 Jul 19 15:20 .
    dr-xr-xr-x 13 root root 0 Jul 19 15:20 ..
    lrwxrwxrwx  1 root root 0 Jul 19 15:20 dm-0 -> ../devices/virtual/block/dm-0
    lrwxrwxrwx  1 root root 0 Jul 19 15:20 dm-1 -> ../devices/virtual/block/dm-1
    lrwxrwxrwx  1 root root 0 Jul 19 15:20 dm-2 -> ../devices/virtual/block/dm-2
    lrwxrwxrwx  1 root root 0 Jul 19 15:20 nvme0n1 -> ../devices/pci0000:00/0000:00:1d.2/0000:3e:00.0/nvme/nvme0/nvme0n1
    lrwxrwxrwx  1 root root 0 Jul 19 15:20 sda -> ../devices/pci0000:00/0000:00:14.0/usb2/2-3/2-3:1.0/host0/target0:0:0/0:0:0:0/block/sda
Comment 27 Carmen Bianca Bakker 2017-07-19 13:24:15 UTC
(In reply to Franck Bui from comment #24)
> (In reply to Carmen Bianca Bakker from comment #3)
> > Created attachment 731520 [details]
> > Boot log after upgrading Leap 42.2 to Tumbleweed
> 
> Hi Carmen, could you please tell us how did you generate this logs exactly ?
> 
> Did you generate it from the dracut emergency shell after upgrading and
> rebooting from 42.2 to TW ?

I generated it from the dracut emergency shell, yes.  `journalctl -b > file`, and then transfer the file onto a USB pendrive.

> Also which version of systemd/udev was 42.2 using before upgrading to TW ?

I did not upgrade any packages in Leap before the upgrade to TW as I remember it.  So whatever version it ships with.
Comment 28 Franck Bui 2017-07-19 13:33:25 UTC
(In reply to Carmen Bianca Bakker from comment #7)
> 
> - Edit /etc/crypttab. Change `/dev/disk/by-id/-INTEL-gibberish-part3` to
> `/dev/disk/by-id/nvme-INTEL-gibberish-part3`.
> 

Please don't change/shortcut the name of the symlinks like you did.

Do you remember if the "gibberish" part was preserved when you prepended "nvme" ?
Comment 29 Thomas Blume 2017-07-19 13:36:48 UTC
(In reply to Carmen Bianca Bakker from comment #26)
> (In reply to Thomas Blume from comment #25)
> > (In reply to Carmen Bianca Bakker from comment #23)
> > > Created attachment 732836 [details]
> > > udevadm test --action=add /block/nvme0n1
> > > 
> > > /block/nvme0n1, not /block/nvme0n1p3
> > 
> > Hm, but your encrypted device is partition 3, e.g. /dev/nvme0n1p3 on this
> > disk, no?
> 
> Correct, but there's no such thing as /block/nvme0n1p3.

Can you run:

find /sys -name nvme0n1p3

and do the udevadm test using the output?
Comment 30 Carmen Bianca Bakker 2017-07-19 15:09:05 UTC
(In reply to Franck Bui from comment #28)
> (In reply to Carmen Bianca Bakker from comment #7)
> > 
> > - Edit /etc/crypttab. Change `/dev/disk/by-id/-INTEL-gibberish-part3` to
> > `/dev/disk/by-id/nvme-INTEL-gibberish-part3`.
> > 
> 
> Please don't change/shortcut the name of the symlinks like you did.
> 
> Do you remember if the "gibberish" part was preserved when you prepended
> "nvme" ?

nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part3

All that missed was "nvme"
Comment 31 Carmen Bianca Bakker 2017-07-19 15:11:23 UTC
Created attachment 732992 [details]
udevadm test --action=add /sys/class/block/nvme0n1p3
Comment 32 Franck Bui 2017-07-19 15:52:31 UTC
(In reply to Carmen Bianca Bakker from comment #30)
> nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part3
> 
> All that missed was "nvme"

Ok then bug #1048679 might not be a duplicate of this bug. At least if we consider the case where you did an installation of TW directly (no upgrade from 42.2).

If the link without the "nvme" prefix was used by the installer then it seems that the compat symlinks are enabled by the installation system and that's incorrect.

Could anybody tell which version of udev is used by the installation system and how udev is installed on it ?

IOW does the installation system has the following file:
/usr/lib/udev/compat-symlink-generation

Steffen, maybe you have an idea ?
Comment 33 Steffen Winterfeldt 2017-07-19 15:58:47 UTC
The installation system contains files /.packages.* that list the exact package
versions used to build it.
Comment 34 Franck Bui 2017-07-19 16:57:25 UTC
I looked at the iso I have (Snapshot20170620).

The installation system was using udev 233 and the compat rule was also installed.

But the file which should contain the generation compat symlink number is not installed. IOW /usr/lib/udev/compat-symlink-generation is not present.

udev has:

  %pre -n udev%{?mini}
  [...]
  if [ $1 -eq 1 ]; then
          echo "COMPAT_SYMLINK_GENERATION=2">/usr/lib/udev/compat-symlink-generation
  fi

Any idea why this file is missing in the installation system ?
Comment 35 Carmen Bianca Bakker 2017-07-19 20:15:58 UTC
(In reply to Franck Bui from comment #32)
> (In reply to Carmen Bianca Bakker from comment #30)
> > nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part3
> > 
> > All that missed was "nvme"
> 
> Ok then bug #1048679 might not be a duplicate of this bug. At least if we
> consider the case where you did an installation of TW directly (no upgrade
> from 42.2).

No, I did an installation of Tumbleweed and then edited /etc/crypttab.  I have no idea why Leap failed booting after upgrading to Tumbleweed.
Comment 36 Carmen Bianca Bakker 2017-07-19 20:18:03 UTC
(In reply to Carmen Bianca Bakker from comment #35)
> (In reply to Franck Bui from comment #32)
> > (In reply to Carmen Bianca Bakker from comment #30)
> > > nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part3
> > > 
> > > All that missed was "nvme"
> > 
> > Ok then bug #1048679 might not be a duplicate of this bug. At least if we
> > consider the case where you did an installation of TW directly (no upgrade
> > from 42.2).
> 
> No, I did an installation of Tumbleweed and then edited /etc/crypttab.  I
> have no idea why Leap failed booting after upgrading to Tumbleweed.

Forget that I said anything.  I completely misread that.
Comment 37 Franck Bui 2017-07-20 08:57:30 UTC
Steffen any idea ?
Comment 38 Steffen Winterfeldt 2017-07-20 09:22:21 UTC
prein script was not run; I've created

https://github.com/openSUSE/installation-images/pull/190

for this.

While we're at it: Franck, need I also run postin?
Comment 39 Franck Bui 2017-07-20 09:34:10 UTC
(In reply to Steffen Winterfeldt from comment #38)
> prein script was not run; I've created
> 
> https://github.com/openSUSE/installation-images/pull/190
> 
> for this.

ouch... not sure why this needs to be manually listed but it's good to know :)

> 
> While we're at it: Franck, need I also run postin?

udev uses the following scriptlets: %pre, %post, %postun, %posttrans
Comment 40 Steffen Winterfeldt 2017-07-20 09:45:44 UTC
Yes, but in most cases they are not needed. The udev postin script also looks
like a no-op to me.
Comment 41 Steffen Winterfeldt 2017-07-20 13:09:12 UTC
patch submitted
Comment 42 Franck Bui 2017-07-21 06:52:48 UTC
udev has no %postin.

Could a testing ISO be provided so we're sure that the provided fix addresses this bug ?

Should this bug be re-assigned to you, Steffen ?
Comment 43 Steffen Winterfeldt 2017-07-21 08:43:38 UTC
It's easy to create a test iso:

# mkdir -p /tmp/foo/usr/lib/udev
# echo COMPAT_SYMLINK_GENERATION=2 > /tmp/foo/usr/lib/udev/compat-symlink-generation
# mksusecd -c new.iso --initrd /tmp/foo old.iso
Comment 44 Steffen Winterfeldt 2017-07-21 08:47:36 UTC
Carmen, could you test is this works?
Comment 45 Carmen Bianca Bakker 2017-07-21 13:41:00 UTC
(In reply to Steffen Winterfeldt from comment #44)
> Carmen, could you test is this works?

Following instructions in comment #43: It works.  I flashed the altered Tumbleweed DVD ISO to a USB drive, booted from it, entered rescue mode and checked the output of `ls -la /dev/disk/by-id`.  It correctly lists the nvme drives.

To make sure, I also flashed an unaltered ISO and checked the same command.  There, the output looked a little (but not exactly!) like this:

    -INTEL_SSDPEKKF256G7L_BTPY7136185T256D -> ../../nvme0n1
    -INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part1 -> ../../nvme0n1p1
    -INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part2 -> ../../nvme0n1p2
    -INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part3 -> ../../nvme0n1p3
    nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D -> ../../nvme0n1
    nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part1 -> ../../nvme0n1p1
    nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part2 -> ../../nvme0n1p2
    nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part3 -> ../../nvme0n1p3

Thank you for your work :)
Comment 46 Franck Bui 2017-07-21 13:47:29 UTC
(In reply to Carmen Bianca Bakker from comment #45)
> (In reply to Steffen Winterfeldt from comment #44)
> > Carmen, could you test is this works?
> 
> Following instructions in comment #43: It works.  I flashed the altered
> Tumbleweed DVD ISO to a USB drive, booted from it, entered rescue mode and
> checked the output of `ls -la /dev/disk/by-id`.  It correctly lists the nvme
> drives.
> 

Thanks for testing.

What did you mean by "it correctly lists the nvme drivers" ?

Did you mean that the symlinks without the "nvme" prefix such as:

>     -INTEL_SSDPEKKF256G7L_BTPY7136185T256D -> ../../nvme0n1
>     -INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part1 -> ../../nvme0n1p1
>     -INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part2 -> ../../nvme0n1p2
>     -INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part3 -> ../../nvme0n1p3

are no more generated ?
Comment 47 Carmen Bianca Bakker 2017-07-21 14:04:25 UTC
(In reply to Franck Bui from comment #46)
> What did you mean by "it correctly lists the nvme drivers" ?
> 
> Did you mean that the symlinks without the "nvme" prefix such as:
> 
> >     -INTEL_SSDPEKKF256G7L_BTPY7136185T256D -> ../../nvme0n1
> >     -INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part1 -> ../../nvme0n1p1
> >     -INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part2 -> ../../nvme0n1p2
> >     -INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part3 -> ../../nvme0n1p3
> 
> are no more generated ?

Correct.  I should have been clearer, sorry.
Comment 48 Steffen Winterfeldt 2017-07-21 14:06:04 UTC
Not so fast!

If you mean by 'rescue mode' that you booted the rescue system from the boot menu then this test proves nothing, as there the compat file did already exist.

This is only about starting a regular install.
Comment 49 Carmen Bianca Bakker 2017-07-21 14:17:39 UTC
(In reply to Steffen Winterfeldt from comment #48)
> Not so fast!
> 
> If you mean by 'rescue mode' that you booted the rescue system from the boot
> menu then this test proves nothing, as there the compat file did already
> exist.
> 
> This is only about starting a regular install.

Strange.  Even so there was a difference in output between the two ISOs (one altered, one unaltered) I flashed.  The unaltered ISO listed the nvme partitions doubly as I specified above, and the altered ISO listed only the correct IDs (e.g., nvme-INTEL_[...])

I cannot do an additional installation on this machine to check, however.  I am sorry.  Is there something I can do that does not involve actually (re)installing openSUSE?
Comment 50 Steffen Winterfeldt 2017-07-21 14:20:16 UTC
Sure, just boot with startshell=1. This leaves you at a shell prompt before the
installer starts. You can check the device names there.
Comment 51 Steffen Winterfeldt 2017-07-21 14:24:12 UTC
> Strange.  Even so there was a difference in output between the two ISOs (one ltered,

The device tree is carried over from the initial startup phase without the compat stuff. So you'd see both versions with the old iso.
Comment 52 Carmen Bianca Bakker 2017-07-21 14:34:56 UTC
(In reply to Steffen Winterfeldt from comment #50)
> Sure, just boot with startshell=1. This leaves you at a shell prompt before
> the
> installer starts. You can check the device names there.

I did that.  This dropped me into a shell.  `ls -la /dev/disk/by-id` listed only correct output:

    nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D -> ../../nvme0n1
    nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part1 -> ../../nvme0n1p1
    nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part2 -> ../../nvme0n1p2
    nvme-INTEL_SSDPEKKF256G7L_BTPY7136185T256D-part3 -> ../../nvme0n1p3
    # and then some for other partitions/devices

I also checked the /usr/lib/udev folder, and it contained a compat-symlink-generation file that contained:

    COMPAT_SYMLINK_GENERATION=2

I have not checked the unaltered ISO.  Ought I?
Comment 53 Steffen Winterfeldt 2017-07-21 14:36:41 UTC
No, not necessary.

I think we can close this bug.

Thanks a lot for your help, Carmen!
Comment 54 Franck Bui 2017-07-21 14:38:30 UTC
Thanks Carmen for your help.
Comment 55 Forgotten User gf5Dn4arcK 2017-07-27 12:58:26 UTC
Out of curiosity, what version of TW was the patch released in? I'm curious because I installed 20170724 and encountered the exact same issue (I'd had it create an encrypted home volume.)

In my case, I could still log in as root, just not any regular user.

After updating the /etc/crypttab entry to add the "nvme" to the beginning of the volume name, it works as expected.
Comment 56 Steffen Winterfeldt 2017-07-28 13:40:02 UTC
It's still being evaluated with other fixes in Staging:D:

https://build.opensuse.org/project/show/openSUSE:Factory:Staging:D