Bugzilla – Bug 1216070
Assignment of /dev/sd* has altered dramatically
Last modified: 2023-11-07 19:59:27 UTC
After recent updates several users have reported that the assignment of /dev/sd* has changed. Whatever has perturbed the allocation of /dev/sd* is recent. It's well documented that the order of assignment of /dev/sd* cannot be relied upon. But it has been 100% stable for years. It turns out that some configs, scripts, and examples based on utilities such as hdparm and smartd, may have been depending on the ordering being stable. Additionally where admins had hard coded /dev/sd, such as /etc/fstab and grub.conf, these changes had also been successful for years. It would be desirable to provide some breathing space to allow systems to boot and run while purged of persistent references to /dev/sd*.
I believe that this issue has affected both my TW and now my Leap 15.6 grub ordering . . . such that systems that are let's say installed in sdb drive are now showing as being in sdbc drive, and vice versa. Running grub2-mkconfig xxxx sometimes shows proper location and function, but this morning systems were again booting to ER mode, and running grub2-mkconfig brought disordered listings . . . and dysfunctional booting to OS, etc. As Michael mentions, this is a new phenomenon . . . whether it is based in "history" . . . up until several days back grub could find and boot my 7 linux installs w/o losing track of them with each boot or restart. Now, with each restart grub is showing different system locations.
Without changing anything else, I tried rebooting to 6.5.3-1-default (no rollbacks of TW - I use ext4 for /). I booted/rebooted 6.5.3-1-default around 5 times, every boot resulted in the old assignments of /dev/sd*. So perhaps this issue is something that arrived in 6.5.4-1-default. Additionally, for 6.5.4-1-default, I tried adding the following to the boot line: scsi_mod.disable_async_probing=sd,ahci,pcieport But the ordering of /dev/sd* still varied from one boot to the next. There's a bit of luck involved as to what the ordering will be, multiple boots need to be attempted, both cold and hot. It's possible I was just lucky with 6.5.3-1-default, but it was five times with all the drives in the old ordering. So it's probably worth seeing if anything relevant changed after 6.5.3-1-default.
Comparing the journals from 6.5.3-1-default and 6.5.4-1-default boots. On 6.5.3-1-default, the initialisations of /dev/sd* appears to happen very early in the boot and pretty much on its own. On the other hand in 6.5.4-1-default, the same initialisation appears sometime later in the boot, it's interleaved with initialising usb, xhci_hcd, and others.
According to a recent forum post, the cause is the change to load SCSI drivers as modules, this has changed the loading order. That's what I see in the boot logs, they now load later. That change was for the following bug: https://bugzilla.suse.com/show_bug.cgi?id=1215182
(In reply to Michael Hamilton from comment #4) > According to a recent forum post, the cause is the change to load SCSI > drivers as modules, this has changed the loading order. That's what I see > in the boot logs, they now load later. > > That change was for the following bug: > https://bugzilla.suse.com/show_bug.cgi?id=1215182 That should read "According to a recent mailing list post..."
For tumbleweed, the relevant change is SUSE kernel commit 08fb080 ("Update config files: make SCSI/ATA drivers modules again"), which was merged in 6.5.4. This patch converted scsi_mod, sd_mod ahci, and some other modules from built-in to modules. This affects the order in which drivers are initialized. In particular, USB devices may now be scanned first. Partial workaround: use modprobe.d to make sure ahci is loaded before usb-storage, like this: softdep usb_storage pre: ahci This softdep needs to be added in the initrd, too, so run dracut after changing it. Anyway, it has been mentioned multiple times in this discussion, but once more: ** don't rely on /dev/sd* device names ** This has been unreliable for years, and you have simply been lucky if you haven't seen issues so far. Closing as WONTFIX.
The Leap 15.6 issues reported by Fritz must have a different reason. Please open a separate bug.
(In reply to Martin Wilck from comment #7) > The Leap 15.6 issues reported by Fritz must have a different reason. Please > open a separate bug. @Martin: The "problem" hit my TW system first, and out of frustration with the problems I had been having with TW over several weeks I moved grub/osprober services over to Leap . . . where this disordering of grub continued. Haven't had any more time to pursue exploration of yet another problem, but if "TW" is the common parameter in this report, it was happening there. I'll tag along on this bug report and see if anything pans out on a fix for it.
(In reply to Fritz Hudnut from comment #9) > (In reply to Martin Wilck from comment #7) > > The Leap 15.6 issues reported by Fritz must have a different reason. Please > > open a separate bug. > > @Martin: > > The "problem" hit my TW system first, and out of frustration with the > problems I had been having with TW over several weeks I moved grub/osprober > services over to Leap . . . where this disordering of grub continued. > > Haven't had any more time to pursue exploration of yet another problem, but > if "TW" is the common parameter in this report, it was happening there. > I'll tag along on this bug report and see if anything pans out on a fix for > it. I'm not that familiar with Leap, but according to https://bugzilla.suse.com/show_bug.cgi?id=1215182#c9 "TW has built-in SCSI drivers while SLE/Leap has modules". Now that TW also changed to use modules, both TW and Leap will both exhibit similar "disordering". So isn't the cause the same (but implemented at different times)?
(In reply to Michael Hamilton from comment #10) > Now that TW also changed to use modules, both TW and Leap will both exhibit > similar "disordering". So isn't the cause the same (but implemented at > different times)? It's possible. But on Leap, these "problems" would have existed basically forever. It can't be a 15.6 regression.
Hm, looking again at https://forums.opensuse.org/t/problem-with-disks-order-after-snapshot-20230921/169324/13: > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-0HP_LOGICAL_VOLUME_00000000 -> ../../sdb > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-0HP_LOGICAL_VOLUME_01000000 -> ../../sda > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-0HP_LOGICAL_VOLUME_02000000 -> ../../sdc > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-3600508b1001c21d488c454a46bc39f22 -> ../../sdc > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-3600508b1001c99233458581ffb65cc88 -> ../../sdb > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-3600508b1001cd3e7527deb3b931a6639 -> ../../sda > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-SHP_LOGICAL_VOLUME_0014380280B60D0 -> ../../sdb > lrwxrwxrwx 1 root root 9 Sep 23 14:44 wwn-0x600508b1001c21d488c454a46bc39f22 -> ../../sdc > lrwxrwxrwx 1 root root 9 Sep 23 14:44 wwn-0x600508b1001c99233458581ffb65cc88 -> ../../sdb > lrwxrwxrwx 1 root root 9 Sep 23 14:44 wwn-0x600508b1001cd3e7527deb3b931a6639 -> ../../sda The workaround I suggested in comment 6 won't help here. All these disks are logical volumes on the same controller. No USB is involved. It's puzzling that the order of these disks changes because some drivers have been converted from built-in to modules, at least at first sight. My hand-waving guess is that if `ahci`, `scsi_mod` and `sd_mod` are built-in, the sd devices are probed in the order the hardware devices (PCI controllers, SATA ports) are detected, and because the hardware detection takes some time, the probing order appears deterministic (async probing of first sd device is finished before the 2nd SATA port appears, etc). OTOH, if `sd_mod` is a module, it will be requested and loaded by the kernel when ahci detects the first SCSI disk device. The loading will take some time, and it's not unlikely that ahci will have finished probing all hardware before sd_mod is ready to probe devices. These devices will then be probed asynchronously, and consequently the ordering will appear non-deterministic. If this theory is correct, you should be able to work around the issue by adding sd_mod to /etc/modules-load.d/ (and rebuilding the initrd). Please test that.
(In reply to Martin Wilck from comment #12) > Hm, looking again at > https://forums.opensuse.org/t/problem-with-disks-order-after-snapshot- > 20230921/169324/13: > > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-0HP_LOGICAL_VOLUME_00000000 -> ../../sdb > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-0HP_LOGICAL_VOLUME_01000000 -> ../../sda > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-0HP_LOGICAL_VOLUME_02000000 -> ../../sdc > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-3600508b1001c21d488c454a46bc39f22 -> ../../sdc > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-3600508b1001c99233458581ffb65cc88 -> ../../sdb > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-3600508b1001cd3e7527deb3b931a6639 -> ../../sda > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-SHP_LOGICAL_VOLUME_0014380280B60D0 -> ../../sdb > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 wwn-0x600508b1001c21d488c454a46bc39f22 -> ../../sdc > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 wwn-0x600508b1001c99233458581ffb65cc88 -> ../../sdb > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 wwn-0x600508b1001cd3e7527deb3b931a6639 -> ../../sda > > The workaround I suggested in comment 6 won't help here. All these disks are > logical volumes on the same controller. No USB is involved. > > It's puzzling that the order of these disks changes because some drivers > have been converted from built-in to modules, at least at first sight. > > My hand-waving guess is that if `ahci`, `scsi_mod` and `sd_mod` are > built-in, the sd devices are probed in the order the hardware devices (PCI > controllers, SATA ports) are detected, and because the hardware detection > takes some time, the probing order appears deterministic (async probing of > first sd device is finished before the 2nd SATA port appears, etc). OTOH, if > `sd_mod` is a module, it will be requested and loaded by the kernel when > ahci detects the first SCSI disk device. The loading will take some time, > and it's not unlikely that ahci will have finished probing all hardware > before sd_mod is ready to probe devices. These devices will then be probed > asynchronously, and consequently the ordering will appear non-deterministic. > > If this theory is correct, you should be able to work around the issue by > adding sd_mod to /etc/modules-load.d/ (and rebuilding the initrd). > > Please test that. I got a bugzilla needinfo email. Your quote from the forum was from forum-user phil524, I'm not sure they are participating in this bugzilla bug. Are you suggesting this new proposed work around is also applicable more generally to my system? Given the delays due to participants being in different timezones, I will proceed and report back in case that's of any help, so this is what I will shortly do: echo sd_mod > /etc/modules-load.d/sd_mod.conf dracut -f --regenerate-all In respect to timings, looking at the journals: with the modules built-in, logging about /dev/sda happens in the first second of logged messages; where as modules not built in doesn't log anything about /dev/sda until about 3 seconds into the boot and a lot of other things have happened as well by then. BTW, after doing: echo 'softdep usb_storage pre: ahci' >/etc/modprobe.d/10-ahci-scsi.conf dracut -f --regenerate-all three boots have all come up with the same old ordering of /dev/sd*
(In reply to Michael Hamilton from comment #13) > (In reply to Martin Wilck from comment #12) > > Hm, looking again at > > https://forums.opensuse.org/t/problem-with-disks-order-after-snapshot- > > 20230921/169324/13: > > > > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-0HP_LOGICAL_VOLUME_00000000 -> ../../sdb > > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-0HP_LOGICAL_VOLUME_01000000 -> ../../sda > > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-0HP_LOGICAL_VOLUME_02000000 -> ../../sdc > > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-3600508b1001c21d488c454a46bc39f22 -> ../../sdc > > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-3600508b1001c99233458581ffb65cc88 -> ../../sdb > > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-3600508b1001cd3e7527deb3b931a6639 -> ../../sda > > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 scsi-SHP_LOGICAL_VOLUME_0014380280B60D0 -> ../../sdb > > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 wwn-0x600508b1001c21d488c454a46bc39f22 -> ../../sdc > > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 wwn-0x600508b1001c99233458581ffb65cc88 -> ../../sdb > > > lrwxrwxrwx 1 root root 9 Sep 23 14:44 wwn-0x600508b1001cd3e7527deb3b931a6639 -> ../../sda > > > > The workaround I suggested in comment 6 won't help here. All these disks are > > logical volumes on the same controller. No USB is involved. > > > > It's puzzling that the order of these disks changes because some drivers > > have been converted from built-in to modules, at least at first sight. > > > > My hand-waving guess is that if `ahci`, `scsi_mod` and `sd_mod` are > > built-in, the sd devices are probed in the order the hardware devices (PCI > > controllers, SATA ports) are detected, and because the hardware detection > > takes some time, the probing order appears deterministic (async probing of > > first sd device is finished before the 2nd SATA port appears, etc). OTOH, if > > `sd_mod` is a module, it will be requested and loaded by the kernel when > > ahci detects the first SCSI disk device. The loading will take some time, > > and it's not unlikely that ahci will have finished probing all hardware > > before sd_mod is ready to probe devices. These devices will then be probed > > asynchronously, and consequently the ordering will appear non-deterministic. > > > > If this theory is correct, you should be able to work around the issue by > > adding sd_mod to /etc/modules-load.d/ (and rebuilding the initrd). > > > > Please test that. > > I got a bugzilla needinfo email. Your quote from the forum was from > forum-user phil524, I'm not sure they are participating in this bugzilla > bug. Are you suggesting this new proposed work around is also applicable > more generally to my system? > > Given the delays due to participants being in different timezones, I will > proceed and report back in case that's of any help, so this is what I will > shortly do: > > echo sd_mod > /etc/modules-load.d/sd_mod.conf > dracut -f --regenerate-all > > In respect to timings, looking at the journals: with the modules built-in, > logging about /dev/sda happens in the first second of logged messages; where > as modules not built in doesn't log anything about /dev/sda until about 3 > seconds into the boot and a lot of other things have happened as well by > then. > > BTW, after doing: > > echo 'softdep usb_storage pre: ahci' >/etc/modprobe.d/10-ahci-scsi.conf > dracut -f --regenerate-all > > three boots have all come up with the same old ordering of /dev/sd* What I did: mv /etc/modprobe.d/10-ahci-scsi.conf ~/ # disable previous fix echo sd_mod > /etc/modules-load.d/sd_mod.conf dracut -f --regenerate-all lsinitrd | less # check for /etc/modules-load.d/sd_mod.conf I then booted several times, hot and cold. Each time the system has ordered the disks as they used be ordered, so this work around seems to work (or I've been exceptionally lucky). Looking athe the journal, I see that /dev/sda is still initialised later in the boot, in the 2nd second, after other stuff, like some of the usb modules. It seems like a nicer work around because it doesn't reference usb_storage.
(In reply to Michael Hamilton from comment #13) > I got a bugzilla needinfo email. Your quote from the forum was from > forum-user phil524, I'm not sure they are participating in this bugzilla > bug. Are you suggesting this new proposed work around is also applicable > more generally to my system? > I had the impression that "phil524" was you. I am not sure any more why, I guess I was just confused. Sorry about that. > BTW, after doing: > > echo 'softdep usb_storage pre: ahci' >/etc/modprobe.d/10-ahci-scsi.conf > dracut -f --regenerate-all > > three boots have all come up with the same old ordering of /dev/sd* So in your case, the changed ordering had been caused by USB drives listed before others? Otherwise, I still don't understand why this would help. (In reply to Michael Hamilton from comment #14) > What I did: > > mv /etc/modprobe.d/10-ahci-scsi.conf ~/ # disable previous fix > echo sd_mod > /etc/modules-load.d/sd_mod.conf > dracut -f --regenerate-all > lsinitrd | less # check for /etc/modules-load.d/sd_mod.conf > > I then booted several times, hot and cold. Each time the system has ordered > the disks as they used be ordered, so this work around seems to work (or > I've been exceptionally lucky). Looking athe the journal, I see that > /dev/sda is still initialised later in the boot, in the 2nd second, after > other stuff, like some of the usb modules. It seems like a nicer work > around because it doesn't reference usb_storage. Very good, thanks. I'll post this alternative workaround in the forum, too. I don't think it will always be effective, but it will be helpful in a larger set of cases than the USB-specific workaround. I think so, at least.
(In reply to Martin Wilck from comment #15) > ... > > > BTW, after doing: > > > > echo 'softdep usb_storage pre: ahci' >/etc/modprobe.d/10-ahci-scsi.conf > > dracut -f --regenerate-all > > > > three boots have all come up with the same old ordering of /dev/sd* > > So in your case, the changed ordering had been caused by USB drives listed > before others? Otherwise, I still don't understand why this would help. > I suspect the intent of the softdep is just to force an earlier initialisation of /dev/sd* rather than having anything specific to do with usb. For some reason moving the /dev/sd* stuff earlier resulted in the old behaviour. Certainly, after applying this fix, the journal logged /dev/sd* related messages about as early as the older built-in kernel. As a mentioned earlier, your latest suggest fix, didn't move the /dev/sd* to the front of the order of events, but it did have the desired outcome.
(In reply to Michael Hamilton from comment #17) > > I suspect the intent of the softdep is just to force an earlier > initialisation of /dev/sd* rather than having anything specific to do with > usb. A softdep usb-storage -> ahci only affects the loading order of these drivers. "sd_mod" is independent, and it will only be loaded if SCSI disks (as opposed to other devices like tapes) are encountered on the SCSI bus. So if there are no USB SCSI disks present, I wouldn't expect an effect. But I am obviously overlooking something.
Adding sd_mod to /etc/modules-load.d/ has restored the behaviour in my setup too. (I know now the enumerating isn't deterministic/guaranteed and I do use labels, but it's still useful to have the disks enumerated the same way it's been for the past 16 years, easier debugging and fixing from a live cd, rescue system ...etc :)). Thanks.
[QUOTE]What I did: mv /etc/modprobe.d/10-ahci-scsi.conf ~/ # disable previous fix echo sd_mod > /etc/modules-load.d/sd_mod.conf dracut -f --regenerate-all lsinitrd | less # check for /etc/modules-load.d/sd_mod.conf [/QUOTE] I did these steps, did not initially seem to make a difference to my multiboot desktop . . . then I remembered that the last time I had efibootmgr problems due to interaction with OSX and linux systems in a Mac . . . I removed the 4th drive. Removing the 4th drive formatted with Mac's APFS system seemed to allow grub to reorder the linux systems in their proper places. So far I have been able to cold boot and restart into a few of the 7 installs with consistency AND, more importantly the GParted data within each system seems to again be showing the correct sdb vs sdc locations. Possibly this is all due to running the "sd_mod" data OR both that and removing the drive OR removing the drive. Once I get through booting into each system I **might** try to add the 4th internal drive back into the machine and see if the new grub can handle the additional drive in its capacity.
I'd like to ask everyone to test yet a different workaround. 1. Remove any previous workarounds, in particular the one from comment 12 2. echo 'softdep scsi_mod post: sd_mod sg' >/etc/modprobe.d/sd.conf 3. (optional) ln -s /dev/null /etc/modules-load.d/sg.conf 4. (optional, and only!! if you don't need dm-multipath) ln -s /dev/null /etc/modules-load.d/multipath.conf 5. make a backup copy of the current initrd, just in case 6. dracut -f Then reboot and verify that a) disk ordering is as expected b) sd_mod and sg are loaded below any low-level SCSI drivers such as ahci or usb-storage in /proc/modules If this works, it would be much nicer than the current "modules-load.d" approach, because it would make it possible to do entirely without SCSI modules on systems that don't need it (NVMe/M.2 laptops, for example). I just tested it on my laptop, it works nicely. Still, if I connect an usb-storage drive, the SCSI modules required for it are properly auto-loaded. Previously, we couldn't use this approach because scsi_mod was built-in on TW. (bug 1036463, comment 46). But now we can, if this test is successful.
The workaround in #26 works for me, what I did: Reversed the previous suggestion (as root): 1) rm /etc/modules-load.d/sd_mod.conf 2) dracut -f --regenerate-all 3) Rebooted and confirmed /dev/sd* was reordered/unordered Applied the minimum from the new suggestion: 1) echo 'softdep scsi_mod post: sd_mod sg' >/etc/modprobe.d/sd.conf 2) dracut -f 3) Rebooted and found /dev/sd* was ordered as it used to be. 4) Rebooted again, just in case I was lucky in the last step. 5) Check that boot times were satisfactory: systemd-analyze critical-chain Looks good. Under what circumstances should I consider using either of the optional steps?
(In reply to Michael Hamilton from comment #27) Thanks a lot!! > > Under what circumstances should I consider using either of the optional > steps? This is for me, and for future improvement of openSUSE ;-) If you don't use dm-multipath, you should be able to take these steps with no risk. The rationale is as follows: Step 3 from comment 26 disables /usr/lib/modules-load.d/sg.conf, which loads the "sg" driver early on (which pulls in scsi_mod). Step 4 disables /usr/lib/modules-load.d/multipath.conf, which also causes scsi_mod to be loaded very early, as a dependency of the multipath modules. By disabling these, we test the "clean" case where some real SCSI controller (say ahci or usb-storage) is found, scsi_mod is loaded, and sd_mod is pulled in via the softdep. This is what I wanted to test; in particular whether the loading order is correct and thus the ordering of sd devices is what you expect (to the extent in which this is possible, we discussed that before). Currently every openSUSE system loads a bunch of SCSI drivers (sg, scsi_mod, scsi_dh_...) always, no matter whether these are actually needed on the system or not. By replacing the "modules-load.d" entries with softdeps on scsi_mod, it's possible to avoid that. Systems that have SCSI hardware will load scsi_mod and the other drivers that come with it. Other systems will not. This allows us to keep the system nice and lean in the non-SCSI case, without loosing functionality for SCSI systems.
(In reply to Martin Wilck from comment #28) > (In reply to Michael Hamilton from comment #27) > > Thanks a lot!! > > > > > Under what circumstances should I consider using either of the optional > > steps? > > This is for me, and for future improvement of openSUSE ;-) If you don't use > dm-multipath, you should be able to take these steps with no risk. Oh, OK, happy to test these too. I've now applied all of them, specifically: echo 'softdep scsi_mod post: sd_mod sg' >/etc/modprobe.d/sd.conf ln -s /dev/null /etc/modules-load.d/multipath.conf ln -s /dev/null /etc/modules-load.d/sg.conf dracut -f And I've done a boot and still see the good-old stable ordering. I'm not that familiar with all the drivers, I will state my interpretations of your explanations below. I now know that sd_mod is the middle level scsi driver responsible for /dev/sd* - good to know. So we want that pulled in if scsi_mod, the core scsi module is loaded. > > The rationale is as follows: > > Step 3 from comment 26 disables /usr/lib/modules-load.d/sg.conf, which loads > the "sg" driver early on (which pulls in scsi_mod). As detailed in the man page, linking to /dev/null disables/overrides any other vendor supplied config file - but it's not clearly stated what the link does to the actual driver. I think this means sg (general scsi) no longer loads early, but may load on demand if needed. Because the sd_mod driver already covers disks, sg would only be needed for other scsi devices like CDROM drives, so that should work fine for later on-demand loading (what about booting from CDROM?). > Step 4 disables /usr/lib/modules-load.d/multipath.conf, which also causes > scsi_mod to be loaded very early, as a dependency of the multipath modules. Similar, but in this case the multi-path I/O mapper used by fancy I/O load balanced systems with multiple I/O channels will also not load early - which I guess might be problem if a system has some kind of fancy I/O spreading for the root volume? But most of us aren't in that league. > > By disabling these, we test the "clean" case where some real SCSI controller > (say ahci or usb-storage) is found, scsi_mod is loaded, and sd_mod is pulled > in via the softdep. This is what I wanted to test; in particular whether the > loading order is correct and thus the ordering of sd devices is what you > expect (to the extent in which this is possible, we discussed that before). So my interpretation of the softdep is correct. > > Currently every openSUSE system loads a bunch of SCSI drivers (sg, scsi_mod, > scsi_dh_...) always, no matter whether these are actually needed on the > system or not. By replacing the "modules-load.d" entries with softdeps on > scsi_mod, it's possible to avoid that. Systems that have SCSI hardware will > load scsi_mod and the other drivers that come with it. Other systems will > not. This allows us to keep the system nice and lean in the non-SCSI case, > without loosing functionality for SCSI systems. So this could be a win for more than just the /dev/sd* order fixated amongst us, our pedantry benefits everyone :-)
(In reply to Michael Hamilton from comment #29) > > Step 3 from comment 26 disables /usr/lib/modules-load.d/sg.conf, which loads > > the "sg" driver early on (which pulls in scsi_mod). > > As detailed in the man page, linking to /dev/null disables/overrides any > other vendor supplied config file - but it's not clearly stated what the > link does to the actual driver. I think this means sg (general scsi) no > longer loads early, but may load on demand if needed. Almost. the scsi-mod softdep that we added would cause "sg" to be loaded whenever scsi_mod is loaded. This is not exactly "on demand". > Because the sd_mod > driver already covers disks, sg would only be needed for other scsi devices > like CDROM drives, so that should work fine for later on-demand loading > (what about booting from CDROM?). This is yet another difficult topic. "sg" is a special driver; it attaches to SCSI devices _besides_ the actual driver (which is "sd_mod" for disks, "st" for tapes, etc.), and allows sending low-level SCSI commands to the devices from user space. This matters most for devices that have no default driver, like tape libraries with cartridge-changing features. Historically, sg was also necessary to send SCSI commands to standard devices like disks. This isn't necessary any more, because "sd" devices support the sg driver's ioctls, too. But we are still loading "sg" early on, because there is some uncertainty whether some 3rd party tools might rely on it being present. Last time this has been discussed was in 2019 (bug 1036463), and we concluded that we aren't aware of any users but we aren't certain, so we kept loading sg early. Perhaps we should re-evaluate this topic now, too. By moving "sg" from modules-load.d to a softdep, we can at least make sure that "sg" isn't loaded when there are no SCSI devices at all. "On demand" loading of sg itself is not possible, because the logic would be "load sg if a scsi device is found for which we have _no_ specific driver", and the kernel doesn't have an autoload mechanism like that. (Kernel autoloading of SCSI drivers works by the "scsi:t-*" modaliases, like "scsi:t-0x01*" for "st"). For SCSI CD/DVD devices, sg isn't required these days, sr_mod has all necessary features. > > Step 4 disables /usr/lib/modules-load.d/multipath.conf, which also causes > > scsi_mod to be loaded very early, as a dependency of the multipath modules. > > Similar, but in this case the multi-path I/O mapper used by fancy I/O load > balanced systems with multiple I/O channels will also not load early - which > I guess might be problem if a system has some kind of fancy I/O spreading > for the root volume? But most of us aren't in that league. This is true. The dm-multipath driver will still need to be loaded if multipath is in use, and it will pull in scsi_mod. But multipath-tools can just be uninstalled on systems that don't need it. Ideally, we'd find a way to load the dm-multipath kernel module only if multipathd.service is enabled. > [...] > > So this could be a win for more than just the /dev/sd* order fixated amongst > us, our pedantry benefits everyone :-) Exactly ;-)
(In reply to Martin Wilck from comment #30) > (In reply to Michael Hamilton from comment #29) > ... > > > > So this could be a win for more than just the /dev/sd* order fixated amongst > > us, our pedantry benefits everyone :-) > > Exactly ;-) Thanks for taking the time to clarify what the work-around is actually doing. I booted a couple more times today, all seems well.
Gents: I followed the most recent suggestions provided here, and so far grub ordering seems to be "OK" . . . have not booted through all of the now 6 OSs that grub is controlling via Leap 15.6 install. It does not appear to be "worse" than it was; have yet to plug in an ext drive, or plug the 4th internal drive back in again. These would be the further testing yet to be done. Not sure if any of the original changes made had anything to do with my Leap install failing to revive from suspend . . . but that has persisted through 3 or 4 kernel upgrades. I even tried to install sddm, which the "qt6" version fixed a problem of "black screen" and "slow to log in" in my TW install . . . I was hoping that it would get Leap back in the game . . . did not prevail. @Michael: Thanks for your explanations and hints provided to get this sdX problem figured and resolved . . . . We possibly might be out of the woods on it??
@All, an experimental suse-module-tools package with the workaround from comment 26 (without the multipath stuff) is available on OBS: https://download.opensuse.org/repositories/home:/mwilck:/suse-module-tools/openSUSE_Tumbleweed/ Installing this should have the same effect as steps 1.-3., 6. in comment 26 (so backup your initrd before installing it!). Currently TW only. The expected effect is that the ordering of sd devices stabilizes again. Testers welcome (please undo manual workarounds that you may have applied so far).
On systems with only NVMe hardware (e.g. laptops with M.2 SSDs) where multipath-tools is not installed (if you aren’t using dm-multipath, you can just uninstall it with “zypper rm multipath-tools”), an additional expected effect is that the system would come up without any SCSI drivers loaded (check /pproc/modules). If an USB stick or some other SCSI device is added to such a system, SCSI modules should be loaded on demand, and the stick should be mounted as usual.
(In reply to Martin Wilck from comment #33) > @All, > > an experimental suse-module-tools package with the workaround from comment > 26 (without the multipath stuff) is available on OBS: > > https://download.opensuse.org/repositories/home:/mwilck:/suse-module-tools/ > openSUSE_Tumbleweed/ > > Installing this should have the same effect as steps 1.-3., 6. in comment 26 > (so backup your initrd before installing it!). > > Currently TW only. > > The expected effect is that the ordering of sd devices stabilizes again. > Testers welcome (please undo manual workarounds that you may have applied so > far). Thanks heaps for stepping back and taking a harder look at this issue. I wasn't sure exactly how to proceed, here is did: I reversed out the previous workaround: 1 rm /etc/modprobe.d/sd.conf 2 rm /etc/modules-load.d/multipath.conf /etc/modules-load.d/sg.conf 3 dracut -f --regenerate-all 4 Rebooted to confirm past unpredictable assignment, I found had to use --regenerate-all. After just dracut -f, two boots followed where the order remained stable. Maybe I was just unlucky, anyway, after adding --regenerate-all, the follow boot showed the old unpredictable behaviour. I installed the new workaround as follows: 0 Copied the working initrd. My recovery plan was to boot an older kernel and put it back. 1 Manually downloaded the two rpms from the URL to a local folder 2 rpm --upgrade workaround/suse-module-tools-*.rpm 3 dracut -f --regenerate-all 4 rebooted and confirmed expected/predictable order Looks good. Maybe I should have used zypper. Would that have done the dracut for me? I don't use multipath. I assumed removing multipath isn't something that needs to be done. What advantage would this have over leaving it in place?
(The postinstall scriptlets are baked into the rpm package, so installing it with zypper or with the rpm command directly should have the same effect).
(In reply to Ahmad Samir from comment #36) > (The postinstall scriptlets are baked into the rpm package, so installing it > with zypper or with the rpm command directly should have the same effect). The rpm install ran too quickly to have done a dracut - instant. I confirmed this just now by restoring the original suse-mod packages and the backup of the initrd. After booting into the restored system, booting the /dev/sd order was again unstable. I did an md5sum of the initrd. I then reinstalled the workaround packages and checked whether the initrd updated, it had not, dates and md5sums were as before. I decided to reboot anyway, and /dev/sd order was still disordered. Then I did a dracut -f and rebooted, and /dev/sd was in stable order. If the suse-mod packages are supposed to post-install run dracut, they didn't for me, at least not when using rpm.
Manually running dracut -f --regenerate-all was correct. --regenerate-all is only needed if you want to recreate *every* initial RAM disk. For the test, it should be sufficient to regenerate just the one for the current kernel, which is what a simple "dracut -f" does. Therefore I didn't ask you to use this parameter, but you can of course if you prefer. Currently suse-module-tools does not run dracut in the %posttrans scriptlet. I am unsure if it should. One potential issue is that the regenerate-initrd-posttrans script itself is part of suse-module-tools, so if we aren't extremely careful, we'll run into an hen-and-egg issue. In general, I think that on openSUSE we have rather too many packages that want to rebuild the initrd. Every initrd rebuild poses an (arguably small) risk that the subsequent reboot will fail. In the case of suse-module-tools, an initrd rebuild is rarely necessary when it's updated. In this specific case, in particular for running a meaningful test, it's necessary.
I've been out of town and haven't had time to read any of the recent posts. Last week, before I left, I did a fresh install of Slow Roll, which traditionally I have used the fresh install of the bootloader as a way to "reorder" grub, and on first couple of reboots it seemed to be "working" as far as listing the 7 installs by order of their location in sda, sdb, and then finally the sdc drives. Today, however, the "sdc7" ubuntu install was showing higher up in the "sda7" location . . . which I do have a / in sda7, so I figured it would boot something . . . rather than ubuntu, it went to TTY ER mode . . . rebooted from that and it went to top listing of SloRoll . . . . I'll have to read through to see if there is a new solution to what appears to be a continuing problem with the grub ordering. I did not run any of the suggested "fixes" provided here as I did in the Leap 15.6 install, so possibly that is the reason for the "disordered grub condition" that has now afflicted the Slo Roll install??? I haven't booted 15.6 system since the new install, soo I haven't disabled os-prober over there yet, conceivably I could boot that system and run "update-bootloader" and see if that would "re-order" grub?? From my impressions of scanning the latest posts, the apparent fix that Michael was using have been revised???
So, I did try to get back to Leap 15.6 . . . this time grub was showing it in "sdc7" which is NOT the correct location, but it did boot to Leap. I ran "update-bootloader" and shut down. On cold boot the list was still "disordered." I then ran the "grub2-mkconfig xxxxx" command [in Leap] and I got the following list, which is showing the correct sdX locations . . . but now is missing the new Slo Roll install, which I think should be in sdb somewhere. I note the inclusion of the "81.589146 multipath driver not loaded" line. This was after I had run a zypper dup that installed two new kernel options and a "macros"???? package. Am about to shut down and see if the grub2-mkconfig in Leap will return "order" to grub OR if I have to figure out how to install the "multipath" driver that seems was discussed in this bug report by Martin . . . . [CODE] Generating grub configuration file ... Found linux image: /boot/vmlinuz-6.5.9-lp155.4.g1823166-default Found initrd image: /boot/initrd-6.5.9-lp155.4.g1823166-default Found linux image: /boot/vmlinuz-6.5.8-lp155.3.g51baea8-default Found initrd image: /boot/initrd-6.5.8-lp155.3.g51baea8-default Found linux image: /boot/vmlinuz-6.5.7-lp155.5.g4e7c99b-default Found initrd image: /boot/initrd-6.5.7-lp155.5.g4e7c99b-default Found linux image: /boot/vmlinuz-6.5.7-lp155.3.g9fcbf99-default Found initrd image: /boot/initrd-6.5.7-lp155.3.g9fcbf99-default Found linux image: /boot/vmlinuz-5.14.21-150500.55.28-default Found initrd image: /boot/initrd-5.14.21-150500.55.28-default Warning: os-prober will be executed to detect other bootable partitions. Its output will be used to detect bootable binaries on them and create new boot entries. 81.589146 | DM multipath kernel driver not loaded Found Debian GNU/Linux trixie/sid on /dev/sda6 Found openSUSE Tumbleweed on /dev/sda7 Found Manjaro Linux (23.0.4) on /dev/sdb7 Found Mac OS X on /dev/sdc2 Found Debian GNU/Linux trixie/sid on /dev/sdc6 Found Ubuntu 23.10 (23.10) on /dev/sdc7 done [/CODE]
(In reply to Fritz Hudnut from comment #40) > So, I did try to get back to Leap 15.6 . . . this time grub was showing it > in "sdc7" which is NOT the correct location, but it did boot to Leap. > ... The only relevant comments with fixes are currently: Comment #29 which should work for Leap, TW, and Slowroll. Comment #33+#35 for Tumbleweed only (for now), it tests the approach for a more "official" fix. I would not bother trying anything other than the above.
(In reply to Michael Hamilton from comment #41) > (In reply to Fritz Hudnut from comment #40) > > So, I did try to get back to Leap 15.6 . . . this time grub was showing it > > in "sdc7" which is NOT the correct location, but it did boot to Leap. > > ... > > The only relevant comments with fixes are currently: > > Comment #29 which should work for Leap, TW, and Slowroll. > > Comment #33+#35 for Tumbleweed only (for now), it tests the approach for a > more "official" fix. > > I would not bother trying anything other than the above. @Michael: Thanks for the reply . . . I did do the #29 process in my Leap 15.6 install, which first seemed to work, but then didn't. I'm not quite getting the "rpm" command to download and install the latest packages . . . slightly beyond my paygrade. BUT, I did go back to TW Slo Roll . . . ran a zypper and then ran the "grub2-mkconfig xxxxx" command and that did show everybody in their proper places, and all of the installs, and nothing about a missing multipath driver. I ran #update-bootloader just for good measure ans shut down. On cold boot everything was listed properly in grub . . . . So, for now Slo Roll grub is OK . . . I guess that is an intermittent condition until we try to install these "multipath" drivers? Or, will zypper pull them in at some point??
(In reply to Fritz Hudnut from comment #40) > ... > > Thanks for the reply . . . I did do the #29 process in my Leap 15.6 install, > which first seemed to work, but then didn't. One aspect of the problem is that the effect is random, so you can just be lucky and on one boot things boot the way you prefer, but they may differ on the next. I haven't experienced that fix failing, so I can't comment on that. > > I'm not quite getting the "rpm" command to download and install the latest > packages . . . slightly beyond my paygrade. I manually downloaded the rpms, then used rpm on the downloaded files (rpm doesn't do downloads). > > BUT, I did go back to TW Slo Roll . . . ran a zypper and then ran the > "grub2-mkconfig xxxxx" command and that did show everybody in their proper > places, and all of the installs, and nothing about a missing multipath > driver. I ran #update-bootloader just for good measure ans shut down. On > cold boot everything was listed properly in grub . . . . So, for now Slo > Roll grub is OK . . . I guess that is an intermittent condition until we try > to install these "multipath" drivers? Or, will zypper pull them in at some > point?? I've no idea what kernel Slowroll is currently using, but Slowroll is an older version of TW. So doing anything on Slowroll may have no effect because it may not yet have this issue and doesn't need "fixing". Multipath has nothing to do with achieving a fix, so don't go there. (Dm-multipath is a special case, and, at one point, it was stated that the fix was only definitely safe for non multipath users).
(In reply to Michael Hamilton from comment #43) > (In reply to Fritz Hudnut from comment #40) > > ... > > > > Thanks for the reply . . . I did do the #29 process in my Leap 15.6 install, > > which first seemed to work, but then didn't. > > One aspect of the problem is that the effect is random, so you can just be > lucky and on one boot things boot the way you prefer, but they may differ on > the next. > > I haven't experienced that fix failing, so I can't comment on that. > > > > > I'm not quite getting the "rpm" command to download and install the latest > > packages . . . slightly beyond my paygrade. > > I manually downloaded the rpms, then used rpm on the downloaded files (rpm > doesn't do downloads). > > > > > BUT, I did go back to TW Slo Roll . . . ran a zypper and then ran the > > "grub2-mkconfig xxxxx" command and that did show everybody in their proper > > places, and all of the installs, and nothing about a missing multipath > > driver. I ran #update-bootloader just for good measure ans shut down. On > > cold boot everything was listed properly in grub . . . . So, for now Slo > > Roll grub is OK . . . I guess that is an intermittent condition until we try > > to install these "multipath" drivers? Or, will zypper pull them in at some > > point?? > > I've no idea what kernel Slowroll is currently using, but Slowroll is an > older version of TW. So doing anything on Slowroll may have no effect > because it may not yet have this issue and doesn't need "fixing". > > Multipath has nothing to do with achieving a fix, so don't go there. > (Dm-multipath is a special case, and, at one point, it was stated that the > fix was only definitely safe for non multipath users). @Michael: Thanks for the conversation . . . I am experiencing the intermittent nature of the problem. It did seem to hit Slow Roll, that's why I tried to revive grub on Leap edition, but that didn't "work" even though I believe I had applied #29 fixes previously, etc. I haven't been kicking in SUSE long enough to know what "rpm" is, is that something like "wget" in Deb/Ubun, except it doesn't "get" the files, but just runs them by searching for .rpm files??? and then installing them??? : - )) Thanks for the comment about "multipath" . . . thought about it, but time is still very limited for playing around with basic function stuff.
(In reply to Fritz Hudnut from comment #44) > (In reply to Michael Hamilton from comment #43) > > (In reply to Fritz Hudnut from comment #40) > > > ... > > > > > > Thanks for the reply . . . I did do the #29 process in my Leap 15.6 install, > > > which first seemed to work, but then didn't. > > > > One aspect of the problem is that the effect is random, so you can just be > > lucky and on one boot things boot the way you prefer, but they may differ on > > the next. > > > > I haven't experienced that fix failing, so I can't comment on that. > > > > > > > > I'm not quite getting the "rpm" command to download and install the latest > > > packages . . . slightly beyond my paygrade. > > > > I manually downloaded the rpms, then used rpm on the downloaded files (rpm > > doesn't do downloads). > > > > > > > > BUT, I did go back to TW Slo Roll . . . ran a zypper and then ran the > > > "grub2-mkconfig xxxxx" command and that did show everybody in their proper > > > places, and all of the installs, and nothing about a missing multipath > > > driver. I ran #update-bootloader just for good measure ans shut down. On > > > cold boot everything was listed properly in grub . . . . So, for now Slo > > > Roll grub is OK . . . I guess that is an intermittent condition until we try > > > to install these "multipath" drivers? Or, will zypper pull them in at some > > > point?? > > > > I've no idea what kernel Slowroll is currently using, but Slowroll is an > > older version of TW. So doing anything on Slowroll may have no effect > > because it may not yet have this issue and doesn't need "fixing". > > > > Multipath has nothing to do with achieving a fix, so don't go there. > > (Dm-multipath is a special case, and, at one point, it was stated that the > > fix was only definitely safe for non multipath users). > > @Michael: > > Thanks for the conversation . . . I am experiencing the intermittent nature > of the problem. It did seem to hit Slow Roll, that's why I tried to revive > grub on Leap edition, but that didn't "work" even though I believe I had > applied #29 fixes previously, etc. > > I haven't been kicking in SUSE long enough to know what "rpm" is, is that > something like "wget" in Deb/Ubun, except it doesn't "get" the files, but > just runs them by searching for .rpm files??? and then installing them??? : > - )) > > Thanks for the comment about "multipath" . . . thought about it, but time is > still very limited for playing around with basic function stuff. This is probably not the place for walking through the basics, we're just cluttering up the bug comments and spamming maintainers :-) Maybe take the questions back to the real forum.
This is an autogenerated message for OBS integration: This bug (1216070) was mentioned in https://build.opensuse.org/request/show/1121300 Factory / suse-module-tools
(In reply to Michael Hamilton from comment #45) > > > This is probably not the place for walking through the basics, we're just > cluttering up the bug comments and spamming maintainers :-) Maybe take the > questions back to the real forum. Indeed. Well, this problem is a continuing problem, yesterday running grub2-mkconfig in Slo Roll brought "proper order" to the list, and then this morning on cold boot . . . back to disorder. The last two months or so have been a continuing saga of engaging one problem after another in my various SUSE installs . . . haven't had time to follow the ins and outs of this one, but it is definitely messing with "work flow." I'm "OK" with reporting that the fix applied from the earlier comment has not prevailed in maintaining proper grub order in this bug report.
I'm closing this bug now, because the fix for the /dev/sdX ordering issue, which was the topic of this bug has reached factory. I won't comment on Fritz' grub2 issues, which are unrelated. If there is an actual problem that's worth reporting as a bug, please open a new one, and provide a concise, precise description of what you experience and why you believe it's broken, together with detailed system information, one system after the other.