(In reply to Nick Singer from comment #0)
> With the drbd-utils update to 9.19.0 on Mon Oct 11 09:43:25 UTC 2021 the
> /lib/drbd/scripts/drbd directory was moved to /usr (see line#130 in
> https://build.opensuse.org/package/view_file/network:ha-clustering:Factory/
> drbd-utils/drbd-utils.changes?expand=1).

on factory, drbd-utils had been udpated to v9.25.0. above line number is wrong.
but below info is still available.

> 
> Soon later a fix for bsc#1203220 was introduced because drbd.service now
> fails with `drbd.service: Failed at step EXEC spawning
> /lib/drbd/scripts/drbd: No such file or directory` and a symlink to /usr was
> created. Apparently this fix never made it into Leap15.4 backports:
> 
> ```
> osiris-1:~ # cat /etc/os-release
> NAME="openSUSE Leap"
> VERSION="15.4"
> ID="opensuse-leap"
> ID_LIKE="suse opensuse"
> VERSION_ID="15.4"
> PRETTY_NAME="openSUSE Leap 15.4"
> ANSI_COLOR="0;32"
> CPE_NAME="cpe:/o:opensuse:leap:15.4"
> BUG_REPORT_URL="https://bugs.opensuse.org"
> HOME_URL="https://www.opensuse.org/"
> DOCUMENTATION_URL="https://en.opensuse.org/Portal:Leap"
> LOGO="distributor-logo-Leap"
> osiris-1:~ # LC_ALL=c zypper info drbd-utils
> Retrieving repository 'Update repository with updates from SUSE Linux
> Enterprise 15' metadata
> ... ...
> 
> Workaround: `mkdir -p /lib/drbd/ && ln -s /usr/lib/drbd/scripts
> /lib/drbd/scripts`

I don't know how & when back port SLE15-sp4 patches to leap 15.4.

@roger, do you know who is the correct bug owner for this bug.

It was fixed (In reply to Nick Singer from comment #0)
> With the drbd-utils update to 9.19.0 on Mon Oct 11 09:43:25 UTC 2021 the
> /lib/drbd/scripts/drbd directory was moved to /usr (see line#130 in
> https://build.opensuse.org/package/view_file/network:ha-clustering:Factory/
> drbd-utils/drbd-utils.changes?expand=1).
> 
> Soon later a fix for bsc#1203220 was introduced because drbd.service now
> fails with `drbd.service: Failed at step EXEC spawning
> /lib/drbd/scripts/drbd: No such file or directory` and a symlink to /usr was
> created. Apparently this fix never made it into Leap15.4 backports:
> 
> ```
> osiris-1:~ # cat /etc/os-release
> NAME="openSUSE Leap"
> VERSION="15.4"
> ID="opensuse-leap"
> ID_LIKE="suse opensuse"
> VERSION_ID="15.4"
> PRETTY_NAME="openSUSE Leap 15.4"
> ANSI_COLOR="0;32"
> CPE_NAME="cpe:/o:opensuse:leap:15.4"
> BUG_REPORT_URL="https://bugs.opensuse.org"
> HOME_URL="https://www.opensuse.org/"
> DOCUMENTATION_URL="https://en.opensuse.org/Portal:Leap"
> LOGO="distributor-logo-Leap"


> osiris-1:~ # LC_ALL=c zypper info drbd-utils
> Retrieving repository 'Update repository with updates from SUSE Linux
> Enterprise 15' metadata
> .............................................................................
> .........................................................[done]
> Building repository 'Update repository with updates from SUSE Linux
> Enterprise 15' cache
> .............................................................................
> ..............................................................[done]
> Loading repository data...
> Reading installed packages...
> 
> 
> Information for package drbd-utils:
> -----------------------------------
> Repository     : Update repository with updates from SUSE Linux Enterprise 15
> Name           : drbd-utils
> Version        : 9.19.0-150400.3.17.1

I wonder, before upgrade drbd-utils "/lib/drbd" directory was there for some reason?

drbd-utils-9.19.0-150400.3.17.1 is the latest binary and will populate the symlink. Your rpm should have the following.

15sp4-5:~ # rpm -q --scripts drbd-utils

if [ -d /lib/drbd ]; then
  rm -rf /lib/drbd.rpmmoved
  mv /lib/drbd /lib/drbd.rpmmoved
elif [ ! -e /usr/lib/drbd ] && [ -L /lib/drbd ]; then
  rm /lib/drbd
fi
posttrans scriptlet (using /bin/sh):
if [ ! -e /lib/drbd ]; then
  ln -sf /usr/lib/drbd /lib/drbd
fi

(In reply to heming zhao from comment #1)
> (In reply to Nick Singer from comment #0)

> 
> I don't know how & when back port SLE15-sp4 patches to leap 15.4.
> 
> @roger, do you know who is the correct bug owner for this bug.

From my understanding, Leap 15.4 is automatically populated to collect SLE15SP4 maintenance update binaries in the public. That says, the identical drbd-utils binary is used for both distributions.

I download leap 15.4 drbd-utils source code rpm, and differ with latest 15sp4.
the source code is same with 15sp4.

I installed a leap 15.4 VM, then 'zypper in drbd-utils'.
After zypper installing job, I could find /lib/drbd symbol link.

please provide your reproducible steps.

(In reply to heming zhao from comment #5)
> I installed a leap 15.4 VM, then 'zypper in drbd-utils'.
> After zypper installing job, I could find /lib/drbd symbol link.
> 
> please provide your reproducible steps.

I just had an old installation of Leap15.4, did a zypper ref && zypper dup and after a reboot I encountered the issue. The only thing I can imagine causing this is that the (real) folder was still present from the previous running version and the update then failed to create the symlink. So could you maybe try to install an older version, start the drbd.service and then update to check if the symlink is properly created?

(In reply to Nick Singer from comment #0)
> 
> Workaround: `mkdir -p /lib/drbd/ && ln -s /usr/lib/drbd/scripts
> /lib/drbd/scripts`

the correct style of "/lib/drbd" is symbol file. in drbd-utils.spec, the script:
```
%posttrans
%if 0%{?suse_version} < 1550
if [ ! -e /lib/drbd ]; then
  ln -sf %{libdir}/drbd /lib/drbd
fi
%endif
```

in my VM:
```
# ls -l /lib/drbd
lrwxrwxrwx 1 root root 13 Sep 20 16:27 /lib/drbd -> /usr/lib/drbd
```
So please create a symbol file as above

===============================================
I still can't trigger this bug in my VM.
1> download a very beginning drbd version from SLE15-sp4 (drbd-utils-9.19.0-150400.1.7.x86_64.rpm), which creates a real folder "/lib/drbd", and there is no "/usr/lib/drbd" folder.
2> use zypper update drbd-utils to install latest drbd-utils (drbd-utils-9.19.0-150400.3.17.1.x86_64.rpm), the result is still correct.

Apparently this bug went away after we created the symlink in the past. However, we just upgraded the exact same host to Leap15.6 now and again face this problem.

I ensured that /lib/drbd is a symlink pointing to -> /usr/lib/drbd and both paths are on the same btrfs subvolume. So I'm not sure why this is happening again now.

Is c#0 Workaround still available?
steps: `mkdir -p /lib/drbd/ && ln -s /usr/lib/drbd/scripts /lib/drbd/scripts`

You (Nick) are the only one who reports symlink issue since bsc#1203220 was closed.
I suspect you env includes some special settings which blocking drbd running.

could you upload supportconfig file?

```
install: zypper in supportutils

ref:
47.2.4 Creating a supportconfig archive from command line
https://documentation.suse.com/sles/15-SP4/html/SLES-all/cha-adm-support.html
```