Bugzilla – Bug 1213788
systemd units under /usr/local don't work properly
Last modified: 2023-08-16 11:00:20 UTC
/usr/local/lib/systemd/system is a location where systemd sources system units, as stated in systemd.unit(5). On a default install of openSUSE Tumbleweed, with a root filesystem of type btrfs, and with snapshots, /usr/local is mounted as a subvolume to avoid it being modified in case of an OS rollback. This combination causes systemd units under /usr/local/lib/systemd/system to not work properly. Service and timer units stored there are found, and can be enabled, but they don't actually run as expected on reboot. If ext4 is used as root filesystem, it works as expected. Tested on Tumbleweed 20230728.
Just to be clear: This isn't an issue with btrfs or snapshotting in itself. The same issue occurs if a plain partition is mounted at /usr/local. I only describe the subvolume mount in the error description because it's what you get after a standard install.
I guess units cannot be installed in /usr/local if it's a separate mount point: - Enabling a unit creates a link in /etc/systemd/system to the unit path (/usr/local/systemd/system in this case) - Mount units are created in /run/systemd/generator (we care about usr-local.mount in this case). - systemd looks for unit files in (higher priority first): > Aug 04 12:53:07 nm systemd[1]: Looking for unit files in (higher priority first): > Aug 04 12:53:07 nm systemd[1]: /etc/systemd/system.control > Aug 04 12:53:07 nm systemd[1]: /run/systemd/system.control > Aug 04 12:53:07 nm systemd[1]: /run/systemd/transient > Aug 04 12:53:07 nm systemd[1]: /run/systemd/generator.early > Aug 04 12:53:07 nm systemd[1]: /etc/systemd/system ---> multi-user.target.wants/test-usr-local.service > Aug 04 12:53:07 nm systemd[1]: /etc/systemd/system.attached > Aug 04 12:53:07 nm systemd[1]: /run/systemd/system > Aug 04 12:53:07 nm systemd[1]: /run/systemd/system.attached > Aug 04 12:53:07 nm systemd[1]: /run/systemd/generator ---> usr-local.mount > Aug 04 12:53:07 nm systemd[1]: /usr/local/lib/systemd/system > Aug 04 12:53:07 nm systemd[1]: /usr/lib/systemd/system > Aug 04 12:53:07 nm systemd[1]: /run/systemd/generator.late Hence, the unit installed in /usr/local/lib/systemd/system is not found when parsing /etc/systemd/system, because usr-local.mount runs later. I'm not sure if this is something that can be fixed or it's just a documentation issue.
On transactional systems, /etc is a fairly complicated mounted filesystem, yet it is still possible to put unit files in it. So if it can be done for /etc, why not for /usr/local, too?
(In reply to Olav Reinert from comment #0) > /usr/local/lib/systemd/system is a location where systemd sources system > units, as stated in systemd.unit(5). If you put unit files in /usr/local you need to tag this directory to be mounted earlier in initrd as described in https://www.suse.com/releasenotes/x86_64/SUSE-SLES/15-SP2/index.html#systemsManagement. For some reasons this release note hasn't been forward ported to more recent versions of SLE, that's probably a bug. systemd could automatically mount /usr/local in initrd like it does for /usr but upstream didn't accept it: https://github.com/systemd/systemd/pull/19408.
(In reply to Franck Bui from comment #4) > If you put unit files in /usr/local you need to tag this directory to be > mounted earlier in initrd as described in > https://www.suse.com/releasenotes/x86_64/SUSE-SLES/15-SP2/index. > html#systemsManagement. Thank you for the tip, this solves the issue for me. As far as I am concerned, this bug can be closed now. > For some reasons this release note hasn't been forward ported to more recent > versions of SLE, that's probably a bug. If you wish, you can also keep this bug report open and repurpose it to track the documentation bug.
(In reply to Olav Reinert from comment #6) > If you wish, you can also keep this bug report open and repurpose it to > track the documentation bug. Yep and I asked the question to someone who might be able to shed some light. Thanks.
Having read the discussion on the upstream bug report, I agree with Pöttering that /usr/local should not automatically be mounted in initrd, but only when admins ask for it (because, supposedly, they know what they're doing).
It seems I must have done something wrong when I tested the proposed solution (from the SLE 15 SP2 release notes), because it doesn't actually work. The correct mount option is "x-initrd.mount", as stated in the systemd.mount(5) man page.
(In reply to Olav Reinert from comment #9) > The correct mount option is "x-initrd.mount", as stated in the > systemd.mount(5) man page. Thanks for spotting, we'll fix the release notes.
I opened bsc#1214328 to fix the documentation issues. Closing this report. Thanks Olav for the report.