Bug 1226415 - data loss: update 15.5 => 15.6 deletes old journal entries
Summary: data loss: update 15.5 => 15.6 deletes old journal entries
Status: RESOLVED FIXED
Alias: None
Product: openSUSE Distribution
Classification: openSUSE
Component: Basesystem (show other bugs)
Version: Leap 15.6
Hardware: Other Other
: P1 - Urgent : Critical (vote)
Target Milestone: ---
Assignee: systemd maintainers
QA Contact: E-mail List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-06-17 05:18 UTC by Stefan Seyfried
Modified: 2024-07-12 09:50 UTC (History)
4 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
/var/log/journal filesystem statistics (16.14 KB, image/png)
2024-06-17 05:18 UTC, Stefan Seyfried
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Stefan Seyfried 2024-06-17 05:18:26 UTC
Created attachment 875512 [details]
/var/log/journal filesystem statistics

I updated my server from 15.5 to 15.6 yesterday and encountered a critical data loss:

My journal is configured to have its own file system for /var/log/journal (8GB) to keep it from eating too much disk space, the config in /etc/systemd/journald.conf is like this:

[Journal]
SystemMaxFileSize=8M
SystemMaxFiles=1024
SystemMaxUse=6666M
SystemKeepFree=10%

After the update, the journal was pruned from 6,5GB to 850MB, removing all files older than one months (see attached screenshot)

server:~ # ls -l /var/log/journal/63ab585aac31189cc7872956493ee6b3/ -t | tail
-rw-r-----  1 root systemd-journal 8388608 Mai 21 07:54 system@62ccf578371040008472a383bf2e369b-00000000004f25f0-000618e2f318a881.journal
-rw-r-----+ 1 root systemd-journal 8388608 Mai 21 07:54 user-10329@00a919c7569e4d03a5a88dc16264a4d4-00000000004f264f-000618e347c0f293.journal
-rw-r-----  1 root systemd-journal 8388608 Mai 20 15:48 system@62ccf578371040008472a383bf2e369b-00000000004f1160-000618d40ffb0c6d.journal
-rw-r-----+ 1 root systemd-journal 8388608 Mai 20 15:48 user-10329@00a919c7569e4d03a5a88dc16264a4d4-00000000004f117c-000618d422c788ee.journal
-rw-r-----+ 1 root systemd-journal 8388608 Mai 19 22:02 user-10329@00a919c7569e4d03a5a88dc16264a4d4-00000000004f0154-000618cbae280621.journal
-rw-r-----  1 root systemd-journal 8388608 Mai 19 22:02 system@62ccf578371040008472a383bf2e369b-00000000004efaa0-000618cbaa041e71.journal
-rw-r-----  1 root systemd-journal 8388608 Mai 19 12:01 system@62ccf578371040008472a383bf2e369b-00000000004ee2ab-000618bd84364db7.journal
-rw-r-----+ 1 root systemd-journal 8388608 Mai 19 12:01 user-10329@00a919c7569e4d03a5a88dc16264a4d4-00000000004ee2f8-000618bda6f3a682.journal
-rw-r-----  1 root systemd-journal 8388608 Mai 18 19:08 system@62ccf578371040008472a383bf2e369b-00000000004ecd0a-000618ade47d8e5b.journal
-rw-r-----+ 1 root systemd-journal 8388608 Mai 18 19:08 user-10329@00a919c7569e4d03a5a88dc16264a4d4-00000000004ecd92-000618ae6f97b723.journal
Comment 1 Stefan Seyfried 2024-06-17 05:24:50 UTC
This is "journalctl -u systemd-journald.service" from the boot doing the update and the one after the update:

-- Boot e02bb8c9070e4203a25fff312819c90a --
Jun 11 10:10:07 server systemd-journald[178]: Journal started
Jun 11 10:10:07 server systemd-journald[178]: Runtime Journal (/run/log/journal/63ab585aac31189cc7872956493ee6b3) is 8.0M, max 792.4M, 784.4M free.
Jun 11 10:10:10 server systemd-journald[178]: Journal stopped
Jun 11 10:10:10 server systemd-journald[659]: Journal started
Jun 11 10:10:10 server systemd-journald[659]: Runtime Journal (/run/log/journal/63ab585aac31189cc7872956493ee6b3) is 8.0M, max 792.4M, 784.4M free.
Jun 11 10:10:10 server systemd[1]: systemd-journald.service: Deactivated successfully.
Jun 11 10:10:11 server systemd-journald[659]: Time spent on flushing to /var/log/journal/63ab585aac31189cc7872956493ee6b3 is 17.007ms for 1194 entries.
Jun 11 10:10:11 server systemd-journald[659]: System Journal (/var/log/journal/63ab585aac31189cc7872956493ee6b3) is 6.5G, max 6.5G, 0B free.
Jun 16 09:55:13 server systemd-journald[659]: Journal stopped
Jun 16 09:55:13 server systemd-journald[24981]: Collecting audit messages is disabled.
Jun 16 09:55:13 server systemd-journald[24981]: File /var/log/journal/63ab585aac31189cc7872956493ee6b3/system.journal corrupted or uncleanly shut down, renaming and replacing.
Jun 16 09:55:13 server systemd-journald[24981]: Journal started
Jun 16 09:55:13 server systemd-journald[24981]: System Journal (/var/log/journal/63ab585aac31189cc7872956493ee6b3) is 800.4M, max 818.2M, 17.7M free.
Jun 16 10:00:16 server systemd-journald[24981]: File /var/log/journal/63ab585aac31189cc7872956493ee6b3/user-10329.journal corrupted or uncleanly shut down, renaming and replacing.
-- Boot 8b04d845e4144a64a73b3479dac75401 --
Jun 16 10:06:33 server systemd-journald[189]: /etc/systemd/journald.conf:52: Failed to parse size value, ignoring: 10%
Jun 16 10:06:33 server systemd-journald[189]: Collecting audit messages is disabled.
Jun 16 10:06:33 server systemd-journald[189]: Journal started
Jun 16 10:06:33 server systemd-journald[189]: Runtime Journal (/run/log/journal/63ab585aac31189cc7872956493ee6b3) is 8.0M, max 792.6M, 784.6M free.
Jun 16 10:06:36 server systemd-journald[189]: Journal stopped
Jun 16 10:06:37 server systemd-journald[691]: /etc/systemd/journald.conf:52: Failed to parse size value, ignoring: 10%
Jun 16 10:06:37 server systemd-journald[691]: Collecting audit messages is disabled.
Jun 16 10:06:37 server systemd-journald[691]: Journal started
Jun 16 10:06:37 server systemd-journald[691]: Runtime Journal (/run/log/journal/63ab585aac31189cc7872956493ee6b3) is 8.0M, max 792.6M, 784.6M free.
Jun 16 10:06:37 server systemd[1]: systemd-journald.service: Deactivated successfully.
Jun 16 10:06:40 server systemd-journald[691]: Time spent on flushing to /var/log/journal/63ab585aac31189cc7872956493ee6b3 is 78.220ms for 1279 entries.
Jun 16 10:06:40 server systemd-journald[691]: System Journal (/var/log/journal/63ab585aac31189cc7872956493ee6b3) is 817.2M, max 6.5G, 5.7G free.
Jun 16 10:06:40 server systemd-journald[691]: Received client request to flush runtime journal.
Jun 16 10:06:40 server systemd-journald[691]: /var/log/journal/63ab585aac31189cc7872956493ee6b3/system.journal: Journal file uses a different sequence number ID, rotating.
Jun 16 10:06:40 server systemd-journald[691]: Rotating system journal.
Jun 16 10:06:54 server systemd-journald[691]: /var/log/journal/63ab585aac31189cc7872956493ee6b3/user-10329.journal: Journal file uses a different sequence number ID, rotating.

Note that at June 15, 9:55:13 it does not even complain about the invalid 10% value, so it seems to ignore the system config during the update.
Comment 2 Stefan Seyfried 2024-06-17 05:37:47 UTC
This is the journal leading to and shortly after the catastrophic event:

Jun 16 09:55:07 server [RPM][24457]: Transaction ID 666e9a5b started
Jun 16 09:55:07 server [RPM][24457]: erase systemd-249.17-150400.8.40.1.x86_64: success
Jun 16 09:55:07 server dbus-daemon[1017]: [system] Reloaded configuration
Jun 16 09:55:07 server dbus-daemon[1017]: [system] Reloaded configuration
Jun 16 09:55:07 server systemd[1]: Reexecuting.
Jun 16 09:55:07 server systemd[1]: systemd 254.10+suse.84.ge8d77af424 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA -SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Jun 16 09:55:07 server systemd[1]: Detected architecture x86-64.
Jun 16 09:55:07 server systemd[1]: bpf-lsm: BPF LSM hook not enabled in the kernel, BPF LSM not supported
Jun 16 09:55:07 server systemd-sysv-generator[24483]: SysV service '/etc/init.d/after.local' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust.
Jun 16 09:55:07 server systemd-sysv-generator[24483]: SysV service '/etc/init.d/boot.local' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust.
Jun 16 09:55:07 server systemd[1]: /usr/lib/systemd/system/rpc-statd.service:14: PIDFile= references a path below legacy directory /var/run/, updating /var/run/rpc.statd.pid → /run/rpc.statd.pid; please update the unit file accordingly.
Jun 16 09:55:08 server systemd[1]: user@10329.service: Sent signal SIGRTMIN+25 to main process 1719 (systemd) on client request.
Jun 16 09:55:08 server systemd[1719]: Received SIGRTMIN+25 from PID 1 (systemd).
Jun 16 09:55:08 server systemd[1]: user@0.service: Sent signal SIGRTMIN+25 to main process 15704 (systemd) on client request.
Jun 16 09:55:08 server systemd[15704]: Received SIGRTMIN+25 from PID 1 (systemd).
Jun 16 09:55:08 server systemd[15704]: Reexecuting.
Jun 16 09:55:08 server systemd[1719]: Reexecuting.
Jun 16 09:55:08 server systemd-xdg-autostart-generator[24494]: /home/seife/.config/autostart/signal-desktop.desktop: stat() failed, ignoring: No such file or directory
Jun 16 09:55:08 server systemd-xdg-autostart-generator[24494]: Exec binary '/usr/bin/setxkbmap' does not exist: No such file or directory
Jun 16 09:55:08 server systemd-xdg-autostart-generator[24494]: /home/seife/.config/autostart/nocapslock.desktop: not generating unit, executable specified in Exec= does not exist.
Jun 16 09:55:08 server systemd[1]: Reloading requested from client PID 24496 ('systemctl') (unit session-5.scope)...
Jun 16 09:55:08 server systemd[1]: Reloading...
Jun 16 09:55:08 server systemd-sysv-generator[24561]: SysV service '/etc/init.d/after.local' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust.
Jun 16 09:55:08 server systemd-sysv-generator[24561]: SysV service '/etc/init.d/boot.local' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust.
Jun 16 09:55:08 server systemd[1]: /usr/lib/systemd/system/rpc-statd.service:14: PIDFile= references a path below legacy directory /var/run/, updating /var/run/rpc.statd.pid → /run/rpc.statd.pid; please update the unit file accordingly.
Jun 16 09:55:09 server systemd[1]: Reloading finished in 619 ms.
Jun 16 09:55:09 server systemd[1]: Reloading requested from client PID 24564 ('systemctl') (unit session-5.scope)...
Jun 16 09:55:09 server systemd[1]: Reloading...
Jun 16 09:55:09 server systemd-sysv-generator[24625]: SysV service '/etc/init.d/after.local' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust.
Jun 16 09:55:09 server systemd-sysv-generator[24625]: SysV service '/etc/init.d/boot.local' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust.
Jun 16 09:55:09 server systemd[1]: /usr/lib/systemd/system/rpc-statd.service:14: PIDFile= references a path below legacy directory /var/run/, updating /var/run/rpc.statd.pid → /run/rpc.statd.pid; please update the unit file accordingly.
Jun 16 09:55:09 server systemd[1]: Reloading finished in 623 ms.
Jun 16 09:55:09 server systemd[1]: Reloading requested from client PID 24627 ('systemctl') (unit session-5.scope)...
Jun 16 09:55:09 server systemd[1]: Reloading...
Jun 16 09:55:10 server syncthing[1797]: [HOJT3] INFO: Connection from KERB3P5-MIHC6UE-3FYFUME-L6HTVSX-ZN35Z5Y-FKXUSMD-BPPHAJW-S7JG6AD at 83.240.5.132:22067 (relay-client) rejected: already connected to this device
Jun 16 09:55:10 server systemd-sysv-generator[24688]: SysV service '/etc/init.d/after.local' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust.
Jun 16 09:55:10 server systemd-sysv-generator[24688]: SysV service '/etc/init.d/boot.local' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust.
Jun 16 09:55:10 server systemd[1]: /usr/lib/systemd/system/rpc-statd.service:14: PIDFile= references a path below legacy directory /var/run/, updating /var/run/rpc.statd.pid → /run/rpc.statd.pid; please update the unit file accordingly.
Jun 16 09:55:10 server systemd[1]: Reloading finished in 676 ms.
Jun 16 09:55:10 server systemd[1]: Reloading requested from client PID 24691 ('systemctl') (unit session-5.scope)...
Jun 16 09:55:10 server systemd[1]: Reloading...
Jun 16 09:55:10 server systemd-sysv-generator[24751]: SysV service '/etc/init.d/after.local' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust.
Jun 16 09:55:10 server systemd-sysv-generator[24751]: SysV service '/etc/init.d/boot.local' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust.
Jun 16 09:55:10 server systemd[1]: /usr/lib/systemd/system/rpc-statd.service:14: PIDFile= references a path below legacy directory /var/run/, updating /var/run/rpc.statd.pid → /run/rpc.statd.pid; please update the unit file accordingly.
Jun 16 09:55:11 server systemd[1]: Reloading finished in 620 ms.
Jun 16 09:55:11 server systemd[1]: Reloading requested from client PID 24760 ('systemctl') (unit session-5.scope)...
Jun 16 09:55:11 server systemd[1]: Reloading...
Jun 16 09:55:11 server systemd-sysv-generator[24826]: SysV service '/etc/init.d/after.local' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust.
Jun 16 09:55:11 server systemd-sysv-generator[24826]: SysV service '/etc/init.d/boot.local' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust.
Jun 16 09:55:11 server systemd[1]: /usr/lib/systemd/system/rpc-statd.service:14: PIDFile= references a path below legacy directory /var/run/, updating /var/run/rpc.statd.pid → /run/rpc.statd.pid; please update the unit file accordingly.
Jun 16 09:55:11 server systemd[1]: Reloading finished in 649 ms.
Jun 16 09:55:11 server systemd[1]: Reloading requested from client PID 24828 ('systemctl') (unit session-5.scope)...
Jun 16 09:55:11 server systemd[1]: Reloading...
Jun 16 09:55:12 server systemd-sysv-generator[24889]: SysV service '/etc/init.d/after.local' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust.
Jun 16 09:55:12 server systemd-sysv-generator[24889]: SysV service '/etc/init.d/boot.local' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust.
Jun 16 09:55:12 server systemd[1]: /usr/lib/systemd/system/rpc-statd.service:14: PIDFile= references a path below legacy directory /var/run/, updating /var/run/rpc.statd.pid → /run/rpc.statd.pid; please update the unit file accordingly.
Jun 16 09:55:12 server systemd[1]: Reloading finished in 726 ms.
Jun 16 09:55:12 server [RPM][24457]: install systemd-254.10-150600.2.3.x86_64: success
Jun 16 09:55:12 server dbus-daemon[1017]: [system] Reloaded configuration
Jun 16 09:55:12 server dbus-daemon[1017]: [system] Reloaded configuration
Jun 16 09:55:12 server systemd[1]: Reloading requested from client PID 24919 ('systemctl') (unit session-5.scope)...
Jun 16 09:55:12 server systemd[1]: Reloading...
Jun 16 09:55:12 server (sd-execu[24966]: /usr/lib/systemd/system-generators/systemd-bless-boot-generator failed with exit status 127.
Jun 16 09:55:13 server systemd[1]: /usr/lib/systemd/system/rpc-statd.service:14: PIDFile= references a path below legacy directory /var/run/, updating /var/run/rpc.statd.pid → /run/rpc.statd.pid; please update the unit file accordingly.
Jun 16 09:55:13 server systemd[1]: systemd-rfkill.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functional until restarted.
Jun 16 09:55:13 server systemd[1]: systemd-initctl.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functional until restarted.
Jun 16 09:55:13 server systemd[1]: Reloading finished in 662 ms.
Jun 16 09:55:13 server systemd-journald[659]: Journal stopped
Jun 16 09:55:13 server systemd-journald[659]: Received SIGTERM from PID 1 (systemd).
Jun 16 09:55:13 server systemd[1]: Stopping Journal Service...
Jun 16 09:55:13 server systemd[1]: systemd-journald.service: Deactivated successfully.
Jun 16 09:55:13 server systemd[1]: Stopped Journal Service.
Jun 16 09:55:13 server systemd[1]: Starting Journal Service...
Jun 16 09:55:13 server systemd-journald[24981]: Collecting audit messages is disabled.
Jun 16 09:55:13 server systemd-journald[24981]: File /var/log/journal/63ab585aac31189cc7872956493ee6b3/system.journal corrupted or uncleanly shut down, renaming and replacing.
Jun 16 09:55:13 server systemd-journald[24981]: Journal started
Jun 16 09:55:13 server systemd-journald[24981]: System Journal (/var/log/journal/63ab585aac31189cc7872956493ee6b3) is 800.4M, max 818.2M, 17.7M free.
Jun 16 09:55:13 server systemd[1]: Started Journal Service.
Jun 16 09:55:13 server systemd[1]: Reloading requested from client PID 24982 ('systemctl') (unit session-5.scope)...
Jun 16 09:55:13 server systemd[1]: Reloading...
Jun 16 09:55:13 server (sd-execu[25030]: /usr/lib/systemd/system-generators/systemd-bless-boot-generator failed with exit status 127.
Jun 16 09:55:14 server systemd[1]: /usr/lib/systemd/system/rpc-statd.service:14: PIDFile= references a path below legacy directory /var/run/, updating /var/run/rpc.statd.pid → /run/rpc.statd.pid; please update the unit file accordingly.
Jun 16 09:55:14 server systemd[1]: systemd-rfkill.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functional until restarted.
Jun 16 09:55:14 server systemd[1]: systemd-initctl.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functional until restarted.
Jun 16 09:55:14 server systemd[1]: Reloading finished in 596 ms.
Jun 16 09:55:14 server [RPM][24457]: erase systemd-249.17-150400.8.40.1.x86_64: success
Jun 16 09:55:14 server [RPM][24457]: Transaction ID 666e9a5b finished: 0
Comment 3 Andreas Stieger 2024-06-17 08:14:33 UTC
Assign to Base:System/systemd maintainers
Comment 4 Franck Bui 2024-06-20 09:55:58 UTC
Would it be possible that before you upgraded your system, the amount of free space on your disk was relavtively low already and the low limit defined by SystemKeepFree= was reached when zypper was downloading the new versions of the packages ?
Comment 5 Franck Bui 2024-06-20 09:58:01 UTC
Also did you notice the following message ?

> systemd-journald[189]: /etc/systemd/journald.conf:52: Failed to parse size value, ignoring: 10%

Does line 52 correspond to "SystemKeepFree=10%" or something else ?
Comment 6 Stefan Seyfried 2024-06-20 12:26:44 UTC
No.
The config is: extra partiton for /var/log/journal, to not interfere with the rest of the system and not be interfered by e.g. "full root fs" causing loss of journal data.

server:~ # df -m /var/log/journal/
Filesystem                             1M-blocks  Used Available Use% Mounted on
/dev/mapper/m2--256g-var--log--journal      8182   895      7288  11% /var/log/journal

The "SystemKeepFree=10%" was actually an invalid config but that one just got ignored.
"SystemMaxUse" is totally enough to make that work. See the longterm statistics diagram attached to this bug.

I'm pretty sure what happened was:

* systemd got updated
  * journald.conf got removed, preserved as journald.conf.rpmsave
  * journald got restarted in systemd's %post
  * no custom journald.conf => defaults are used.
    >>>> DATA LOSS HAPPENS HERE <<<
    because the defaults are so that max 10% of the filesystem is used...
    ...80% of the old logs were destroyed.
  * journald.conf.rpmsave is renamed to journald.conf in %posttrans
    => this is way too late, data is lost.

This is extracted from my typescript created during update:

( 152/1152) Installing: systemd-devel-254.10-150600.2.3.x86_64 ...........[done]
/usr/lib/tmpfiles.d/systemd-suse.conf:10: Duplicate line for path "/run/lock", ignoring.
after-local.service is no more provided by systemd but a copy has been installed in /etc
warning: /etc/systemd/logind.conf saved as /etc/systemd/logind.conf.rpmsave
warning: /etc/systemd/journald.conf saved as /etc/systemd/journald.conf.rpmsave
Failed to try-restart systemd-timesyncd.service: Unit systemd-timesyncd.service not found.
( 153/1152) Installing: systemd-254.10-150600.2.3.x86_64 .................[done]
...
(1152/1152) Installing: virt-manager-4.1.0-150600.10.1.noarch ............[done]
%posttrans(openssl-1_1-1.1.1w-150600.3.10.x86_64) script output:
mv: cannot stat '/etc/ssl/engines1.1.d.rpmsave/*': No such file or directory
mv: cannot stat '/etc/ssl/engdef1.1.d.rpmsave/*': No such file or directory
%posttrans(systemd-254.10-150600.2.3.x86_64) script output:
Restoring /etc/systemd/journald.conf. Please consider moving your customizations in a drop-in instead.
For more details, visit https://en.opensuse.org/Systemd#Configuration.
renamed '/etc/systemd/journald.conf.rpmsave' -> '/etc/systemd/journald.conf'
Restoring /etc/systemd/logind.conf. Please consider moving your customizations in a drop-in instead.
For more details, visit https://en.opensuse.org/Systemd#Configuration.
renamed '/etc/systemd/logind.conf.rpmsave' -> '/etc/systemd/logind.conf'
%posttrans(udev-254.10-150600.2.3.x86_64) script output:
...

systemd-journald gets try-restarted in %postun, so this cannot even be fixed in the new package, as it is done by the old package already... 

You probably should refuse to update the package completely if the config has been changed from the default with the warning about "You need to move your customizations to  a drop-in"
Comment 7 Stefan Seyfried 2024-06-20 12:27:24 UTC
(In reply to Franck Bui from comment #5)
> Also did you notice the following message ?
> 
> > systemd-journald[189]: /etc/systemd/journald.conf:52: Failed to parse size value, ignoring: 10%
> 
> Does line 52 correspond to "SystemKeepFree=10%" or something else ?

yes
Comment 8 Franck Bui 2024-06-20 15:16:01 UTC
(In reply to Stefan Seyfried from comment #6)
> I'm pretty sure what happened was:

I think you're right.

> 
> * systemd got updated
>   * journald.conf got removed, preserved as journald.conf.rpmsave

Unfortunately journald.conf.rpmsave is created during the removal of the old package (at least last time I checked). But I need to double check whether it's done before or after the execution of %postun

>   * journald got restarted in systemd's %post
>   * no custom journald.conf => defaults are used.
>     >>>> DATA LOSS HAPPENS HERE <<<
>     because the defaults are so that max 10% of the filesystem is used...
>     ...80% of the old logs were destroyed.

or maybe because SystemMaxFiles= dropped from 1024 to 100.

>   * journald.conf.rpmsave is renamed to journald.conf in %posttrans
>     => this is way too late, data is lost.
> 
> systemd-journald gets try-restarted in %postun, so this cannot even be fixed
> in the new package, as it is done by the old package already... 
> 
> You probably should refuse to update the package completely if the config
> has been changed from the default with the warning about "You need to move
> your customizations to  a drop-in"

That would be pretty rude.

The only way to fix it I can see currently would be to restart journald from %posttrans after restoring journald.conf.

Or another option is to restore /etc/systemd/journald.conf (maybe as a ghost file) in the new package and suggests the user to convert it to a drop-in.
Comment 9 Franck Bui 2024-06-20 15:17:43 UTC
(In reply to Franck Bui from comment #8)
> The only way to fix it I can see currently would be to restart journald from
> %posttrans after restoring journald.conf.


Argh no since as you already mentioned %postun of the old package can't be modified.
Comment 10 Franck Bui 2024-06-20 15:30:22 UTC
Maybe another option: in %pre, copy journald.conf in /run/systemd and clean it up from %posttran of the new package.
Comment 11 Franck Bui 2024-06-24 09:00:51 UTC
(In reply to Franck Bui from comment #8)
> Or another option is to restore /etc/systemd/journald.conf (maybe as a ghost
> file) in the new package and suggests the user to convert it to a drop-in.

Finally we decided to follow this path as it's the safest one (there's no more risk to screw it up a second time).

I'll submit the fix today.
Comment 14 Franck Bui 2024-06-24 12:27:00 UTC
FTR, I just submitted the fix via sr#336505.
Comment 15 Franck Bui 2024-06-24 12:28:37 UTC
Stefan, I'm sorry for the troubles and hope that you didn't loose too important logs.

The problem should be fix with the next update thanks to your report.

Again sorry.
Comment 16 Joao Walter Bruno Filho 2024-06-24 12:40:01 UTC
Re-prioritizing the bug as it seems it affects 15 SP6.
Comment 18 OBSbugzilla Bot 2024-06-24 15:35:04 UTC
This is an autogenerated message for OBS integration:
This bug (1226415) was mentioned in
https://build.opensuse.org/request/show/1183029 Factory / systemd
Comment 19 Stefan Seyfried 2024-06-25 04:45:52 UTC
(In reply to Franck Bui from comment #15)
> Stefan, I'm sorry for the troubles and hope that you didn't loose too
> important logs.

I have a profound distrust in systemd-related things, so I keep rsyslog around, for reliable logging ;-)

> The problem should be fix with the next update thanks to your report.

Thanks, I still have a 15.5 system to update where I will test this ;-)

> Again sorry.

bugs happen, that's life :-) Thanks for fixing it.

This will, even more important for me with my SLES customer hat on, save trouble on the upcoming updates to SLES15-SP6 a few months from now.
Comment 20 Maintenance Automation 2024-06-26 12:30:02 UTC
SUSE-RU-2024:2239-1: An update that has one fix can now be installed.

Category: recommended (critical)
Bug References: 1226415
Maintenance Incident: [SUSE:Maintenance:34275](https://smelt.suse.de/incident/34275/)
Sources used:
openSUSE Leap 15.6 (src):
 systemd-mini-254.13-150600.4.5.1, systemd-254.13-150600.4.5.1
Basesystem Module 15-SP6 (src):
 systemd-254.13-150600.4.5.1
SUSE Package Hub 15 15-SP6 (src):
 systemd-254.13-150600.4.5.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 21 Franck Bui 2024-07-01 12:19:41 UTC
The fix has been released hence I'm closing the bug.

Thanks again Stefan for reporting this issue back.