Bugzilla – Bug 1223090
[Build 20240417] openQA test fails in await_install: Installation of package filesystem failed
Last modified: 2024-07-12 08:00:04 UTC
Created attachment 874361 [details] y2log ## Observation openQA test in scenario opensuse-Tumbleweed-DVD-s390x-textmode:investigate:retry@s390x-zVM-vswitch-l2 fails in [await_install](https://openqa.opensuse.org/tests/4091989/modules/await_install/steps/87) relevant log: >2024-04-18 09:39:16 <1> o3zvm001(3595) [Pkg] Package.cc(searchPackage):595 Package 'filesystem' installed: false >2024-04-18 09:39:16 <1> o3zvm001(3595) [Pkg] modules/SlideShowCallbacks.rb:305 Pkg Builtin called: PkgDU >2024-04-18 09:39:16 <1> o3zvm001(3595) [Pkg] modules/SlideShowCallbacks.rb:371 Pkg Builtin called: TargetAvailable >2024-04-18 09:39:16 <1> o3zvm001(3595) [librpmDb] RpmDb.cc(doInstallPackage):1705 RpmDb::installPackage(/mnt/var/cache/zypp/packages/openSUSE-20240417-0/s390x/filesystem-84.87-15.3.s390x.rpm,0x0000000c) >2024-04-18 09:39:16 <1> o3zvm001(3595) [zypp::exec++] forkspawnengine.cc(start):274 Executing[C] 'rpm' '--root' '/mnt' '--dbpath' '/var/lib/rpm' '-U' '--percent' '--noglob' '--force' '--nodeps' '--' '/mnt/var/cache/zypp/packages/openSUSE-20240417-0/s390x/filesystem-84.87-15.3.s390x.rpm' >2024-04-18 09:39:16 <1> o3zvm001(3595) [zypp::exec++] forkspawnengine.cc(start):427 pid 5145 launched >2024-04-18 09:39:16 <2> o3zvm001(3595) [zypp::exec] abstractspawnengine.cc(checkStatus):182 Pid 5145 exited with status 1 >2024-04-18 09:39:16 <5> o3zvm001(3595) [zypp-core] Exception.cc(log):202 RpmDb.cc(doInstallPackage):1846 THROW: Subprocess failed. Error: RPM failed: Command exited with status 1. >2024-04-18 09:39:16 <5> o3zvm001(3595) [zypp-core] Exception.cc(log):202 History: >2024-04-18 09:39:16 <5> o3zvm001(3595) [zypp-core] Exception.cc(log):202 - error: lua script failed: [string "%prein(filesystem-84.87-15.3.s390x)"]:22: exit >2024-04-18 09:39:16 <5> o3zvm001(3595) [zypp-core] Exception.cc(log):202 error: filesystem-84.87-15.3.s390x: install failed >2024-04-18 09:39:16 <5> o3zvm001(3595) [zypp-core] Exception.cc(log):202 >2024-04-18 09:39:16 <1> o3zvm001(3595) [Ruby] modules/PackageCallbacks.rb(DonePackage):506 DonePackage(error: 3, reason: 'Subprocess failed. Error: RPM failed: Command exited with status 1. >History: > - error: lua script failed: [string "%prein(filesystem-84.87-15.3.s390x)"]:22: exit > error: filesystem-84.87-15.3.s390x: install failed >') ## Test suite description Installation in textmode and selecting the textmode "desktop" during installation. ## Reproducible Fails since (at least) Build [](https://openqa.opensuse.org/tests/4091989) (current job) ## Expected result Last good: [lastgood](https://openqa.opensuse.org/tests/4054622) (or more recent) ## Further details Always latest result in this scenario: [latest](https://openqa.opensuse.org/tests/latest?arch=s390x&distri=opensuse&flavor=DVD&machine=s390x-zVM-vswitch-l2&test=textmode%3Ainvestigate%3Aretry&version=Tumbleweed)
In what world is this a YaST problem? >2024-04-18 09:39:16 <1> o3zvm001(3595) [zypp::exec++] forkspawnengine.cc(start):274 Executing[C] 'rpm' '--root' '/mnt' '--dbpath' '/var/lib/rpm' '-U' '--percent' '--noglob' '--force' '--nodeps' '--' '/mnt/var/cache/zypp/packages/openSUSE-20240417-0/s390x/filesystem-84.87-15.3.s390x.rpm' >2024-04-18 09:39:16 <5> o3zvm001(3595) [zypp-core] Exception.cc(log):202 RpmDb.cc(doInstallPackage):1846 THROW: Subprocess failed. Error: RPM failed: Command exited with status 1. And why are you attaching the one /var/log/YaST/y2log file?
FWICT this happens if it tries to execute /usr/libexec/convertfs, which it must not do for fresh installs. According to the y2log, filesystem is not installed as one of the first few packages, which might cause this. When the installation fails, please do in an xterm (Ctrl-Alt-Shift-X): rpm --root /mnt -qal | grep -E '^(/bin|/sbin|/lib64|/lib)' To figure out what created files in pre-usrmerge locations. That should be fixed and additionally filesystem should be installed earlier somehow.
And no y2logs tarball at the test case. As you can see, this failed at installing the very first RPM already; which may mean that creating the root filesystem failed, or that there was a problem with that selected DASD. But I am not going to piece that together from data fragments.
Yes, I agree - I was in a rush, sorry for the poor bug report. I was only able to recover this one file quickly enough, before the test died, as for some reason it couldn’t upload logs. I can provide more information tomorrow when I am back in the office.
Please first try what Fabian suggested.
Got a message: 4:o3zvm004:~ # rpm --root /mnt -qal | grep -E '^(/bin|/sbin|/lib64|/lib)' /lib/s390-tools/stage3.bin This is from s390-genprotimg-data, which only contains data files and has no dependencies, not even glibc or filesystem, so it's technically valid to install it before them. It must use /usr/lib/ here instead of the legacy /lib/. Reassigning.
(In reply to Fabian Vogt from comment #6) > Got a message: > > 4:o3zvm004:~ # rpm --root /mnt -qal | grep -E '^(/bin|/sbin|/lib64|/lib)' > /lib/s390-tools/stage3.bin > > This is from s390-genprotimg-data, which only contains data files and has no > dependencies, not even glibc or filesystem, so it's technically valid to > install it before them. > > It must use /usr/lib/ here instead of the legacy /lib/. Reassigning. By some reason (unknown), the stage3.bin was always in /lib/ until now. Checked in other distros, too. It is in /lib/s390-tools/.
The package filesystem can not also be installed with the test suite autoyast zvm: https://openqa.opensuse.org/tests/4104434#step/installation/48
Nikolay, why did you assign this to yast2-maintainers? Where exactly is YaST involved in this? That "convertfs" call is in the .spec file of that filesystem package; not our responsibility. https://build.opensuse.org/projects/openSUSE:Factory/packages/filesystem/files/filesystem.spec?expand=1 s390-tools is not ours: osc maintainer s390-tools Defined in package: Base:System/s390-tools bugowner of s390-tools : ngueorguiev maintainer of s390-tools : ngueorguiev Defined in project: Base:System bugowner of s390-tools : - maintainer of s390-tools : dirkmueller, msmeissn, oertel, a_jaeger, seife, trenn, WernerFink, dmolkentin, group:factory-maintainers Defined in project: Base bugowner of s390-tools : - maintainer of s390-tools : adrianSuSE, janblunck, rguenther And the order of package installation is determined by libzypp, not by YaST. I also don't know if something like "X needs to be installed before Y" shouldn't be expressed as RPM dependencies. So, why did this get assigned back to YaST, and what is even left to do here?
(In reply to Stefan Hundhammer from comment #9) > Nikolay, why did you assign this to yast2-maintainers? > > Where exactly is YaST involved in this? > > That "convertfs" call is in the .spec file of that filesystem package; not > our responsibility. > > https://build.opensuse.org/projects/openSUSE:Factory/packages/filesystem/ > files/filesystem.spec?expand=1 > > > s390-tools is not ours: > > osc maintainer s390-tools > Defined in package: Base:System/s390-tools > bugowner of s390-tools : > ngueorguiev > > maintainer of s390-tools : > ngueorguiev > > Defined in project: Base:System > bugowner of s390-tools : > - > > maintainer of s390-tools : > dirkmueller, msmeissn, oertel, a_jaeger, seife, trenn, WernerFink, > dmolkentin, group:factory-maintainers > > Defined in project: Base > bugowner of s390-tools : > - > > maintainer of s390-tools : > adrianSuSE, janblunck, rguenther > > > And the order of package installation is determined by libzypp, not by YaST. > I also don't know if something like "X needs to be installed before Y" > shouldn't be expressed as RPM dependencies. > > > So, why did this get assigned back to YaST, and what is even left to do here? I have re-assigned it to the 'initial' group. If you know which group would be better, please feel free to reassign it. From the ticket, I don't see anything about s390-tools, except the 'recommended' directory for the /lib/s390-tools/stage3.bin. The root cause is something else, not that the data file is in wrong directory.
(In reply to Nikolay Gueorguiev from comment #10) > (In reply to Stefan Hundhammer from comment #9) > > Nikolay, why did you assign this to yast2-maintainers? > > > > Where exactly is YaST involved in this? > > > > That "convertfs" call is in the .spec file of that filesystem package; not > > our responsibility. > > > > https://build.opensuse.org/projects/openSUSE:Factory/packages/filesystem/ > > files/filesystem.spec?expand=1 > > > > > > s390-tools is not ours: > > > > osc maintainer s390-tools > > Defined in package: Base:System/s390-tools > > bugowner of s390-tools : > > ngueorguiev > > > > maintainer of s390-tools : > > ngueorguiev > > > > Defined in project: Base:System > > bugowner of s390-tools : > > - > > > > maintainer of s390-tools : > > dirkmueller, msmeissn, oertel, a_jaeger, seife, trenn, WernerFink, > > dmolkentin, group:factory-maintainers > > > > Defined in project: Base > > bugowner of s390-tools : > > - > > > > maintainer of s390-tools : > > adrianSuSE, janblunck, rguenther > > > > > > And the order of package installation is determined by libzypp, not by YaST. > > I also don't know if something like "X needs to be installed before Y" > > shouldn't be expressed as RPM dependencies. > > > > > > So, why did this get assigned back to YaST, and what is even left to do here? > > I have re-assigned it to the 'initial' group. > > If you know which group would be better, please feel free to reassign it. > > From the ticket, I don't see anything about s390-tools, except the > 'recommended' directory for the /lib/s390-tools/stage3.bin. The root cause > is something else, not that the data file is in wrong directory. 2024-04-18 09:39:16 <5> o3zvm001(3595) [zypp-core] Exception.cc(log):202 RpmDb.cc(doInstallPackage):1846 THROW: Subprocess failed. Error: RPM failed: Command exited with status 1. 2024-04-18 09:39:16 <5> o3zvm001(3595) [zypp-core] Exception.cc(log):202 History: 2024-04-18 09:39:16 <5> o3zvm001(3595) [zypp-core] Exception.cc(log):202 - error: lua script failed: [string "%prein(filesystem-84.87-15.3.s390x)"]:22: exit 2024-04-18 09:39:16 <5> o3zvm001(3595) [zypp-core] Exception.cc(log):202 error: filesystem-84.87-15.3.s390x: install failed 2024-04-18 09:39:16 <5> o3zvm001(3595) [zypp-core] Exception.cc(log):202 2024-04-18 09:39:16 <1> o3zvm001(3595) [Ruby] modules/PackageCallbacks.rb(DonePackage):506 DonePackage(error: 3, reason: 'Subprocess failed. Error: RPM failed: Command exited with status 1.
So it's down to guessing again... >> 09:39:13 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/apparmor-docs-3.1.7-6.1.noarch.rpm,0x0000000c) >> 09:39:13 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/bash-doc-5.2.26-12.1.noarch.rpm,0x0000000c) >> 09:39:13 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/boost-license1_84_0-1.84.0-3.2.noarch.rpm,0x0000000c) >> 09:39:13 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/branding-openSUSE-84.87.20240405-1.1.noarch.rpm,0x0000000c) >> 09:39:13 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/btrfsprogs-udev-rules-6.8-1.2.noarch.rpm,0x0000000c) >> 09:39:13 <1> RpmDb::installPackage(.../openSUSE-20240417-0/s390x/busybox-static-1.36.1-4.2.s390x.rpm,0x0000000c) >> 09:39:13 <1> RpmDb::installPackage(.../openSUSE-20240417-0/s390x/compat-usrmerge-tools-84.87-5.21.s390x.rpm,0x0000000c) >> 09:39:13 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/coreutils-doc-9.5-1.1.noarch.rpm,0x0000000c) >> 09:39:13 <1> RpmDb::installPackage(.../openSUSE-20240417-0/s390x/cracklib-dict-full-2.8.12-66.16.s390x.rpm,0x0000000c) >> 09:39:14 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/cryptsetup-doc-2.7.2-1.1.noarch.rpm,0x0000000c) >> 09:39:14 <1> RpmDb::installPackage(.../openSUSE-20240417-0/s390x/debuginfod-profile-0.191-1.1.s390x.rpm,0x0000000c) >> 09:39:14 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/distribution-logos-openSUSE-Tumbleweed-20240404-1.1.noarch.rpm,0x0000000c) >> 09:39:14 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/file-magic-5.45-2.2.noarch.rpm,0x0000000c) >> 09:39:14 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/hicolor-icon-theme-0.17-4.4.noarch.rpm,0x0000000c) >> 09:39:14 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/info-std-7.0.3-1.7.noarch.rpm,0x0000000c) >> 09:39:14 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/libX11-data-1.8.9-1.1.noarch.rpm,0x0000000c) >> 09:39:14 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/libicu73-bedata-73.2-3.3.noarch.rpm,0x0000000c) >> 09:39:14 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/libldap-data-2.6.6-1.4.noarch.rpm,0x0000000c) >> 09:39:14 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/libnl-config-3.9.0-1.3.noarch.rpm,0x0000000c) >> 09:39:14 <1> RpmDb::installPackage(.../openSUSE-20240417-0/s390x/libpaper-2.1.3-1.2.s390x.rpm,0x0000000c) >> 09:39:14 <1> RpmDb::installPackage(.../openSUSE-20240417-0/s390x/libsemanage-conf-3.6-1.4.s390x.rpm,0x0000000c) >> 09:39:15 <1> RpmDb::installPackage(.../openSUSE-20240417-0/s390x/libssh-config-0.10.6-2.1.s390x.rpm,0x0000000c) >> 09:39:15 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/libstorage-ng-lang-4.5.202-1.1.noarch.rpm,0x0000000c) >> 09:39:15 <1> RpmDb::installPackage(.../openSUSE-20240417-0/s390x/libthai-data-0.1.29-1.10.s390x.rpm,0x0000000c) >> 09:39:15 <1> RpmDb::installPackage(.../openSUSE-20240417-0/s390x/libtirpc-netconfig-1.3.4-1.3.s390x.rpm,0x0000000c) >> 09:39:15 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/man-pages-6.7-1.2.noarch.rpm,0x0000000c) >> 09:39:15 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/man-pages-posix-2017a-1.10.noarch.rpm,0x0000000c) >> 09:39:15 <1> RpmDb::installPackage(.../openSUSE-20240417-0/s390x/openSUSE-release-ftp-20240417-1880.1.s390x.rpm,0x0000000c) >> 09:39:15 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/pam-manpages-1.6.1-1.1.noarch.rpm,0x0000000c) >> 09:39:15 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/pkgconf-m4-2.1.1-1.2.noarch.rpm,0x0000000c) >> 09:39:15 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/publicsuffix-20240410-1.1.noarch.rpm,0x0000000c) >> 09:39:15 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/readline-doc-8.2.10-1.3.noarch.rpm,0x0000000c) >> 09:39:15 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/s390-tools-genprotimg-data-2.31.0-7.1.noarch.rpm,0x0000000c) >> 09:39:15 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/shared-python-startup-0.1-6.11.noarch.rpm,0x0000000c) >> 09:39:15 <1> RpmDb::installPackage(.../openSUSE-20240417-0/s390x/snmp-mibs-5.9.4-2.4.s390x.rpm,0x0000000c) >> 09:39:15 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/system-user-root-20190513-2.15.noarch.rpm,0x0000000c) >> 09:39:16 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/unzip-doc-6.00-42.1.noarch.rpm,0x0000000c) >> 09:39:16 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/vim-data-common-9.1.0330-1.1.noarch.rpm,0x0000000c) >> 09:39:16 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/wireless-regdb-20231201-1.3.noarch.rpm,0x0000000c) >> 09:39:16 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/xbitmaps-1.1.3-2.3.noarch.rpm,0x0000000c) >> 09:39:16 <1> RpmDb::installPackage(.../openSUSE-20240417-0/s390x/yast2-alternatives-5.0.0-1.2.s390x.rpm,0x0000000c) >> 09:39:16 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/yast2-nfs-common-5.0.0-1.2.noarch.rpm,0x0000000c) >> 09:39:16 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/yast2-trans-stats-2.19.0-17.32.noarch.rpm,0x0000000c) >> 09:39:16 <1> RpmDb::installPackage(.../openSUSE-20240417-0/noarch/filesystem-84.87-15.3.s390x.rpm, 0x0000000c) >> 09:39:16 <5> [zypp-core] Exception.cc(log):202 RpmDb.cc(doInstallPackage):1846 THROW: Subprocess failed. Error: RPM failed: Command exited with status 1. >> 09:39:16 <5> [zypp-core] Exception.cc(log):202 History: >> 09:39:16 <5> [zypp-core] Exception.cc(log):202 - error: lua script failed: [string "%prein(filesystem-84.87-15.3.s390x)"]:22: exit >> 09:39:16 <5> [zypp-core] Exception.cc(log):202 error: filesystem-84.87-15.3.s390x: install failed ...and as Fabian remarked in comment #2, the filesystem package obviously started 'convertfs' in one of its scripts, and that caused problems. Because the filesystem package should be installed first? I don't know. I would expect it to be installed first, not after 43 other packages. Was it always installed first in the past, and now no more? I don't know. I hope the libzypp maintainers know more about this.
Created attachment 874499 [details] The code of that script in question seems to be this See comment #9 Source https://build.opensuse.org/projects/openSUSE:Factory/packages/filesystem/files/filesystem.spec?expand=1
Created attachment 874500 [details] y2logs tarball
The change that made this blow up is https://build.opensuse.org/request/show/1168394 until then, the file in /lib was part of s390-tools, which is a package with some more dependencies (like coreutils, gawk) With that request, the files living in /lib moved to a package without any dependencies at all, which makes it seem like a good candidate for early installation. As we can see here, this in turn trips up on the filesystem package; this package has code in to merge /lib into /usr/lib for migrations - but in this case trips as a FRESH install on a HDD without pre-existing /lib installs files into /lib, and file-system comes in early on and would NOW need to migrate lib to /usr/lib - even though before the transaction started, this was not needed. The safe-guard before was s390x-tools having a runtime dependency on glibc - which un turn requires filesystem, making for an order to make this behave as expected Solutions: * move the files from /lib to /usr/lib and make the package UsrMerge compatible without relying on compact symlinks * as a workaround, if above is not an option, Require filesystem or glibc
Probably a good quick fix would be to require the filesytem package, which is a very reasonable assumption anyway (no filesytem, no fun), and then carefully moving the files from /lib to /usr/lib.
The SR was accepted. https://build.opensuse.org/request/show/1170172
For reference: the test passed openQA: https://openqa.opensuse.org/tests/4109573 IMHO the package should still be changed to install its files to /usr/lib directly instead of relying other packages timely putting the lib -> usr/lib symlink in place