Bug 1218684 - systemd-testsuite depends on qemu-kvm on aarch64, but package is not found
Summary: systemd-testsuite depends on qemu-kvm on aarch64, but package is not found
Status: RESOLVED FIXED
Alias: None
Product: PUBLIC SUSE Linux Enterprise Server 15 SP6
Classification: openSUSE
Component: Other (show other bugs)
Version: unspecified
Hardware: aarch64 Other
: P2 - High : Normal
Target Milestone: ---
Assignee: E-mail List
QA Contact: Antoine Ginies
URL: https://openqa.suse.de/tests/13223289...
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-01-10 08:11 UTC by Andrea Manzini
Modified: 2024-02-29 18:50 UTC (History)
6 users (show)

See Also:
Found By: openQA
Services Priority:
Business Priority:
Blocker: Yes
Marketing QA Status: ---
IT Deployment: ---


Attachments
full serial terminal log from testsuite (12.51 KB, text/plain)
2024-01-10 08:11 UTC, Andrea Manzini
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Andrea Manzini 2024-01-10 08:11:45 UTC
Created attachment 871737 [details]
full serial terminal log from testsuite

## Observation

openQA test in scenario sle-15-SP6-Online-aarch64-upstream_systemd_tests-amanzini@aarch64 fails in
[prepare_systemd_and_testsuite](https://openqa.suse.de/tests/13223289/modules/prepare_systemd_and_testsuite/steps/87)

## Test suite description



## Reproducible

Fails since (at least) Build [45.1](https://openqa.suse.de/tests/13215711)


## Expected result

Last good: (unknown) (or more recent)


## Further details

Always latest result in this scenario: [latest](https://openqa.suse.de/tests/latest?arch=aarch64&distri=sle&flavor=Online&machine=aarch64&test=upstream_systemd_tests-amanzini&version=15-SP6)


conflicting dependencies trying to install qemu-kvm 

Processing conflicts -  0
0| 2024-01-10 02:37:43 <3> susetest(4532) [zypp::solver] SATResolver.cc(solving):849 Solverrun finished with an ERROR
0| 2024-01-10 02:37:43 <2> susetest(4532) [zypp::solver] SATResolver.cc(resolvePool):921 SATResolver::resolvePool() done. Ret:0
0| 2024-01-10 02:37:43 <1> susetest(4532) [zypp::solver] Resolver.cc(problems):414 Resolver::problems()
0| 2024-01-10 02:37:43 <1> susetest(4532) [zypp::solver] SATResolver.cc(problems):1353 Encountered problems! Here are the solutions:
0| 2024-01-10 02:37:43 <1> susetest(4532) [zypp::solver] SATResolver.cc(problems):1353 
0| 2024-01-10 02:37:43 <1> susetest(4532) [zypp::solver] SATResolver.cc(problems):1357 Problem 1:
0| 2024-01-10 02:37:43 <1> susetest(4532) [zypp::solver] SATResolver.cc(problems):1358 ====================================
0| 2024-01-10 02:37:43 <1> susetest(4532) [zypp::solver] SATResolver.cc(problems):1362 nothing provides 'qemu-kvm' needed by the to be installed systemd-testsuite-254.5-150600.2.4.aarch64
0| 2024-01-10 02:37:43 <1> susetest(4532) [zypp::solver] SATResolver.cc(problems):1363 ------------------------------------
0| 2024-01-10 02:37:43 <1> susetest(4532) [libsolv++] PoolImpl.cc(logSat):131 solver statistics: 1 learned rules, 0 unsolvable, 0 minimization steps
EOL
SCRIPT_FINISHEDsUnaF-0-
Comment 1 Claudio Fontana 2024-01-11 07:46:06 UTC
It seems that the compatibility package qemu-kvm is not available on aarch64, I see the same issue on 15 SP5.
Comment 2 Antoine Ginies 2024-01-11 08:58:54 UTC
yes, the package is not available on aarch64. The pattern has been fixed for marble (sle micro6.0).
Comment 3 Dario Faggioli 2024-01-11 11:56:25 UTC
(In reply to Claudio Fontana from comment #1)
> It seems that the compatibility package qemu-kvm is not available on
> aarch64, I see the same issue on 15 SP5.

It never (well, at least since 6 years, when I joined) has been.

We've always been building that package only for '%ix86 x86_64 s390x'. I don't know why that's the case... Historical reasons, probably, as the qemu-kvm fork was available for those arch-es, or something like that.

We can change that, if we want or need. Or we can just see if we can get rid of that package which, by the way, contains *ONLY* a symlink from /usr/bin/qemu-system-x86_64 (or _s390x) to /usr/bin/qemu-kvm.

Dependency wise, it only Requires: qemu-x86 (or -s390x).

So, it's basically useless and it looks to me that it's doing more harm than good. But maybe there's something I'm missing...
Comment 4 Dario Faggioli 2024-01-11 11:58:24 UTC
(In reply to Andrea Manzini from comment #0)
> ## Further details
> 
> Always latest result in this scenario:
> [latest](https://openqa.suse.de/tests/
> latest?arch=aarch64&distri=sle&flavor=Online&machine=aarch64&test=upstream_sy
> stemd_tests-amanzini&version=15-SP6)
> 
> 
> conflicting dependencies trying to install qemu-kvm 
>
Can we change whatever it is that is trying to install and/or depends on qemu-kvm, to install/depend on qemu instead?
Comment 5 Dario Faggioli 2024-01-11 11:59:40 UTC
(In reply to Antoine Ginies from comment #2)
> yes, the package is not available on aarch64. The pattern has been fixed for
> marble (sle micro6.0).
>
Do we have a list of places and patterns that depends from it (or a way to figure that out)? If yes, I'm happy to go and change all of them myself to depend on qemu, which seems much saner...
Comment 6 Claudio Fontana 2024-01-11 12:16:21 UTC
(In reply to Dario Faggioli from comment #4)
> (In reply to Andrea Manzini from comment #0)
> > ## Further details
> > 
> > Always latest result in this scenario:
> > [latest](https://openqa.suse.de/tests/
> > latest?arch=aarch64&distri=sle&flavor=Online&machine=aarch64&test=upstream_sy
> > stemd_tests-amanzini&version=15-SP6)
> > 
> > 
> > conflicting dependencies trying to install qemu-kvm 
> >
> Can we change whatever it is that is trying to install and/or depends on
> qemu-kvm, to install/depend on qemu instead?

If it works and we can be sure to capture all dependencies, yes, it would be preferable, since qemu-kvm is an obsolete, compatibility-only package.
Comment 7 Antoine Ginies 2024-01-11 13:51:22 UTC
There is no list of patterns which depends on it.
As far as i know there is the ALP pattern manage by ALP RM, and there is our pattern from SLES, and that's all.
Comment 8 Claudio Fontana 2024-01-11 14:17:28 UTC
(In reply to Antoine Ginies from comment #7)
> There is no list of patterns which depends on it.
> As far as i know there is the ALP pattern manage by ALP RM, and there is our
> pattern from SLES, and that's all.

I currently see on TW x86:

# zypper se --requires qemu-kvm

kanku-worker               | Worker daemon for kanku                 | package
openQA-devel               | Development package pulling in all bu-> | package
os-autoinst-qemu-kvm       | Convenience package providing os-auto-> | package
python3-avocado-plugins-vt | Avocado Virt Test Plugin                | package
qemu-headless              | Minimum set of packages for having a -> | package
systemd-testsuite          | Testsuite for systemd                   | package

# zypper se --recommends qemu-kvm

build                   | A Script to Build SUSE Linux RPMs | package
minikube                | Tool to run Kubernetes locally    | package
patterns-media-rest_dvd | Remaining Software                | package

interestingly enough I also see a conflict with:

# zypper se --conflicts qemu-kvm

kvm_stat | Monitoring Tool for KVM guests | package


On TW AArch64 I see:

# zypper se --requires qemu-kvm

  | python3-avocado-plugins-vt | Avocado Virt Test Plugin | package
  | systemd-testsuite          | Testsuite for systemd    | package

# zypper se --recommends qemu-kvm

  | build                   | A Script to Build SUSE Linux RPMs | package
  | minikube                | Tool to run Kubernetes locally    | package
  | patterns-media-rest_dvd | Remaining Software                | package

# zypper se --conflicts qemu-kvm
  | kvm_stat | Monitoring Tool for KVM guests | package


On 15SP5 AArch64 I see:

# zypper se --requires qemu-kvm

  | kanku-worker               | Worker daemon for kanku  | package
  | python3-avocado-plugins-vt | Avocado Virt Test Plugin | package

# zypper se --recommends qemu-kvm
  | build | A Script to Build SUSE Linux RPMs | package

# zypper se --conflicts qemu-kvm
  | kvm_stat | Monitoring Tool for KVM guests | package

On 15SP5 x86 I see:

# zypper se --requires qemu-kvm

?? nothing ??

# zypper se --recommends qemu-kvm
  | build | A Script to Build SUSE Linux RPMs | package

curiously enough:

# zypper se --conflicts qemu-kvm

 | kvm_stat                                             | Monitoring -> |package
 | SUSE-SLE-Module-Server-Applications-15-SP5-2023-2549 | Recommended-> | patch
 | SUSE-SLE-Module-Server-Applications-15-SP5-2023-3082 | Security up-> | patch
 | SUSE-SLE-Module-Server-Applications-15-SP5-2023-4662 | Security up-> | patch
Comment 9 James Fehlig 2024-01-11 15:17:37 UTC
(In reply to Dario Faggioli from comment #3)
> We've always been building that package only for '%ix86 x86_64 s390x'. I
> don't know why that's the case... Historical reasons, probably, as the
> qemu-kvm fork was available for those arch-es, or something like that.

I think that's the case. The original qemu-kvm package (created when the fork existed, i.e. before kvm support was merged into upstream qemu) only supported x86 and s390.

> We can change that, if we want or need. Or we can just see if we can get rid
> of that package which, by the way, contains *ONLY* a symlink from
> /usr/bin/qemu-system-x86_64 (or _s390x) to /usr/bin/qemu-kvm.

I'm a little worried about dropping it, or at least the symlink. There could be old VMs out there with <emulator>/usr/bin/qemu-kvm</emulator> in their config. IIRC, our install tools (vm-install and/or virt-install) used to create such config.

(In reply to Claudio Fontana from comment #8)
> # zypper se --requires qemu-kvm
...
> systemd-testsuite          | Testsuite for systemd                   |
> package

IMO, this bug is fixed by fixing the systemd-testsuite package. We should also evaluate whether the other packages requiring/recommending qemu-kvm need adjusted.
Comment 10 Claudio Fontana 2024-01-11 15:38:22 UTC
(In reply to James Fehlig from comment #9)
> (In reply to Dario Faggioli from comment #3)
> > We've always been building that package only for '%ix86 x86_64 s390x'. I
> > don't know why that's the case... Historical reasons, probably, as the
> > qemu-kvm fork was available for those arch-es, or something like that.
> 
> I think that's the case. The original qemu-kvm package (created when the
> fork existed, i.e. before kvm support was merged into upstream qemu) only
> supported x86 and s390.
> 
> > We can change that, if we want or need. Or we can just see if we can get rid
> > of that package which, by the way, contains *ONLY* a symlink from
> > /usr/bin/qemu-system-x86_64 (or _s390x) to /usr/bin/qemu-kvm.
> 
> I'm a little worried about dropping it, or at least the symlink. There could
> be old VMs out there with <emulator>/usr/bin/qemu-kvm</emulator> in their
> config. IIRC, our install tools (vm-install and/or virt-install) used to
> create such config.

I agree. I think we need to keep the qemu-kvm package,

but we should avoid having dependencies to it (or even conflicts with it, the conflict with kvm_stat is very strange).

> 
> (In reply to Claudio Fontana from comment #8)
> > # zypper se --requires qemu-kvm
> ...
> > systemd-testsuite          | Testsuite for systemd                   |
> > package
> 
> IMO, this bug is fixed by fixing the systemd-testsuite package. We should
> also evaluate whether the other packages requiring/recommending qemu-kvm
> need adjusted.
Comment 11 Dario Faggioli 2024-01-12 11:44:05 UTC
(In reply to James Fehlig from comment #9)
> I'm a little worried about dropping it, or at least the symlink. There could
> be old VMs out there with <emulator>/usr/bin/qemu-kvm</emulator> in their
> config. IIRC, our install tools (vm-install and/or virt-install) used to
> create such config.
> 
Well, I think I can make the qemu-system-%{arch} packages create the link themselves.

Like, if you install qemu-system-x86 (which ships the qemu-system-x86_64 binary) and we are on x86_64, create the qemu-kvm -> qemu-system-x86_64 symlink. If you install qemu-system-s390x and we are on x86_64, create the qemu-kvm -> qemu-system-s390x symlink.

I only have to see if I need to do this in the %postinst phase of such packages (and, if yes, I don't know if it's possible to check the architecture of the machine where I'm installing the pacakge), or if it's fine during %build.

At that point (and if it works), I can even do something like, I think, "Obsoletes: qemu-kvm", still in those packages, and we don't even have to change the patterns...
Comment 12 Dario Faggioli 2024-01-12 11:47:47 UTC
(In reply to James Fehlig from comment #9)
> ...
> > systemd-testsuite          | Testsuite for systemd                   |
> > package
> 
> IMO, this bug is fixed by fixing the systemd-testsuite package. We should
> also evaluate whether the other packages requiring/recommending qemu-kvm
> need adjusted.
>
Problem is, it's probably not only the packages. It's what you said above about the link.

In fact, I don't know if we have VMs around with qemu-kvm set as their emulator. I'm, on the other hand, quite sure that `osc`, and probably even some tool/script in this systemd-testsuite is actually calling `qemu-kvm` _directly_.

So, if just we change the Requires: in the spec files, packages will be installable but then things will likely fail at runtime. :-/
Comment 13 Claudio Fontana 2024-01-12 11:57:22 UTC
(In reply to Dario Faggioli from comment #12)
> (In reply to James Fehlig from comment #9)
> > ...
> > > systemd-testsuite          | Testsuite for systemd                   |
> > > package
> > 
> > IMO, this bug is fixed by fixing the systemd-testsuite package. We should
> > also evaluate whether the other packages requiring/recommending qemu-kvm
> > need adjusted.
> >
> Problem is, it's probably not only the packages. It's what you said above
> about the link.
> 
> In fact, I don't know if we have VMs around with qemu-kvm set as their
> emulator. I'm, on the other hand, quite sure that `osc`, and probably even
> some tool/script in this systemd-testsuite is actually calling `qemu-kvm`
> _directly_.
> 
> So, if just we change the Requires: in the spec files, packages will be
> installable but then things will likely fail at runtime. :-/

Let us keep the qemu-kvm package containing the symlink on the x86 architecture.

People may have getting used to installing it. That is fine and we don't want to hurt existing users.

Let us however not add dependencies to it from other packages,
and let us hunt down the existing dependencies in existing packages for new products so that they do not depend anymore on this obsolete package and path that is there for backward compatibility only.
Comment 14 James Fehlig 2024-01-12 17:44:55 UTC
(In reply to Claudio Fontana from comment #13)
> Let us keep the qemu-kvm package containing the symlink on the x86
> architecture.

I tend to prefer this as well, especially for SLE15 SPx. For Factory, we could drop the package altogether, and discover/fix the fallout.
Comment 15 Andrea Manzini 2024-01-23 07:21:45 UTC
reporting same behavior on ppc64le: 
https://openqa.suse.de/tests/13316367#step/prepare_systemd_and_testsuite/71

can this be solved by fixing the systemd-testsuite.rpm package dependencies, or maybe just disable build for non-x86_64 archs ?
Comment 16 Swayammitra Tripathy 2024-02-05 13:54:10 UTC
Hello Antonio,

Could you please the update the status.
Comment 17 Franck Bui 2024-02-16 16:25:03 UTC
(In reply to Dario Faggioli from comment #12)
> In fact, I don't know if we have VMs around with qemu-kvm set as their
> emulator. I'm, on the other hand, quite sure that `osc`, and probably even
> some tool/script in this systemd-testsuite is actually calling `qemu-kvm`
> _directly_.

I think it should be easily fixable as long as /usr/bin/qemu-system-x86_64 and /usr/bin/kvm behave the same way.

If you can confirm this I'll turn the dep:

  Requires: qemu-kvm

into

  Requires: qemu

Thanks.
Comment 19 OBSbugzilla Bot 2024-02-17 03:15:01 UTC
This is an autogenerated message for OBS integration:
This bug (1218684) was mentioned in
https://build.opensuse.org/request/show/1147256 Factory / qemu
Comment 20 Dario Faggioli 2024-02-17 08:43:30 UTC
(In reply to Franck Bui from comment #17)
> (In reply to Dario Faggioli from comment #12)
> > In fact, I don't know if we have VMs around with qemu-kvm set as their
> > emulator. I'm, on the other hand, quite sure that `osc`, and probably even
> > some tool/script in this systemd-testsuite is actually calling `qemu-kvm`
> > _directly_.
> 
> I think it should be easily fixable as long as /usr/bin/qemu-system-x86_64
> and /usr/bin/kvm behave the same way.
> 
/usr/bin/kvm does not exist anywhere (as far as I know... And I really hope that to be the case, or that's another problem/bug! :-O)

/usr/bin/qemu-kvm, where it exists (i.e., only on x86_64 and s390x, for now) is a symlink to /usr/bin/qemu-system_x86_64, on x86_64, and to qemu-system_s390x, on s390x.

So, yes, they're identical.

> If you can confirm this I'll turn the dep:
> 
>   Requires: qemu-kvm
> 
> into
> 
>   Requires: qemu
> 
This is the right thing to do, and IMO, it should be done everywhere.

However, this may not be enough. In fact, if QEMU is invoked via 'qemu-kvm' somewhere in the code (of this, or of any other of the packages that depend from 'qemu-kvm'), things will fail "at runtime"... :-(

And that's why I did, with https://build.suse.de/request/show/321910, which should be enough a workaround/transition solution.

Basically, with that change, it's the 'qemu' package that creates the symlink, so it will exist on all arches (and, in each arch, it will point to the proper '/usr/bin/qemu-system-%{arch} binary). This way, any script/software/whatever invoking '/usr/bin/qemu-kvm' will continue to work.

At the packaging level, the 'qemu' package itself is now Providing and Obsoleting 'qemu-kvm', so dependencies should also be satisfied (even if/until the switch between 'Requires: qemu-kvm' and 'Requires: qemu' isn't done).
Comment 21 Dario Faggioli 2024-02-17 08:44:41 UTC
Can anyone provide feedback and/or test if what I did in https://build.suse.de/request/show/321910, and explained in comment 20 works?
Comment 22 Dario Faggioli 2024-02-17 08:46:18 UTC
(All that said, I must confess I really don't understand how we got into this situation... In fact, qemu-kvm has _NEVER_ been available for, e.g., aarch64... How is it possible that we have packages depending from it?!?! :-O )
Comment 23 Franck Bui 2024-02-19 08:22:04 UTC
(In reply to Dario Faggioli from comment #20)
> /usr/bin/kvm does not exist anywhere (as far as I know... And I really hope
> that to be the case, or that's another problem/bug! :-O)
> 

Sorry, typo here, should be read /usr/bin/qemu-kvm.

> /usr/bin/qemu-kvm, where it exists (i.e., only on x86_64 and s390x, for now)
> is a symlink to /usr/bin/qemu-system_x86_64, on x86_64, and to
> qemu-system_s390x, on s390x.
> 
> So, yes, they're identical.

My concern was about any trick in qemu that would have checked the name of the binary it was called from and did some special treatments when argv[0]=="qemu-kvm".

But from what I understood, it's not the case.

> However, this may not be enough. In fact, if QEMU is invoked via 'qemu-kvm'
> somewhere in the code (of this, or of any other of the packages that depend
> from 'qemu-kvm'), things will fail "at runtime"... :-(

As far as systemd is concerned, if the testsuite fails due to the use of qemu-kvm symlink, we'll fix it. Hence it's not worth keeping it if it's only for systemd.
Comment 24 Franck Bui 2024-02-19 08:26:32 UTC
(In reply to Dario Faggioli from comment #22)
> (All that said, I must confess I really don't understand how we got into
> this situation... In fact, qemu-kvm has _NEVER_ been available for, e.g.,
> aarch64... How is it possible that we have packages depending from it?!?!
> :-O )

Has this package been documented as obsolete somewhere or was there any related announcement ?
Comment 25 Claudio Fontana 2024-02-19 09:06:15 UTC
(In reply to Franck Bui from comment #24)
> (In reply to Dario Faggioli from comment #22)
> > (All that said, I must confess I really don't understand how we got into
> > this situation... In fact, qemu-kvm has _NEVER_ been available for, e.g.,
> > aarch64... How is it possible that we have packages depending from it?!?!
> > :-O )
> 
> Has this package been documented as obsolete somewhere or was there any
> related announcement ?

Package has been deprecated in multiple distros starting in 2013, but it just won't die, because there is always that one script that continues to use qemu-kvm from back when qemu mainline could not do kvm.
Comment 26 OBSbugzilla Bot 2024-02-19 10:15:04 UTC
This is an autogenerated message for OBS integration:
This bug (1218684) was mentioned in
https://build.opensuse.org/request/show/1147539 Factory / systemd
Comment 28 Dario Faggioli 2024-02-19 15:57:13 UTC
(In reply to Franck Bui from comment #23) 
> My concern was about any trick in qemu that would have checked the name of
> the binary it was called from and did some special treatments when
> argv[0]=="qemu-kvm".
> 
> But from what I understood, it's not the case.
> 
Ah, ok, I see.

No, there used to be something like that in QEMU's Linux user space emulation, but we right now should be fine there as well (and, even if we're not, it'd be a different issue, as qemu-linux-user binaries don't use or know about the qemu-kvm link).

> > However, this may not be enough. In fact, if QEMU is invoked via 'qemu-kvm'
> > somewhere in the code (of this, or of any other of the packages that depend
> > from 'qemu-kvm'), things will fail "at runtime"... :-(
> 
> As far as systemd is concerned, if the testsuite fails due to the use of
> qemu-kvm symlink, we'll fix it. Hence it's not worth keeping it if it's only
> for systemd.
>
Good to hear, and thanks for that. Hopefully, I'v managed to find a way of getting rid of the package, which is already something. The symlink will stay, for now, but I'm not loosing hope of being able to nuke it at some point... :-O
Comment 34 Radoslav Tzvetkov 2024-02-26 10:24:21 UTC
A SR mentioning this bug was successfully integrated into Build 58.1. Please, if needed, set it to RESOLVED FIXED for QE to verify it.
Comment 35 Dario Faggioli 2024-02-28 15:09:01 UTC
(In reply to Radoslav Tzvetkov from comment #34)
> A SR mentioning this bug was successfully integrated into Build 58.1.
> Please, if needed, set it to RESOLVED FIXED for QE to verify it.
>
I've tested installing systemd-testsuite in Tumbleweed and on SP6, with the Devel:Virt:SLE-15-SP6 repository enabled, and it works.

I'm not sure about Build 58.1, but if the bots says we're fine, I guess we are.

Marking is as fixed.