Bug 1216274 - [Build 26.14] 15.3 to 15.6 migration fails due to missing library files
Summary: [Build 26.14] 15.3 to 15.6 migration fails due to missing library files
Status: RESOLVED FIXED
Alias: None
Product: PUBLIC SUSE Linux Enterprise Server 15 SP6
Classification: openSUSE
Component: Other (show other bugs)
Version: unspecified
Hardware: Other Other
: P3 - Medium : Major
Target Milestone: ---
Assignee: E-mail List
QA Contact:
URL: https://openqa.suse.de/tests/12507218...
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-10-16 09:52 UTC by Sofia Syrianidou
Modified: 2024-02-05 08:50 UTC (History)
7 users (show)

See Also:
Found By: openQA
Services Priority:
Business Priority:
Blocker: Yes
Marketing QA Status: ---
IT Deployment: ---


Attachments
zypper and y2logs from different migration tests, compressed (15.19 MB, application/x-xz)
2023-10-16 09:52 UTC, Sofia Syrianidou
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Sofia Syrianidou 2023-10-16 09:52:28 UTC
Created attachment 870220 [details]
zypper and y2logs from different migration tests, compressed

Different migration tests from sles 15 SP3 with ltss fail during migration with zypper or yast, due errors related with missing shared library objects. 

- Error 1 : "nothing provides 'libnbd.so.0()(64bit)' needed by the to be installed libvirt-daemon-driver-qemu-9.8.0-150600.1.1.x86_64"
https://openqa.suse.de/tests/12507218#step/zypper_migration/8
- Error 2 :"nothing provides 'libjson-c.so.5()(64bit)' needed by the to be installed bluez-5.69-150600.1.2.aarch64"
https://openqa.suse.de/tests/12507141#step/zypper_migration/8
also found during yast migration: 
https://openqa.suse.de/tests/12507132#step/yast2_migration/35

Each error is causing migration failure on different openqa tests.
Currently, only 2 openqa test for migration from 15.3 to 15.6 are successful:
high availability online migration: https://openqa.suse.de/tests/12507269
online migration from image with minimal installation, including addons:	basesystem,server applications and ltss: https://openqa.suse.de/tests/12507292


The failed tests are concerning all system architectures.

Attached: zypper logs for each error and yast logs for libjson error
Comment 1 Sofia Syrianidou 2023-10-16 11:54:59 UTC
Also affects 15.4 to 15.6 migrations: 
https://openqa.suse.de/tests/12507153#step/yast2_migration/12
Adding "nothing rpovides qemu-img: error mesage:
https://openqa.suse.de/tests/12507153#step/yast2_migration/12
Comment 2 Antoine Ginies 2023-10-16 14:10:31 UTC
online zypper or yast migration?
I don't think this is a supported scenario. We support offline migration for this kind of scenario.
Comment 3 Antoine Ginies 2023-10-16 14:12:08 UTC
lower down prio... this is not a P1.
Comment 4 James Fehlig 2023-10-16 16:43:14 UTC
(In reply to Sofia Syrianidou from comment #0)
> - Error 1 : "nothing provides 'libnbd.so.0()(64bit)' needed by the to be
> installed libvirt-daemon-driver-qemu-9.8.0-150600.1.1.x86_64"

Starting with libvirt 9.8.0, the qemu driver prefers using nbdkit for network disks. For more details see the 'Improvements' section of the 9.8.0 news

https://libvirt.org/news.html#v9-8-0-2023-10-02

libnbd (and nbdkit) are available for SLE15 SP6

https://build.suse.de/package/show/SUSE:SLE-15-SP6:GA/libnbd
https://build.suse.de/package/show/SUSE:SLE-15-SP6:GA/nbdkit

As you can see, they are actually inherited from older service packs. But AFAIK the packages are not distributed. Until the recent libvirt change, they were only used internally for building up the CDI container used by kubevirt. With SP6 we'll need to start distributing libnbd and nbdkit.

BTW, I'd like to update both for SP6. Question is whether to do a MR on the existing packages or a new SR to SP6 only?
Comment 5 Sofia Syrianidou 2023-10-17 08:23:06 UTC
(In reply to Antoine Ginies from comment #2)
> online zypper or yast migration?
> I don't think this is a supported scenario. We support offline migration for
> this kind of scenario.

It is difficult to tell about offline scenarios, because we have another failure there, earlier than the migration step.
Online migration tests fail, using both zypper and yast. Not only for migrating from 15.3 ltss to 15.6, but also 15.4->15.6 and 15.5->15.6
Comment 6 Sofia Syrianidou 2023-10-17 08:23:50 UTC
(In reply to Sofia Syrianidou from comment #5)
> (In reply to Antoine Ginies from comment #2)
> > online zypper or yast migration?
> > I don't think this is a supported scenario. We support offline migration for
> > this kind of scenario.
> 
> It is difficult to tell about offline scenarios, because we have another
> failure there, earlier than the migration step.
> Online migration tests fail, using both zypper and yast. Not only for
> migrating from 15.3 ltss to 15.6, but also 15.4->15.6 and 15.5->15.6

15.5->15.6 fail when desktop module is included
Comment 7 Antoine Ginies 2023-10-17 12:10:48 UTC
(In reply to James Fehlig from comment #4)
> (In reply to Sofia Syrianidou from comment #0)
> > - Error 1 : "nothing provides 'libnbd.so.0()(64bit)' needed by the to be
> > installed libvirt-daemon-driver-qemu-9.8.0-150600.1.1.x86_64"
> 
> Starting with libvirt 9.8.0, the qemu driver prefers using nbdkit for
> network disks. For more details see the 'Improvements' section of the 9.8.0
> news
> 
> https://libvirt.org/news.html#v9-8-0-2023-10-02
> 
> libnbd (and nbdkit) are available for SLE15 SP6
> 
> https://build.suse.de/package/show/SUSE:SLE-15-SP6:GA/libnbd
> https://build.suse.de/package/show/SUSE:SLE-15-SP6:GA/nbdkit
> 
> As you can see, they are actually inherited from older service packs. But
> AFAIK the packages are not distributed. Until the recent libvirt change,
> they were only used internally for building up the CDI container used by
> kubevirt. With SP6 we'll need to start distributing libnbd and nbdkit.
> 
> BTW, I'd like to update both for SP6. Question is whether to do a MR on the
> existing packages or a new SR to SP6 only?

MR will update SP5 also, do we need this update?
Comment 8 James Fehlig 2023-10-17 16:56:23 UTC
(In reply to James Fehlig from comment #4)
> Starting with libvirt 9.8.0, the qemu driver prefers using nbdkit for
> network disks. For more details see the 'Improvements' section of the 9.8.0
> news

After a bit more snooping, I think we should disable the functionality. It's only enabled for a future Fedora release in the upstream spec file, which has the accompanying comment:

# We should only enable nbdkit support if the OS ships a SELinux policy that
# allows libvirt to launch it. Right now that's not the case anywhere, but
# things should be fine by the time Fedora 40 is released.

I suspect apparmor profiles also need adjusted for this to work. But more problematic is the functionality cannot be loaded at runtime if available. The feature was introduced as a build-time option and would be non-trivial to convert to a dynamically loaded runtime option. The additional hard dependency is undesirable in use cases such as kubevirt, where the functionality is not needed and we want to keep the virt-launcher container as small as possible.

The qemu package has already modularized the network block backends, e.g. qemu-block-ssh, qemu-block-curl, qemu-block-nfs, etc. I don't think we want libvirt to undo some of that hard work by introducing a hard dependency on functionality that is questionable to support anyhow.
Comment 9 James Fehlig 2023-10-17 17:04:31 UTC
(In reply to Antoine Ginies from comment #7)
> > BTW, I'd like to update both for SP6. Question is whether to do a MR on the
> > existing packages or a new SR to SP6 only?
> 
> MR will update SP5 also, do we need this update?

Probably not. libnbd had a recent CVE (bug#1215799) which I fixed in Factory and ALP, but apparently it wasn't needed in SLE. Vasily would have to verify if CDI is affected in any way by the libnbd CVE. Regardless, I should probably update both libnbd and nbdkit in SP6.
Comment 10 James Fehlig 2023-10-18 23:02:38 UTC
(In reply to James Fehlig from comment #8)
> After a bit more snooping, I think we should disable the functionality. It's
> only enabled for a future Fedora release in the upstream spec file, which
> has the accompanying comment:

FYI, I've disabled it now and submitted an updated libvirt package to Factory, ALP, and SLE15 SP6.
Comment 12 OBSbugzilla Bot 2023-10-18 23:25:03 UTC
This is an autogenerated message for OBS integration:
This bug (1216274) was mentioned in
https://build.opensuse.org/request/show/1118744 Factory / libvirt
Comment 14 Claudio Fontana 2024-02-05 08:50:57 UTC
closing as fixed, if it isn't go ahead and reopen, thanks!