Bug 1217964

Hey, so we cloned the test to another OpenQA instance so I could take a look at it after the failure. It also failed at the same step, but after that I could just start the named service as usual, so everything seems to be working to me. Sorry, I'm not familiar with the migration OpenQA tests, but to me the service seems to just not be enabled and thus not starting or am I missing something here?

(In reply to Jorik Cronenberg from comment #2)
> but to me the service seems to just not be enabled and thus not starting or
> am I missing something here?

(In reply to Ming Li from comment #0)
> ## Test steps:
> 1. Install sles12sp5
> 2. systemctl enable named
>    systemctl start named
>    systemctl status named: Active: active (running)
>    Please check https://openqa.suse.de/tests/13047458#step/install_service/74

Ah I overlooked this part. But is this really a bind issue then? Or more of an issue with the migration that the service doesn't get enabled again? Because I don't see how this would be caused by the bind package.

(In reply to Jorik Cronenberg from comment #3)
> (In reply to Jorik Cronenberg from comment #2)
> > but to me the service seems to just not be enabled and thus not starting or
> > am I missing something here?
> 
> (In reply to Ming Li from comment #0)
> > ## Test steps:
> > 1. Install sles12sp5
> > 2. systemctl enable named
> >    systemctl start named
> >    systemctl status named: Active: active (running)
> >    Please check https://openqa.suse.de/tests/13047458#step/install_service/74
> 
> Ah I overlooked this part. But is this really a bind issue then? Or more of
> an issue with the migration that the service doesn't get enabled again?
> Because I don't see how this would be caused by the bind package.

The fact is before migration we enabled the named, then after migration the named hasn't start automatically. Maybe something during migration caused this issue, not sure. If you need other info or log please contact me to provide. Thanks.

(In reply to Jorik Cronenberg from comment #3)
> (In reply to Jorik Cronenberg from comment #2)
> > but to me the service seems to just not be enabled and thus not starting or
> > am I missing something here?
> 
> (In reply to Ming Li from comment #0)
> > ## Test steps:
> > 1. Install sles12sp5
> > 2. systemctl enable named
> >    systemctl start named
> >    systemctl status named: Active: active (running)
> >    Please check https://openqa.suse.de/tests/13047458#step/install_service/74
> 
> Ah I overlooked this part. But is this really a bind issue then? Or more of
> an issue with the migration that the service doesn't get enabled again?
> Because I don't see how this would be caused by the bind package.

Added 'is-enabled' to verify the enabled status before migration, http://openqa.nue.suse.com/tests/13076940#step/install_service/76

Hello Ming ,
Please retest with upcoming snapshot

(In reply to Swayammitra Tripathy from comment #6)
> Hello Ming ,
> Please retest with upcoming snapshot

Ok, I will test it and upload the latest test result.

The same issue happens on s390x testing:

https://openqa.suse.de/tests/13356248#step/check_upgraded_service/18

1. Install sles12sp5
2. systemctl enable named
   systemctl start named
   systemctl status named: Active: active (running)
https://openqa.suse.de/tests/13356248#step/install_service/76
3. Offline migration to sles15sp6 via proxySCC
4. Check named status with upgraded system:
   systemctl status named: Active: inactive
https://openqa.suse.de/tests/13356248#step/check_upgraded_service/16

Expected result: 
   named should be auto-active after migration

Hello Ming,

Could you please update the status of this issue

(In reply to Swayammitra Tripathy from comment #9)
> Hello Ming,
> 
> Could you please update the status of this issue

Hi Swayammitra, the same issue still can be found on build 50.2, https://openqa.suse.de/tests/13395752#step/check_upgraded_service/16

Adding systemd-maintainers.
Could you take a quick look at this if this is a bug with systemd, or maybe I'm doing something wrong in the spec file with regards to the service file https://build.suse.de/package/view_file/SUSE:SLE-15-SP6:GA/bind/bind.spec?expand=1 but I don't really see anything that could cause this, plus I don't think anything really changed in this regard from SP5.

Has any service name changed between the 2 versions ?

(In reply to Franck Bui from comment #12)
> Has any service name changed between the 2 versions ?

No, both are named.service. There is a lwresd.service in 12SP5 which isn't in 15SP6, but again nothing changed here. This service hasn't been there for a long time.

Except that named is a SysV init script with SLE12-SP5 ;)

May I have access to a system where the problem occured ?

Otherwise Ming, can you check that insserv-compat package is still installed after the upgrade ?

(In reply to Franck Bui from comment #16)
> Otherwise Ming, can you check that insserv-compat package is still installed
> after the upgrade ?

Hi Frank, yes, the insserv-compat package is still installed after upgrade. https://openqa.suse.de/tests/13542804#step/check_upgraded_service/18

I cloned an openQA job and you can use vncviewer to connect it in developer mode, I will tell you how to access in slack. https://openqa.suse.de/tests/13542963#live

(In reply to Ming Li from comment #17)
> I cloned an openQA job and you can use vncviewer to connect it in developer
> mode, I will tell you how to access in slack.
> https://openqa.suse.de/tests/13542963#live

Thanks.

It seems that the problem is due to the order of the package updates during the migration.

Basically systemd-sysvcompat has been introduced in SP6 and is the package that now contains the specific scripts that deals with the SysV init scripts.

The problem is that this package is installed after the update of bind. Hence when bind is updated, named init script is converted into a systemd unit but the scripts handling the enablement of the new unit file (shipped by systemd-sysvcompat) are missing.

A potential fix consists in including "Suggests: systemd-sysvcompat" in the %systemd_{ordering,requires} rpm macro so all packages depending on systemd will also be ordered after systemd-sysvcompat.

This means that besides the fix in the systemd rpm macros, bind package will need to be rebuilt.

Jorik could you make sure that this will happen if that's not done automatically ?

So the fix has been submitted to SLE-15:Update (see above) but it will probably take a while before it will be released.

Radoslav, we see 2 options here:

 - create a *temporary* fork of systemd-rpm-macros for SP6 until the fix is released in SLE15:Update. Hence the fix should be available more quickly. We just need to be sure that the fork will be dropped before the final release of SP6.

 - increase the priority of the SR to SLE15 so it get released quickly enough.

In both cases, bind will need to be rebuilt against the new version of systemd-rpm-macros (v16).

WDYT ?

It's been decided to temporary fork the package.

@Eugenio, I'm handing over this bug to you in order to track the progress of sr#322363.

Please note that bind needs to be rebuilt with the new version of systemd-rpm-macros.

(In reply to Franck Bui from comment #21)
> It's been decided to temporary fork the package.

To be on the safe side: please make sure to unfork the package when sr#322363 will be released.

Thanks.

I reassigned the bug to Eugenio to do the unfork.

The SR quoting this bug was integrated into Build 59.2; please set it as resolved fixed so QE can verify it if you consider it fixed.

Waiting for https://smelt.suse.de/incident/32678/ to go in

Still not released ; so RC1 will go out with this package still forked in SP6.

SUSE-RU-2024:0982-1: An update that has one fix can now be installed.

Category: recommended (moderate)
Bug References: 1217964
Maintenance Incident: [SUSE:Maintenance:32678](https://smelt.suse.de/incident/32678/)
Sources used:
openSUSE Leap Micro 5.3 (src):
 systemd-rpm-macros-15-150000.7.39.1
openSUSE Leap Micro 5.4 (src):
 systemd-rpm-macros-15-150000.7.39.1
openSUSE Leap 15.5 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Linux Enterprise Micro for Rancher 5.3 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Linux Enterprise Micro 5.3 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Linux Enterprise Micro for Rancher 5.4 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Linux Enterprise Micro 5.4 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Linux Enterprise Micro 5.5 (src):
 systemd-rpm-macros-15-150000.7.39.1
Basesystem Module 15-SP5 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Linux Enterprise High Performance Computing 15 SP2 LTSS 15-SP2 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Linux Enterprise High Performance Computing LTSS 15 SP3 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Linux Enterprise High Performance Computing ESPOS 15 SP4 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Linux Enterprise High Performance Computing LTSS 15 SP4 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Linux Enterprise Desktop 15 SP4 LTSS 15-SP4 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Linux Enterprise Server 15 SP2 LTSS 15-SP2 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Linux Enterprise Server 15 SP3 LTSS 15-SP3 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Linux Enterprise Server 15 SP4 LTSS 15-SP4 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Linux Enterprise Server for SAP Applications 15 SP2 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Linux Enterprise Server for SAP Applications 15 SP3 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Linux Enterprise Server for SAP Applications 15 SP4 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Manager Proxy 4.3 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Manager Retail Branch Server 4.3 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Manager Server 4.3 (src):
 systemd-rpm-macros-15-150000.7.39.1
SUSE Enterprise Storage 7.1 (src):
 systemd-rpm-macros-15-150000.7.39.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

Please indicate the current expectation for fixing the bug using the Target Milestone field. What do we believe is possible? 

In the cases where we do not plan to deliver or cannot guess (!?) please do not enter anything, but you can comment if you wish to provide more details.

Released in maintenance ; deletion request ready at https://build.suse.de/request/show/324941

Will be merged once we sync Maintenance updates (which is a manual operation after RC1, and we scheduled that for next week)

Maintenance sync complete ; deletion request accepted, so systemd-rpm-macros is now going through again via inheritance.