Bug 1224249 - [Build 90.1] zypper migration failed because of python3-argcomplete installation warning: %post(python3-argcomplete-1.9.2-150000.3.5.1.noarch) scriptlet failed, exit status 2
Summary: [Build 90.1] zypper migration failed because of python3-argcomplete installat...
Status: REOPENED
Alias: None
Product: PUBLIC SUSE Linux Enterprise Server 15 SP6
Classification: openSUSE
Component: Basesystem (show other bugs)
Version: unspecified
Hardware: Other Other
: P2 - High : Major
Target Milestone: ---
Assignee: Daniel Garcia
QA Contact:
URL: https://openqa.suse.de/tests/14308798...
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-05-15 05:25 UTC by Chenzi Cao
Modified: 2024-07-11 19:23 UTC (History)
10 users (show)

See Also:
Found By: openQA
Services Priority:
Business Priority:
Blocker: Yes
Marketing QA Status: ---
IT Deployment: ---


Attachments
zypper patch output (228.12 KB, text/x-log)
2024-06-20 13:25 UTC, Michele Pagot
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Chenzi Cao 2024-05-15 05:25:24 UTC
This issue happens on zypper migration to sles15sp6 build90.1

Steps:
- Boot SLES15SP3/SP4/SP5 images
- Register system and its modules via SCC
- Fully patch the packages and reboot the system
- Set proxySCC url (http://all-90.1.proxy.scc.suse.de) to /etc/SUSEConnect
- Perform online migration by "zypper migration"

Observed:
After all the packages got upgraded, it prints "migration failed", and then it started to do a migration again. 

Please find the serial0 log:
https://openqa.suse.de/tests/14308798/logfile?filename=serial0.txt

I cut part of the log here:
======================================
(1315/1315) Installing: patterns-server-kvm_tools-20180302-150600.24.1.x86_64 [..done]
Running post-transaction scripts [...
%posttrans(ucode-amd-20240201-150600.1.2.noarch) script output:
dracut[I]: Executing: /usr/bin/dracut --kver=5.14.21-150500.55.59-default -f
...
...
...
View the notifications now? [y/n] (n): There are running programs which still use files and libraries deleted or updated by recent upgrades. They should be restarted to benefit from the latest updates. Run 'zypper ps -s' to list these programs.
 
Since the last system boot core libraries or services have been updated.
Reboot is suggested to ensure that your system benefits from these updates
=======================================

[my comment] Until here, it looks good, but on the screen,it prints "Migration failed". 
Then it started to do a upgrade twice, log as below:

========================================
command '/usr/bin/zypper --releasever 15.6 --no-refresh dist-upgrade --no-allow-vendor-change' failed
Error: zypper returned 107 with 'Warning: Enforced setting: $releasever=15.6
Loading repository data...
Reading installed packages...
Warning: You are about to do a distribution upgrade with all enabled repositories. Make sure these repositories are compatible before you continue. See 'man zypper' for more information about this command.
Computing distribution upgrade...

The following 1087 packages are going to be upgraded:
==========================================

Please find the openqa test results as bellow:
https://openqa.suse.de/tests/14308796#step/zypper_migration/21 (15sp3 to 15sp6)
https://openqa.suse.de/tests/14308797#step/zypper_migration/18 (15sp4 to 15sp6)
https://openqa.suse.de/tests/14308798#step/zypper_migration/18 (15sp5 to 15sp6)

And this is the autoinst log:
https://openqa.suse.de/tests/14308798/logfile?filename=autoinst-log.txt

## Observation

openQA test in scenario sle-15-SP6-Migration-from-SLE15-SPx-x86_64-migr_sles15sp5_desktop-dev_all@64bit fails in
[zypper_migration](https://openqa.suse.de/tests/14308798/modules/zypper_migration/steps/19)

## Test suite description
Online migration from sles15 sp5 with addons desk,dev. Migration via zypper. Origin system has system role gnome and all patterns. Additionally, rollback is performed after migration.


## Reproducible

Fails since (at least) Build [90.1](https://openqa.suse.de/tests/14307367)


## Expected result

Last good: [89.1](https://openqa.suse.de/tests/14286963) (or more recent)


## Further details

Always latest result in this scenario: [latest](https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Migration-from-SLE15-SPx&machine=64bit&test=migr_sles15sp5_desktop-dev_all&version=15-SP6)
Comment 1 Benjamin Zeller 2024-05-15 06:45:36 UTC
> Error: zypper returned 107 with 'Warning: Enforced setting: $releasever=15.6

107 is a informal error code meaning: "some rpm %post configuration script failed"

So one of the packages that was installed had a failure in a rpm post script. This is not a zypp bug. Maybe a bug for the package that failed installing.
Comment 2 Benjamin Zeller 2024-05-15 07:00:41 UTC
From the zypper logs:

> (  39/1315) Installing: python3-argcomplete-1.9.2-150000.3.5.1.noarch [....
> update-alternatives: error: alternative python-argcomplete-check-easy-install-script can't be slave of register-python-argcomplete: it is a master alternative
>warning: %post(python3-argcomplete-1.9.2-150000.3.5.1.noarch) scriptlet failed, exit status 2
>.done]

So python3-argcomplete-1.9.2-150000.3.5.1 produces the failure. It's debatable if zypper-migration should
treat informal exit codes as error in the first place though.
Comment 3 Chenzi Cao 2024-05-15 07:28:56 UTC
Thank you for checking this, I reassign this bug report now.
Comment 4 Robert Schweikert 2024-05-15 12:43:09 UTC
Well if argcomplete falling is the issue that would be the Python packaging team.
Comment 5 Daniel Garcia 2024-05-15 12:47:56 UTC
(In reply to Robert Schweikert from comment #4)
> Well if argcomplete falling is the issue that would be the Python packaging
> team.

The python-argcomplete update-alternatives problem was solved the past week with this request: https://build.suse.de/request/show/329803, and as far as I can tell it was released yesterday https://smelt.suse.de/incident/33805/
Comment 6 Radoslav Tzvetkov 2024-05-16 08:54:40 UTC
build 90.1
Comment 7 Chenzi Cao 2024-05-16 09:30:07 UTC
Hi Daniel, is the fix also submitted to 15SP6 please? 

I rerun the job this afternoon, the issue is still existing:
https://openqa.suse.de/tests/14317958#step/zypper_migration/18
Comment 8 Daniel Garcia 2024-05-16 10:26:52 UTC
(In reply to Chenzi Cao from comment #7)
> Hi Daniel, is the fix also submitted to 15SP6 please? 
> 
> I rerun the job this afternoon, the issue is still existing:
> https://openqa.suse.de/tests/14317958#step/zypper_migration/18

It should be inherited, in 15SP6 the python3-argcomplete package should be inherited from SUSE:SLE-15:Update, and the fix was done there.
Comment 9 Chenzi Cao 2024-05-17 09:05:05 UTC
Hi, I have to reopen this bug report, the issue still happening on GMC candidate build91.1.

The python3-argcomplete is downgraded, see the output as below:

The following 30 packages are going to be downgraded:
  389-ds aaa_base aaa_base-extras coreutils coreutils-doc coreutils-lang ghostscript ghostscript-x11 lib389 libgck-1-0 libgcr-3-1 libjavascriptcoregtk-4_0-18 libjavascriptcoregtk-4_1-0 libsvrcore0 libwebkit2gtk-4_0-37 libwebkit2gtk-4_1-0 libwireshark15 libwiretap12 libwsutil13 libzypp python3-argcomplete python3-rpm rpm rpm-32bit rpm-build rpm-devel webkit2gtk-4_0-injected-bundles webkit2gtk-4_1-injected-bundles wireshark wireshark-ui-qt
(Log from https://openqa.suse.de/tests/14340398#step/zypper_migration/6)

And please see the log when installing the package of python3-argcomplete:

Retrieving: python3-argcomplete-1.9.2-150000.3.5.1.noarch (SLE-Module-Basesystem15-SP6-Pool) (37/1283),  56.7 KiB    
Retrieving: python3-argcomplete-1.9.2-150000.3.5.1.noarch.rpm [..done]

(  42/1319) Installing: python3-argcomplete-1.9.2-150000.3.5.1.noarch [.[  764.891930] [RPM][9667]: Transaction ID 6647158a started
.[  765.188474] [RPM][9667]: erase python3-argcomplete-1.9.2-150000.3.8.1.noarch: success
.[  765.367752] [RPM][9667]: scriptlet %post(python3-argcomplete-1.9.2-150000.3.5.1.noarch) failure: 2
[  765.369136] [RPM][9667]: install python3-argcomplete-1.9.2-150000.3.5.1.noarch: success
[  765.380665] [RPM][9667]: erase python3-argcomplete-1.9.2-150000.3.8.1.noarch: success

update-alternatives: error: alternative python-argcomplete-check-easy-install-script can't be slave of register-python-argcomplete: it is a master alternative
warning: %post(python3-argcomplete-1.9.2-150000.3.5.1.noarch) scriptlet failed, exit status 2
[  765.480042] [RPM][9667]: install python3-argcomplete-1.9.2-150000.3.5.1.noarch: success
[  765.481402] [RPM][9667]: 0 elements failed, 1 scripts failed
[  765.483038] [RPM][9667]: Transaction ID 6647158a finished: 0
done]
(log from https://openqa.suse.de/tests/14340398#step/zypper_migration/11)

And I checked the SLES15SP6 build91.1: the version of python3-argcomplete is still 1.9.2-150000.3.5.1.

zypper se -s python3-argcomplete
Refreshing service 'Basesystem_Module_15_SP6_x86_64'.
Refreshing service 'SUSE_Linux_Enterprise_Server_15_SP6_x86_64'.
Refreshing service 'Server_Applications_Module_15_SP6_x86_64'.
Loading repository data...
Reading installed packages...

S | Name                | Type    | Version            | Arch   | Repository
--+---------------------+---------+--------------------+--------+------------------------------------
  | python3-argcomplete | package | 1.9.2-150000.3.5.1 | noarch | SLE-Module-Basesystem15-SP6-Pool
  | python3-argcomplete | package | 1.9.2-150000.3.5.1 | noarch | SLE-Module-Basesystem15-SP6-Updates

    Note: For an extended search including not yet activated remote resources please use 'zypper
    search-packages'.
Comment 10 Eugenio Paolantonio 2024-05-17 09:15:35 UTC
SP6 indeed ships with


python3-argcomplete-1.9.2-150000.3.5.1

(as a context, maintenance updates are synced weekly, and are actually frozen in the late stage of development, so even if the build service shows that the package is inherited it might not be there unfortunately)
Comment 12 Radoslav Tzvetkov 2024-05-21 08:29:24 UTC
Can we have a workaround for this in order not to block the testing?
Comment 13 Chenzi Cao 2024-05-21 08:53:57 UTC
(In reply to Radoslav Tzvetkov from comment #12)
> Can we have a workaround for this in order not to block the testing?

I would try to remove the package before migration as a workaround. 
While I have a question that, I just checked the last 15SP6 PublicRC milestone build, there's higher version of the package in SLE-Module-Basesystem15-SP6-Updates channel:

# cat /etc/issue

Welcome to SUSE Linux Enterprise Server 15 SP6 PublicRC-202404 (x86_64) - Kernel \r (\l).

eth0: \4{eth0} \6{eth0}

# zypper se -s python3-argcomplete
Refreshing service 'Basesystem_Module_15_SP6_x86_64'.
Refreshing service 'Python_3_Module_15_SP6_x86_64'.
Refreshing service 'SUSE_Linux_Enterprise_Server_15_SP6_x86_64'.
Refreshing service 'Server_Applications_Module_15_SP6_x86_64'.
Building repository 'SLE-Module-Basesystem15-SP6-Pool' cache .....................................................................[done]
Building repository 'SLE-Module-Basesystem15-SP6-Updates' cache ..................................................................[done]
Building repository 'SLE-Module-Python3-15-SP6-Pool' cache .......................................................................[done]
Building repository 'SLE-Module-Python3-15-SP6-Updates' cache ....................................................................[done]
Building repository 'SLE-Module-Server-Applications15-SP6-Pool' cache ............................................................[done]
Building repository 'SLE-Module-Server-Applications15-SP6-Updates' cache .........................................................[done]
Retrieving repository 'scc_rmt' metadata .........................................................................................[done]
Building repository 'scc_rmt' cache ..............................................................................................[done]
Loading repository data...
Reading installed packages...

S | Name                | Type    | Version            | Arch   | Repository
--+---------------------+---------+--------------------+--------+------------------------------------
  | python3-argcomplete | package | 1.9.2-150000.3.8.1 | noarch | SLE-Module-Basesystem15-SP6-Updates
  | python3-argcomplete | package | 1.9.2-150000.3.5.1 | noarch | SLE-Module-Basesystem15-SP6-Pool

    Note: For an extended search including not yet activated remote resources please use 'zypper
    search-packages'.

================================================================================

So the issue happens because when testing with proxySCC, the update channel is not opened please?
Comment 15 Chenzi Cao 2024-05-22 11:17:23 UTC
Added a workaround to remove the package before migration by submitting PR: 
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/19350,
and it works fine now:

https://openqa.suse.de/tests/14413778#step/zypper_migration/11
https://openqa.suse.de/tests/14413786#step/zypper_migration/14
https://openqa.suse.de/tests/14413783#step/zypper_migration/14

And filed a ticket to remove the workaround after the 15SP6 GM released and the package python3-argcomplete-1.9.2-150000.3.8.1 is synced to 15SP6 update channel:
https://progress.opensuse.org/issues/160724
Comment 16 Michele Pagot 2024-05-29 12:56:08 UTC
zypper patch fails in Azure cloud image 15SP5 for MU 33974

Test sequence:
1. create a VM using Azure cloud image suse:sles-sap-15-sp5-byos:gen2:latest (PAYG) or suse:sles-sap-15-sp5-byos:gen2:latest (BYOS)
2. register the image in case on BYOS
3. add the repo http://download.suse.de/ibs/SUSE:/Maintenance:/33974/SUSE_Updates_SLE-Module-Basesystem_15-SP5_x86_64,http://download.suse.de/ibs/SUSE:/Maintenance:/33974/SUSE_Updates_SLE-Module-Development-Tools_15-SP5_x86_64
4. run `zypper patch` with "/usr/bin/zypper --quiet --non-interactive --xmlout patch --auto-agree-with-licenses --no-recommends"

Test fails as zypper exit code is 107

From the /var/log/zyppy/history I can see

# 2024-05-24 13:56:30 python311-argcomplete-3.3.0-150400.12.12.2.noarch.rpm installed ok
# Additional rpm output:
# update-alternatives: error: alternative python-argcomplete-check-easy-install-script can't be master: it is a slave of register-python-argcomplete
# warning: %post(python311-argcomplete-3.3.0-150400.12.12.2.noarch) scriptlet failed, exit status 2
# 

Issue is only present on 15sp5 and not on sp4 or sp3
sp6 is no tested yet (no available official cloud image to use)
Comment 17 Marcus Meissner 2024-06-11 15:18:36 UTC
the azure 15 yp5 image needs to be refreshed to fix it i fear.

currently planned for August 2024, until critical issue found first.
Comment 18 Michele Pagot 2024-06-20 13:23:10 UTC
I manage to reproduce the issue about zypper returning 107 also manually (and not using Ansible in any way)

Create an Azure VM using catalog base image SUSE:sles-sap-15-sp5:gen2:latest

Add repo "http://download.suse.de/ibs/SUSE:/Maintenance:/33974/SUSE_Updates_SLE-Module-Basesystem_15-SP5_x86_64"

zypper ref
zypper patch -y (I also tested zypper --non-interactive patch --auto-agree-with-licenses --no-recommends)

Get Zypper exit with 107

Log of the zypper patch output in attach


For the automation used by QE-SAP in openQA, Alvaro Carvaja created workaround in https://github.com/SUSE/qe-sap-deployment/pull/245 that is based on the fact to run update/install python3-argcomplete before to run zypper patch
Comment 19 Michele Pagot 2024-06-20 13:25:16 UTC
Created attachment 875602 [details]
zypper patch output

refer to https://bugzilla.suse.com/show_bug.cgi?id=1224249#c18
Comment 20 Sergio Rafael Lemke 2024-06-25 13:00:48 UTC
Hello, python3-argcomplete was fixed as this was also a problem on upgrading 15-SP4 to 15-SP5, here: https://smelt.suse.de/incident/33805/

Is the procedure happening like this:
1- Fresh 15-SP4 or 15-SP5, then add maintenance repos and update.
2- Then attach repos. of the next product and update/upgrade again.

Dunno if this procedure is done with some tool, but if the python3-argcomplete being used is the one from GA or any before the one released on 33805(python-argcomplete-1.9.2-150000.3.8.1) then you will get that zypper error. 
IIRC there is one more such package, which was fixed the same way.

The procedure should always be to upgrade from a fully up-to-date system to the next GA, eg:
15-SP4 + LTSS and fully updates, only then attach 15-SP5 product./repos and update again.
The reason is, that is the scenario customer likely have.

The python problem itself is that the python3-xxx packages leaves some leftovers (alternatives), and later when installing the python311 of the same package zypper errors out. Update 33805 fixed the python3 packages.
Comment 21 Sergio Rafael Lemke 2024-06-25 13:33:56 UTC
One other package with similar effects is python3-docutils, which we update here:
https://smelt.suse.de/incident/32673/

The former  python3-docutils has some leftovers (alternatives), that make zypper exit on error when trying to install python311-docutils.
Comment 22 Robert Schweikert 2024-07-11 19:23:39 UTC
(In reply to Marcus Meissner from comment #17)
> the azure 15 yp5 image needs to be refreshed to fix it i fear.
> 
> currently planned for August 2024, until critical issue found first.

I am not sure what makes you draw this conclusion. python3-argcomplete which has the issue with the alternatives is completely independent of Azure or AWS or GCE.

If I have a SLES 15 SP5 system, laptop or server with the buggy python3-argcomplete installed I will run into this problem. I do not see where refreshed images solve any of this.

We need to document that we had bugs in python3-argcomplete and python3-docutils and that on any system these need to be updated to their latest version before migration to 15 SP6 or pulling in the latest Python 3.6 interpreter via "zypper patch"