Bug 1221005 - [Build 62.1] Can't login system as root after migration or minimal installation
Summary: [Build 62.1] Can't login system as root after migration or minimal installation
Status: RESOLVED DUPLICATE of bug 1223486
Alias: None
Product: PUBLIC SUSE Linux Enterprise Server 15 SP6
Classification: openSUSE
Component: Other (show other bugs)
Version: unspecified
Hardware: S/390-64 SLES 15
: P2 - High : Normal
Target Milestone: Public RC
Assignee: Eugenio Paolantonio
QA Contact:
URL: https://openqa.suse.de/tests/13713500...
Whiteboard: On the staging; Jan Stehlik to talek ...
Keywords:
Depends on:
Blocks:
 
Reported: 2024-03-06 03:27 UTC by Lemon Li
Modified: 2024-07-17 12:40 UTC (History)
15 users (show)

See Also:
Found By: openQA
Services Priority:
Business Priority:
Blocker: Yes
Marketing QA Status: ---
IT Deployment: ---


Attachments
logs_from_installation_system-y2logs.tar.bz2 (572.53 KB, application/x-bzip)
2024-03-06 03:27 UTC, Lemon Li
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Lemon Li 2024-03-06 03:27:58 UTC
Created attachment 873238 [details]
logs_from_installation_system-y2logs.tar.bz2

## Observation
on build 62.1, some migration test failed to login as root after migration on s390x, https://openqa.suse.de/tests/13715356/modules/system_prepare/steps/3

And this issue only happened on offline migration test with default modules.

Same issue:
https://openqa.suse.de/tests/13713291#step/system_prepare/3
https://openqa.suse.de/tests/13715297#step/system_prepare/3
https://openqa.suse.de/tests/13715356#step/system_prepare/3

I have tried to do manual check before and after migration:

Before migration: Root login enabled, https://openqa.suse.de/tests/13715804#step/check_system_info/2

After migration: Root login disabled:

susetest:/etc/ssh # grep -nr PermitRootLogin
sshd_config:39:#PermitRootLogin prohibit-password
sshd_config:87:# the setting of "PermitRootLogin prohibit-password".

susetest:/etc/ssh/sshd_config.d # rpm -qf /etc/ssh/sshd_config
openssh-server-9.3p2-150600.1.1.s390x

openQA test in scenario sle-15-SP6-Migration-from-SLE15-SPx-s390x-migr_sles15sp5_role_minimal@s390x-kvm fails in
[system_prepare](https://openqa.suse.de/tests/13715356/modules/system_prepare/steps/3)

## Test suite description
Offline migration from sles 15 sp5 with default addons. Origin system has system role minimal and default patterns.


## Reproducible

Fails since (at least) Build [59.2](https://openqa.suse.de/tests/13636205)


## Expected result

Last good: [57.1](https://openqa.suse.de/tests/13549345) (or more recent)


## Further details

Always latest result in this scenario: [latest](https://openqa.suse.de/tests/latest?arch=s390x&distri=sle&flavor=Migration-from-SLE15-SPx&machine=s390x-kvm&test=migr_sles15sp5_role_minimal&version=15-SP6)
Comment 1 Lemon Li 2024-03-06 07:22:49 UTC
This should be the same issue https://openqa.suse.de/tests/13713749#step/hostname/3

This is an installation test that skip registration with system role of minimal, after installation and reboot it failed to login as root.
Comment 2 Roy Cai 2024-03-12 06:07:50 UTC
also hit the same issue on aarch64 platform:
http://openqa.qa2.suse.asia/tests/70496#step/reboot_and_wait_up_upgrade/15
Comment 3 Marcela Maslanova 2024-03-13 10:05:00 UTC
Is it possible that you hit the openssh version, which switched off the option to login as a root? It was there as default for a while and it was removed later on. Can you re-run the test with latest build and let me know?

I have a similar report from one of VM on the s390x machine.
Comment 4 Lemon Li 2024-03-14 01:17:06 UTC
(In reply to Marcela Maslanova from comment #3)
> Is it possible that you hit the openssh version, which switched off the
> option to login as a root? It was there as default for a while and it was
> removed later on. Can you re-run the test with latest build and let me know?
> 
> I have a similar report from one of VM on the s390x machine.

Till now there is no new build and still the 62.1, I think you mean test it on next new build, I will update the test result when new build comes.

Anyway, I cloned the job on build 62.1 and got the same issue. http://openqa.suse.de/tests/13770654#step/system_prepare/3
Comment 5 Eugenio Paolantonio 2024-03-14 18:04:24 UTC
Thanks for the bug report! Heads up: this should still happen in the upcoming 64.2.
Comment 6 Lemon Li 2024-03-18 01:37:33 UTC
(In reply to Eugenio Paolantonio from comment #5)
> Thanks for the bug report! Heads up: this should still happen in the
> upcoming 64.2.

Yes, same issue on build 64.2, https://openqa.suse.de/tests/13784831#step/system_prepare/3
Comment 7 Lemon Li 2024-03-22 02:02:01 UTC
Update status, we have the same issue on latest build 67.1 https://openqa.suse.de/tests/13838804#step/system_prepare/3
Comment 8 Roy Cai 2024-03-25 00:39:58 UTC
the same issue on build 71.1 (aarch64 platform)
http://openqa.qa2.suse.asia/tests/70728#step/reboot_and_wait_up_upgrade/1
Comment 9 Lemon Li 2024-03-28 02:02:04 UTC
After some investigation on the bug, I think it failed for ssh related package haven't installed. I tried to add pattern 'enhanced_base' to minimal role profile(by default patterns are 'apparmor, base, and minimal_base), then run installation( https://openqa.suse.de/tests/13886356# ) then migration test, it passed https://openqa.suse.de/tests/13887630#.

So the minimal role do not have the needed package installed, I found this test passed on build 57.1 https://openqa.suse.de/tests/13549345, I guess after that build the related package moved out of the minimal role. Please help to confirm that.
Comment 10 Joaquín Rivera 2024-03-28 04:55:45 UTC
So the question is, would be acceptable for the customer to discover that when a minimal system is installed now, the system needs to be adjusted post-installation to be accessed with ssh root if needed?
Comment 11 Radoslav Tzvetkov 2024-04-02 10:04:32 UTC
We are tightening the SUSE distribution security, so please, it is not a problem. 

Eventually, it'll need to be announced in the release notes. I'll discuss downgrading this to P3.
Comment 12 Radoslav Tzvetkov 2024-04-02 10:09:49 UTC
Please indicate the current expectation for fixing the bug using the Target Milestone field. What do we believe is possible? 

In the cases where we do not plan to deliver or cannot guess (!?) please do not enter anything, but you can comment if you wish to provide more details.
Comment 13 Joaquín Rivera 2024-04-03 08:36:53 UTC
Radoslav you mentioned "it is not a problem", does it mean that in QE we need to workaround it?
On the other hand it looks like we are waiting for information about how can this be fixed, so I got confused. Could you please clarify? thanks.
Comment 14 Radoslav Tzvetkov 2024-04-04 08:59:53 UTC
Michal, this is one of the exception cases. Shall we take this exception in behavior that needs to be documented for the customer on the basis of improved security?
Comment 15 Radoslav Tzvetkov 2024-04-04 13:00:58 UTC
We consider it a corner case in minimal installation, and when we do minimal installation, SUSE does not recommend upgrading.

Since the PublicBeta, no customer has reacted to it. Repairing it would require rolling back the previous behavior and diverging from the Tumbleweed and Leap distributions. Allowing default root access is arguably substandard.  

It was discussed on the PublicBeta as an acceptable corner case.
Comment 16 Michal Svec 2024-04-05 11:16:01 UTC
We should have the same behavior as we had with the previous SPs.
(Without rolling-back)
Comment 17 Joaquín Rivera 2024-04-11 06:54:39 UTC
@Radoslav, I wasn't aware of that, so we will remove our test coverage for migration with system with minimal role. Is that written somewhere that you can point me? thanks.

then for installation it will not be possible to validate the system without root access for remote workers in openQA, like s390x. https://openqa.suse.de/tests/13957007#step/system_prepare/3.
What would be the workaround? do we need to install some specific package during installation or should we drop any validation in this scenario?
Comment 18 Radoslav Tzvetkov 2024-04-11 12:06:44 UTC
Hi Ming Li, are only the migration tests broken? 

Also, there is some discussion on #team-lsg-qe-yam, which I see can alleviate this. So that the yast installs this additional package.
Comment 19 Michal Svec 2024-04-11 13:21:51 UTC
FTR this should be fixed. I'm not sure how to interpret comment #17
so I just want to emphasize that.
Comment 20 Radoslav Tzvetkov 2024-04-11 15:01:34 UTC
Why the test does not install «Recommends» by default?
Comment 21 Joaquín Rivera 2024-04-12 04:31:46 UTC
comment #17 is about:
(1) SUSE does not recommend upgrading when minimal installation, so in QE we will no cover then this scenario, a migration with minimal role doesn't make sense then. My question here was if that is reflected on documentation or where is that information?

(2) For installation with minimal role it fails (answering comment #18) only for those architecture where for testing we need to connect via ssh as root, s390x and ppc64le:
https://openqa.suse.de/tests/13957007#step/system_prepare/3
https://openqa.suse.de/tests/13960148#step/system_prepare/2
so if this is not fixed or workaround somehow we cannot validate further steps, for example if the system installed what was supposed to install, we cannot interact with the system post-installation.
Comment 22 Lemon Li 2024-04-12 06:23:10 UTC
(In reply to Radoslav Tzvetkov from comment #18)
> Hi Ming Li, are only the migration tests broken? 
> 
> Also, there is some discussion on #team-lsg-qe-yam, which I see can
> alleviate this. So that the yast installs this additional package.

Not only migration test but also some installation test with minimal role on s390x.

I'm talking in yam channel for this, and trying to find which pattern/packages need to be added during installation with default patterns to workaround this issue on SLE12SP5.

Lemon
Comment 24 Antonio Larrosa 2024-04-12 18:06:09 UTC
https://build.suse.de/request/show/326223 should fix this.
Comment 26 Eugenio Paolantonio 2024-04-18 09:31:09 UTC
Checking in 80.1 looks like these testcases got fixed. @Melon, @Jan could you please confirm (perhaps I missed some others)? Thanks in advance!
Comment 27 Eugenio Paolantonio 2024-04-18 09:32:06 UTC
Sorry, I wanted to write Lemon! Apologies :)
Comment 28 Lemon Li 2024-04-18 10:05:29 UTC
(In reply to Eugenio Paolantonio from comment #26)
> Checking in 80.1 looks like these testcases got fixed. @Melon, @Jan could
> you please confirm (perhaps I missed some others)? Thanks in advance!

I checked the latest build 80.1 and haven't found the same issue. I think it got fixed. Thanks.

Lemon
Comment 29 Eugenio Paolantonio 2024-04-18 10:23:31 UTC
Thank you, closing then.
Comment 30 Lemon Li 2024-04-19 07:46:16 UTC
Verified on build 80.1.
Comment 31 Roy Cai 2024-04-29 03:15:08 UTC
After upgrading from sles15sp5 to sles15sp6 latest, Still hit the same issue on aarch64 platform with build 80.1 and 82.1
http://openqa.qa2.suse.asia/tests/71454#step/reboot_and_wait_up_upgrade/15

Cannot login with correct password remotely.
# ssh root@vh-aarch64-01.qa2.suse.asia
Warning: Permanently added 'vh-aarch64-01.qa2.suse.asia' (ED25519) to the list of known hosts.
(root@vh-aarch64-01.qa2.suse.asia) Password: 
(root@vh-aarch64-01.qa2.suse.asia) Password: 
(root@vh-aarch64-01.qa2.suse.asia) Password: 


then login by ipmi, get such information:
vh-aarch64-01:~ # cat /etc/ssh/sshd_config | grep prohibit
#PermitRootLogin prohibit-password
# the setting of "PermitRootLogin prohibit-password".

vh-aarch64-01:~ # rpm -qa | grep ssh
openssh-server-9.6p1-150600.2.1.aarch64
libssh2-1-1.11.0-150600.18.1.aarch64
sshpass-1.05-qa.3.5.aarch64
openssh-clients-9.6p1-150600.2.1.aarch64
libssh-config-0.9.8-150600.9.1.aarch64
openssh-common-9.6p1-150600.2.1.aarch64
openssh-9.6p1-150600.2.1.aarch64
openssh-askpass-gnome-9.6p1-150600.2.1.aarch64
libssh4-0.9.8-150600.9.1.aarch64


Workaround:
PermitRootLogin prohibit-password" should be changed to "PermitRootLogin yes" to allow root login with a password.

Please help confirm it on aarch64 platform, thanks.
Comment 32 Jan Stehlik 2024-04-29 08:13:35 UTC
Since it was confirmed to still be present on ARM, please open a duplicate of this bug for aarch64.
Comment 33 Lemon Li 2024-04-29 08:22:46 UTC
I noticed that this issue is a bit different from our Yast & migration issue, this is migration from KVM Virtualization Host role (http://openqa.qa2.suse.asia/tests/71454#step/system_role/9) while not minimal role. Just for your reference. 

Lemon
Comment 34 Radoslav Tzvetkov 2024-04-29 08:38:40 UTC
Hi Ming Li, can you duplicate this bug by changing the architecture for aarch64? 

Thank you
Comment 35 Lemon Li 2024-04-29 09:29:57 UTC
Roy already filed the bug on aarch64, https://bugzilla.suse.com/show_bug.cgi?id=1223486
Comment 36 Lemon Li 2024-04-29 09:55:50 UTC

*** This bug has been marked as a duplicate of bug 1223486 ***