Bug 1220594

Summary: [Build 59.2] openQA test fails in system_prepare - susetest sshd[5179]: error: PAM: Authentication failure for root from 10.145.10.5
Product: [openSUSE] PUBLIC SUSE Linux Enterprise Server 15 SP6 Reporter: yutao wang <yuwang>
Component: OtherAssignee: Eugenio Paolantonio <eugenio.paolantonio>
Status: VERIFIED FIXED QA Contact:
Severity: Normal    
Priority: P3 - Medium CC: dawei.pang, eugenio.paolantonio, ihno, jan.stehlik, leli, llzhao, pdostal, qe-virt, richard.fan, rtsvetkov, santiago.zarate, wchen, xlai, yuwang
Version: unspecified   
Target Milestone: ---   
Hardware: S/390-64   
OS: SLES 15   
URL: https://openqa.suse.de/tests/13636339/modules/system_prepare/steps/6
Whiteboard:
Found By: openQA Services Priority:
Business Priority: Blocker: Yes
Marketing QA Status: --- IT Deployment: ---
Bug Depends on:    
Bug Blocks: 1220661    
Attachments: installation system logs
ha_cluster_join-SSH-failure

Description yutao wang 2024-02-29 01:50:31 UTC
Hardware: s390x
Software: sles15sp4 migration to sles15sp6
***************************************************************************
## Observation

openQA test in scenario sle-15-SP6-Migration-from-SLE15-SPx-s390x-migr_sles15sp4_lp@s390x-kvm fails in
[system_prepare](https://openqa.suse.de/tests/13636339/modules/system_prepare/steps/6)

## Test suite description
Offline migration from sles 15 sp4 with addons live. Origin system has system role textmode and default patterns.


## Reproducible

Fails since (at least) Build [59.2](https://openqa.suse.de/tests/13636339) (current job)


## Further details

Always latest result in this scenario: [latest](https://openqa.suse.de/tests/latest?arch=s390x&distri=sle&flavor=Migration-from-SLE15-SPx&machine=s390x-kvm&test=migr_sles15sp4_lp&version=15-SP6)
*************************************************************************
Test steps:
1. Installed sles15sp4 with addons live. Origin system has system role textmode and default patterns.
2. Offline migration to sles15sp6
3. After upgrade, relogin the system.
Got error message:
2024-02-28T17:30:21.084902-05:00 susetest sshd[5179]: error: PAM: Authentication failure for root from 10.145.10.5
Comment 1 yutao wang 2024-02-29 02:01:06 UTC
Created attachment 873099 [details]
installation system logs
Comment 2 Richard Fan 2024-02-29 02:58:29 UTC
The issue can be found in fresh installed system as well https://openqa.suse.de/tests/13637002#step/system_prepare/5
Comment 3 Richard Fan 2024-02-29 03:53:14 UTC
The issue is gone with below configuration:

localhost:/etc/ssh/sshd_config.d # cat root.conf
PermitRootLogin yes
Comment 4 lili zhao 2024-02-29 07:42:48 UTC
Not sure if it the same issue as the error msg are not the same, it happened on ppc64le/x86_64/aarch64 in HA test cases.

"ha-cluster-join" command reports: 
"
INFO: Configuring SSH passwordless with root@$node02
...
ERROR: cluster.join: Failed to login to root@$node01. Please check the credentials.
"
See attached pic for more info.

For example:
  ppc64le-2g: https://openqa.suse.de/tests/13643431#step/ha_cluster_join/10
  x86_64: https://openqa.suse.de/tests/13643427#step/ha_cluster_join/10
  aarch64: https://openqa.suse.de/tests/13643434#step/ha_cluster_join/10
Comment 5 lili zhao 2024-02-29 07:44:05 UTC
Created attachment 873104 [details]
ha_cluster_join-SSH-failure
Comment 6 Richard Fan 2024-02-29 09:29:20 UTC
(In reply to Richard Fan from comment #3)
> The issue is gone with below configuration:
> 
> localhost:/etc/ssh/sshd_config.d # cat root.conf
> PermitRootLogin yes

The same issue/workaround can be found on x86_64 as well
Comment 8 Dawei Pang 2024-02-29 14:35:23 UTC
Encountered the similar issue when ssh login with root account

http://10.67.129.4/tests/69186#step/login_console/15
Comment 10 Lemon Li 2024-03-01 10:27:59 UTC
I cloned a migration job(migration from SLES15SP5 to SLES15SP6) to do two manual test:

1. Before migration, don't change anything for /etc/ssh/sshd_config, the root login is enabled by default.
   'PermitRootLogin yes'
   After migration found the /etc/ssh/sshd_config changed:
   '#PermitRootLogin prohibit-password'

2. Before migration, change the /etc/ssh/sshd_config to disable the root login:
   '#PermitRootLogin yes'
   After migration, the /etc/ssh/sshd_config changed:
   '#PermitRootLogin prohibit-password'

It seems the sshd_config will be updated after migration and not depend on the file content before migration.

Just for you reference.
Comment 11 Radoslav Tzvetkov 2024-03-01 14:21:33 UTC
*** Bug 1220661 has been marked as a duplicate of this bug. ***
Comment 12 Richard Fan 2024-03-05 08:59:33 UTC
Hello Eugenio,

With the latest sle15sp6 with fresh installation. I can see the new package 

# rpm -ql openssh-server-config-rootlogin-9.3p2-150600.1.1.x86_64
/etc/ssh/sshd_config.d/50-permit-root-login.conf

# cat  /etc/ssh/sshd_config.d/50-permit-root-login.conf
PermitRootLogin yes

===================================

However, I have one more question here.

Without this package, I can add this line to /etc/ssh/sshd_config file, and it can work as well. is it by design? the reason why I asked this question is that we used to enable  root ssh for sle version <=sle15sp6

echo "PermitRootLogin yes" >> /etc/ssh/sshd_config
Comment 13 Santiago Zarate 2024-03-05 09:12:08 UTC
(In reply to Richard Fan from comment #12)
> Hello Eugenio,
> 
> With the latest sle15sp6 with fresh installation. I can see the new package 
> 
> # rpm -ql openssh-server-config-rootlogin-9.3p2-150600.1.1.x86_64
> /etc/ssh/sshd_config.d/50-permit-root-login.conf
> 
> # cat  /etc/ssh/sshd_config.d/50-permit-root-login.conf
> PermitRootLogin yes
> 
> ===================================
> 
> However, I have one more question here.
> 
> Without this package, I can add this line to /etc/ssh/sshd_config file, and
> it can work as well. is it by design? the reason why I asked this question
> is that we used to enable  root ssh for sle version <=sle15sp6
> 
> echo "PermitRootLogin yes" >> /etc/ssh/sshd_config

We should unify the test code, so we use the same file everywhere, instead of sshd_config

https://github.com/search?q=repo%3Aos-autoinst%2Fos-autoinst-distri-opensuse+permitroot+language%3APerl&type=code&l=Perl

> I can add this line to /etc/ssh/sshd_config file, and it can work as well. is it by design?

I would think so, it's the same behavior that there is for ALP, and for others (IIRC mircoos too)
Comment 14 Richard Fan 2024-03-05 09:51:48 UTC
(In reply to Santiago Zarate from comment #13)
> (In reply to Richard Fan from comment #12)
> > Hello Eugenio,
> > 
> > With the latest sle15sp6 with fresh installation. I can see the new package 
> > 
> > # rpm -ql openssh-server-config-rootlogin-9.3p2-150600.1.1.x86_64
> > /etc/ssh/sshd_config.d/50-permit-root-login.conf
> > 
> > # cat  /etc/ssh/sshd_config.d/50-permit-root-login.conf
> > PermitRootLogin yes
> > 
> > ===================================
> > 
> > However, I have one more question here.
> > 
> > Without this package, I can add this line to /etc/ssh/sshd_config file, and
> > it can work as well. is it by design? the reason why I asked this question
> > is that we used to enable  root ssh for sle version <=sle15sp6
> > 
> > echo "PermitRootLogin yes" >> /etc/ssh/sshd_config
> 
> We should unify the test code, so we use the same file everywhere, instead
> of sshd_config
> 
> https://github.com/search?q=repo%3Aos-autoinst%2Fos-autoinst-distri-
> opensuse+permitroot+language%3APerl&type=code&l=Perl
> 
> > I can add this line to /etc/ssh/sshd_config file, and it can work as well. is it by design?
> 
> I would think so, it's the same behavior that there is for ALP, and for
> others (IIRC mircoos too)

Thanks Santiago, I will try to fix our test code then
Comment 15 Lemon Li 2024-03-05 10:36:55 UTC
Hi, on build 62.1 we still can't login as root after migration on s390x, https://openqa.suse.de/tests/13713500#step/system_prepare/3

And this issue only happened on 15SP5 offline migration test with minimal role.

I have tried to do manual check before and after migration:

Before migration: Root login enabled, https://openqa.suse.de/tests/13715804#step/check_system_info/2

After migration: Root login disabled:

susetest:/etc/ssh # grep -nr PermitRootLogin
sshd_config:39:#PermitRootLogin prohibit-password
sshd_config:87:# the setting of "PermitRootLogin prohibit-password".

susetest:/etc/ssh/sshd_config.d # rpm -qf /etc/ssh/sshd_config
openssh-server-9.3p2-150600.1.1.s390x

So do you think this is a bug or by design? I will create a new bug if needed. Thanks.
Comment 16 Eugenio Paolantonio 2024-03-05 10:59:12 UTC
Thanks everyone for testing!


@Richard


yes, they both work. /etc/ssh/sshd_config.d/*.conf is read first, then /usr/etc/ssh/sshd_config.d/*.conf and then the rest of the contents of /etc/ssh/sshd_config.

From a rapid check in SP5 looks like that the ssh version there doesn't search those paths, so I'm afraid that to have one single test working on every SLE 15 Service Pack, changing /etc/ssh/sshd_config is the only way.


@Ming


yes, please open a new bug. Feel free to assign that to me. We need to evaluate that in the project side.
Comment 17 Richard Fan 2024-03-08 06:41:02 UTC
(In reply to Eugenio Paolantonio from comment #16)
> Thanks everyone for testing!
> 
> 
> @Richard
> 
> 
> yes, they both work. /etc/ssh/sshd_config.d/*.conf is read first, then
> /usr/etc/ssh/sshd_config.d/*.conf and then the rest of the contents of
> /etc/ssh/sshd_config.
> 
> From a rapid check in SP5 looks like that the ssh version there doesn't
> search those paths, so I'm afraid that to have one single test working on
> every SLE 15 Service Pack, changing /etc/ssh/sshd_config is the only way.
> 
> 

Thanks much! then I can change my test code base on your kindly confirmation.

> @Ming
> 
> 
> yes, please open a new bug. Feel free to assign that to me. We need to
> evaluate that in the project side.
Comment 18 Radoslav Tzvetkov 2024-03-11 11:05:14 UTC
Is this resolved? Please put it in resolved fixed when done. Thanks
Comment 19 yutao wang 2024-03-12 01:53:22 UTC
It has been fixed.
Verified jobs: http://openqa.suse.de/tests/13713541#
 http://openqa.suse.de/tests/13713541#
Comment 20 Jan Stehlik 2024-03-26 12:25:28 UTC
According to comment yutao wang 2024-03-12 01:53:22 UTC this issue has been resolved and verified.
Comment 21 Wayne Chen 2024-03-27 02:40:40 UTC
Also verified/fixed in virtualization test.