Bug 1223107

Created attachment 874367 [details]
failed to verify certificate: xx509: certificate signed by unknown authority

I asked SCC team for this issue and got analysis from Thomas that :
'Hi Lemon,
the error message says that your SMT/RMT are running with self-signed certificates.
That's expected behavior. But Yast should ask the user at this point if she wants
to accept & import the certificate to proceed. That seems not to happen here,
so maybe that's a bug in yast for 15SP6. 

Thomas Schmidt (tschmidt [at] suse.de) '

Thanks Thomas, so it is better to let yast team to have a look on this issue. there is no y2log for this is a libyui test for installation, but if more logs needed please tell me to support, thanks.

Lemon

> the error message says that your SMT/RMT are running with self-signed
> certificates.
> That's expected behavior. But Yast should ask the user at this point if she
> wants
> to accept & import the certificate to proceed. That seems not to happen here,
> so maybe that's a bug in yast for 15SP6. 

As from this statement, I do not see the bug as a test blocker.

Created attachment 874411 [details]
Screenshot: Expected certificate handling in SLE-15-SP5

https://openqa.suse.de/tests/14058366#step/rmt_registration_trust_import/1

Created attachment 874412 [details]
Screenshot: Actual certificate handling in SLE-15-SP6: Error

https://openqa.suse.de/tests/13923984#step/rmt_registration_trust_import/1

There are no y2logs in the failing OpenQA test case:

  https://openqa.suse.de/tests/13923984#downloads

Not sure what we are expected to do here without any logs. We need something to work with.

Created attachment 874414 [details]
diff of the yast-registration source code between the SLE-15-SP5 branch and the SLE-15-SP6 branch

This is the result of

>> git diff upstream/SLE-15-SP5..upstream/SLE-15-SP6 src


There is no difference in the certificate handling.

Yet the error handling for that part comes from here:

  https://github.com/yast/yast-registration/blob/SLE-15-SP6/src/lib/registration/connect_helpers.rb#L130-L136

That code is unchanged for 7 years, the whole file is unchanged since 2 years.

According to the source code, that dialog to import the SSL certificate is shown for those error codes:

https://github.com/yast/yast-registration/blob/SLE-15-SP6/src/lib/registration/ssl_error_codes.rb#L38-L39

    SELF_SIGNED_CERT,
    SELF_SIGNED_CERT_IN_CHAIN

where

    # "self signed certificate"
    SELF_SIGNED_CERT = 18

    # "self signed certificate in certificate chain"
    SELF_SIGNED_CERT_IN_CHAIN = 19

which is consistent with the error code that we see in the SLE-15-SP5 screenshot: That one was reported as "Self signed certificate in certificate chain".

But here we have a different error: "certificate signed by unknown authority"

I couldn't find that error message "certificate signed by unknown authority" in any YaST code. Most likely, it comes from a rubygem or an SSL library for low-level SSL handling.

It is entirely possible that the rubygem or SSL library changed between SLE-15-SP5 and SLE-15-SP6, and that we need to adapt to that changed behavior.

But it is also possible that the certificate really has that different error in SP6.

And BTW that other source file ssl_error_codes.rb is also unchanged since many years, in this case 5 years:

https://github.com/yast/yast-registration/blame/SLE-15-SP6/src/lib/registration/ssl_error_codes.rb

(In reply to Ming Li from comment #2)
> the error message says that your SMT/RMT are running with self-signed
> certificates.
> That's expected behavior. But Yast should ask the user at this point if she
> wants
> to accept & import the certificate to proceed. That seems not to happen here,
> so maybe that's a bug in yast for 15SP6. 


Let's ask Ladislav, if this is the expected behavior.

BTW, is it documented anywhere? In RMT documentation?

I found the message in /usr/lib64/libsuseconnect.so which is part of package libsuseconnect, a subpackage of source package suseconnect-ng.

I see a mapping of error codes (probably from libOpenSSL?) here:

https://github.com/SUSE/connect-ng/blob/main/third_party/libsuseconnect/libsuseconnect.go#L274-L282

which includes x509.UnknownAuthorityError, which is mapped to error code 19 above, SELF_SIGNED_CERT_IN_CHAIN.

I suspect that mapping doesn't work anymore. Did that change in libOpenSSL maybe?

The error codes are defined in /usr/include/openssl/x509_vfy.h from package libopenssl-3-devel:

# define X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT          18
# define X509_V_ERR_SELF_SIGNED_CERT_IN_CHAIN            19

My Leap 15.5  has libopenssl-3-devel-3.0.8-150500.5.27.1.x86_64,
my Tumbleweed has libopenssl-3-devel-3.1.4-5.2.x86_64,
but there doesn't seem to be a significant difference in those headers.

The only obvious recent code change were libsuseconnect.go:

https://github.com/SUSE/connect-ng/commits/main/third_party/libsuseconnect/libsuseconnect.go


That doesn't necessarily mean that the problem must have come from there, but it would be the first obvious place to check.

Tom, please have a look.

How to reproduce: 

$ podman run --rm -ti registry.suse.com/suse/sle15:15.6 /bin/bash
# zypper rm -y container-suseconnect
# zypper ar -f http://updates.suse.de/SUSE/Products/SLE-Module-Basesystem/15-SP6/x86_64/product/ 156
# zypper in yast2-registration
# yast2 registration # -> register to https://rmt.scc.suse.de

Hi Stefan,
The github link you attached hasn't been released yet. But even there, we haven't made changes to the ssl error handling in over 3 years. This is the link to the release suseconnect error handling. https://github.com/SUSE/connect-ng/blob/99084c9ed4a133ea0569360b2c5d679ba8409470/libsuseconnect/libsuseconnect.go#L267

> I see a mapping of error codes (probably from libOpenSSL?) here:
> https://github.com/SUSE/connect-ng/blob/main/third_party/libsuseconnect/libsuseconnect.go#L274-L282
> which includes x509.UnknownAuthorityError, which is mapped to error code 19 above, SELF_SIGNED_CERT_IN_CHAIN.
> I suspect that mapping doesn't work anymore. Did that change in libOpenSSL maybe?

I agree. I've tested the same suseconnect version on 15SP5, and it works as expected. 
On SP6, it seems we don't receive x509.UnknownAuthorityError, but tls.CertificateVerificationError from the system.

See also bug #1223107 (very likely a duplicate?)

*** Bug 1223528 has been marked as a duplicate of this bug. ***

Bug #1223528 is indeed a duplicate of this, and that one was upgrade to P1 CRITICAL by the PM.

I don't think we can handle this on the YaST side: We'd probably open a lot of security problems if we simply treated all 'tls.CertificateVerificationError' values that same way.

IMHO this is a task for the openSSL maintainers.

osc maintainer -e openssl

Defined in project:  security:tls
  bugowner of openssl : 
   pmonrealgonzalez@suse.com, otto.hollmann@suse.com

  maintainer of openssl : 
   meissner@suse.com, mpluskal@suse.com, pmonrealgonzalez@suse.com, otto.hollmann@suse.com, -, openssl-maintainers@suse.de

Defined in project:  security
  bugowner of openssl : 
   meissner@suse.com

  maintainer of openssl : 
   meissner@suse.com, lrupp@suse.com, varkoly@suse.com, ahodgkinson@novell.com, pth@suse.com, jjolly@suse.com, duwe@suse.com, werner@suse.com, bitdealer@gmail.com, duane.buss@microfocus.com, mseben@gmail.com, alexander_naumov@opensuse.org, Greg.Freemyer@gmail.com, vpereira@suse.com, jsegitz@suse.com, -, security-team@suse.de

JFYI, the releveant YaST part has not been changed in the last 5 years: https://github.com/yast/yast-registration/blame/SLE-15-SP6/src/lib/registration/ssl_error_codes.rb

I'm assigning the bug to Otto.

the underlying openssl was changed, so it might return different error codes now.

I'm not sure what to do from the openssl side. There might be other moving parts involved. For example, here are some recent commits in yast-registration that change the behavior in SP6+:
  * https://github.com/yast/yast-registration/commit/a407c4bb
  * https://github.com/yast/yast-registration/commit/1e0b40b1
Also, I'm not familiar with the go1.XX-openssl package an the versions involved.

Not sure what has changed between [0] (worked 1 month ago) and [1] (failed 27 days ago). Looking at openssl, it looks like the code hasn't changed in the last 2 months in SP6:GA

HTH

[0] https://openqa.suse.de/tests/13818573#step/rmt_registration_trust_import/1
[1] https://openqa.suse.de/tests/13923984#step/rmt_registration_trust_import/1

(In reply to Pedro Monreal Gonzalez from comment #26)
> I'm not sure what to do from the openssl side. There might be other moving
> parts involved. For example, here are some recent commits in
> yast-registration that change the behavior in SP6+:
>   * https://github.com/yast/yast-registration/commit/a407c4bb
>   * https://github.com/yast/yast-registration/commit/1e0b40b1
> Also, I'm not familiar with the go1.XX-openssl package an the versions
> involved.

See comment #10 and the comments around that one.

suseconnect is not running into this condition (https://github.com/SUSE/connect-ng/blob/main/third_party/libsuseconnect/libsuseconnect.go#L274-L282) and not returning code 19, because it doesn't receive x509.UnknownAuthorityError anymore in SP6, but gets tls.CertificateVerificationError.

So, what is the next step?

I found that exactly this behaviour is described here
> https://github.com/golang/go/issues/59226

And it corresponds to release notes for Go 1.20
> https://go.dev/doc/go1.20#crypto/tls

> For a handshake failure due to a certificate verification failure, the TLS
> client and server now return an error of the new type
> CertificateVerificationError, which includes the presented certificates.

Is it possible that Go has been upgraded in the meantime?

I rerun the test on build 82.1 and got the same issue. http://openqa.suse.de/tests/14212028#step/rmt_registration_trust_import/1

As [comments #35] (https://bugzilla.suse.com/show_bug.cgi?id=1223107#c35) and [comments #36](https://bugzilla.suse.com/show_bug.cgi?id=1223107#c36), the issue seems comes from Go script.

Thomas, can you handle this tls.CertificateVerificationError  

in thirdparty/libsuseconnect/libsuseconnect.go

same as SSLError and return error 19

Yes, we're on it: https://trello.com/c/zzQHvy5z/5110-fix-suseconnect-ssl-certificate-handling-with-golang-121

Could you run a test with the package from https://build.opensuse.org/package/show/systemsmanagement:SCC/suseconnect-ng? We fixed the certificate issue in that build.

(In reply to Thomas Schmidt from comment #41)
> Could you run a test with the package from
> https://build.opensuse.org/package/show/systemsmanagement:SCC/suseconnect-
> ng? We fixed the certificate issue in that build.

Hi Thomas, I tried to install the updated pkg suseconnect-ng-1.9.0-150300.92.1.x86_64 http://openqa.suse.de/tests/14233856#step/register_system/6, but it still failed to verify certification http://openqa.suse.de/tests/14233856#step/register_system/11 . Is there anything wrong for my test?

Lemon

Hi @Ming Li, the fix is in the package libsuseconnect-1.9.0. Please update all the packages suseconnect-ruby-bindings-1.9.0, suseconnect-ng-1.9.0, libsuseconnect-1.9.0.

(In reply to Thomas Schmidt from comment #43)
> Hi @Ming Li, the fix is in the package libsuseconnect-1.9.0. Please update
> all the packages suseconnect-ruby-bindings-1.9.0, suseconnect-ng-1.9.0,
> libsuseconnect-1.9.0.

Hi Thomas, after updated these pkgs, the certification for RMT server works now. http://openqa.suse.de/tests/14235073#step/register_system/15

fixed package submitted to 15SP6: https://build.suse.de/request/show/329584

I think this can be closed as FIXED now.

Fixed in libsuseconnect-1.9.0

SUSE-RU-2024:1792-1: An update that has two fixes can now be installed.

Category: recommended (important)
Bug References: 1220679, 1223107
Maintenance Incident: [SUSE:Maintenance:33842](https://smelt.suse.de/incident/33842/)
Sources used:
SUSE Linux Enterprise Server 15 SP2 (src):
 suseconnect-ng-1.9.0-150100.3.26.3
SUSE Linux Enterprise High Performance Computing 15 SP2 LTSS 15-SP2 (src):
 suseconnect-ng-1.9.0-150100.3.26.3
SUSE Linux Enterprise High Performance Computing LTSS 15 SP3 (src):
 suseconnect-ng-1.9.0-150100.3.26.3
SUSE Linux Enterprise Server 15 SP2 LTSS 15-SP2 (src):
 suseconnect-ng-1.9.0-150100.3.26.3
SUSE Linux Enterprise Server 15 SP3 LTSS 15-SP3 (src):
 suseconnect-ng-1.9.0-150100.3.26.3
SUSE Linux Enterprise Server for SAP Applications 15 SP2 (src):
 suseconnect-ng-1.9.0-150100.3.26.3
SUSE Linux Enterprise Server for SAP Applications 15 SP3 (src):
 suseconnect-ng-1.9.0-150100.3.26.3
SUSE Enterprise Storage 7.1 (src):
 suseconnect-ng-1.9.0-150100.3.26.3

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

SUSE-RU-2024:1800-1: An update that has two fixes can now be installed.

Category: recommended (moderate)
Bug References: 1220679, 1223107
Maintenance Incident: [SUSE:Maintenance:33841](https://smelt.suse.de/incident/33841/)
Sources used:
SUSE Linux Enterprise Micro 5.1 (src):
 suseconnect-ng-1.9.0-150300.3.20.2

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

SUSE-RU-2024:1799-1: An update that has two fixes can now be installed.

Category: recommended (moderate)
Bug References: 1220679, 1223107
Maintenance Incident: [SUSE:Maintenance:33832](https://smelt.suse.de/incident/33832/)
Sources used:
openSUSE Leap 15.5 (src):
 suseconnect-ng-1.9.0-150500.3.21.2
SUSE Linux Enterprise Micro 5.5 (src):
 suseconnect-ng-1.9.0-150500.3.21.2
Basesystem Module 15-SP5 (src):
 suseconnect-ng-1.9.0-150500.3.21.2

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

SUSE-RU-2024:1811-1: An update that has two fixes can now be installed.

Category: recommended (moderate)
Bug References: 1220679, 1223107
Maintenance Incident: [SUSE:Maintenance:33836](https://smelt.suse.de/incident/33836/)
Sources used:
SUSE Linux Enterprise Micro 5.2 (src):
 suseconnect-ng-1.9.0-150300.3.15.2
SUSE Linux Enterprise Micro for Rancher 5.2 (src):
 suseconnect-ng-1.9.0-150300.3.15.2

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

SUSE-RU-2024:1839-1: An update that has two fixes can now be installed.

Category: recommended (important)
Bug References: 1220679, 1223107
Maintenance Incident: [SUSE:Maintenance:33843](https://smelt.suse.de/incident/33843/)
Sources used:
SUSE Linux Enterprise High Performance Computing 12 SP5 (src):
 suseconnect-ng-1.9.0-3.15.4
SUSE Linux Enterprise Server 12 SP5 (src):
 suseconnect-ng-1.9.0-3.15.4
SUSE Linux Enterprise Server for SAP Applications 12 SP5 (src):
 suseconnect-ng-1.9.0-3.15.4

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

SUSE-RU-2024:1940-1: An update that has two fixes can now be installed.

Category: recommended (moderate)
Bug References: 1220679, 1223107
Maintenance Incident: [SUSE:Maintenance:33835](https://smelt.suse.de/incident/33835/)
Sources used:
openSUSE Leap 15.4 (src):
 suseconnect-ng-1.9.0-150400.3.31.2
openSUSE Leap Micro 5.3 (src):
 suseconnect-ng-1.9.0-150400.3.31.2
openSUSE Leap Micro 5.4 (src):
 suseconnect-ng-1.9.0-150400.3.31.2
SUSE Linux Enterprise Micro for Rancher 5.3 (src):
 suseconnect-ng-1.9.0-150400.3.31.2
SUSE Linux Enterprise Micro 5.3 (src):
 suseconnect-ng-1.9.0-150400.3.31.2
SUSE Linux Enterprise Micro for Rancher 5.4 (src):
 suseconnect-ng-1.9.0-150400.3.31.2
SUSE Linux Enterprise Micro 5.4 (src):
 suseconnect-ng-1.9.0-150400.3.31.2
SUSE Linux Enterprise High Performance Computing ESPOS 15 SP4 (src):
 suseconnect-ng-1.9.0-150400.3.31.2
SUSE Linux Enterprise High Performance Computing LTSS 15 SP4 (src):
 suseconnect-ng-1.9.0-150400.3.31.2
SUSE Linux Enterprise Desktop 15 SP4 LTSS 15-SP4 (src):
 suseconnect-ng-1.9.0-150400.3.31.2
SUSE Linux Enterprise Server 15 SP4 LTSS 15-SP4 (src):
 suseconnect-ng-1.9.0-150400.3.31.2
SUSE Linux Enterprise Server for SAP Applications 15 SP4 (src):
 suseconnect-ng-1.9.0-150400.3.31.2
SUSE Manager Proxy 4.3 (src):
 suseconnect-ng-1.9.0-150400.3.31.2
SUSE Manager Retail Branch Server 4.3 (src):
 suseconnect-ng-1.9.0-150400.3.31.2
SUSE Manager Server 4.3 (src):
 suseconnect-ng-1.9.0-150400.3.31.2

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.