Bug 1184326 - zypper dup from Leap 15.2 to 15.3 beta: rpm verification keys not installed, rpms installed unverified
zypper dup from Leap 15.2 to 15.3 beta: rpm verification keys not installed, ...
Status: RESOLVED FIXED
Classification: openSUSE
Product: openSUSE Distribution
Classification: openSUSE
Component: Upgrade Problems
Leap 15.3
Other Other
: P5 - None : Major (vote)
: ---
Assigned To: E-mail List
Jiri Srain
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2021-04-04 08:45 UTC by Dirk Weber
Modified: 2021-11-24 02:30 UTC (History)
14 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
/var/log/zypp/history of the zypper dup from 15.2 to 15.3 (635.94 KB, text/plain)
2021-04-09 17:56 UTC, Dirk Weber
Details
build-rsa2048-39db7c82-5f68629b.asc (972 bytes, text/plain)
2021-05-21 14:29 UTC, Ruediger Oertel
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Dirk Weber 2021-04-04 08:45:27 UTC
Doing a zypper dup from a fully up to date Leap 15.2 to Leap 15.3 Beta (Build117.13) results in the following output of rpm for most packages being installed:

...
2021-04-04 09:30:26|install|perl-Sub-Install|0.928-1.24|noarch||openSUSE-15.3-OSS|364d1c9df44c9557376552a2cf1917736f492d03fd4fe51484b385fafd0632b1|
# 2021-04-04 09:30:27 perl-Sub-Exporter-Progressive-0.001013-1.24.noarch.rpm installed ok
# Additional rpm output:
# warning: /var/cache/zypp/packages/openSUSE-15.3-OSS/noarch/perl-Sub-Exporter-Progressive-0.001013-1.24.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 39db7c82: NOKEY
#
2021-04-04 09:30:27|install|perl-Sub-Exporter-Progressive|0.001013-1.24|noarch||openSUSE-15.3-OSS|82cb65a07ded38c2e14b223c2d3df3ae279dd7488942ff5dd18caa0ce63ccd07|
# 2021-04-04 09:30:27 perl-Perl-Tidy-20200110-bp153.1.17.noarch.rpm installed ok
# Additional rpm output:
# warning: /var/cache/zypp/packages/openSUSE-15.3-OSS/noarch/perl-Perl-Tidy-20200110-bp153.1.17.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 65176565: NOKEY
...

This indicates that the keys which are available in the repo

https://download.opensuse.org/distribution/leap/15.3/repo/oss/gpg-pubkey-39db7c82-5847eb1f.asc

https://download.opensuse.org/distribution/leap/15.3/repo/oss/gpg-pubkey-65176565-59787af5.asc

are not installed automatically and the packages are installed without verification. This could result in the system being compromised.

After the keys above are downloaded and imported into rpm manually package verification works.
Comment 1 Michael Andres 2021-04-09 13:56:14 UTC
Regarding gpg-pubkey-*.asc files in the medias root directory: 

  They are not installed automatically. 
  In fact they are not even visible to zypper, because they are not
  part of the repository (i.e. not mentioned in the repositories master
  index file repodata/repomd.xml).

  

Nevertheless the system is secure:

The repositories content is defined by the master index repodata/repomd.xml and
repodata/repomd.xml.asc is expected to provide the ascii armored signature for it.
 
In case the signing key is also provided as repodata/repomd.xml.key, and it is not in the database, zypp asks whether you want to trust and import it.

If the repo metadata are not signed, zypper will ask whether you want to accept the repo at all.


The packages are secured by their checksum, which is mentioned in the repos primary.xml. The primary.xml is secured via it's checksum mentioned in the repoindex.xml.


Zyppers default package verification now depends on whether the repo metadata have been signed or not: 
 
  For signed metadata we accept packages verified by their checksum 
  in the metadata.

  For unsigned metadata we require the package to be signed and the
  key to be imported in the database. We enforce the rpm signature
  check then.


The repos verification setting are visible in e.g. with 'zypper lr':

    #  | Alias  | ... | GPG Check | ...
    ---+--------+ ...-+-----------+-...
     1 | KDE3   | ... | (r ) Yes  | ...
     2 | LOCAL  | ... | ( p) Yes  | ...

'r' tells you that the repos metadata are signed and checked; 
'p' that the package signature check is enforced at download/install time.

This default behavior can be amended by applying more strict (or more relaxed) rules. Either per repo (see --gpgcheck* options to zypper ar/mr) or globally in /etc/zypp/zypp.conf (*gpgcheck options).
Comment 2 Dirk Weber 2021-04-09 16:04:08 UTC
Thanks for the comment, still I think this current behavior is worrying.
For many versions I already update my systems with zypper dup, and I am quite sure I never got this warning before.
Now it comes for most packages, maybe 2000 times during the upgrade of a system.

If there is no feasible way to automate the import of the key during the upgrade before package installation then I think at least there needs to be an information in the release notes and upgrade description to inform users to manually import the keys into rpm.

To tell them to ignore the warning would be a really bad example.
Comment 3 Michael Andres 2021-04-09 17:17:47 UTC
I just checked the 2 keys from https://download.opensuse.org/distribution/leap/15.3/repo/oss/gpg-pubkey-...

> === gpg-pubkey-39db7c82-5847eb1f.asc{- 0644 216/50 size 972}
> [SuSE Package Signing Key <build@suse.de>]
>  +----[RSA 2048]-----+
>  |   . . ^           |  fpr FEAB502539D846DB2C0961CA70AF9E8139DB7C82
>  |    : : .          |   id 70AF9E8139DB7C82
>  |     ^ ^ .         |  alg RSA 2048
>  |    ^ . l i        |  cre 1481108255 Wed Dec  7 11:57:35 2016
>  |   : ^ . f :       |  exp 1607252255 Sun Dec  6 11:57:35 2020 (EXPIRED)
>  |    ? ^ .Sl        |  ttl -125
>  |   E i ...         |  rpm 39db7c82-5847eb1f
>  |      ^ ..         |
>  |       .  .        |
>  |        .  .       |
>  |         ....      |
>  +----[39DB7C82]-----+

This is an old (expired) version of the SuSE Package Signing Key. I actually wonder that the key is not in your DB. The repo metadata are signed with the updated version of 39db7c82 (repomd.xml.key):

> === ./repomd.xml.key{- 0644 216/50 size 972}
> [SuSE Package Signing Key <build@suse.de>]
>  +----[RSA 2048]-----+
>  |   . . ^           |  fpr FEAB502539D846DB2C0961CA70AF9E8139DB7C82
>  |    : : .          |   id 70AF9E8139DB7C82
>  |     ^ ^ .         |  alg RSA 2048
>  |    ^ . l i        |  cre 1600676507 Mon Sep 21 10:21:47 2020
>  |   : ^ . f :       |  exp 1726820507 Fri Sep 20 10:21:47 2024
>  |    ? ^ .Sl        |  ttl 1259
>  |   E i ...         |  rpm 39db7c82-5f68629b
>  |      ^ ..         |
>  |       .  .        |
>  |        .  .       |
>  |         ....      |
>  +----[39DB7C82]-----+

It's a bit strange, that rpm complains the key is missing. The database should even contain this updated version (imported when the repo was refreshed). Even my oldest 15.0 has this key.

Could you please attach the /var/log/zypper.log showing the dup?
I'd realy like to check this. 




The 2nd key is from the Backports OBS Project. 

> === gpg-pubkey-65176565-59787af5.asc{- 0644 216/50 size 1113}
> [openSUSE:Backports OBS Project <openSUSE:Backports@build.opensuse.org>]
>  +----[RSA 2048]-----+
>  |      ...^ ^^^.E   |  fpr 637B32FF3D83F07A7AE1C40A9C214D4065176565
>  |       .. . . .    |   id 9C214D4065176565
>  |        .          |  alg RSA 2048
>  |         .         |  cre 1570022273 Wed Oct  2 15:17:53 2019
>  |        ^          |  exp 1639142273 Fri Dec 10 14:17:53 2021
>  |       . S         |  ttl 244
>  |        : l .      |  rpm 65176565-5d94a381
>  |         ? ^ :     |
>  |          i i ^    |
>  |           ^ i.^   |
>  |           ^i. .^  |
>  +----[65176565]-----+
Comment 4 Michael Andres 2021-04-09 17:44:29 UTC
> It's a bit strange, that rpm complains the key is missing. The database
> should even contain this updated version (imported when the repo was
> refreshed). Even my oldest 15.0 has this key.

Sorry I oversaw that you're on LEAP!


The SuSE Package Signing Key is from SLES. And now it makes sense that he key is missing.
For 15.3 there's been an attempt to close the gap between Leap and SLES. It may be that now packages are taken from SLES directly (signed with 39db7c82) while older Leap packages were usually signed by 3dbdc284 (openSUSE Project Signing Key <opensuse@opensuse.org>).
Comment 5 Dirk Weber 2021-04-09 17:56:55 UTC
Created attachment 848192 [details]
/var/log/zypp/history of the zypper dup from 15.2 to 15.3

At the beginning the attached log contains the last update of the previous installed 15.2 to show that this was on current updates from the repo.

Then the removal of the 15.2 repos and the adding of the 15.3 repos.

Then the execution of zypper dup with the warnings about NOKEY for most packages.

At 09:33 I interrupted the dup process with ^C and manually imported the two missing keys. Then I continued by executing again "zypper dup" - no more warnings about the missing keys.

On none of the 15.2 systems I checked I have these two keys installed.

Here the list of keys which were installed on the system before the update: 
rpm -qa |grep gpg-pubkey
gpg-pubkey-0dfb3188-41ed929b
gpg-pubkey-307e3d54-4be01a65
gpg-pubkey-3dbdc284-53674dd4
gpg-pubkey-56b4177a-4be18cab
gpg-pubkey-7e2e3b05-4be037ca
gpg-pubkey-9c800aca-4be01999
gpg-pubkey-a1912208-446a0899

and the list after the update and after manually importing the two missing keys:
rpm -qa |grep gpg-pubkey
gpg-pubkey-0dfb3188-41ed929b
gpg-pubkey-307e3d54-4be01a65
gpg-pubkey-39db7c82-5847eb1f
gpg-pubkey-3dbdc284-53674dd4
gpg-pubkey-56b4177a-4be18cab
gpg-pubkey-65176565-5d94a381
gpg-pubkey-7e2e3b05-4be037ca
gpg-pubkey-9c800aca-4be01999
gpg-pubkey-a1912208-446a0899
Comment 6 Michael Andres 2021-04-09 18:30:17 UTC
Sorry for the confusion. If found a note in the Leap 15.3 release notes:

  https://doc.opensuse.org/release-notes/x86_64/openSUSE/Leap/15.3/

  2.1 Seamless upgrade from openSUSE Leap 15.2

  Unlike 15.2, the default installation of openSUSE Leap 15.3 contains the
  majority of rpms from SUSE Linux Enterprise Server. These rpms are signed by
  SUSE LLC instead of using the openSUSE key....


Looks like Leap 15.3 does not yet pay attention to the key import. I'll try raise this issue...
Comment 7 Dirk Weber 2021-04-10 07:25:38 UTC
Thanks for the hints related to gpgcheck repository options.

I modified my repos to use --gpgcheck-strict.

I definitely want the installation to fail instead of issuing a warning if a package checksum can not be verified. I admit I thought this would be the default behavior.
Comment 8 Michael Andres 2021-04-11 15:45:40 UTC
(In reply to Dirk Weber from comment #7)
> Thanks for the hints related to gpgcheck repository options.
> 
> I modified my repos to use --gpgcheck-strict.

If you want this always to be the default, consider setting it in /etc/zypp/zypp.conf:

##
## Signature checking (repo metadata and downloaded rpm packages)
##
...
##
# repo_gpgcheck = unset -> according to gpgcheck
# pkg_gpgcheck =  unset -> according to gpgcheck

repo_gpgcheck = on
pkg_gpgcheck  = on
Comment 9 Dirk Weber 2021-04-11 16:54:45 UTC
(In reply to Michael Andres from comment #8)
> If you want this always to be the default, consider setting it in
> /etc/zypp/zypp.conf:
I thought of it. But changing this file probably requires to maintain 
/etc/zypp/zypp.conf.rpmnew
after each update of libzypp, and since Leap 15.2 was released there were 7.
So it is probably less effort to put it in the repo configuration.
Comment 10 Lubos Kocman 2021-04-14 15:19:12 UTC
I did send email to Max, Adrian, and Michael, hopefully we'll be able to find solution quickly.

Lubos
Comment 11 Lubos Kocman 2021-04-15 09:38:54 UTC
Copy paste from Michael's email:

Question is whether we find a way to ship additional keys with a repo,
so Leap could ship the SLES buildkeys to import them. One option could
be to add additional keyfiles to the repos master index file
(repodata/repomd.xml) or allow to ship multiple keys in
repodata/repomd.xml.key and change zypp to ask importing all of them
(we currently just ask for the key signing the metadata).

Adding the keys to repodata/repomd.xml is probably more effort. Tags for the 
files must be negotiated and the keys have to be available at the time the repo 
metadata are generated. But the keyfiles would be guarded by cheksum and the 
repos signature.

Simply adding all keys to the repodata/repomd.xml.key file is far simpler. The 
file however is not protected, so adding malicious keys to the file is easy. 
The user is required to check each key and can not simply rely on us having 
added only save and needed keys.


A compromise could be to somehow list only the additional key-ids (e.g. as repo 
keywords) in the repodata/repomd.xml and to process only those keys. The id 
data we provide as keyword have to lead us to the file on the medium, which 
contains the key.

A repo keyword could e.g. mention 'gpg-pubkey-0dfb3188-41ed929b' 
then we check for a key with id 0dfb3188' being in the database.
If not, we'd try the download the key from e.g. the repositories
(root)/gpg-pubkey-0dfb3188-41ed929b.asc.

Don't know if security would be fine with such a workflow.
Comment 12 Lubos Kocman 2021-04-15 09:40:51 UTC
Security team could would you be okay witch Michael's proposal to key-ids in repomd.xml and zypper fetching given key from repodata? It seems to be a compromise which would work well for our problem with packages signed by multiple keys.
Comment 13 Lubos Kocman 2021-04-15 11:23:06 UTC
Response from Rudi:

well, it's not too easy. for the repos that are directly created by the buildservice there can only be the key in the repo metadata that is used for the project (and all rpms built or aggregated in that project would be resigned with the projects key anyway). but things are different for a built product where product builder combines rpms from multiple repositories anyway
but I'm not completely sure the product builder actually puts the key to the repomd.xml.key file or if it just puts a placeholder there and the buildservice then replaces it with the actual key (which in turn would go back to one project one key and no access to the other pubkeys)
IMHO the sane approach would be to change the openSUSE-build-key package to include the missing (SLE and Backports) pubkeys
Comment 14 Michael Andres 2021-04-15 12:53:51 UTC
Including key in a package does not help while the package is not installed.

The idea would not be to include the key itself in the metadata. Just being able to mentioning the additionally desired key IDs in the repo keywords. E.g.:

  [repomd.xml]
    <repomd>
      <tags>

        <content>gpg-pubkey-0dfb3188-41ed929b</content>

or maybe just the ID (without 'release'/'date' -41ed929b) so the key can easily be updated if it expired.

        <content>gpg-pubkey-0dfb3188</content>

      </tags>

      <data ....


In this form we know the key 0dfb3188 would be desired.

- We can check if it is in the database.

- If it is not we can look for an [optional] file 'gpg-pubkey-0dfb3188' or 'gpg-pubkey-0dfb3188.asc' (or gpg-pubkey-0dfb3188-41ed929b.asc if with -release) in the medias root directory. Download it and if it actually contains a key with ID 0dfb3188, we ask whether to import this as well.


The point is that zypp does not want to ask whether to trust/import arbitrary keys found at some unsigned/unprotected location on a media.

We ask for the key signing the metadata, and then we could ask for additional keys mentioned in the signed metadata. For this it sufficient if the key data in the repo hint us to the keyid and location (optional filename) on the meduim. 

If a key is present we fetch it, if not we could at least tell that the repo suggest 0dfb3188 and ther we did not find it.

The packages are secured by the checksum in the signed metadata, but preferably the gpg key  is somehow available as well. Specially for custom/SUMA repos with 3rd party packages.


(most probably even createrepo could create this abstract of keyids used to sign the packages included in the primary.xml).
Comment 15 Michael Andres 2021-04-15 13:39:27 UTC
Chatted with Rudi: Bringing the 'gpg-pubkey-0dfb3188-41ed929b.asc' strings into the repos content keywords seems to be doable.
Comment 16 OBSbugzilla Bot 2021-04-15 16:10:05 UTC
This is an autogenerated message for OBS integration:
This bug (1184326) was mentioned in
https://build.opensuse.org/request/show/885718 Tools / instsource-susedata
https://build.opensuse.org/request/show/885720 Factory / instsource-susedata
https://build.opensuse.org/request/show/885723 Tools / product-builder-plugin-Tumbleweed
https://build.opensuse.org/request/show/885724 Factory / product-builder-plugin-Tumbleweed
https://build.opensuse.org/request/show/885725 15.3 / product-builder-plugin-Tumbleweed
https://build.opensuse.org/request/show/885726 15.3 / instsource-susedata
Comment 17 Dirk Weber 2021-04-28 19:19:09 UTC
Current state: update of a fully up to date Leap 15.2 to Leap 15.3 Build142.1 ( 2021-04-27T02:49:40 ) 

Leap 15.2 update status:
tail /var/log/zypp/history
...
2021-04-28 20:53:59|patch  |openSUSE-2021-622|1|noarch|openSUSE_15.2_Update|important|recommended|needed|applied|
2021-04-28 20:53:59|patch  |openSUSE-2021-625|1|noarch|openSUSE_15.2_Update|moderate|recommended|needed|applied|


Change the repos to 15.3 

zypper lr -d
# | Alias                        | Name                         | Enabled   | GPG Check       | Refresh        | Priority  | Type   | URI                                                                | Serv->
--+------------------------------+------------------------------+-----------+-----------------+----------------+-----------+--------+--------------------------------------------------------------------+-------
1 | openSUSE-15.3-OSS            | openSUSE-15.3-OSS            | Ja        | (rp) Ja         | Nein           |   99      | rpm-md | https://download.opensuse.org/distribution/leap/15.3/repo/oss/     | 
2 | openSUSE-15.3-non-OSS        | openSUSE-15.3-non-OSS        | Ja        | (rp) Ja         | Nein           |   99      | rpm-md | https://download.opensuse.org/distribution/leap/15.3/repo/non-oss/ | 
3 | openSUSE_15.3-update-non-oss | openSUSE_15.3-update-non-oss | Ja        | (rp) Ja         | Ja             |   99      | rpm-md | https://download.opensuse.org/update/leap/15.3/non-oss/            | 
4 | openSUSE_15.3_Update         | openSUSE_15.3_Update         | Ja        | (rp) Ja         | Ja             |   99      | rpm-md | https://download.opensuse.org/update/leap/15.3/oss/                | 


perform zypper dup:
...
when the download starts the related key is not available, as gpgcheck-strict is configured the installation fails:

(Press 'q' to exit the pager.)
Retrieving package GeoIP-data-1.6.12-6.3.1.noarch                                                                                         (1/3261),  13.9 KiB (    0   B unpacked)
Retrieving: GeoIP-data-1.6.12-6.3.1.noarch.rpm .............................................................................................................................[done]
GeoIP-data-1.6.12-6.3.1.noarch.rpm:
    Header V3 RSA/SHA256 Signature, key ID 39db7c82: NOKEY

Looking for gpg key ID 39DB7C82 in cache /var/cache/zypp/pubkeys.
Repository openSUSE-15.3-OSS does not define additional 'gpgkey=' URLs.
GeoIP-data-1.6.12-6.3.1.noarch (openSUSE-15.3-OSS): Signature verification failed [4-Signatures public key is not available]
Abort, retry, ignore? [a/r/i] (a): 



That the installation stopps is ok for me, but I would like to have information how to get the key and be able to check the fingerprints of the key.
Comment 18 Michael Andres 2021-04-29 08:19:21 UTC
Download to key from 
https://download.opensuse.org/distribution/leap/15.3/repo/oss/gpg-pubkey/39db7c82-5847eb1f.asc

- Create the directory /var/cache/zypp/pubkeys if it does not exist and put the file there.
         IMPORTANT: rename the file so it ends on '.key'
  
  (you can put multiple .key filest here, also concatenated key files containing multiple keys)


If you do the 'zypper dup' interactively, zypp will show and ask to import the missing key if is found in there. Even if the file would contain multiple keys, zypp will import just the one required.

> Retrieving: zoo-2.10-lp152.3.5.x86_64.rpm ........................[done]
> zoo-2.10-lp152.3.5.x86_64.rpm:
>     Header V3 RSA/SHA256 Signature, key ID 3dbdc284: NOKEY
> 
> Looking for gpg key ID 3DBDC284 in cache /var/cache/zypp/pubkeys.
> 
> New repository or package signing key received:
> 
>   Repository:       repo-oss
>   Key Name:         openSUSE Project Signing Key <opensuse@opensuse.org>
>   Key Fingerprint:  22C07BA5 34178CD0 2EFE22AA B88B2FD4 3DBDC284
>   Key Created:      Mon 05 May 2014 10:37:40 AM CEST
>   Key Expires:      Thu 02 May 2024 10:37:40 AM CEST
>   Rpm Name:         gpg-pubkey-3dbdc284-53674dd4
> 
> 
> Do you want to reject the key, or trust always? [r/a/?] (r)


- Without zypp: download the file. Use e.g. `gpg2 --list-packets FILE` to see the key(s) inside. If multimple keys are inside, extract the one you want to import into a file and call 'rpm --import KEYFILE'.
(The files are ascii-armored, so you can view then e.g. with 'less')
Comment 19 Michal Suchanek 2021-04-29 08:23:19 UTC
(In reply to Michael Andres from comment #14)
> Including key in a package does not help while the package is not installed.
> 
> The idea would not be to include the key itself in the metadata. Just being
> able to mentioning the additionally desired key IDs in the repo keywords.
> E.g.:
> 
>   [repomd.xml]
>     <repomd>
>       <tags>
> 
>         <content>gpg-pubkey-0dfb3188-41ed929b</content>
> 
> or maybe just the ID (without 'release'/'date' -41ed929b) so the key can
> easily be updated if it expired.

If the key expires then the index should be regenerated.

Can't it include enough information to verify the key so that it can be imported automatically without asking the user?

> 
>         <content>gpg-pubkey-0dfb3188</content>
> 
>       </tags>
> 
>       <data ....
> 
> 
> In this form we know the key 0dfb3188 would be desired.
> 
> - We can check if it is in the database.
> 
> - If it is not we can look for an [optional] file 'gpg-pubkey-0dfb3188' or
> 'gpg-pubkey-0dfb3188.asc' (or gpg-pubkey-0dfb3188-41ed929b.asc if with
> -release) in the medias root directory. Download it and if it actually
> contains a key with ID 0dfb3188, we ask whether to import this as well.
> 
> 
> The point is that zypp does not want to ask whether to trust/import
> arbitrary keys found at some unsigned/unprotected location on a media.

Indeed. Where is the security in importing random keys asking the user or not.

Unless the keys that are used in the repository are verified by the index signature there is no point to have keys.
Comment 20 Max Lin 2021-04-29 10:05:01 UTC
With Rudi's change, I can see these tags/content be added to repomd.xml

<content>pool</content>
<content>gpg-pubkey-39db7c82-5847eb1f.asc</content>
<content>gpg-pubkey-307e3d54-5aaa90a5.asc</content>
<content>gpg-pubkey-65176565-59787af5.asc</content>
<content>gpg-pubkey-3dbdc284-53674dd4.asc</content>

* _pool_ appears there seems weird since it look like the build service repository name to me.

According to the latest Leap build test result on openQA, I still see NOKEY warning while zypper dup from Leap 15.1 to Leap 15.3[1], like

Additional rpm output:
warning: /var/cache/zypp/packages/repo1/noarch/farstream-data-0.2.8-bp153.1.46.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 65176565: NOKEY

I don't see the warning against the SUSE packaging key:39db7c82, all warnings are about Backports key 65176565.

do we still missing something for this topic?

[1] https://openqa.opensuse.org/tests/1715646/file/serial0.txt
Comment 21 Michael Andres 2021-04-29 10:15:32 UTC
(In reply to Max Lin from comment #20)
> do we still missing something for this topic?

Yes. The implementation in libzypp/zypper evaluating, downloading and importing the provided data. The security teams audit of the sugested workflow is fine so far, and I'm about to implement it for libzypp-17.25.11.
Comment 22 Dirk Weber 2021-04-29 19:27:41 UTC
Another unexpected behavior:

I have downloaded the Leap 15.3 RPMs on one system with zypper dup -d.

The downloading system had the SUSE package signing keys imported into the rpm database by
rpm --import.

I used the downloaded packages to upgrade another Leap 15.2 system by copying them into the /var/cache/zypp/packages directory.

I switched the repos of the system to 15.3 including --gpgcheck-strict

I provided the keyfiles like descriped in comment 18:

ll /var/cache/zypp/pubkeys/
insgesamt 8
-rw-r--r-- 1 root root  972 29. Apr 20:54 gpg-pubkey-39db7c82-5847eb1f.key
-rw-r--r-- 1 root root 1113 29. Apr 20:54 gpg-pubkey-65176565-59787af5.key


zypper lr -d
# | Alias                        | Name                         | Enabled   | GPG Check       | Refresh        | Priority  | Type   | URI                                                                | Serv->
--+------------------------------+------------------------------+-----------+-----------------+----------------+-----------+--------+--------------------------------------------------------------------+-------
1 | openSUSE-15.3-OSS            | openSUSE-15.3-OSS            | Ja        | (rp) Ja         | Nein           |   99      | rpm-md | https://download.opensuse.org/distribution/leap/15.3/repo/oss/     | 
2 | openSUSE-15.3-non-OSS        | openSUSE-15.3-non-OSS        | Ja        | (rp) Ja         | Nein           |   99      | rpm-md | https://download.opensuse.org/distribution/leap/15.3/repo/non-oss/ | 
3 | openSUSE_15.3-update-non-oss | openSUSE_15.3-update-non-oss | Ja        | (rp) Ja         | Ja             |   99      | rpm-md | https://download.opensuse.org/update/leap/15.3/non-oss/            | 
4 | openSUSE_15.3_Update         | openSUSE_15.3_Update         | Ja        | (rp) Ja         | Ja             |   99      | rpm-md | https://download.opensuse.org/update/leap/15.3/oss/   

rpm -q zypper libzypp
zypper-1.14.43-lp152.2.18.1.x86_64
libzypp-17.25.8-lp152.2.22.1.x86_64

When I executed zypper dup the keys were not imported and the RPMs were installed even when the signature could not be verified by rpm.

Checking for file conflicts: ...............................................................................................................................................[done]
(   1/3326) Removing kde-l10n-de-17.08.3-lp152.5.2.noarch ..................................................................................................................[done]
(   2/3326) Removing kde-l10n-de-data-17.08.3-lp152.5.2.noarch .............................................................................................................[done]
(   3/3326) Removing kde-l10n-de-doc-17.08.3-lp152.5.2.noarch ..............................................................................................................[done]
(   4/3326) Removing patterns-base-enhanced_base_opt-20171206-lp152.34.2.x86_64 ............................................................................................[done]
(   5/3326) Installing: GeoIP-data-1.6.12-6.3.1.noarch .....................................................................................................................[done]
Additional rpm output:
warning: /var/cache/zypp/packages/openSUSE-15.3-OSS/noarch/GeoIP-data-1.6.12-6.3.1.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 39db7c82: NOKEY


(   6/3326) Installing: MozillaFirefox-branding-openSUSE-68-lp153.4.22.x86_64 ..............................................................................................[done]
(   7/3326) Installing: alsa-ucm-conf-1.2.4-4.13.noarch ....................................................................................................................[done]
Additional rpm output:
warning: /var/cache/zypp/packages/openSUSE-15.3-OSS/noarch/alsa-ucm-conf-1.2.4-4.13.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 39db7c82: NOKEY


Should not either the keys be imported automatically from the /var/cache/zypp/pubkeys/ directory or the package installation fail when rpm can not verify the package signature?
Comment 23 Michael Andres 2021-04-29 19:37:11 UTC
@Dirk: Please attach the /var/log/zypper.log showing the dup command, thanks.
Comment 24 Michael Andres 2021-04-29 19:52:06 UTC
(In reply to Dirk Weber from comment #22)
> I used the downloaded packages to upgrade another Leap 15.2 system by
> copying them into the /var/cache/zypp/packages directory.

Sorry, I oversaw the above comment. The log is not needed.

We do the signature check and the import of missing keys upon download. Only checked packages or those confirmed by the user are stored in our cache and are later used for installation.

If you manually inject packages into the cache, they are considered to be trustworthy. We pass them to rpm directly. Rpm however does not reject packages with an unknown key. It just writes out he message you see, but does not consider this as an error.
Comment 25 Dirk Weber 2021-04-29 20:07:10 UTC
(In reply to Michael Andres from comment #24)
> (In reply to Dirk Weber from comment #22)
> > I used the downloaded packages to upgrade another Leap 15.2 system by
> > copying them into the /var/cache/zypp/packages directory.
> 
> Sorry, I oversaw the above comment. The log is not needed.
I dumped the image already and was just about to redo the case to provide the zypper.log.

> We do the signature check and the import of missing keys upon download. Only
> checked packages or those confirmed by the user are stored in our cache and
> are later used for installation.
> 
> If you manually inject packages into the cache, they are considered to be
> trustworthy. We pass them to rpm directly. Rpm however does not reject
> packages with an unknown key. It just writes out he message you see, but
> does not consider this as an error.
Thanks for the explanation. 

So it seems the procedure which works in most cases is to import the SUSE package signing keys in advance into the rpm database.
Comment 26 Michael Andres 2021-04-29 20:15:05 UTC
(In reply to Dirk Weber from comment #25)
> So it seems the procedure which works in most cases is to import the SUSE
> package signing keys in advance into the rpm database.

Currently, yes. 

The repo metadata generator is already fixed, so the information about additional keys is already present (see s#20).

Once libzypp is fixed to evaluate the data, the missing keys will be imported when you add/refresh the repo. If this is done, injecting the packages into the cache will also work (provided the repo metadata list all needed keys).
Comment 27 Lubos Kocman 2021-04-30 11:21:37 UTC
Anything we'd like to keep in release notes?
Comment 28 Dirk Weber 2021-04-30 15:50:45 UTC
I repeated the procedure described in comment 22 and just deleted the two first installed packages with the warning about the missing key from the cache before executing zypper dup in order to trigger zypper to ask about importing the key.

put the keys in advance into /var/cache/zypp/pubkeys/

switched the repos to 15.3 with --gpgcheck-strict

deleted the 2 packages from the cache
GeoIP-data-1.6.12-6.3.1.noarch
alsa-ucm-conf-1.2.4-4.13.noarch

executed zypper dup

and (after ending the pager showing the license) got this:
(Press 'q' to exit the pager.)
Retrieving package GeoIP-data-1.6.12-6.3.1.noarch                                                         (1/3260),  13.9 KiB (    0   B unpacked)
Retrieving: GeoIP-data-1.6.12-6.3.1.noarch.rpm .............................................................................................[done]
GeoIP-data-1.6.12-6.3.1.noarch.rpm:
    Header V3 RSA/SHA256 Signature, key ID 39db7c82: NOKEY

Looking for gpg key ID 39DB7C82 in cache /var/cache/zypp/pubkeys.

New repository or package signing key received:

  Repository:       openSUSE-15.3-OSS
  Key Name:         SuSE Package Signing Key <build@suse.de>
  Key Fingerprint:  FEAB5025 39D846DB 2C0961CA 70AF9E81 39DB7C82
  Key Created:      Wed Dec  7 11:57:35 2016
  Key Expires:      Sun Dec  6 11:57:35 2020 (EXPIRED)
  Rpm Name:         gpg-pubkey-39db7c82-5847eb1f


Do you want to reject the key, or trust always? [r/a/?] (r):

and I found this key with fingerprint also on
https://www.suse.com/de-de/support/security/download-verification/

For me this is now a work flow which is quite ok. 
Thanks for the improvements and the explanations.

Related to comment 27: 
the release notes could mention that due to the change how the distribution is built now (CTLG) it makes sense for the update of the running system with zypper dup to put the additional keys into the /var/cache/zypp/pubkeys/ directory with filenames ending with .key in advance.
Comment 29 Michael Andres 2021-04-30 16:28:26 UTC
Thanks for sharing the link.
Comment 30 Michal Suchanek 2021-04-30 17:15:39 UTC
(In reply to Dirk Weber from comment #28)
> I repeated the procedure described in comment 22 and just deleted the two
> first installed packages with the warning about the missing key from the
> cache before executing zypper dup in order to trigger zypper to ask about
> importing the key.
> 
> put the keys in advance into /var/cache/zypp/pubkeys/
> 
How is the user supposed to know to do this?
> .............................................................................
> ................[done]
> GeoIP-data-1.6.12-6.3.1.noarch.rpm:
>     Header V3 RSA/SHA256 Signature, key ID 39db7c82: NOKEY
> 
> Looking for gpg key ID 39DB7C82 in cache /var/cache/zypp/pubkeys.
> 
> New repository or package signing key received:
> 
>   Repository:       openSUSE-15.3-OSS
>   Key Name:         SuSE Package Signing Key <build@suse.de>
>   Key Fingerprint:  FEAB5025 39D846DB 2C0961CA 70AF9E81 39DB7C82
>   Key Created:      Wed Dec  7 11:57:35 2016
>   Key Expires:      Sun Dec  6 11:57:35 2020 (EXPIRED)
>   Rpm Name:         gpg-pubkey-39db7c82-5847eb1f
> 
> 
> Do you want to reject the key, or trust always? [r/a/?] (r):

How am I as a user supposed to answer this question without seeing this bug beforehand?

This is ridiculous UX and leans into this 'we do cryptography but make it totally useless for actually securing anything' I have seen in (open)SUSE so far.

There is surely a way to distribute the key securely so that there is no need to ask the user. But nobody ever wants to do that. 

Ask the user to accept random keys without any explanation all the time is the UX we strive for?
Comment 31 Michael Andres 2021-04-30 17:32:33 UTC
(In reply to Michal Suchanek from comment #30)
> (In reply to Dirk Weber from comment #28)
> > I repeated the procedure described in comment 22 and just deleted the two
> > first installed packages with the warning about the missing key from the
> > cache before executing zypper dup in order to trigger zypper to ask about
> > importing the key.
> > 
> > put the keys in advance into /var/cache/zypp/pubkeys/
> > 
> How is the user supposed to know to do this?

That's why we're about to fix libzypp to evaluate the tags Rudi brought into the repo metadata (c#20). Once this is ready, libzypp will know where to find the missing key.

The above is only needed with an unfixed libzypp or if the repo metadata do not mention additionally key needed.
Comment 32 Michal Suchanek 2021-04-30 17:42:40 UTC
(In reply to Michael Andres from comment #31)
> (In reply to Michal Suchanek from comment #30)
> > (In reply to Dirk Weber from comment #28)
> > > I repeated the procedure described in comment 22 and just deleted the two
> > > first installed packages with the warning about the missing key from the
> > > cache before executing zypper dup in order to trigger zypper to ask about
> > > importing the key.
> > > 
> > > put the keys in advance into /var/cache/zypp/pubkeys/
> > > 
> > How is the user supposed to know to do this?
> 
> That's why we're about to fix libzypp to evaluate the tags Rudi brought into
> the repo metadata (c#20). Once this is ready, libzypp will know where to
> find the missing key.

It will know where to find it but it is not signed so we are still asking the user to import random keys without any verification whatsoever.
Comment 39 OBSbugzilla Bot 2021-05-04 13:20:04 UTC
This is an autogenerated message for OBS integration:
This bug (1184326) was mentioned in
https://build.opensuse.org/request/show/890354 Factory / instsource-susedata
https://build.opensuse.org/request/show/890355 15.3 / instsource-susedata
Comment 40 Lubos Kocman 2021-05-17 11:16:21 UTC
Hello zypp-maintainers? What is left to be done?
I see referenced submissions but what's the overall status? We're bit concerned as the GM is scheduled for end of this week.

Thank you
Comment 41 Michael Andres 2021-05-20 17:28:04 UTC
@Rudi: Do we need to configure something on https://download.opensuse.org/ to get the gpg keys delivered? 

I'm persistenly redirected to http:// sites, but zypp does not follow https to http redirects. So I don't get the keys. E.g.

- https://download.opensuse.org/distribution/leap/15.3/repo/oss/gpg-pubkey-307e3d54-5aaa90a5.asc

- location: http://ftp.uni-erlangen.de/pub/mirrors/opensuse/distribution/leap/15.3/repo/oss/gpg-pubkey-307e3d54-5aaa90a5.asc

And this although e.g. https://ftp.uni-erlangen.de/.. would also work.
Comment 42 Michael Andres 2021-05-20 19:18:33 UTC
@Rudi (2nd Q):

 Retrieving: gpg-pubkey-39db7c82-5847eb1f.asc .....................[done]
 [70AF9E8139DB7C82-5847eb1f] [SuSE Package Signing Key <build@suse.de>] 
 [expired: 2020-12-06]

Is it intended to ship expired keys?
(in https://download.opensuse.org/distribution/leap/15.3/repo/oss/)
Comment 43 Ruediger Oertel 2021-05-21 12:30:43 UTC
this bug is in openSUSE:Leap:15.3/openSUSE-build-key
the pubkeys in this package need to be updated to the most current versions.
Comment 44 Ruediger Oertel 2021-05-21 12:34:16 UTC
for the redirects, I've asked on the teamlist
Andrii, any good idea ?
Comment 46 Max Lin 2021-05-21 13:29:23 UTC
(In reply to Ruediger Oertel from comment #43)
> this bug is in openSUSE:Leap:15.3/openSUSE-build-key
> the pubkeys in this package need to be updated to the most current versions.

There are 2 SUSE package signing key in openSUSE-build-key[1]

* gpg-pubkey-307e3d54-5aaa90a5.asc
* gpg-pubkey-39db7c82-5847eb1f.asc

I don't know what is the difference in between, if gpg-pubkey-39db7c82-5847eb1f.asc need to be updated to a recent version hen where can I get it?

[1] https://build.opensuse.org/package/show/openSUSE:Leap:15.3/openSUSE-build-key
Comment 48 Ruediger Oertel 2021-05-21 14:28:15 UTC
build-rsa-307e3d54-5aaa90a5.asc looks up to date.
Comment 49 Ruediger Oertel 2021-05-21 14:29:54 UTC
Created attachment 849567 [details]
build-rsa2048-39db7c82-5f68629b.asc
Comment 50 Ruediger Oertel 2021-05-21 14:30:19 UTC
updated pub key for sle has been attached
Comment 51 Max Lin 2021-05-21 20:19:38 UTC
(In reply to Ruediger Oertel from comment #50)
> updated pub key for sle has been attached

done in openSUSE-build-key https://build.opensuse.org/package/rdiff/openSUSE:Leap:15.3/openSUSE-build-key?linkrev=base&rev=3
Comment 53 Lubos Kocman 2021-05-26 11:14:53 UTC
Michael could you please review the Release Notes entry and suggest any corrections?

https://github.com/openSUSE/release-notes-openSUSE/pull/108

Thank you
Comment 54 Andrii Nikitin 2021-05-26 11:25:54 UTC
(In reply to Ruediger Oertel from comment #44)
> for the redirects, I've asked on the teamlist
> Andrii, any good idea ?

(Sorry for missing it,) so download.opensuse.org cannot handle https requests properly https://github.com/openSUSE/mirrorbrain/issues/3
I had a PR, but wasn't brave enough to manage it alone, so decided to provide an alternative approach. Currently https://mirrorcache.opensuse.org/download should able to handle https/IPv6 requests properly and the team is working on having it as a new backend behind download.opensuse.org .
Comment 55 Michael Andres 2021-05-26 12:32:57 UTC
(In reply to Andrii Nikitin from comment #54)
> (Sorry for missing it,) so download.opensuse.org cannot handle https
> requests properly https://github.com/openSUSE/mirrorbrain/issues/3

I updated https://bugzilla.suse.com/show_bug.cgi?id=1184808 accordingly.
We'll need to add an exception and allow https->http redirects from download.opensuse.org. Otherwise we will have trouble to get the keys downloaded.
Comment 56 Michael Andres 2021-06-02 14:29:26 UTC
Fixed for libzypp-17.26.0 / zypper-1.14.44
Looks liek this:

> Retrieving repository '153' metadata -----------------------------------------------------------------------[\]
> 
> Note: Received 3 new package signing keys from repository 153:
> 
>   Those additional keys are usually used to sign packages shipped by the repository. In order to
>   validate those packages upon download and installation the new keys will be imported into the rpm
>   database.
> 
>   New:
>   Key Fingerprint:  4E98 E675 19D9 8DC7 362A 5990 E3A5 C360 307E 3D54
>   Key Name:         SuSE Package Signing Key <build@suse.de>
>   Key Algorithm:    RSA 1024
>   Key Created:      Thu 15 Mar 2018 04:26:29 PM CET
>   Key Expires:      Mon 14 Mar 2022 04:26:29 PM CET
>   Rpm Name:         gpg-pubkey-307e3d54-5aaa90a5
> 
>   New:
>   Key Fingerprint:  637B 32FF 3D83 F07A 7AE1 C40A 9C21 4D40 6517 6565
>   Key Name:         openSUSE:Backports OBS Project <openSUSE:Backports@build.opensuse.org>
>   Key Algorithm:    RSA 2048
>   Key Created:      Wed 02 Oct 2019 03:17:53 PM CEST
>   Key Expires:      Fri 10 Dec 2021 02:17:53 PM CET
>   Rpm Name:         gpg-pubkey-65176565-5d94a381
> 
>   New:
>   Key Fingerprint:  FEAB 5025 39D8 46DB 2C09 61CA 70AF 9E81 39DB 7C82
>   Key Name:         SuSE Package Signing Key <build@suse.de>
>   Key Algorithm:    RSA 2048
>   Key Created:      Wed 07 Dec 2016 11:57:35 AM CET
>   Key Expires:      Sun 06 Dec 2020 11:57:35 AM CET (EXPIRED)
>   Rpm Name:         gpg-pubkey-39db7c82-5847eb1f
> 
>   The repository metadata introducing the new keys have been signed and validated by the trusted
>   key:
> 
>   Repository:       153
>   Key Fingerprint:  22C0 7BA5 3417 8CD0 2EFE 22AA B88B 2FD4 3DBD C284
>   Key Name:         openSUSE Project Signing Key <opensuse@opensuse.org>
>   Key Algorithm:    RSA 2048
>   Key Created:      Mon 05 May 2014 10:37:40 AM CEST
>   Key Expires:      Thu 02 May 2024 10:37:40 AM CEST
>   Rpm Name:         gpg-pubkey-3dbdc284-53674dd4
> 
> Retrieving repository '153' metadata ....................................................................[done]
Comment 57 Michael Andres 2021-06-02 14:38:07 UTC
closing
Comment 59 Michal Suchanek 2021-06-02 15:34:36 UTC
Why is the key fingerprint used for key verification rather than the same hash as the other files?

Clearly the fingerprint function provides a shorter hash than the SHA256 so it is likely weaker. It is meant to simplify key management of user keys, not to securely identify a key.
Comment 60 Marcus Meissner 2021-06-07 10:57:07 UTC
I now tested using the "update script" feature.

openSUSE-build-key with a update script

home:msmeissn:branches:openSUSE:Leap:15.2:Update/openSUSE-build-key.openSUSE_Leap_15.2_Update$ cat openSUSE-build-key.sh 
#!/bin/sh
# import SLE 15 key
rpm --import /usr/lib/rpm/gnupg/keys/gpg-pubkey-39db7c82-5f68629b.asc

# import opensuse backports key
rpm --import /usr/lib/rpm/gnupg/keys/gpg-pubkey-65176565-59787af5.asc


This seems to import the 2 keys without RPM locking issues.
Comment 61 OBSbugzilla Bot 2021-06-07 12:40:12 UTC
This is an autogenerated message for OBS integration:
This bug (1184326) was mentioned in
https://build.opensuse.org/request/show/898068 15.2 / openSUSE-build-key
Comment 62 Swamp Workflow Management 2021-06-08 10:20:09 UTC
SUSE-RU-2021:1879-1: An update that has four recommended fixes can now be installed.

Category: recommended (important)
Bug References: 1184326,1184399,1184997,1185325
CVE References: 
JIRA References: 
Sources used:
SUSE MicroOS 5.0 (src):    libsolv-0.7.19-6.1, libzypp-17.26.0-9.1, zypper-1.14.45-10.1
SUSE Linux Enterprise Module for Development Tools 15-SP3 (src):    libsolv-0.7.19-6.1
SUSE Linux Enterprise Module for Development Tools 15-SP2 (src):    libsolv-0.7.19-6.1
SUSE Linux Enterprise Module for Basesystem 15-SP3 (src):    libsolv-0.7.19-6.1, libzypp-17.26.0-9.1, zypper-1.14.45-10.1
SUSE Linux Enterprise Module for Basesystem 15-SP2 (src):    libsolv-0.7.19-6.1, libzypp-17.26.0-9.1, zypper-1.14.45-10.1
SUSE Linux Enterprise Installer 15-SP2 (src):    libsolv-0.7.19-6.1, libzypp-17.26.0-9.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 63 Swamp Workflow Management 2021-06-09 13:20:18 UTC
openSUSE-RU-2021:0859-1: An update that has two recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1181344,1184326
CVE References: 
JIRA References: 
Sources used:
openSUSE Leap 15.2 (src):    openSUSE-build-key-1.0-lp152.5.3.1
Comment 65 Swamp Workflow Management 2021-06-16 19:23:57 UTC
openSUSE-RU-2021:0871-1: An update that has four recommended fixes can now be installed.

Category: recommended (important)
Bug References: 1184326,1184399,1184997,1185325
CVE References: 
JIRA References: 
Sources used:
openSUSE Leap 15.2 (src):    libsolv-0.7.19-lp152.2.25.1, libzypp-17.26.0-lp152.2.34.1, zypper-1.14.45-lp152.2.24.1
Comment 66 Martin Wilck 2021-06-24 08:05:38 UTC
Overall, the end user experience is still pretty bad. I had libzypp-17.26.0-9.1 and zypper-1.14.45-10.1 installed an was still seeing the NOKEY warnings for 39db7c82.

While the key 39db7c82 is available under https://download.opensuse.org/distribution/leap/15.3/repo/oss/, it is still missing from
http://download.opensuse.org/update/leap/15.3/sle/, where I'd expected to find it too. 

Rationale: as a user, I see a message like this:

> warning: /var/cache/zypp/packages/repo-sle-update/sle-sp3/x86_64/qemu-block-rbd-5.2.0-17.1.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 39db7c82: NOKEY

Using zypper commands, I can figure out that this package was obtained from the http://download.opensuse.org/update/leap/15.3/sle repo. But the key can't be found there. The key should be available in every repo from which packages signed with that key can be downloaded.

General:

We ship a package called "distribution-gpg-keys", which contains all sorts of fedora and rpmfusion keys, but not the ones we are actually using for our own packages, except the openSUSE key. Instead, our keys are shipped in the package "openSUSE-build-key". But that's a specific piece of information that's non-obvious even to long-time SUSE/openSUSE users. Once the user has found openSUSE-build-key, the solution is rather simple,

> rpm -import /usr/lib/rpm/gnupg/keys/gpg-pubkey-39db7c82-5847eb1f.asc

But security-aware users who hit this issue might find it strange that the openSUSE-build-key package itself is signed with 39db7c82, the very key which is missing.
Comment 67 Marcus Meissner 2021-06-24 08:11:31 UTC
did you install all 15.2 online updates before migration?

I specifically released an openSUSE-build-key update on 15.2 that imports the SLE abd Backports key.
Comment 68 Martin Wilck 2021-06-24 08:37:33 UTC
(In reply to Marcus Meissner from comment #67)
> did you install all 15.2 online updates before migration?
> 
> I specifically released an openSUSE-build-key update on 15.2 that imports
> the SLE abd Backports key.

Right - I saw the messages on one system, where I'd forgotten to do this.
The last "zypper up" on that system had been run on May 17th, ~2 weeks before the dup to 15.3, and openSUSE-build-key-1.0-lp152.4.2 was installed why I did the dup. Good to know. 

But relying on users not to forget this step is fragile, and once the issue exists, figuring out where to obtain the key is non-obvious.

Even with access to SUSE internal data I found it difficult. Googling for "39db7c82 NOKEY" took me to a forum page that included mostly clueless remarks (as usual on forum pages - can't verify this right now as forum.opensuse.org is down :-( ).

I would suggest the following to increase chances that users find the key quickly:

 - add the key to every repo where packages signed with it can be downloaded
 - add our own keys to distribution-gpg-keys, even if it's redundant with openSUSE-build-key
 - push our package signing keys to public key servers.
 - add the import script to openSUSE-build-key on 15.3 as well.
Comment 69 Martin Wilck 2021-06-24 09:22:07 UTC
(In reply to Martin Wilck from comment #68)

>  - push our package signing keys to public key servers.

This seems to be the case already. Not sure why I didn't find them in the first place. sorry.
Comment 70 Benjamin Zeller 2021-06-24 09:36:21 UTC
I just did a dup from 15.2 -> 15.3 yesterday with:

> zypper dup --allow-vendor-change 

This gave me no signature warnings at all
Comment 72 Swamp Workflow Management 2021-07-10 10:16:51 UTC
openSUSE-RU-2021:1879-1: An update that has four recommended fixes can now be installed.

Category: recommended (important)
Bug References: 1184326,1184399,1184997,1185325
CVE References: 
JIRA References: 
Sources used:
openSUSE Leap 15.3 (src):    libsolv-0.7.19-6.1, libzypp-17.26.0-9.1, zypper-1.14.45-10.1
Comment 76 Swamp Workflow Management 2021-11-24 02:21:19 UTC
SUSE-RU-2021:3780-1: An update that has 31 recommended fixes and contains one feature can now be installed.

Category: recommended (moderate)
Bug References: 1153687,1182372,1183268,1183589,1184326,1184399,1184997,1185325,1186447,1186503,1186602,1187224,1187425,1187466,1187738,1187760,1188156,1188435,1189031,1190059,1190199,1190356,1190465,1190712,1190815,1191286,1191324,1191370,1191609,1192337,1192436
CVE References: 
JIRA References: SLE-18858
Sources used:
SUSE Linux Enterprise Server for SAP 15 (src):    libsolv-0.7.20-3.48.1, libzypp-17.28.8-3.78.1, zypper-1.14.50-3.60.1
SUSE Linux Enterprise Server 15-LTSS (src):    libsolv-0.7.20-3.48.1, libzypp-17.28.8-3.78.1, zypper-1.14.50-3.60.1
SUSE Linux Enterprise Installer 15 (src):    libsolv-0.7.20-3.48.1, libzypp-17.28.8-3.78.1, zypper-1.14.50-3.60.1
SUSE Linux Enterprise High Performance Computing 15-LTSS (src):    libsolv-0.7.20-3.48.1, libzypp-17.28.8-3.78.1, zypper-1.14.50-3.60.1
SUSE Linux Enterprise High Performance Computing 15-ESPOS (src):    libsolv-0.7.20-3.48.1, libzypp-17.28.8-3.78.1, zypper-1.14.50-3.60.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 77 Swamp Workflow Management 2021-11-24 02:30:45 UTC
SUSE-RU-2021:3781-1: An update that has 31 recommended fixes and contains one feature can now be installed.

Category: recommended (moderate)
Bug References: 1153687,1182372,1183268,1183589,1184326,1184399,1184997,1185325,1186447,1186503,1186602,1187224,1187425,1187466,1187738,1187760,1188156,1188435,1189031,1190059,1190199,1190356,1190465,1190712,1190815,1191286,1191324,1191370,1191609,1192337,1192436
CVE References: 
JIRA References: SLE-18858
Sources used:
SUSE Linux Enterprise Server for SAP 15-SP1 (src):    libsolv-0.7.20-4.3.1, libzypp-17.28.8-3.61.1, zypper-1.14.50-3.46.1
SUSE Linux Enterprise Server 15-SP1-LTSS (src):    libsolv-0.7.20-4.3.1, libzypp-17.28.8-3.61.1, zypper-1.14.50-3.46.1
SUSE Linux Enterprise Server 15-SP1-BCL (src):    libsolv-0.7.20-4.3.1, libzypp-17.28.8-3.61.1, zypper-1.14.50-3.46.1
SUSE Linux Enterprise Installer 15-SP1 (src):    libsolv-0.7.20-4.3.1, libzypp-17.28.8-3.61.1
SUSE Linux Enterprise High Performance Computing 15-SP1-LTSS (src):    libsolv-0.7.20-4.3.1, libzypp-17.28.8-3.61.1, zypper-1.14.50-3.46.1
SUSE Linux Enterprise High Performance Computing 15-SP1-ESPOS (src):    libsolv-0.7.20-4.3.1, libzypp-17.28.8-3.61.1, zypper-1.14.50-3.46.1
SUSE Enterprise Storage 6 (src):    libsolv-0.7.20-4.3.1, libzypp-17.28.8-3.61.1, zypper-1.14.50-3.46.1
SUSE CaaS Platform 4.0 (src):    libsolv-0.7.20-4.3.1, libzypp-17.28.8-3.61.1, zypper-1.14.50-3.46.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.