Bugzilla – Bug 1046158
[kmp, psign, obs] Building KMPs without # needssslcertforbuild leads to broken Signatures
Last modified: 2020-06-04 12:28:41 UTC
Loading the lustre-client KMP from OpenHPC (http://build.openhpc.community/OpenHPC:/1.3/SLE_12/) on a SLE-12-SP2 system (running any officially released kernel) fails with -ERANGE: # modprobe lustre modprobe: ERROR: could not insert 'lustre': Numerical result out of range strace reveals that the failing syscall is: finit_module(3, "", 0) = -1 ERANGE (Numerical result out of range) Hunting this down in the kernel one finds that in crypto/asymmetric_keys/rsa.c:RSA_I2OSP() the test below fails and returns -ERANGE: x_size = mpi_get_nbits(x); pr_devel("size(x)=%u xLen*8=%zu\n", x_size, xLen * 8); if (x_size != xLen * 8 - 15) return -ERANGE; In this case, no attempt is made to actually load the module unlike when an unsigned module is loaded as long as no strict signature checking is required (ie in the non-secureboot case). This indicates that the signature in this kernel module is bogus. The build was done on an OBS system at OpenHPC. OpenHPC doesn't seem to run a signing server, and the '# needssslcertforbuild' magic was not set in the spec file. Still the OBS voodoo for kernel module signing runs and the test for a cert fails and the fallback is taken as can be seen by the message: warning: No buildservice project certificate found, add warning: # needssslcertforbuild to the specfile warning: Using /usr/lib/rpm/pesign/pesign-cert.x509 as fallback Check: https://build.openhpc.community/build/OpenHPC:1.3/SLE_12/x86_64/lustre-client/_log I don't think this behaviour is intended as the built modules are useless unless a signing server is run. I'm not sure how relevant secure boot and signed kernel modules are in the HPC realm. Right now this looks to me like an issue in pesign-obs-integration, thus the assignment. The same issue can be found when building this package in the openSUSE OBS (as well as in IBS).
The signing service is set up properly, you can see in the build log that it loops over the *.sig files provided by the service. A solution to the issue, besides adding # needssslcertforbuild, is to link the pesign-obs-integration package to the project. That way, it will use the correct fallback public certificate. Reassigning to Joey to decide whether the error should be non-fatal in the non SecureBoot case.
The pesign-obs-integration package is pulled into the OpenHPC build already as can be seen from the line [ 107s] [164/165] installing pesign-obs-integration-10.0-29.1 pulled from the link in the description. I may have misunderstood - I was under the impression that the # needssslcertforbuild should only be used when a signing server is active. I will tell the openHPC folks to branch the package and build it with this line added and see if this helps.
(In reply to Egbert Eich from comment #2) > The pesign-obs-integration package is pulled into the OpenHPC build already > as can be seen from the line > [ 107s] [164/165] installing pesign-obs-integration-10.0-29.1 > pulled from the link in the description. > I may have misunderstood - I was under the impression that the > # needssslcertforbuild should only be used when a signing server is active. > I will tell the openHPC folks to branch the package and build it with this > line added and see if this helps. I cannot test this in our OBS instance as we do have a signing server set up, so, with the above option set, I do get a valid signature.
You are using the pesign-obs-integration binary package from SLE12, with the canned SLE12 certificate. What I mean is to linkpac/copypac openSUSE.org:SUSE:SLE-12-SP2:GA/pesign-obs-integration into OpenHPC:1.3, that way it will get rebuilt and store the OpenHPC certificate. The second option is of course to patch the KMPs to have # needssslcertforbuild.
Ok, I've passed the information to copypac/linpac the pesign-obs-integration integration package into the build project to OpenHPC now. Along with it, I passed information how to interconnect build services - so they can connect directly to oS OBS (which may be useful for other thins as well). Wouldn't it be possible that a module built with the pesign-obs-integration containing the 'canned SLE12 certificate' to spit out a big warning together with an explanation of what to do and fail the build? The only explanation it spits out is to add the '# needssslcertforbuild' magic, which obviously isn't enough. This warning also doesn't go away after adding the ine. So this is a bit confusing already. But let's see what comes out of OpenHPC tests.
(In reply to Egbert Eich from comment #5) > The only explanation it spits out is to add the '# needssslcertforbuild' > magic, which obviously isn't enough. This warning also doesn't go away after > adding the ine. So this is a bit confusing already. Can you double-check this? The OpenHPC:1.3/lustre-client package does not have the # needssslcertforbuild line in the spec file, which is why the build is complaining and which is why the wrong certificate information is recorded in the modules.
(In reply to Michal Marek from comment #6) > (In reply to Egbert Eich from comment #5) > > The only explanation it spits out is to add the '# needssslcertforbuild' > > magic, which obviously isn't enough. This warning also doesn't go away after > > adding the ine. So this is a bit confusing already. > > Can you double-check this? The OpenHPC:1.3/lustre-client package does not > have the # needssslcertforbuild line in the spec file, which is why the > build is complaining and which is why the wrong certificate information is > recorded in the modules. Sure, but this was exactly the difference that I asked Karl to check. I forgot to post the link to the spec file, here it is: https://build.openhpc.community/package/view_file/OpenHPC:1.3:Update2:Factory/lustre-client/_service:extract_file:lustre.spec?expand=1
Weird. It could be that the project is lacking a SSL certificate. Can you paste the output of osc signkey --sslcert OpenHPC:1.3:Update2:Factory ?
Looks like do not in fact have a SSL key for this project. I presume that is now a requirement for SP2 loading? $ osc signkey --sslcert OpenHPC:1.3:Update2:Factory OpenHPC:1.3:Update2:Factory has no key, trying OpenHPC:1.3:Update2
OK, that could be it. In the past, the buildservice would not start the build, now it starts it but it ignores the needssslcertforbuild request and most likely it also does not produce any signatures. As to new SP2 requirement, this has been required since SLE11 SP3 and throughout the whole SLE12 series, but maybe you only started to build kernel modules now?
(In reply to Michal Marek from comment #10) > > As to new SP2 requirement, this has been required since SLE11 SP3 and > throughout the whole SLE12 series, but maybe you only started to build > kernel modules now? Actually, we used OBS to build Lustre kernel modules that loaded fine in SLE12 SP1. This seemed to first crop up for us with SP2.
Did the SLE12 SP1 project have an SSL certificate generated?
(In reply to Michal Marek from comment #12) > Did the SLE12 SP1 project have an SSL certificate generated? No, same setup as we have currently. We have a global GPG key configured for packaging signing using the functionality in BSConfig.pm, but nothing specific using "osc signkey". It does work now if I generate a new gpg key for this current project using "osc signkey --create", but obviously that overrides our global GPG project key which is undesirable. Is there some equivalent way to configure it to leverage our existing GPG key during the kernel module signing process?
OK, then I'm out of ideas. Michael, can you have a look? The specfile linked from comment 7. Despite the # needssslcertforbuild line, there is no _projectcert.crt generated (see the build log in comment 0). Reportedly, this worked with previous project using the same global signing key.
Some clarifications: One difference is that older versions of this package (ie spec file) did not use the %kernel_module_package macro for building a KMP. Therefore the 2nd stage of the module package build which adds the signatures never ran. This version of spec file was used to created packages for SLES-12 SP1. Karl and I got a bit further on this - since the discussion was started on emailI never updated it. Michal's comment #8 and the fact that this key never existed on Karl's system made me google a bit and I cane up with: http://opensuse.14.x6.nabble.com/Build-failure-due-to-a-missing-RSA-key-td5045532.html Running an 'osc signkey --create' in this package directory will create a gpg key for this package which will then be used to generate the ssl certificate when needed. This did work for Karl, however, now he has a different gpg key for this package than the system wide one. So the next question was how to make use of the system wide key. Since I was unable to answer this, I've added Adrian to Cc:, however, I've not seen any feedback from him. I do assume he's on vacation.
(In reply to Egbert Eich from comment #15) > Some clarifications: > > One difference is that older versions of this package (ie spec file) did not > use the %kernel_module_package macro for building a KMP. > Therefore the 2nd stage of the module package build which adds the > signatures never ran. Ah, OK, mystery solved. Thanks. > Running an 'osc signkey --create' in this package directory will create a > gpg key for this package which will then be used to generate the ssl > certificate when needed. Running 'osc signkey --create' indeed is a fix for this. > This did work for Karl, however, now he has a different gpg key for this > package than the system wide one. So the next question was how to make use > of the system wide key. > Since I was unable to answer this, I've added Adrian to Cc:, however, I've > not seen any feedback from him. > I do assume he's on vacation. I have no idea whether you can use different keys for package signing and for module signing. Adrian or Michael have to answer this.
An OBS admin could of course copy the key from one project to another one, but the question is then if you don't break the idea of secure boot kernels. If you really want to copy the key, you need to copy on your source server the _pubkey and _signkey file in /srv/obs/projects/$PROJECT.pkg/ . Afterwards you need to generate the _sslcert using osc signkey --sslcert $project or regenerate it if an old one is there using osc signkey --extend $project However, I think it makes more sense just to disable the secure boot option as this basically breaks the idea.
(In reply to Adrian Schröter from comment #17) > An OBS admin could of course copy the key from one project to another one, > but the question is then if you don't break the idea of secure boot kernels. > > If you really want to copy the key, you need to copy on your source server > the _pubkey and _signkey file in /srv/obs/projects/$PROJECT.pkg/ . > Afterwards you need to generate the _sslcert using > > osc signkey --sslcert $project > > or regenerate it if an old one is there using > > osc signkey --extend $project > > However, I think it makes more sense just to disable the secure boot option > as this basically breaks the idea. Thanks Adrian! So the suggestion now is to disable the signing completely. Karl, would this be an option? I guess, the easiest way to achieve this would be to remove: BuildRequires: pesign-obs-integration in the spec file.
(In reply to Egbert Eich from comment #18) > > So the suggestion now is to disable the signing completely. Karl, would this > be an option? > I guess, the easiest way to achieve this would be to remove: > BuildRequires: pesign-obs-integration > in the spec file. Sure - disabling signing is an option. Recall that it was originally suggested to add BuildRequires: pesign-obs-integration when we first observed the module load failure. So as expected, removing this one line was not sufficient to get it working. However, if I also remove the following line: BuildRequires: %kernel_module_package_buildreqs then, that combination looks to disable the signing attempt and I can indeed load the resulting kernel module under SLE SP2. Thanks everyone for the help getting to the bottom of this! Cheers.
For completeness, there should be a BuildRequires: kernel-default-devel in the specfile, so that a module is built at all. %kernel_module_package_buildreqs pulls in kernel-syms which indeed pulls in pesign-obs-integration for convenience. Another option is to add export BRP_PESIGN_FILES="" to the %install section. This tells the pesign scripts not to sign anything.