Bugzilla – Bug 1194187
openssl broken
Last modified: 2024-02-15 15:50:07 UTC
found this while debugging why my dehydrated job is erroring out. up2date Tumbleweed rpm -qa \*openssl\* libopenssl1_0_0-1.0.2u-6.2.x86_64 libopenssl1_0_0-steam-1.0.2u-6.2.x86_64 libopenssl1_0_0-steam-32bit-1.0.2u-6.2.x86_64 libopenssl1_1-32bit-1.1.1l-4.1.x86_64 openssl-1_1-1.1.1l-4.1.x86_64 libopenssl1_1-1.1.1l-4.1.x86_64 openssl-1.1.1l-1.2.noarch ``` Error configuring OpenSSL modules 139707710848960:error:08064066:object identifier routines:OBJ_create:oid exists:crypto/objects/obj_dat.c:698: 139707710848960:error:0D0AE0AB:asn1 encoding routines:oid_module_init:adding object:crypto/asn1/asn_moid.c:38: 139707710848960:error:0E07606D:configuration file routines:module_run:module initialization error:crypto/conf/conf_mod.c:177:module=oid_section, value=new_oids, retcode=-1 ```
worked around it by commenting out: ``` # oid_section = new_oids # engines = engine_section ```
just commenting out the oid_section statement is enough.
Hmm. I just took the /etc/ssl/openssl.cnf file from the openssl-1_1-1.1.1l-4.1.x86_64 RPM and compared it to my test system. It's identical, and I didn't see that message when I did my testing. Where, exactly, is that "Error configuring OpenSSL modules" message seen?
The call that triggers it is: "openssl req -new -sha256 -key /etc/dehydrated/certs/keyname_rsa/privkey-1640907684.pem -out /etc/dehydrated/certs/keyname_rsa/cert-1640907684.csr -subj /CN=keyname/ -reqexts SAN -config /tmp/dehydrated-DwFNuo" /tmp/dehydrated-DwFNuo is a copy of our /etc/ssl/openssl.cnf with a `[SAN]` section attached.
(In reply to Marcus Rückert from comment #4) > /tmp/dehydrated-DwFNuo is a copy of our /etc/ssl/openssl.cnf with a `[SAN]` > section attached. Thanks. Could you attach a copy of your openssl.cnf file with that section included in it?
cp /etc/ssl/openssl.cnf /tmp/dehydrated-DwFNuo Add something like ``` [SAN] subjectAltName=DNS:host1, DNS:host2 ```
(In reply to Marcus Rückert from comment #6) > cp /etc/ssl/openssl.cnf /tmp/dehydrated-DwFNuo > > > Add something like > > ``` > [SAN] > subjectAltName=DNS:host1, DNS:host2 > ``` Thanks, Marcus. Now I have to ask you to create something I can use for /etc/dehydrated/certs/keyname_rsa/privkey-1640907684.pem to see what's going on. Obviously I don't want you to send me a "real" private key, but something that gets created the same way would be really nice.
it is something like this: "openssl genrsa -out /tmp/dehydrated-YLCP3U 4096"
Thanks again. In playing around, it seems there is some sort of issue with the addition of these two lines: openssl_conf = openssl_init [openssl_init] When I added those, I was following the documentation at https://www.openssl.org/docs/man1.1.1/man5/config.html If I remove them or comment them out, the CSR file gets created, but the engines definitions don't get created, even though the include files are read. It's as though these two lines are being totally ignored without the section header: oid_section = new_oids engines = engine_section Which seems odd, since the documentation says: The first section of a configuration file is special and is referred to as the default section. This section is usually unnamed and spans from the start of file until the first named section. Without those two lines, the old_section and engines lines should be considered as being in the default section. This is starting to look more like an openssl bug to me, rather than the changes I made being somehow wrong. Just what that means for Marcus adding the [SAN] section for his needs is really not clear. It seems as though his particular command only requires the "[ req_distinguished_name ]" section from openssl.cnf. But, none of those values are customized by us, so it's all very generic. I don't know enough about openssl to be sure, but this kind of looks like this is relevant: https://github.com/openssl/openssl/issues/4598#issuecomment-341321065 Pedro, if you could take a look at this, I would appreciate it.
Hi, Mark, Marcus. Sorry for not coming back to this earlier since we are quite busy with the FIPS certification now. Looks like by adding those two lines you are initializing twice both sections and it fails in OBJ_create() when checking the long name from the NID in OBJ_ln2nid() as its already there. Mark, do you need these new sections for bsc#1004463 or could these be removed?
The .include directives could be used instead if needed.
(In reply to Pedro Monreal Gonzalez from comment #10) > Looks like by adding those two lines you are initializing twice both > sections and it fails in OBJ_create() when checking the long name from the > NID in OBJ_ln2nid() as its already there. Well, that's where my confusion comes in. How/where is it being initialized a second time? From what I can tell, the "oid_section = new_oids" statement is not being processed in the previous version of openssl.cnf. It's only being processed after adding openssl_conf = openssl_init [openssl_init] Why is that? > Mark, do you need these new sections for bsc#1004463 or could these be > removed? If I remove either of those two lines, the "engines = engine_section" directive isn't being processed, and so the ibmca engine I'm trying to have recognized, is not recognized.
okay just ran into the same thing, probably everyone running factory with dehydrated for SSL certs (was broken since december, got the message from letsencrypt my cert is running out now, so I noticed)
*** Bug 1196271 has been marked as a duplicate of this bug. ***
Hi, @Mark. The added engine section in your submmission [0] looks fine but the openssl_init new section, namely: openssl_conf = openssl_init [openssl_init] is not needed. Please, let me know if we can remove these two lines or these are somehow needed for bsc#1004463. [0] https://build.opensuse.org/request/show/936137
As I said in comment#12, if those lines are not there, the engine section is not recognized and used by openssl. So yes, they are needed. According to the documentation I've read, they should not be required, but in practice they are.
OK, then adding a new section and following the INI-type file sections, this should work: [...] openssl_conf = openssl_init [openssl_init] engines = engine_section # This include will look through the directory that will contain the # engine declarations for any engines provided by other packages. [ engine_section ] .include /etc/ssl/engines.d/ # This include will look through the directory that will contain the # definitions of the engines declared in the engine section. .include /etc/ssl/engdef.d/ [ oid_section ] # Extra OBJECT IDENTIFIER info: #oid_file = $ENV::HOME/.oid oid_section = new_oids # To use this configuration file with the "-extfile" option of the [...] Would you mind checking?
Jeez, the one in comment#19 was correct.
Using the version from comment#19, I get this: # openssl engine (dynamic) Dynamic engine loading support (ibmca) Ibmca hardware engine support 4396668127008:error:260AC089:engine routines:int_ctrl_helper:invalid cmd name:crypto/engine/eng_ctrl.c:87: 4396668127008:error:260AB089:engine routines:ENGINE_ctrl_cmd_string:invalid cmd name:crypto/engine/eng_ctrl.c:255: 4396668127008:error:260BC066:engine routines:int_engine_configure:engine configuration error:crypto/engine/eng_cnf.c:141:section=ibmca_section, name=oid_section, value=new_oids 4396668127008:error:0E07606D:configuration file routines:module_run:module initialization error:crypto/conf/conf_mod.c:177:module=engines, value=engine_section, retcode=-1 As before, commenting out this oid_section = new_oids makes those errors go away. I'm really thinking there's a bug in the parsing code openssl is using.
Created attachment 856471 [details] openssl.cnf with fixed sections Yes, as you mentioned in comment#9 there is a bug when loading cnf file twice. Its quite an intricate bug and upstream decided to not fixing it. I think the bug described here can be fixed with correct sections to not load the OID twice and getting the error "oid exists". I'm attaching here the fixed openssl.cnf file that works in my setup and I think it should work in all cases. Here is a minimal reproducer: 1.- This fails: openssl req -config /etc/ssl/openssl.cnf -x509 -nodes -out x.pem -keyout x.pem 2.- This works with the new openssl.cnf file in /tmp: openssl req -config /tmp/openssl.cnf -x509 -nodes -out x.pem -keyout x.pem
Interesting. On my test system (s390vsl152.suse.de) this is what happens: # openssl req -config /tmp/openssl.cnf -x509 -nodes -out x.pem -keyout x.pem Error configuring OpenSSL modules 4396961204000:error:26078067:engine routines:engine_list_add:conflicting engine id:crypto/engine/eng_list.c:63: 4396961204000:error:2606906E:engine routines:ENGINE_add:internal list error:crypto/engine/eng_list.c:223: 4396961204000:error:260B6067:engine routines:dynamic_load:conflicting engine id:crypto/engine/eng_dyn.c:502: 4396961204000:error:260BC066:engine routines:int_engine_configure:engine configuration error:crypto/engine/eng_cnf.c:141:section=ibmca_section, name=dynamic_path, value=/usr/lib64/engines-1.1/ibmca.so 4396961204000:error:0E07606D:configuration file routines:module_run:module initialization error:crypto/conf/conf_mod.c:177:module=engines, value=engine_section, retcode=-1 Oddly, if I move the file to /etc/ssl, and leave the --config parameter off, it works. Since Marcus is using the -config /tmp/ syntax, I don't think this will work for him.
Hi @Mark, thanks for testing! I see this is a different error message than the "oid exists" originally reported. Your error message is "conflicting engine id" and this is specific for the engines in ibmca for the s390x architecture. Let me reserve an s390x vm to test the engines part. I may need your guidance to set up the environment. As the error message says, there must be a conflict adding the engines. Probably, this should be left for the user and not include it in the default shipped openssl.cnf file as it was done before. It would be great if @Marcus could test the solution from comment#23 in his x86_64 arch.
Note that the engines configuration is detailed here: * https://www.openssl.org/docs/man1.1.1/man5/config.html
I think I've found a better solution. Instead of using the "-config /tmp/..." syntax, if you use this "OPENSSL_CONF=/tmp/openssl.cnf.test openssl req ..." then it doesn't seem to trigger the parsing bug.
yes we could fix dehydrated. but we can _not_ control how everyone out there is calling openssl. so we really should fix this bug within openssl.
Based on what Pedro said, upstream is not willing to fix it.
I can submit the config fix in comment#23 to fix the "oid exists" error and we can focus on the "conflicting engine id" error for s390x later. Would you be fine with that?
(In reply to Pedro Monreal Gonzalez from comment #31) > I can submit the config fix in comment#23 to fix the "oid exists" error and > we can focus on the "conflicting engine id" error for s390x later. Would you > be fine with that? Not really, no. And the reason I say that is if Marcus were to try and set up dehydrated on an s390x system, it *still* wouldn't work for him. Because the errors from comment#24 would cause his command to fail. I think we need to come up with something that will work for Marcus, and our s390x customers. What that might look like, I have no idea at this point.
OK, I figured out how to get rid of the engine errors, while still using the base openssl.cnf file from comment#23. Based on what I read in https://github.com/openssl/openssl/issues/9481 it seems that we don't need the following statement in the engine definition file for ibmca: dynamic_path = /usr/lib64/engines-1.1/ibmca.so The reason is, we're installing the library in openssl's default location, so it finds it without this statement. Yay. That leaves two tasks: 1. I submit an update to openssl-ibmca to remove that line. 2. You submit an update to openssl-1_1 to use the version of the openssl.cnf from comment#23. I would only ask one more thing... In your version, you have the include statements with a trailing "/" on them. It seems openssl automatically appends a trailing slash on whatever is specified there, so having it in the openssl.cnf is not needed, and it makes strace output look a little weird. That is, instead of .include /etc/ssl/engines.d/ .include /etc/ssl/engdef.d/ do this: .include /etc/ssl/engines.d .include /etc/ssl/engdef.d Yeah, it's picking at a nit, but the OCD in me would appreciate it. Thanks for your help on this.
Thanks, Mark! I have submitted the fix and added the include modifications, here: * https://build.opensuse.org/request/show/960455
Are you also going to be submitting this to SUSE:SLE-15-SP3:Update and SUSE:SLE-15-SP4:GA ?
(In reply to Mark Post from comment #35) > Are you also going to be submitting this to SUSE:SLE-15-SP3:Update and > SUSE:SLE-15-SP4:GA ? No, the engines change was not submitted to SLE, its only in Factory. Do you need it in SLE?
I would like to see it in 15 SP4 at least, yes. This whole change was driven by needing to get rid of a very fragile script that made direct changes to the openssl.cnf file.
OK, I'll add the change in the submission we have planned for next week.
Fix submitted here: * https://build.suse.de/request/show/267605
Submitted to SLE15-SP4:GA also. Closing. Please, reopen if required.
Hi, @Mark. I submitted the fix to Factory and SLE15-SP4:GA. Do you need the engines section change also in openssl-3 config file?
Yes, please.
OK, I'm reopening then.
@Jason, the change needs to be added to the config file also for openssl-3 in both Factory and SL15-SP4. Could you submit? Assigning back to you.
@Mark Post, are you able to reproduce this issue using openssl-3? I prepared a patch (updated config file) and would like to know if you can verify/test it.
Created attachment 864410 [details] openssl-3.cnf with fixed sections
(In reply to Otto Hollmann from comment #48) > @Mark Post, are you able to reproduce this issue using openssl-3? I prepared > a patch (updated config file) and would like to know if you can verify/test > it. What distribution would have openssl3 included, that is also available for s390x?
SLES 15 SP4, Tumbleweed are availab for IBM z
Created attachment 864467 [details] Console output from the "openssl-3 engine" command (In reply to Otto Hollmann from comment #48) > @Mark Post, are you able to reproduce this issue using openssl-3? I prepared > a patch (updated config file) and would like to know if you can verify/test > it. Hi, Otto. I installed openssl-3 on one of my test systems, then replaced the openssl.cnf file with your version. The "openssl-3 engine" command generated the following error messages: # openssl-3 engine FATAL: Startup failure (dev note: apps_startup()) for openssl-3 000003FFBE2F6710:error:12800067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:118:filename(/usr/lib64/engines-3/ibmca.so): /usr/lib64/engines-3/ibmca.so: cannot open shared object file: No such file or directory 000003FFBE2F6710:error:12800067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:152: 000003FFBE2F6710:error:13000084:engine routines:dynamic_load:dso not found:crypto/engine/eng_dyn.c:422: 000003FFBE2F6710:error:13000074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:430:id=ibmca 000003FFBE2F6710:error:13000066:engine routines:int_engine_configure:engine configuration error:crypto/engine/eng_cnf.c:139:section=ibmca_section, name=init, value=1 000003FFBE2F6710:error:0700006D:configuration file routines:module_run:module initialization error:crypto/conf/conf_mod.c:243:module=engines, value=engine_section retcode=-1 That was because there's nothing in /usr/lib64/engines-3/ for openssl-ibmca. Just out of curiosity, I tried creating a symlink there to /usr/lib64/engines-1.1 and re-ran the command. On the terminal, the only messages that came out were: # openssl-3 engine (dynamic) Dynamic engine loading support (ibmca) Ibmca hardware engine support Segmentation fault (core dumped) But on the virtual console of the guest, a "User process fault" mini-dump was displayed. The output from that is in the attached file. So, overall, things aren't looking good. I would have to guess that at least part of the problem is that the openssl-ibmca package is not built against openssl3. That may actually be the whole problem, but I don't know for sure.
SUSE-SU-2023:4635-1: An update that solves one vulnerability and has two security fixes can now be installed. Category: security (important) Bug References: 1194187, 1207472, 1216922 CVE References: CVE-2023-5678 Sources used: openSUSE Leap 15.5 (src): openssl-3-3.0.8-150500.5.19.1 Basesystem Module 15-SP5 (src): openssl-3-3.0.8-150500.5.19.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
SUSE-SU-2023:4649-1: An update that solves one vulnerability and has two security fixes can now be installed. Category: security (important) Bug References: 1194187, 1207472, 1216922 CVE References: CVE-2023-5678 Sources used: openSUSE Leap 15.4 (src): openssl-3-3.0.8-150400.4.42.1 SUSE Linux Enterprise Micro for Rancher 5.3 (src): openssl-3-3.0.8-150400.4.42.1 SUSE Linux Enterprise Micro 5.3 (src): openssl-3-3.0.8-150400.4.42.1 SUSE Linux Enterprise Micro for Rancher 5.4 (src): openssl-3-3.0.8-150400.4.42.1 SUSE Linux Enterprise Micro 5.4 (src): openssl-3-3.0.8-150400.4.42.1 Basesystem Module 15-SP4 (src): openssl-3-3.0.8-150400.4.42.1 openSUSE Leap Micro 5.3 (src): openssl-3-3.0.8-150400.4.42.1 openSUSE Leap Micro 5.4 (src): openssl-3-3.0.8-150400.4.42.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
This is an autogenerated message for OBS integration: This bug (1194187) was mentioned in https://build.opensuse.org/request/show/1142584 Factory / openssl-3
OpenSSL 3 will be default starting SLE15-SP6 and ALP and it will read from directories /etc/ssl/engines3.d and /etc/ssl/engdef3.d/. Also symbolic links /etc/ssl/engines.d/ and /etc/ssl/engdef.d/ to above versioned directories were created. Everything submitted, closing.