Bug 1223152

Summary:	Package openssl-3 built with gcc14 does not pass its testsuite and the build therefore fails
Product:	[openSUSE] openSUSE Tumbleweed	Reporter:	Martin Jambor <mjambor>
Component:	Other	Assignee:	Martin Jambor <mjambor>
Status:	IN_PROGRESS ---	QA Contact:	E-mail List <qa-bugs>
Severity:	Normal
Priority:	P5 - None	CC:	meissner, otto.hollmann, pmonrealgonzalez, rguenther
Version:	Current
Target Milestone:	---
Hardware:	Other
OS:	Other
Whiteboard:
Found By:	---	Services Priority:
Business Priority:		Blocker:	---
Marketing QA Status:	---	IT Deployment:	---
Bug Depends on:
Bug Blocks:	1220574

Description Martin Jambor 2024-04-19 14:44:27 UTC

Package openssl-3, when built with with gcc14, spins infinitely
(probably) in its testsuite and the build therefore fails:

https://build.opensuse.org/package/live_build_log/home:rguenther:plgrnd/openssl-3/openSUSE_Factory/x86_64

This can be reproduced locally with:
  oosc build  --clean --alternative-project home:rguenther:nextgcc

The failure may be a genuine miscompilation, it goes away when LTO is
disabled or when we disable identical code folding (ICF) using:
  %global optflags %{optflags} -fno-ipa-icf

I have also tried disabling strict aliasing in a similar fashion, but
that did not work.

On the other hand, the issue seems to be brought in by our patches, I
could reproduce the issue outside of an osc chroot when I applied all
the patches from the package too but the issue did not reproduce when
I did not.  Unfortunately I was not quite able to pin it to a single
patch, partly because that part of the spec file is somewhat
complicated, partly probably because I quickly confused myself.

It would be nice to figure out what exactly is going on and runs
differently with gcc14 and its ICF.

Comment 1 Otto Hollmann 2024-04-29 11:03:45 UTC

I found that OpenSSL 3.2.1 builds fine
> https://build.opensuse.org/package/show/home:ohollmann:branches:security:tls:gcc/openssl-3


Since this bug is for Tumbleweed, I suggest to resolve it by OpenSSL version upgrade. I will check if other packages are ready for such upgrade.

Comment 2 Martin Jambor 2024-05-22 09:47:36 UTC

(In reply to Otto Hollmann from comment #1)
> I found that OpenSSL 3.2.1 builds fine
> > https://build.opensuse.org/package/show/home:ohollmann:branches:security:tls:gcc/openssl-3
> 
> 
> Since this bug is for Tumbleweed, I suggest to resolve it by OpenSSL version
> upgrade. I will check if other packages are ready for such upgrade.

That would be great!

Comment 3 Martin Jambor 2024-05-22 09:48:01 UTC

Meanwhile, to investigate with CPU time, I resorted to debug counters.
The failure can still be reproduced when adding:

%global optflags %{optflags} -fdbg-cnt=merged_ipa_icf:86-86

to the spec file and conversely goes away with:

%global optflags %{optflags} -fdbg-cnt=merged_ipa_icf:1-85:87-4095

Unfortunately IPA-ICF does not dump stuff differently when not merging
functions because of a debug counter so I hacked it a little to do so
and with just doing the 86th merge and with -fdump-ipa-icf-details the
key message appeared in the following dump files (before I stopped the
stalled test run):

./fuzz/cmp-test.wpa.083i.icf:crypto/o_str.c:225:23: optimized: Semantically equivallent nodes merged.
./fuzz/punycode-test.wpa.083i.icf:crypto/engine/eng_rdrand.c:52:12: optimized: Semantically equivallent nodes merged.
./providers/fips.so.wpa.083i.icf:crypto/dsa/dsa_sign.c:79:5: optimized: Semantically equivallent nodes merged.
./test/algorithmid_test.wpa.083i.icf:crypto/rand/rand_lib.c:917:13: optimized: Semantically equivallent nodes merged.
./test/cmactest.wpa.083i.icf:crypto/rand/rand_lib.c:917:13: optimized: Semantically equivallent nodes merged.
./test/acvp_test.wpa.083i.icf:crypto/params.c:968:12: optimized: Semantically equivallent nodes merged.
./test/asn1_internal_test.wpa.083i.icf:crypto/evp/kdf_meth.c:20:12: optimized: Semantically equivallent nodes merged.
./test/bn_internal_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/asn1_dsa_internal_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/cmp_ctx_test.wpa.083i.icf:providers/implementations/rands/drbg_ctr.c:355:12: optimized: Semantically equivallent nodes merged.
./test/chacha_internal_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/cmp_asn_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/cmp_protect_test.wpa.083i.icf:crypto/o_str.c:225:23: optimized: Semantically equivallent nodes merged.
./test/cmp_server_test.wpa.083i.icf:providers/implementations/rands/drbg_ctr.c:355:12: optimized: Semantically equivallent nodes merged.
./test/cmp_msg_test.wpa.083i.icf:providers/implementations/rands/drbg_ctr.c:355:12: optimized: Semantically equivallent nodes merged.
./test/cmp_hdr_test.wpa.083i.icf:crypto/rand/rand_lib.c:917:13: optimized: Semantically equivallent nodes merged.
./test/cmp_status_test.wpa.083i.icf:crypto/rand/rand_lib.c:917:13: optimized: Semantically equivallent nodes merged.
./test/cmp_client_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/cipher_overhead_test.wpa.083i.icf:crypto/evp/evp_fetch.c:573:13: optimized: Semantically equivallent nodes merged.
./test/context_internal_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/cmp_vfy_test.wpa.083i.icf:providers/implementations/rands/drbg_ctr.c:355:12: optimized: Semantically equivallent nodes merged.
./test/ctype_internal_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/curve448_internal_test.wpa.083i.icf:crypto/o_str.c:225:23: optimized: Semantically equivallent nodes merged.
./test/destest.wpa.083i.icf:crypto/rand/rand_lib.c:917:13: optimized: Semantically equivallent nodes merged.
./test/dhtest.wpa.083i.icf:crypto/rand/rand_lib.c:917:13: optimized: Semantically equivallent nodes merged.
./test/drbgtest.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/dsa_no_digest_size_test.wpa.083i.icf:crypto/rand/rand_lib.c:917:13: optimized: Semantically equivallent nodes merged.
./test/dsatest.wpa.083i.icf:crypto/rand/rand_lib.c:917:13: optimized: Semantically equivallent nodes merged.
./test/ec_internal_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/ecdsatest.wpa.083i.icf:crypto/provider_conf.c:60:20: optimized: Semantically equivallent nodes merged.
./test/ectest.wpa.083i.icf:crypto/evp/legacy_sha.c:61:1: optimized: Semantically equivallent nodes merged.
./test/endecode_test.wpa.083i.icf:include/internal/refcount.h:47:19: optimized: Semantically equivallent nodes merged.
./test/endecoder_legacy_test.wpa.083i.icf:providers/implementations/keymgmt/ecx_kmgmt.c:425:12: optimized: Semantically equivallent nodes merged.
./test/evp_libctx_test.wpa.083i.icf:crypto/o_str.c:225:23: optimized: Semantically equivallent nodes merged.
./test/evp_pkey_provided_test.wpa.083i.icf:crypto/o_str.c:225:23: optimized: Semantically equivallent nodes merged.
./test/ext_internal_test.wpa.083i.icf:test/testutil/options.c:69:5: optimized: Semantically equivallent nodes merged.
./test/ffc_internal_test.wpa.083i.icf:crypto/rand/rand_lib.c:917:13: optimized: Semantically equivallent nodes merged.
./test/hexstr_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/hmactest.wpa.083i.icf:crypto/rand/rand_lib.c:917:13: optimized: Semantically equivallent nodes merged.
./test/ideatest.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/keymgmt_internal_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/modes_internal_test.wpa.083i.icf:crypto/evp/signature.c:295:5: optimized: Semantically equivallent nodes merged.
./test/namemap_internal_test.wpa.083i.icf:crypto/provider_conf.c:60:20: optimized: Semantically equivallent nodes merged.
./test/nodefltctxtest.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/param_build_test.wpa.083i.icf:crypto/provider_core.c:1265:7: optimized: Semantically equivallent nodes merged.
./test/params_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/poly1305_internal_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/property_test.wpa.083i.icf:crypto/o_str.c:225:23: optimized: Semantically equivallent nodes merged.
./test/prov_config_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/provfetchtest.wpa.083i.icf:crypto/evp/evp_fetch.c:573:13: optimized: Semantically equivallent nodes merged.
./test/provider_internal_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/provider_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/punycode_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/rc2test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/rc4test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/rc5test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/rdcpu_sanitytest.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/rsa_mp_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/rsa_sp800_56b_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/rsa_test.wpa.083i.icf:crypto/rand/rand_lib.c:917:13: optimized: Semantically equivallent nodes merged.
./test/siphash_internal_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/sm3_internal_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/sparse_array_test.wpa.083i.icf:crypto/rsa/rsa_backend.c:328:24: optimized: Semantically equivallent nodes merged.
./test/ssl_old_test.wpa.083i.icf:include/internal/packet.h:176:31: optimized: Semantically equivallent nodes merged.
./test/timing_load_creds.wpa.083i.icf:providers/implementations/ciphers/cipher_aes.c:59:1: optimized: Semantically equivallent nodes merged.
./test/tls13encryptiontest.wpa.083i.icf:include/internal/packet.h:536:31: optimized: Semantically equivallent nodes merged.
./test/x509_internal_test.wpa.083i.icf:crypto/rand/rand_lib.c:917:13: optimized: Semantically equivallent nodes merged.

Comment 4 Martin Jambor 2024-05-31 11:27:08 UTC

Strange developments happened yesterday.

First and foremost, the I noticed the issue reported here went away
with revision 24 of openssl-3 (https://build.opensuse.org/package/rdiff/openSUSE:Factory/openssl-3?linkrev=base&rev=24).
Both locally and in the Gcc7 staging project (https://build.opensuse.org/package/live_build_log/openSUSE:Factory:Staging:Gcc7/openssl-3/standard/x86_64 )

We have nevertheless looked at the issue a bit deeper and found out
that ICF is not doing anything wrong on its own, rather unifying
functions i2d_ECDSA_SIG and i2d_DSA_SIG in fips.so leads to inlining
of function BUF_MEM_new because it is now the only call (the function
is also split making the situation even more blurry).  However, moving
the function into a header file and marking it always_inline leads to
test failures even when ICF is switched off.

I'm assigning the issue to myself in order for further analysis of
what is going on.