Bug 1220681

Apache2 is now built against openssl3, so it might need adjustments for FIPS.

So, this seems resolved?

merged

Created attachment 873804 [details]
httpd-prefork core dump

With Build71.1 now getting core dump instead on a similar FIPS setup when starting apache:

Mar 22 14:55:26 susetest systemd[1]: apache2.service: Main process exited, code=dumped, status=11/SEGV
Mar 22 14:55:26 susetest systemd[1]: apache2.service: Failed with result 'core-dump'.

Reopening for still not starting, see attachments.

Could somebody get a back trace with debuginfo files installed? TIA

There are various upstram fixes that could be related (1913912, 1914365, 1916054, 1915067). See also:

https://github.com/apache/httpd/pull/402
https://src.fedoraproject.org/rpms/httpd/blob/rawhide/f/httpd-2.4.58-r1913912+.patch
https://src.fedoraproject.org/rpms/httpd/blob/rawhide/f/httpd-2.4.58-r1914365.patch

Once we get a full bt or access to an SP6 vm in FIPS mode we could circle the problem.

Access to VM provided.

I could not extract more info from the core file. The debuginfo files have been installed now but I haven't been able to reproduce the issue. Maybe if somebody can reproduce now. The command that segfaulted before was something like:

> httpd-prefork -DSYSCONFIG -DSSL -C 'PidFile /run/httpd.pid' -C 'Include /etc/apache2/sysconfig.d/include.conf' -C 'Include /etc/apache2/sysconfig.d/include.conf' -DSYSTEMD -DFORGROUND -k start

The service fails to start but does not dump a core, the error message now is:

> AH00557: httpd: apr_sockaddr_info_get() failed for dev-server
> AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message

I've narrowed down the core dump problem to when using environment variable FIPS mode (not kernel boot parameter fips=1). That is, in /etc/bash.bashrc:

export OPENSSL_FIPS=1
export OPENSSL_FORCE_FIPS_MODE=1

Additionally, "SSLFIPS on" in /etc/apache2/ssl-global.conf.

Re-tested manually on the new 73.1 build.

The lack of good/compatible debuginfo packages remains a problem. I've now a compressed pre-setupped VM image available for debugging.

I'm preparing an updated version with some patches (difficult to know without debuginfo).

@Timo: Let me finish them and I'll let you know. Would you be able to test these new packages once they build?

Sure, today or tomorrow.

Created attachment 873879 [details]
journal and apache logs

I got updated packages from David this morning and tested them. Findings:

- Core dump problem is fixed now!
- However, apache still refuses to start, due to "SSL Library Error: error:0A0000A1:SSL routines::library has no ciphers".
- I also reasserted on this VM if I put fips=1 to kernel parameters, apache starts. Removing it, relying on the environment variables, brings back the problem.

So this is now a third time a bug is being fixed and yet another problem uncovered, I will the change the title of this poor bug report to reflect that.

(In reply to Timo Jyrinki from comment #18)
> Created attachment 873879 [details]
> journal and apache logs
> 
> I got updated packages from David this morning and tested them. Findings:
> 
> - Core dump problem is fixed now!

Phew!

> - However, apache still refuses to start, due to "SSL Library Error:
> error:0A0000A1:SSL routines::library has no ciphers".

I requested some help from crypto/openssl people. Let's see.

> - I also reasserted on this VM if I put fips=1 to kernel parameters, apache
> starts. Removing it, relying on the environment variables, brings back the
> problem.

At least we have a workaround.

> 
> So this is now a third time a bug is being fixed and yet another problem
> uncovered, I will the change the title of this poor bug report to reflect
> that.

Yes, please ;)

For now, I just sent the updates to GA here:
- https://build.suse.de/request/show/325026

Staged on S. Should come next week

A submit request mentioning this bug  was successfully integrated into build 77.1.

Please set it as resolved and eventually verify.

Please indicate the current expectation for fixing the bug using the Target Milestone field. What do we believe is possible? 

In the cases where we do not plan to deliver or cannot guess (!?) please do not enter anything, but you can comment if you wish to provide more details.

Apache2 is still failing to start on 77.1 (with FIPS env vars).

(In reply to David Anes from comment #19)
> I requested some help from crypto/openssl people. Let's see.

Yes, I provided Otto a VM where this can be reproduced last week. I can respin it based on 77.1 if needed.

> > - I also reasserted on this VM if I put fips=1 to kernel parameters, apache
> > starts. Removing it, relying on the environment variables, brings back the
> > problem.
> At least we have a workaround.

That is not a real workaround in the sense that we have two ways of supporting FIPS, the kernel mode and using the environment variables, and both are expected to work. But yes, it's anyway very useful to narrow down where things might go wrong.

@Otto/@Pedro do you think "SSL Library Error: error:0A0000A1:SSL routines::library has no ciphers" is a configuration error on the side of Apache or on the side of OpenSSL/system?

It's interesting that it works enabling fips at kernel level but not via envvars.(In reply to Timo Jyrinki from comment #23)

> That is not a real workaround in the sense that we have two ways of
> supporting FIPS, the kernel mode and using the environment variables, and
> both are expected to work. But yes, it's anyway very useful to narrow down
> where things might go wrong.

I agree, at least we know that apache works in fips mode, therefore it seems something config-related or a silly issue somewhere.

Hi, David. See my comment inline.

(In reply to David Anes from comment #24)
> @Otto/@Pedro do you think "SSL Library Error: error:0A0000A1:SSL
> routines::library has no ciphers" is a configuration error on the side of
> Apache or on the side of OpenSSL/system?

Most likely yes, I have just set up a VM in FIPS mode and I start testing this on my environment.

> It's interesting that it works enabling fips at kernel level but not via
> envvars.(In reply to Timo Jyrinki from comment #23)
> 
> > That is not a real workaround in the sense that we have two ways of
> > supporting FIPS, the kernel mode and using the environment variables, and
> > both are expected to work. But yes, it's anyway very useful to narrow down
> > where things might go wrong.
> 
> I agree, at least we know that apache works in fips mode, therefore it seems
> something config-related or a silly issue somewhere.

Please, note that the support for the kernel FIPS mode can be enabled in two ways:
 1.- Add fips=1 on line GRUB_CMDLINE_LINUX_DEFAULT in file /etc/default/grub and execute following commands: zypper in -y -t pattern fips && grub2-mkconfig -o /boot/grub2/grub.cfg && mkinitrd && reboot

 2.- Use just one command from the crypto-policies-scripts package and reboot: fips-mode-setup --enable

Also, using forced FIPS mode for openssl with the envvar OPENSSL_FORCE_FIPS_MODE=1 should work as well. This is mostly used for testing the FIPS mode before enabling it.

So, from the previous comment it looks that the kernel FIPS mode has not been correctly enabled and that's probably why you mentioned its working there. Could you confirm? TIA

(In reply to Pedro Monreal Gonzalez from comment #25)
> So, from the previous comment it looks that the kernel FIPS mode has not
> been correctly enabled and that's probably why you mentioned its working
> there. Could you confirm? TIA

Was this for me? For me and our testing, we're correctly enabling kernel FIPS mode. In this switching between kernel end env var mode in my manual testing, I had installed the fips pattern and added fips=1 via yast2 bootloader module.

In our automated testing where it also passes, we're using fips-mode-setup --enable from crypto-policies-scripts (reference: https://openqa.suse.de/tests/13914185#step/fips_setup/22).

(In reply to Timo Jyrinki from comment #26)
> (In reply to Pedro Monreal Gonzalez from comment #25)
> > So, from the previous comment it looks that the kernel FIPS mode has not
> > been correctly enabled and that's probably why you mentioned its working
> > there. Could you confirm? TIA
> 
> Was this for me? For me and our testing, we're correctly enabling kernel
> FIPS mode. In this switching between kernel end env var mode in my manual
> testing, I had installed the fips pattern and added fips=1 via yast2
> bootloader module.
> 
> In our automated testing where it also passes, we're using fips-mode-setup
> --enable from crypto-policies-scripts (reference:
> https://openqa.suse.de/tests/13914185#step/fips_setup/22).

Hi, Timo. I just don't understand why it fails in forced FIPS mode by using envvars and not in kernel FIPS mode, I think both should work the same. But, since I'm not familiar with enabling FIPS via yast2, I was wondering if all was done correctly. See also the SP5 documentation [0].

Could you check if doing this fails the same as with 'fips-mode-setup --enable && reboot'?:
  1.- run the command 'fips-mode-setup --disable'
  2.- reboot
  3.- use envvars only and start apache2

Thanks in advance!

[0] https://documentation.suse.com/sles/15-SP5/single-html/SLES-security/#sec-how-to-fips-install

(In reply to Pedro Monreal Gonzalez from comment #27)
> Hi, Timo. I just don't understand why it fails in forced FIPS mode by using
> envvars and not in kernel FIPS mode

Yes, me neither :)

But we are testing both cases just because of the mechanism being different.

> I think both should work the same. But,
> since I'm not familiar with enabling FIPS via yast2, I was wondering if all
> was done correctly. See also the SP5 documentation [0].
> 
> Could you check if doing this fails the same as with 'fips-mode-setup
> --enable && reboot'?:
>   1.- run the command 'fips-mode-setup --disable'
>   2.- reboot
>   3.- use envvars only and start apache2

I was trying to explain we've covered all of these via either manual or automated testing, but just in case of doubt I did what you asked in the VM and the results say the same: after --disable (but env vars present) failing, while with FIPS kernel mode enabled not failing.

Again, the VM is available for easy transfering if you want to play around with it yourself.

Timo, thanks for double checking. Yes, please, give me and Otto access to the VM and we can take a look. TIA

Ack, Otto has had it since last Thursday, now you have access as well.

(In reply to Timo Jyrinki from comment #30)
> Ack, Otto has had it since last Thursday, now you have access as well.

Thanks! I suspect there might be other libraries at play that are not in forced FIPS mode when the envvar for openssl is used. There could be NSS, GnuTLS, Libgcrypt... they can be forced similarly with:

 * LIBGCRYPT_FORCE_FIPS_MODE=1

I'll try this now!

We always set the following env vars in our automated testing when using environment variables (in kernel fips mode, we do not set those):

OPENSSL_FIPS=1
OPENSSL_FORCE_FIPS_MODE=1
LIBGCRYPT_FORCE_FIPS_MODE=1
NSS_FIPS=1
GnuTLS_FORCE_FIPS_MODE=1

(in the VM I provided, I added them both to /etc/profile and /etc/bash.bashrc)

As I understand, Pedro provides the information. If so, I cleave the needinfo flag.

 We'll have an SR for this?

(In reply to Radoslav Tzvetkov from comment #33)
> As I understand, Pedro provides the information. If so, I cleave the
> needinfo flag.
> 
>  We'll have an SR for this?

The original segfault was fixed by David and I think he will submit the final fix once this is clear.

Unfortunately, I can't find the root cause of why it fails in forced FIPS mode and not in kernel FIPS mode. The answer should be in apache2 code or config files but I'm not familiar with the code.

Hello David,

Could please provide SR.

(In reply to Swayammitra Tripathy from comment #35)
> Hello David,
> 
> Could please provide SR.

There is no fix yet to submit.

Do you have an estimate if it can be submitted before 17.04? This is a blocker issue for the PublicRC.

(In reply to Radoslav Tzvetkov from comment #37)
> Do you have an estimate if it can be submitted before 17.04? This is a
> blocker issue for the PublicRC.

Unfortunately I have no estimate. Let me contact Otto, he may know a bit more about FIPS than me.

Also, why is it a blocker? I mean, it works in FIPS mode as far as it's set system-wide via kernel parameter (it's not crashing anymore, as it was, and a MR was already sent that fixed that).

What's missing is using envvars to force OpenSSL to be in FIPS so Apache picks it up, but in this case apache cannot find any valid cipher. I still think it's a config issue, but I didn't find the cause.

Hello David, this bug is a blocker because it blocks QE tests, and not because of the FIPS. FIPS should be ready with the Q1.

Jan, as FIPS is irrelevant for GM, can we remove the blocker?

SR mentioned this was integrated into 15 SP6 PublicRC 04 2024

Summary as of today:

* The original segfault was long fixed and set to all affected codestreams.
* There is still no fix for "works in kernel FIPS mode but using envvars/config then OpenSSL doesn't provide ciphers".

Unfortunately, I don't know how to debug this issue neither I found the root cause.