Bugzilla – Bug 1226217
Regression of security fix: Apache ignores headers sent by CGI scripts
Last modified: 2024-07-17 12:41:08 UTC
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0 Build Identifier: A CGI script sents Status: 200 OK\r\n Content-Length: 422264\r\n Content-type: application/octet-stream\r\n Connection: close\r\n\rn ...data the Apache sends HTTP/1.1 200 OK\r\n Date: Wed, 12 Jun 2024 11:24:45 GMT\r\n Server: Apache\r\n Connection: close\r\n Transfer-Encoding: chunked\r\n Content-Type: application/octet-stream\r\n\r\n This is missing the content length. This again breaks the client software relying on the content length. Same issue is with Transfer-Encoding. We have a script using transfer-encoding chunked and apache ignores this and thus double encodes the output. Also apache now sends everything chunked, also stuff which was previously send without chunks. This happens with apache2-prefork-2.4.51-150400.6.17.1. A downgrade to apache2-prefork-2.4.51-150400.6.14.1 fixes the issue. I thing a security updates is broken. This caused already several hundred € costs on our side until we found the root cause. Please fix this as soon as possible! Reproducible: Always Steps to Reproduce: 1. Start version apache2-prefork-2.4.51-150400.6.17.1 2. Use a CGI script which sets header lines 3. Header lines are not used by the server and also not sent over the air.
* https://build.opensuse.org/package/rdiff/SUSE:SLE-15-SP5:Update/apache2?linkrev=base&rev=5 * apache2-CVE-2024-24795.patch From the description it looks like that could be the root cause.
Probably this: https://bz.apache.org/bugzilla/show_bug.cgi?id=68905
https://bz.apache.org/bugzilla/show_bug.cgi?id=68907(In reply to Petr Gajdos from comment #2) > Probably this: > https://bz.apache.org/bugzilla/show_bug.cgi?id=68905 It should have been: https://security-tracker.debian.org/tracker/CVE-2024-24795
@Petr: Is far as I can see that's not related. The issue here is that apache in the buggy patched version ignores the lines which the CGI script outputs (i.e. sends to apache via stdout) and instead does what it wants on the output side. Contrary to what my initial report says after looking at the patch which seems to be the cause probably only headers content-length and transfer-encoding are affected. As said a downgrade to the previous version 14.1 fixes the issue, so the reason can only be in the applied SUSE security patches.
(In reply to Dirk Stoecker from comment #4) > @Petr: Is far as I can see that's not related. The issue here is that apache > in the buggy patched version ignores the lines which the CGI script outputs > (i.e. sends to apache via stdout) and instead does what it wants on the > output side. > > Contrary to what my initial report says after looking at the patch which > seems to be the cause probably only headers content-length and > transfer-encoding are affected. > > As said a downgrade to the previous version 14.1 fixes the issue, so the > reason can only be in the applied SUSE security patches. Hello Dirk, I'm working on this one right now. My plan is the following: 1. Creating a simple reproducing environment. 2. Test against latest apache version in TW (to validate if this happens with latest published version of Apache). 3. Bisect the changes on the aforementioned patches (there are 3 in total). 4. Debug the issue and evaluate the best course of action. Let me work on it and I'll come back to you asap.
(In reply to Dirk Stoecker from comment #0) > User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:126.0) Gecko/20100101 > Firefox/126.0 > Build Identifier: > > A CGI script sents > Status: 200 OK\r\n > Content-Length: 422264\r\n > Content-type: application/octet-stream\r\n > Connection: close\r\n\rn > ...data > > the Apache sends > HTTP/1.1 200 OK\r\n > Date: Wed, 12 Jun 2024 11:24:45 GMT\r\n > Server: Apache\r\n > Connection: close\r\n > Transfer-Encoding: chunked\r\n > Content-Type: application/octet-stream\r\n\r\n > > This is missing the content length. This again breaks the client software > relying on the content length. > Dirk, to speed up things while I setup everything, can you send us the same information (a sample for headers in/out of the http requests) using the downgraded version that works for you? Thank you.
Here a short test script showing both issues: #!/usr/bin/perl print "Status: 200 OK\r\n"; print "Transfer-Encoding: chunked\r\n"; print "Content-Length: 3\r\n"; print "\r\n"; print "3\r\n"; print "---\r\n"; print "0\r\n"; print "\r\n"; Error (curl -vv, stripped private info) > GET /cgi-bin/error.cgi HTTP/1.1 > User-Agent: curl/8.8.0 > Accept: */* > * Request completely sent off < HTTP/1.1 200 OK < Date: Thu, 13 Jun 2024 08:53:37 GMT < Server: Apache < Transfer-Encoding: chunked < Content-Type: application/x-cgi < 3 --- 0 --- in tcpdump the double encoding is visible better: MHTTP/1.1 200 OK Date: Thu, 13 Jun 2024 08:45:23 GMT Server: Apache Transfer-Encoding: chunked Content-Type: application/x-cgi 8 3 --- 0 Ok: > GET /cgi-bin/error.cgi HTTP/1.1 > User-Agent: curl/8.8.0 > Accept: */* > * Request completely sent off < HTTP/1.1 200 OK < Date: Thu, 13 Jun 2024 08:54:09 GMT < Server: Apache < Transfer-Encoding: chunked < Content-Length: 3 < Content-Type: application/x-cgi < * Connection #0 to host a07upload.gnssonline.eu left intact --- It SHOULD send content length header and not do transfer encoding different to the script (i.e. when script has no transfer encoding, the output also shouldn't - not tested in this example).
(In reply to Dirk Stoecker from comment #7) > Here a short test script showing both issues: > > #!/usr/bin/perl > > print "Status: 200 OK\r\n"; > print "Transfer-Encoding: chunked\r\n"; > print "Content-Length: 3\r\n"; > print "\r\n"; > print "3\r\n"; > print "---\r\n"; > print "0\r\n"; > print "\r\n"; Isn't this wrong as per http spec? > 3. If a message is received with both a Transfer-Encoding and a Content-Length header field, the Transfer-Encoding overrides the Content-Length. Such a message might indicate an attempt to perform request smuggling (Section 11.2) or response splitting (Section 11.1) and ought to be handled as an error. An intermediary that chooses to forward the message MUST first remove the received Content-Length field and process the Transfer-Encoding (as described below) prior to forwarding the message downstream. https://www.rfc-editor.org/rfc/rfc9112#name-message-body-length The problem I see is that our patches are missing a patch that was added later to show an error instead of processing the request (https://github.com/apache/httpd/pull/444) Can you try the same but removing "Transfer-Encoding" from the perl CGI?
For completeness the nochunk test: #!/usr/bin/perl print "Status: 200 OK\r\n"; print "Content-Length: 3\r\n"; print "\r\n"; print "---"; Error: < HTTP/1.1 200 OK < Date: Thu, 13 Jun 2024 09:06:58 GMT < Server: Apache < Transfer-Encoding: chunked < Content-Type: application/x-cgi Ok: < HTTP/1.1 200 OK < Date: Thu, 13 Jun 2024 09:07:48 GMT < Server: Apache < Content-Length: 3 < Content-Type: application/x-cgi
(In reply to Dirk Stoecker from comment #9) > For completeness the nochunk test: > > #!/usr/bin/perl > > print "Status: 200 OK\r\n"; > print "Content-Length: 3\r\n"; > print "\r\n"; > print "---"; > > Error: > < HTTP/1.1 200 OK > < Date: Thu, 13 Jun 2024 09:06:58 GMT > < Server: Apache > < Transfer-Encoding: chunked > < Content-Type: application/x-cgi > > Ok: > < HTTP/1.1 200 OK > < Date: Thu, 13 Jun 2024 09:07:48 GMT > < Server: Apache > < Content-Length: 3 > < Content-Type: application/x-cgi Ok, I see. Let me investigate, thank you. I will come back to you asap.
Just for completeness. Is this your current setup or is there anything in between? client <-> apache <-> cgi
> Isn't this wrong as per http spec? It's not wrong, only superfluous (both state the same value). It get's wrong because apache now always enforces chunked encoding ignoring the input. If it would strip the content-length in the transfer encoding coming from the script case because of the information in the link you provided I can live with this. I don't think that combination is used anywhere that we have a receiver relying on the content length and using transfer encoding. Also doesn't make much sense anyway: If I know the length of the transfer at the beginning chunked transfer is wasted bandwidth.
Yes. curl → apache → cgi. The apache running mainly with defaults (I use different systems for ok/error instead of installing/reinstalling for each test :-).
(In reply to Dirk Stoecker from comment #12) > If it would strip the content-length in the transfer encoding coming from > the script case because of the information in the link you provided I can > live with this. I don't think that combination is used anywhere that we have > a receiver relying on the content length and using transfer encoding. Also > doesn't make much sense anyway: If I know the length of the transfer at the > beginning chunked transfer is wasted bandwidth. I fully agree. Let me investigate now the problem has been narrowed down to a very simple case.
I think the behavior of the patched version is the same as in TW/apache2 (2.4.59).
Ok, after some investigation, I confirm this is deliberate change from upstream to fix the CVEs. I found the following upstream bugs: * https://bz.apache.org/bugzilla/show_bug.cgi?id=68973 Explains the change and the issue Dirk is facing. This happens also in latest Apache2 TW version (thanks for testing Petr!). It's not a bug introduced by the patches (actually, this means the patches work as expected) but obviously this is a change in behavior customers should be aware. * https://bz.apache.org/bugzilla/show_bug.cgi?id=68907 This one proposes a workaround (at least for CGI, but no code yet) so this can be deactivated conditionally via config and implemented in a filter somewhere else. This has not been released yet, so we cannot backport it. Now that mistery is "solved", we need to find out the best course of action (or wait a bit until upstream proposes a change/workaround).
Meanwhile Dirk, you can disable this "filtering" manually if you trust your endpoints, by updating your apache configuration: > SetEnvIf Request_URI "\.pl$" ap_trust_cgilike_cl Adjust it to your needs and it should work as you expect.
It seems the upstream reports misses the second fact that now always chunked transfer encoding is sent out and also that in case of chunked transfer from the script that apache now double encodes which is clearly a bug and can in no case be explained as a behaviour change.
Marcus, can we update CVE pages for Apache mentioning so we can document the change in behavior? Something like this should work: "The security update for CVE-2024-24795 may need updating Apache 2 configuration as it changes how Content-Length and Transfer-Encoding headers are used. If a (Fast)CGI script is being used, then the data is not trust by default and Content-Header is being removed by default. If you trust the endpoint and/or the code behind the CGI script, the following change is needed in your configuration (for example, via htaccess) so it behaves as previously. For example, to fix it for PHP: SetEnvIf Request_URI "\.php$" ap_trust_cgilike_cl Please, adjust that configuration as needed. "
(In reply to Dirk Stoecker from comment #18) > It seems the upstream reports misses the second fact that now always chunked > transfer encoding is sent out and also that in case of chunked transfer from > the script that apache now double encodes which is clearly a bug and can in > no case be explained as a behaviour change. Yes, I agree. That's a bug that's fixed upstream in the following commit, which we need to backport to all codestreams as it was not originally part of the CVE fix: https://github.com/apache/httpd/pull/444 Petr, can you handle the backport of this one, please? It should easy. Thanks
(In reply to David Anes from comment #19) > Marcus, can we update CVE pages for Apache mentioning so we can document the > change in behavior? > > Something like this should work: > > "The security update for CVE-2024-24795 may need updating Apache 2 > configuration as it changes how Content-Length and Transfer-Encoding headers > are used. > > If a (Fast)CGI script is being used, then the data is not trust by default > and Content-Header is being removed by default. If you trust the endpoint > and/or the code behind the CGI script, the following change is needed in > your configuration (for example, via htaccess) so it behaves as previously. > For example, to fix it for PHP: > > SetEnvIf Request_URI "\.php$" ap_trust_cgilike_cl > > Please, adjust that configuration as needed. > " TID proposal: === Title: Apache regression ignores headers sent by CGI scripts introduced by CVE-2024-24795 Description: Because of changes to apache2 introduced by security fix CVE-2024-24799 a configuration change may be required as the fix changes the way how Content-Length and Transfer-Encoding header are used. Resolution: If a (Fast)CGI script is being used, then the data is not trusted by default and Content-Headers are being removed by default. If the endpoint is trusted and/or the code behind the CGI script, the following change is needed in the existing configuration (for example, via htaccess) so it behaves as previously. For example, to fix it for PHP: SetEnvIf Request_URI "\.php$" ap_trust_cgilike_cl Please adjust existing configurations as needed. === Comments/enhalncements welcome! Once agreed I will create a TID for it.
SUSE-RU-2024:2229-1: An update that has one fix can now be installed. Category: recommended (important) Bug References: 1226217 Maintenance Incident: [SUSE:Maintenance:34445](https://smelt.suse.de/incident/34445/) Sources used: openSUSE Leap 15.6 (src): apache2-prefork-2.4.58-150600.5.6.1, apache2-worker-2.4.58-150600.5.6.1, apache2-event-2.4.58-150600.5.6.1, apache2-devel-2.4.58-150600.5.6.1, apache2-manual-2.4.58-150600.5.6.1, apache2-utils-2.4.58-150600.5.6.1, apache2-test_event-2.4.58-150600.5.6.1, apache2-test_prefork-2.4.58-150600.5.6.1, apache2-test_worker-2.4.58-150600.5.6.1, apache2-test_main-2.4.58-150600.5.6.1, apache2-test_devel-2.4.58-150600.5.6.1, apache2-2.4.58-150600.5.6.1 Basesystem Module 15-SP6 (src): apache2-prefork-2.4.58-150600.5.6.1, apache2-2.4.58-150600.5.6.1 SUSE Package Hub 15 15-SP6 (src): apache2-event-2.4.58-150600.5.6.1, apache2-2.4.58-150600.5.6.1 Server Applications Module 15-SP6 (src): apache2-utils-2.4.58-150600.5.6.1, apache2-devel-2.4.58-150600.5.6.1, apache2-worker-2.4.58-150600.5.6.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
SUSE-RU-2024:2228-1: An update that has one fix can now be installed. Category: recommended (important) Bug References: 1226217 Maintenance Incident: [SUSE:Maintenance:34447](https://smelt.suse.de/incident/34447/) Sources used: SUSE Linux Enterprise Software Development Kit 12 SP5 (src): apache2-2.4.51-35.44.1, apache2-tls13-2.4.51-35.44.1 SUSE Linux Enterprise High Performance Computing 12 SP5 (src): apache2-2.4.51-35.44.1, apache2-tls13-2.4.51-35.44.1 SUSE Linux Enterprise Server 12 SP5 (src): apache2-2.4.51-35.44.1, apache2-tls13-2.4.51-35.44.1 SUSE Linux Enterprise Server for SAP Applications 12 SP5 (src): apache2-2.4.51-35.44.1, apache2-tls13-2.4.51-35.44.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
SUSE-RU-2024:2227-1: An update that has one fix can now be installed. Category: recommended (important) Bug References: 1226217 Maintenance Incident: [SUSE:Maintenance:34446](https://smelt.suse.de/incident/34446/) Sources used: SUSE Linux Enterprise High Performance Computing 15 SP2 LTSS 15-SP2 (src): apache2-2.4.51-150200.3.65.1 SUSE Linux Enterprise High Performance Computing LTSS 15 SP3 (src): apache2-2.4.51-150200.3.65.1 SUSE Linux Enterprise Server 15 SP2 LTSS 15-SP2 (src): apache2-2.4.51-150200.3.65.1 SUSE Linux Enterprise Server 15 SP3 LTSS 15-SP3 (src): apache2-2.4.51-150200.3.65.1 SUSE Linux Enterprise Server for SAP Applications 15 SP2 (src): apache2-2.4.51-150200.3.65.1 SUSE Linux Enterprise Server for SAP Applications 15 SP3 (src): apache2-2.4.51-150200.3.65.1 SUSE Enterprise Storage 7.1 (src): apache2-2.4.51-150200.3.65.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
SUSE-RU-2024:2226-1: An update that has one fix can now be installed. Category: recommended (important) Bug References: 1226217 Maintenance Incident: [SUSE:Maintenance:34444](https://smelt.suse.de/incident/34444/) Sources used: Server Applications Module 15-SP6 (src): apache2-2.4.51-150400.6.20.1 SUSE Linux Enterprise High Performance Computing ESPOS 15 SP4 (src): apache2-2.4.51-150400.6.20.1 SUSE Linux Enterprise High Performance Computing LTSS 15 SP4 (src): apache2-2.4.51-150400.6.20.1 SUSE Linux Enterprise Desktop 15 SP4 LTSS 15-SP4 (src): apache2-2.4.51-150400.6.20.1 SUSE Linux Enterprise Server 15 SP4 LTSS 15-SP4 (src): apache2-2.4.51-150400.6.20.1 SUSE Linux Enterprise Server for SAP Applications 15 SP4 (src): apache2-2.4.51-150400.6.20.1 SUSE Manager Proxy 4.3 (src): apache2-2.4.51-150400.6.20.1 SUSE Manager Retail Branch Server 4.3 (src): apache2-2.4.51-150400.6.20.1 SUSE Manager Server 4.3 (src): apache2-2.4.51-150400.6.20.1 openSUSE Leap 15.4 (src): apache2-2.4.51-150400.6.20.1 openSUSE Leap 15.5 (src): apache2-2.4.51-150400.6.20.1 Basesystem Module 15-SP5 (src): apache2-2.4.51-150400.6.20.1 SUSE Package Hub 15 15-SP5 (src): apache2-2.4.51-150400.6.20.1 Server Applications Module 15-SP5 (src): apache2-2.4.51-150400.6.20.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
(In reply to Sascha Wessels from comment #22) > (In reply to David Anes from comment #19) > > Marcus, can we update CVE pages for Apache mentioning so we can document the > > change in behavior? > > > > Something like this should work: > > > > "The security update for CVE-2024-24795 may need updating Apache 2 > > configuration as it changes how Content-Length and Transfer-Encoding headers > > are used. > > > > If a (Fast)CGI script is being used, then the data is not trust by default > > and Content-Header is being removed by default. If you trust the endpoint > > and/or the code behind the CGI script, the following change is needed in > > your configuration (for example, via htaccess) so it behaves as previously. > > For example, to fix it for PHP: > > > > SetEnvIf Request_URI "\.php$" ap_trust_cgilike_cl > > > > Please, adjust that configuration as needed. > > " > > TID proposal: > > === > Title: Apache regression ignores headers sent by CGI scripts introduced by > CVE-2024-24795 > > Description: > Because of changes to apache2 introduced by security fix CVE-2024-24799 a > configuration change may be required as the fix changes the way how > Content-Length and Transfer-Encoding header are used. > > Resolution: > If a (Fast)CGI script is being used, then the data is not trusted by default > and Content-Headers are being removed by default. If the endpoint is trusted > and/or the code behind the CGI script, the following change is needed in > the existing configuration (for example, via htaccess) so it behaves as > previously. For example, to fix it for PHP: > > SetEnvIf Request_URI "\.php$" ap_trust_cgilike_cl > > Please adjust existing configurations as needed. > === > > Comments/enhalncements welcome! Once agreed I will create a TID for it. i dropped the ball on it, sorry. But it looks good to me!
All codestreams were fixed and updates are available. Sending back to security for review (just in case). Thanks Petr for handling the updates.
TID released: https://www.suse.com/support/kb/doc/?id=000021486