Bug 690734

Summary: Internal server error on very long URL
Product: [openSUSE] openSUSE.org Reporter: Pascal Bleser <pascal.bleser>
Component: BuildServiceAssignee: Roman Drahtmueller <draht>
Status: RESOLVED FIXED QA Contact: Adrian Schröter <adrian.schroeter>
Severity: Critical    
Priority: P5 - None CC: adrian.schroeter, andrej.semen, davejplater, detlef, jdd, matthias.sweertvaegher, meissner, pascal.bleser, pgajdos
Version: unspecified   
Target Milestone: ---   
Hardware: Other   
OS: Other   
Whiteboard: maint:released:11.3:41814 maint:released:11.4:41814 maint:released:sle11-sp1:42964 maint:released:sle10-sp4:42965 maint:released:sle10-sp3:42962 maint:released:sle10-sp2:43259
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: shell (curl) script to reproduce the issue
excerpt of src_server.log on the Packman OBS instance
take LimitRequestFieldsize config option into account when parsing headers from backend.

Description Pascal Bleser 2011-04-28 22:39:09 UTC
Created attachment 427187 [details]
shell (curl) script to reproduce the issue

User-Agent:       Mozilla/5.0 (X11; Linux x86_64; rv:2.0.0) Gecko/20100101 Firefox/4.0

Our OBS instance at Packman gets broken builds for no apparent reasons on a regular basis. After tracking down the issue, it appears that our OBS instance gets a 400 HTTP error code back from build.opensuse.org (or, rather, from public.api.opensuse.org) on very long GET query URLs.

Reproducible: Sometimes

Steps to Reproduce:
1. run the following shell script (uses curl):
http://linux01.gwdg.de/~pbleser/files/obs/issues/remote_error/reproduce_with_curl.sh
(also in attachment)
Actual Results:  
HTTP/1.1 404 Not Found
Date: Thu, 28 Apr 2011 22:36:46 GMT
Status: 500 Internal Server Error
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.6
X-Opensuse-Errorcode: not_found
Cache-Control: no-cache
Status: 404
Vary: Accept-Encoding
Content-Type: application/xml; charset=utf-8
Transfer-Encoding: chunked

<?xml version="1.0" encoding="UTF-8"?>
<status code="not_found">
  <summary>No route matches "/error/HTTP_INTERNAL_SERVER_ERROR.html.var" with {:method=&gt;:get}</summary>
  <details></details>
</status>
Comment 1 Pascal Bleser 2011-04-28 22:40:11 UTC
Created attachment 427188 [details]
excerpt of src_server.log on the Packman OBS instance
Comment 2 Pascal Bleser 2011-04-28 22:46:30 UTC
While the HTTP spec and real world tests with Apache show that there is no limitation on the maximum length of an HTTP GET query string, the one that triggers the error is 5k long.

Maybe a bug in Passenger ? (unlikely to be a bug in Apache: real-world tests show that Apache replies with a 413 (Entity too large) and not crash:
http://www.boutell.com/newfaq/misc/urllength.html
Comment 3 Adrian Schröter 2011-05-02 14:38:14 UTC
it is a problem between apache and mod_xforward. disabling xforward in options.yml should help (but may bring down your server in the long run).

Problem is entire clear yet, because mod_xforward gets already a stripped url from apache.
Comment 4 Pascal Bleser 2011-05-02 21:13:19 UTC
Ummm, there's no xforward setting anywhere and, as far as I can see, the issue is on build.opensuse.org, not on our side of things.
Comment 5 Detlef Reichelt 2011-05-08 20:13:47 UTC
Push!

The PackMan repo is broken now...... :(
Comment 6 Forgotten User zOWss6Gs9u 2011-05-09 22:11:19 UTC
*** Bug 692487 has been marked as a duplicate of this bug. ***
Comment 7 Adrian Schröter 2011-05-11 12:28:29 UTC
fixed now, api.opensuse.org should work for you again.

It was a bug in apache dieing on large headers even when configured large enough, patch will be sent upstream.
Comment 8 Adrian Schröter 2011-05-11 12:34:57 UTC
reopen to check if we want this in SLE 11 SP 2.

Maintenance team, do we want this fix ?
Comment 9 Harald Mueller-Ney 2011-05-11 13:54:27 UTC
Please attach the patch and assign to the right package maintainer for his input first
Comment 10 Adrian Schröter 2011-05-11 14:05:22 UTC
Created attachment 429096 [details]
take LimitRequestFieldsize config option into account when parsing headers from backend.
Comment 11 Roman Drahtmueller 2011-06-28 03:03:41 UTC
package submitted for SLE10-SP4 and SLE11-SP1,2, accompanied by libapr1 via https://bugzilla.novell.com/show_bug.cgi?id=693778 and libapr-util1 for SLE11-SP1, not for SLE10-SP4, via  https://bugzilla.novell.com/show_bug.cgi?id=653510 and https://bugzilla.novell.com/show_bug.cgi?id=693778.

openSUSE packages will follow shortly after test.

reassigned to maint-coord@.

Thank you, Pascal and Adrian!
Comment 13 Bernhard Wiedemann 2011-07-26 14:00:58 UTC
This is an autogenerated message for OBS integration:
This bug (690734) was mentioned in
https://build.opensuse.org/request/show/77089 11.3 / apache2
https://build.opensuse.org/request/show/77092 11.4 / apache2
Comment 16 Swamp Workflow Management 2011-07-28 12:17:08 UTC
Update released for: apache2, apache2-debuginfo, apache2-debugsource, apache2-devel, apache2-doc, apache2-event, apache2-event-debuginfo, apache2-example-certificates, apache2-example-pages, apache2-itk, apache2-itk-debuginfo, apache2-prefork, apache2-prefork-debuginfo, apache2-utils, apache2-utils-debuginfo, apache2-worker, apache2-worker-debuginfo, libapr-util1, libapr-util1-dbd-mysql, libapr-util1-dbd-mysql-debuginfo, libapr-util1-dbd-pgsql, libapr-util1-dbd-pgsql-debuginfo, libapr-util1-dbd-sqlite3, libapr-util1-dbd-sqlite3-debuginfo, libapr-util1-debuginfo, libapr-util1-debugsource, libapr-util1-devel, libapr1, libapr1-debuginfo, libapr1-debugsource, libapr1-devel
Products:
openSUSE 11.3 (debug, i586, x86_64)
openSUSE 11.4 (debug, i586, x86_64)
Comment 17 Bernhard Wiedemann 2011-08-31 19:00:20 UTC
This is an autogenerated message for OBS integration:
This bug (690734) was mentioned in
https://build.opensuse.org/request/show/80441 11.4 / apache2
Comment 18 Swamp Workflow Management 2011-09-06 01:32:55 UTC
Update released for: apache2, apache2-debuginfo, apache2-debugsource, apache2-devel, apache2-doc, apache2-event, apache2-example-pages, apache2-prefork, apache2-utils, apache2-worker
Products:
SLE-DEBUGINFO 11-SP1 (i386, ia64, ppc64, s390x, x86_64)
SLE-SDK 11-SP1 (i386, ia64, ppc64, s390x, x86_64)
SLE-SERVER 11-SP1 (i386, ia64, ppc64, s390x, x86_64)
SLE-SERVER 11-SP1-TERADATA (x86_64)
SLES4VMWARE 11-SP1 (i386, x86_64)
Comment 19 Swamp Workflow Management 2011-09-06 05:05:33 UTC
Update released for: apache2, apache2-debuginfo, apache2-devel, apache2-doc, apache2-event, apache2-example-pages, apache2-prefork, apache2-worker
Products:
SLE-DEBUGINFO 10-SP4 (i386, ia64, ppc, s390x, x86_64)
SLE-SDK 10-SP4 (i386, ia64, ppc, s390x, x86_64)
SLE-SERVER 10-SP4 (i386, ia64, ppc, s390x, x86_64)
Comment 20 Swamp Workflow Management 2011-09-06 12:11:21 UTC
Update released for: apache2, apache2-debuginfo, apache2-devel, apache2-doc, apache2-event, apache2-example-pages, apache2-prefork, apache2-worker
Products:
SLE-DEBUGINFO 10-SP3 (i386, ia64, ppc, s390x, x86_64)
SLE-SAP-APL 10-SP3 (x86_64)
SLE-SDK 10-SP3 (i386, ia64, ppc, s390x, x86_64)
SLE-SERVER 10-SP3 (i386, ia64, ppc, s390x, x86_64)
SLE-SERVER 10-SP3-TERADATA (x86_64)
Comment 21 Marcus Meissner 2011-09-06 12:55:00 UTC
released
Comment 22 Andrej Semen 2011-10-17 10:31:39 UTC
did run test case on SLE-10-SP1 ltss

looks like the "internel Sever Error" did not happen on SLE-10-SP1 ltss

it results with "1.1 200 OK"

/suse/rd-qa/testfiles/bug-690734_reproduce_with_curl.sh

dax:/tmp # /suse/rd-qa/testfiles/bug-690734_reproduce_with_curl.sh | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  4 26067    4  1152    0     0   9655      0  0:00:02 --:--:--  0:00:02  9655HTTP/1.1 200 OK
Date: Mon, 17 Oct 2011 10:26:25 GMT
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.8
X-Opensuse-APIVersion: 2.3.0.git201110060711
X-Runtime: 7
Cache-Control: no-cache
Content-Type: text/xml
Cache-Control: no-cache
Content-Length: 26067
Vary: Accept-Encoding
 31 26067   31  8392    0     0  69594      0 --:--:-- --:--:-- --:--:-- 7070k
curl: (23) Failed writing body
Comment 23 Swamp Workflow Management 2011-11-04 04:40:48 UTC
Update released for: apache2, apache2-devel, apache2-doc, apache2-event, apache2-example-pages, apache2-prefork, apache2-worker
Products:
SLE-SERVER 10-SP2-LTSS (i386, s390x, x86_64)
Comment 24 Petr Gajdos 2015-06-25 08:58:06 UTC
Adrian,

it seems that httpd-2.2.x-bnc690734.patch is under serious danger in factory as you as the author seem to not try to upstream it (or am I missing Apache's bug #?). This means it will not be in sle13, probably.
Comment 25 Adrian Schröter 2015-06-25 09:06:20 UTC
Roman, as he was the maintainer of apache said that he will take care of getting it upstream. Maybe Kristyna Streitova <kstreitova@suse.com> can take care of that now?

What is the reason that you say that the patch is under danger? It was IMHO nice from me to provide the patch at all and not to declare openSUSE not to be supported anymore by the OBS due to buggy apache:/
Comment 26 Petr Gajdos 2015-06-25 09:12:17 UTC
(In reply to Adrian Schröter from comment #25)
> What is the reason that you say that the patch is under danger? It was IMHO

I have noticed that someone has commented out it in oS:F during some update.

> nice from me to provide the patch at all and not to declare openSUSE not to
> be supported anymore by the OBS due to buggy apache:/

Every author of the patch is responsible to take it upstream. He knows much more about the patch than we, ordinary packagers.

Take it as friendly reminder ;).

Thanks for understanding