Bug 156931

Summary: dumb behavior of "wget" on server errors
Product: [openSUSE] SUSE LINUX 10.0 Reporter: Ulrich Windl <Ulrich.Windl>
Component: NetworkAssignee: Reinhard Max <max>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Minor    
Priority: P5 - None CC: suse-beta
Version: Final   
Target Milestone: ---   
Hardware: Other   
OS: Other   
Whiteboard:
Found By: Other Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Ulrich Windl 2006-03-10 08:04:33 UTC 
When detecting transfer problems, wget does not retry the existing download, but starts a new one, wasting bandwidth. Example:
> wget ftp://ftp.suse.com/pub/suse/i386/update/10.0/rpm/i586/kernel-smp-nongpl-2.6.13-15.8.i586.rpm
--07:49:59--  ftp://ftp.suse.com/pub/suse/i386/update/10.0/rpm/i586/kernel-smp-nongpl-2.6.13-15.8.i586.rpm
           => `kernel-smp-nongpl-2.6.13-15.8.i586.rpm'
Auflösen des Hostnamen »ftp.suse.com«.... 195.135.221.132
Verbindungsaufbau zu ftp.suse.com|195.135.221.132|:21... verbunden.
Anmelden als anonymous ... Angemeldet!
==> SYST ... fertig.    ==> PWD ... fertig.
==> TYPE I ... fertig.  ==> CWD /pub/suse/i386/update/10.0/rpm/i586 ... fertig.==> PASV ... fertig.    ==> RETR kernel-smp-nongpl-2.6.13-15.8.i586.rpm ... fertig.
Länge: 5.647.636 (5.4M) (unmaßgeblich)

95% [=================================>  ] 5.416.968     --.--K/s    ETA 01:11

08:18:05 (3.15 KB/s) - Daten-Verbindung: Die Wartezeit für die Verbindung ist abgelaufen; Daten-Übertragung abgebrochen.
Erneuter Versuch.

--08:18:06--  ftp://ftp.suse.com/pub/suse/i386/update/10.0/rpm/i586/kernel-smp-nongpl-2.6.13-15.8.i586.rpm
  (Versuch: 2) => `kernel-smp-nongpl-2.6.13-15.8.i586.rpm'
==> CWD nicht erforderlich.
==> SIZE kernel-smp-nongpl-2.6.13-15.8.i586.rpm ...
Fehler in der Antwort des Servers; schließe Kontroll-Verbindung.
Erneuter Versuch.

--08:18:08--  ftp://ftp.suse.com/pub/suse/i386/update/10.0/rpm/i586/kernel-smp-nongpl-2.6.13-15.8.i586.rpm
  (Versuch: 3) => `kernel-smp-nongpl-2.6.13-15.8.i586.rpm'
Verbindungsaufbau zu ftp.suse.com|195.135.221.132|:21... verbunden.
Anmelden als anonymous ... Angemeldet!
==> SYST ... fertig.    ==> PWD ... fertig.
==> TYPE I ... fertig.  ==> CWD /pub/suse/i386/update/10.0/rpm/i586 ... fertig.==> PASV ... fertig.    ==> RETR kernel-smp-nongpl-2.6.13-15.8.i586.rpm ... fertig.
»kernel-smp-nongpl-2.6.13-15.8.i586.rpm« ist plötzlich entstanden.
Erneuter Versuch.

--08:18:15--  ftp://ftp.suse.com/pub/suse/i386/update/10.0/rpm/i586/kernel-smp-nongpl-2.6.13-15.8.i586.rpm
  (Versuch: 4) => `kernel-smp-nongpl-2.6.13-15.8.i586.rpm.1'
Verbindungsaufbau zu ftp.suse.com|195.135.221.132|:21... verbunden.
Anmelden als anonymous ... Angemeldet!
==> SYST ... fertig.    ==> PWD ... fertig.
==> TYPE I ... fertig.  ==> CWD /pub/suse/i386/update/10.0/rpm/i586 ... fertig.==> PASV ... fertig.    ==> RETR kernel-smp-nongpl-2.6.13-15.8.i586.rpm ... fertig.
Länge: 5.647.636 (5.4M) (unmaßgeblich)

83% [============================>       ] 4.692.968     --.--K/s    ETA 05:37

08:46:03 (2.76 KB/s) - Daten-Verbindung: Die Wartezeit für die Verbindung ist abgelaufen; Daten-Übertragung abgebrochen.
Erneuter Versuch.

--08:46:07--  ftp://ftp.suse.com/pub/suse/i386/update/10.0/rpm/i586/kernel-smp-nongpl-2.6.13-15.8.i586.rpm
  (Versuch: 5) => `kernel-smp-nongpl-2.6.13-15.8.i586.rpm.1'
==> CWD nicht erforderlich.
==> SIZE kernel-smp-nongpl-2.6.13-15.8.i586.rpm ...
Fehler in der Antwort des Servers; schließe Kontroll-Verbindung.
Erneuter Versuch.

--08:46:12--  ftp://ftp.suse.com/pub/suse/i386/update/10.0/rpm/i586/kernel-smp-nongpl-2.6.13-15.8.i586.rpm
  (Versuch: 6) => `kernel-smp-nongpl-2.6.13-15.8.i586.rpm.1'
Verbindungsaufbau zu ftp.suse.com|195.135.221.132|:21... verbunden.
Anmelden als anonymous ... Angemeldet!
==> SYST ... fertig.    ==> PWD ... fertig.
==> TYPE I ... fertig.  ==> CWD /pub/suse/i386/update/10.0/rpm/i586 ... fertig.==> PASV ... fertig.    ==> RETR kernel-smp-nongpl-2.6.13-15.8.i586.rpm ... fertig.
»kernel-smp-nongpl-2.6.13-15.8.i586.rpm.1« ist plötzlich entstanden.
Erneuter Versuch.

--08:46:21--  ftp://ftp.suse.com/pub/suse/i386/update/10.0/rpm/i586/kernel-smp-nongpl-2.6.13-15.8.i586.rpm
  (Versuch: 7) => `kernel-smp-nongpl-2.6.13-15.8.i586.rpm.2'
Verbindungsaufbau zu ftp.suse.com|195.135.221.132|:21... verbunden.
Anmelden als anonymous ... Angemeldet!
==> SYST ... fertig.    ==> PWD ... fertig.
==> TYPE I ... fertig.  ==> CWD /pub/suse/i386/update/10.0/rpm/i586 ... fertig.==> PASV ... fertig.    ==> RETR kernel-smp-nongpl-2.6.13-15.8.i586.rpm ... fertig.
Länge: 5.647.636 (5.4M) (unmaßgeblich)

67% [=======================>            ] 3.808.240      2.64K/s    ETA 05:01
^C

Then I used "wget -c ftp://ftp.suse.com/pub/suse/i386/update/10.0/rpm/i586/kernel-smp-nongpl-2.6.13-15.8.i586.rpm" with the following result:
--08:57:33--  ftp://ftp.suse.com/pub/suse/i386/update/10.0/rpm/i586/kernel-smp-nongpl-2.6.13-15.8.i586.rpm
           => `kernel-smp-nongpl-2.6.13-15.8.i586.rpm'
Auflösen des Hostnamen »ftp.suse.com«.... 195.135.221.132
Verbindungsaufbau zu ftp.suse.com|195.135.221.132|:21... verbunden.
Anmelden als anonymous ... Angemeldet!
==> SYST ... fertig.    ==> PWD ... fertig.
==> TYPE I ... fertig.  ==> CWD /pub/suse/i386/update/10.0/rpm/i586 ... fertig.==> SIZE kernel-smp-nongpl-2.6.13-15.8.i586.rpm ... fertig.
==> PASV ... fertig.    ==> REST 5416968 ... fertig.
==> RETR kernel-smp-nongpl-2.6.13-15.8.i586.rpm ... fertig.
Länge: 5.647.636 (5.4M), 230.668 (225K) sind noch übrig

100%[++++++++++++++++++++++++++++++++++=>] 5.647.636      5.96K/s    ETA 00:00

08:58:40 (8.02 KB/s) - »kernel-smp-nongpl-2.6.13-15.8.i586.rpm« gespeichert [5647636]


Why isn't wget trying a "reget" after errors?
Comment 1 Michael Gross 2006-03-13 15:24:07 UTC 
Following the man page (even for a way older version) regetting should be supported, however the server must offer this feature. Reassigning.
Comment 2 Hendrik Vogelsang 2006-03-13 18:44:53 UTC 
Reinhard is that the same problem as Bug #144410

can you please check?
Comment 3 Reinhard Max 2006-03-27 07:54:26 UTC 
Might be.

Ulrich, can you please try the patch from http://bugzilla.novell.com/show_bug.cgi?id=144410#c7 or the wget package from the 10.1 betas, to see if fixes your problem as well?
Comment 4 Ulrich Windl 2006-03-27 10:58:44 UTC 
The problem with "trying" is this: How do I find a server that guarantees to time out or abort the connection while I try. The "looking at the source" approach seems better here.
Comment 5 Reinhard Max 2006-03-27 12:03:58 UTC 
To test the patch when I created it, I set up a vsftpd on my test machine, started downloading a large file, and then killed the worker process for the connection. To enforce a timeout situation, you could just just send a SIGSTOP to the worker process and then wait until wget times out.
Comment 6 Ulrich Windl 2006-03-27 12:11:53 UTC 
OK, the package from 10.1 doesn't work with 10.0 (libssl.so.0.9.8 not found). Is there a build for 10.0 already, or just the patch?
Comment 7 Reinhard Max 2006-03-29 14:54:17 UTC 
Sorry for the delay. I just uploaded a patched RPM for 10.0 to
ftp://ftp.suse.com/pub/people/max/
Comment 8 Ulrich Windl 2006-03-30 08:11:31 UTC 
wget from wget-1.10.2-10 (2006-03-29) looks good so far.
Comment 9 Reinhard Max 2006-04-13 08:03:43 UTC 
So, can I close this one?
Comment 10 Ulrich Windl 2006-04-13 09:48:29 UTC 
(In reply to comment #9: "So, can I close this one?")
Yes, I found no new problems with the new version, and the old problem looks fixed.
Comment 11 Reinhard Max 2006-04-13 10:35:48 UTC 
thanks.