Bug 1174885 - download.opensuse.org is unreliable for multiple hours in AU/NZ
Summary: download.opensuse.org is unreliable for multiple hours in AU/NZ
Status: NEW
Alias: None
Product: openSUSE.org
Classification: openSUSE
Component: BuildService (show other bugs)
Version: unspecified
Hardware: Other Other
: P5 - None : Major (vote)
Target Milestone: ---
Assignee: Bernhard Wiedemann
QA Contact: Adrian Schröter
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-08-05 00:57 UTC by William Brown
Modified: 2021-04-19 23:57 UTC (History)
1 user (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description William Brown 2020-08-05 00:57:39 UTC 
While using zypper with tumbleweed the following errors are commonly recieved:

# zypper -vv ref
Verbosity: 3
Initializing Target
Specified repositories:
Skipping disabled repository 'openSUSE-Tumbleweed-Debug'
Checking whether to refresh metadata for openSUSE-Tumbleweed-Non-Oss
Retrieving repository 'openSUSE-Tumbleweed-Non-Oss' metadata [.error]
Checking whether to refresh metadata for openSUSE-Tumbleweed-Oss
Retrieving repository 'openSUSE-Tumbleweed-Oss' metadata [.Repository 'openSUSE-Tumbleweed-Non-Oss' is invalid.
[repo-non-oss|http://download.opensuse.org/tumbleweed/repo/non-oss/] Valid metadata not found at specified URL
History:
 - [|] Error trying to read from 'http://download.opensuse.org/tumbleweed/repo/non-oss/'
 - Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/non-oss/content':
Error code: Connection failed
Error message: Could not resolve host: download.opensuse.org


Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'openSUSE-Tumbleweed-Non-Oss' because of the above error.
error]
Repository 'openSUSE-Tumbleweed-Oss' is invalid.
[repo-oss|http://download.opensuse.org/tumbleweed/repo/oss/] Valid metadata not found at specified URL
History:
 - [|] Error trying to read from 'http://download.opensuse.org/tumbleweed/repo/oss/'
 - Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/content':
Error code: Connection failed
Error message: Could not resolve host: download.opensuse.org


Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'openSUSE-Tumbleweed-Oss' because of the above error.
Skipping disabled repository 'openSUSE-Tumbleweed-Source'
Checking whether to refresh metadata for openSUSE-Tumbleweed-Update
Retrieving repository 'openSUSE-Tumbleweed-Update' metadata [.error]
Repository 'openSUSE-Tumbleweed-Update' is invalid.
[repo-update|http://download.opensuse.org/update/tumbleweed/] Valid metadata not found at specified URL
History:
 - [|] Error trying to read from 'http://download.opensuse.org/update/tumbleweed/'
 - Download (curl) error for 'http://download.opensuse.org/update/tumbleweed/content':
Error code: Connection failed
Error message: Could not resolve host: download.opensuse.org


Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'openSUSE-Tumbleweed-Update' because of the above error.
Could not refresh the repositories because of errors.




This happens every few days for multiple hours at a time. This would indicate a problem with the mirrors in this region.

Here is the metalink output:

[william@pyrite 10:53] ~ > curl http://download.opensuse.org/tumbleweed/repo/oss/CHECKSUMS.metalink
<?xml version="1.0" encoding="UTF-8"?>
<metalink version="3.0" xmlns="http://www.metalinker.org/"
  origin="http://download.opensuse.org/tumbleweed/repo/oss/CHECKSUMS.metalink"
  generator="MirrorBrain 2.19.0 (see http://mirrorbrain.org/)"
  type="dynamic"  pubdate="Wed, 05 Aug 2020 00:53:59 GMT"  refreshdate="Wed, 05 Aug 2020 00:53:59 GMT">

  <publisher>
    <name>openSUSE</name>
    <url>http://download.opensuse.org</url>
  </publisher>

  <files>
  <file name="CHECKSUMS">
    <size>37050</size>

    <!-- <mtime>1596544245</mtime> -->

    <resources>



    <!-- Found 83 mirrors: 0 in the same network prefix, 0 in the same autonomous system,
         0 handling this country, 1 in the same region, 74 elsewhere -->

    <!-- Mirrors in the same network (2001:44b8:2100::/40): -->

    <!-- Mirrors in the same AS (4739): -->

    <!-- Mirrors which handle this country (AU): -->

    <!-- Mirrors in the same continent (OC): -->
    <url type="http" location="nz" preference="100">http://opensuse.mirrors.theom.nz/tumbleweed/repo/oss/CHECKSUMS</url>

    <!-- Mirrors in the rest of the world: -->
    <url type="http" location="id" preference="99">http://mirror.telkomuniversity.ac.id/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="jp" preference="98">http://ftp.riken.jp/Linux/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="jp" preference="97">http://ftp.kddilabs.jp/Linux/packages/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="kr" preference="96">http://ftp.kaist.ac.kr/pub/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="cn" preference="95">http://mirrors.ustc.edu.cn/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="cn" preference="94">http://mirrors.tuna.tsinghua.edu.cn/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="cn" preference="93">http://mirror.bjtu.edu.cn/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="cn" preference="92">http://mirror.lzu.edu.cn/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="ru" preference="91">http://mirror.tspu.ru/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="uz" preference="90">http://mirror.dc.uz/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="ir" preference="89">http://repo.iut.ac.ir/repo/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="ke" preference="88">http://opensuse.mirror.liquidtelecom.com/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="am" preference="87">http://opensuse.ucom.am/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="cy" preference="86">http://mirror.library.ucy.ac.cy/linux/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="ru" preference="85">http://mirror.linux-ia64.org/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="ru" preference="84">http://mirror.yandex.ru/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="gr" preference="83">http://ftp.cc.uoc.gr/mirrors/linux/opensuse/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="ro" preference="82">http://mirrors.leadhosts.com/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="bg" preference="81">http://mirrors.netix.net/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="bg" preference="80">http://mirror.host.ag/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="ro" preference="79">http://mirrors.nxthost.com/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="hu" preference="78">http://quantum-mirror.hu/mirrors/pub/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="sk" preference="77">http://tux.rainside.sk/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="pl" preference="76">http://ftp.icm.edu.pl/pub/Linux/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="hu" preference="75">http://ftp.novell.hu/pub/mirrors/ftp.opensuse.org/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="lv" preference="74">http://opensuse.koyanet.lv/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="at" preference="73">http://mirror.easyname.at/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="cz" preference="72">http://mirror.karneval.cz/pub/linux/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="fi" preference="71">http://ftp.funet.fi/pub/mirrors/ftp.opensuse.com/pub/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="at" preference="70">http://suse.lagis.at/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="se" preference="69">http://ftp.acc.umu.se/mirror/opensuse.org/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="de" preference="68">http://ftp.tu-chemnitz.de/pub/linux/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="se" preference="67">http://ftp.acc.umu.se/mirror/opensuse.org/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="de" preference="66">http://widehat.opensuse.org/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="de" preference="65">http://suse.uni-leipzig.de/pub/ftp.opensuse.org/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="de" preference="64">http://ftp.uni-bayreuth.de/linux/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="de" preference="63">http://ftp.uni-erlangen.de/pub/mirrors/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="se" preference="62">http://ftp.lysator.liu.se/pub/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="de" preference="61">http://ftp.rz.uni-wuerzburg.de/linux/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="de" preference="60">http://mirror1.hs-esslingen.de/pub/Mirrors/ftp.opensuse.org/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="ch" preference="59">http://mirror.hostsuisse.com/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="https" location="de" preference="58">https://ftp.gwdg.de/pub/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="dk" preference="57">http://mirrors.dotsrc.org/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="de" preference="56">http://opensuse.schlundtech.de/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="de" preference="55">http://mirror.de.leaseweb.net/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="de" preference="54">http://mirror.linux-schulserver.de/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="fr" preference="53">http://fr2.rpmfind.net/linux/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="de" preference="52">http://ftp.halifax.rwth-aachen.de/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="no" preference="51">http://ftp.uninett.no/pub/linux/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="nl" preference="50">http://ftp1.nluug.nl/os/Linux/distr/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="nl" preference="49">http://ftp2.nluug.nl/os/Linux/distr/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="nl" preference="48">http://opensuse.mirror.liteserver.nl/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="nl" preference="47">http://mirror.nl.leaseweb.net/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="fr" preference="46">http://mirrors.standaloneinstaller.com/opensuse-stable/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="gb" preference="45">http://mirror.ox.ac.uk/sites/ftp.opensuse.org/pub/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="gb" preference="44">http://www.mirrorservice.org/sites/download.opensuse.org/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="br" preference="43">http://opensuse.c3sl.ufpr.br/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="uy" preference="42">http://espejito.fder.edu.uy/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="jp" preference="41">http://nrt.edge.kernel.org/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="us" preference="40">http://sjc.edge.kernel.org/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="us" preference="39">http://mirror.siena.edu/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="us" preference="38">http://ewr.edge.kernel.org/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="us" preference="37">http://mirror.clarkson.edu/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="us" preference="36">http://mirror.umd.edu/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="us" preference="35">http://mirror.us.leaseweb.net/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="us" preference="34">http://plug-mirror.rcac.purdue.edu/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="us" preference="33">http://mirror.dal10.us.leaseweb.net/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="nl" preference="32">http://ams.edge.kernel.org/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="https" location="us" preference="31">https://provo-mirror.opensuse.org/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="https" location="us" preference="30">https://provo-mirror.opensuse.org/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="us" preference="29">http://mirror.sjc02.svwh.net/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="us" preference="28">http://mirror.sfo12.us.leaseweb.net/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="us" preference="27">http://sfo-korg-mirror.kernel.org/opensuse/tumbleweed/repo/oss/CHECKSUMS</url>
    <url type="http" location="us" preference="26">http://suse.mobile-central.org/tumbleweed/repo/oss/CHECKSUMS</url>
    </resources>
    </file>
  </files>
</metalink>


It appears at least that http://mirror.telkomuniversity.ac.id has not been updated in multiple days, and http://ftp-srv2.kddilabs.jp/ is more than 1 month out of date.

This happens frequently, to the point that every few days I'm unable to update or install packages on opensuse tumbleweed from Australia due to the mirrors being in this state.

Thanks,
Comment 1 Bernhard Wiedemann 2021-03-08 01:27:32 UTC 
> Error message: Could not resolve host: download.opensuse.org

could also indicate problems with your DNS resolution.
Is this still happening?

ns1.opensuse.org.
to
ns4.opensuse.org.
all have IPv6+4 addrs these days.
Comment 2 William Brown 2021-03-08 02:10:06 UTC 
It's not an issue with DNS. DNS works for this duration.

Yes, the download services still have frequent down times in Australian afternoons.
Comment 3 Bernhard Wiedemann 2021-04-19 19:37:59 UTC 
Around 2021-04-09 we fixed some issues with download.o.o

If it is still bad, you can try to replace in /etc/zypp/repos.d/*repo
http://download.opensuse.org/ with
http://mirrorcache.opensuse.org/download/

MirrorCache is a new experimental redirection service.


On the opensuse-factory ML, there was also a thread
"DNS reliability issues on Tumbleweed"
where multiple users confirmed the behaviour that the first DNS request often fails

https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/WSUUYT2LT5PEJWY4HOIOXYAKRB4WWQBU/

So it may be a more general problem with Tumbleweed DNS
and not about our infra.
Comment 4 William Brown 2021-04-19 23:57:42 UTC 
It seems the issue isn't dns for me, but the fact that the metadata and the rpm content on the mirrors is out of step, so I'm probably stepping on the period of time when an rsync is happening in some cases. But in other cases the mirrors are just really flakey :(