[gentoo-dev] [RFC] New third party mirrors

Hello,

As part of my portage-janitor scripts (https://github.com/iksaif/portage-janitor)
I created a tool that counts URIs prefixes and sort them.
Here is the result of this script on the gentoo-x86 tree:

$ ./mirrors.py --all --count
1262 http://dev.gentoo.org
483 http://oss.tresys.com
482 http://oss.tresys.com/files
482 http://oss.tresys.com/files/refpolicy
473 http://xorg.freedesktop.org
473 http://xorg.freedesktop.org/releases
473 http://xorg.freedesktop.org/releases/individual
313 http://dev.gentoo.org/~swift
313 http://dev.gentoo.org/~swift/patches
309 http://dev.gentoo.org/~swift/patches/selinux-base-policy
297 http://pear.php.net
297 http://pear.php.net/get
296 http://hackage.haskell.org/packages
296 http://hackage.haskell.org/packages/archive
296 http://hackage.haskell.org
209 http://launchpad.net
206 http://ftp.xemacs.org
197 https://github.com
196 http://ftp.xemacs.org/pub
196 http://ftp.xemacs.org/pub/xemacs
193 http://ftp.xemacs.org/pub/xemacs/packages
179 http://gstreamer.freedesktop.org
179 http://gstreamer.freedesktop.org/src
176 http://github.com
175 http://linuxgazette.net/ftpfiles
175 http://linuxgazette.net
139 http://xorg.freedesktop.org/release...vidual/app
130 http://pear.horde.org/get
130 http://pear.horde.org
126 http://xorg.freedesktop.org/release...ual/driver
115 http://dev.gentoo.org/~vapier
112 http://dev.gentoo.org/~vapier/dist
96 ftp://sources.redhat.com/pub
96 ftp://sources.redhat.com
94 http://savannah.nongnu.org/download
94 http://savannah.nongnu.org
92 http://xorg.freedesktop.org/release...vidual/lib
88 http://pecl.php.net
88 http://download.gna.org
88 http://pecl.php.net/get
84 http://dev.gentoo.org/~caster/distfiles
84 http://dev.gentoo.org/~caster
80 https://fedorahosted.org
74 http://get.qt.nokia.com
72 http://components.ez.no/get
72 http://components.ez.no
71 http://get.qt.nokia.com/qt
70 http://get.qt.nokia.com/qt/source
67 http://www.phrack.org/archives
67 http://www.phrack.org/archives/tgz

And here are some mirrors that can be deduced from that (except freebsd-jp
which I found reading the linphone ebuild):

freebsd-jp ftp://ports.jp.FreeBSD.org/pub/FreeBSD-jp/ ftp://ftp.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp1.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp2.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp3.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp4.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp5.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp6.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp7.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp8.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp9.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp.ics.es.osaka-u.ac.jp/pub/FreeBSD-jp/ ftp://ftp.ics.es.osaka-u.ac.jp/pub/...reeBSD-jp/ ftp://ftp.allbsd.org/pub/FreeBSD-jp/ ftp://ftp.nagaokaut.ac.jp/pub/FreeBSD-jp/ ftp://ftp.nara.wide.ad.jp/pub/FreeBSD-jp/ ftp://ftp.jaist.ac.jp/pub/FreeBSD-jp/
pear http://pear.php.net/ http://de.pear.php.net/ http://us.pear.php.net/
redhat ftp://sources.redhat.com/pub/ ftp://mirrors.kernel.org/sources.redhat.com/ ftp://mirrors.kernel.org/sources.redhat.com/ http://mirrors.kernel.org/sources.redhat.com/ http://sources-redhat.mirrors.airband.net/ ftp://gd.tuwien.ac.at/gnu/sourceware/ http://gd.tuwien.ac.at/gnu/sourceware/ ftp://ftp.gwdg.de/pub/linux/sources.redhat.com/ http://ftp.gwdg.de/pub/linux/sources.redhat.com/ ftp://ftp-stud.fht-esslingen.de/pub...eware.org/ http://ftp-stud.fht-esslingen.de/pu...eware.org/ http://bo.mirror.garr.it/mirrors/sourceware.org/ ftp://bo.mirror.garr.it/mirrors/sourceware.org/ ftp://ftp.mirrorservice.org/sites/s...t.com/pub/ http://www.mirrorservice.org/sites/...t.com/pub/
xemacs http://ftp.xemacs.org/pub/ ftp://ftp.sa.xemacs.org/pub/
xorg http://xorg.freedesktop.org/pub/ http://ftp.x.org/pub/

That would affect ~1000 URIs.
Unfortunatly I didn't found any mirror for:
- pecl.php.net
- hackage.haskell.org
- get.qt.nokia.com
- launchpad

But maybe we should still add them to thirdpartymirrors ? Look at 3dgamers, beyondunreal, bitbucket, github, gnome, kernel, xfce, vdrfiles and vdr-developerorg: they all only have one mirror !

For those afraid to add these mirrors and patch all affected ebuilds: no worry, mirrors.py can do it for you, and I already generated splitted patches generated from various thirdpartymirrors here: http://xf.iksaif.net/dev/gentoo/

Note that as part of https://bugs.gentoo.org/show_bug.cgi?id=405533 mgorny already commited a massive amount of patches adding mirror:// when the URI prefix was already present in thirdpartymirrors.

Thanks,

Corentin Chary
http://xf.iksaif.net/