Improving Geo-IP based mirror selection ?

[Flekain]

Hey Archers,

I'm using alarm on one of my Pi, and I encounter one annoying error quite frequently, and I bet I'm not the only one in that case: error: failed retrieving file 'core.db' from mirror.archlinuxarm.org : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
$this->bbcode_second_pass_code('', '[root@alarmpi ~]$ pacman -Syu
:: Synchronizing package databases...
error: failed retrieving file 'core.db' from mirror.archlinuxarm.org : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed to update core (download library error)
error: failed retrieving file 'extra.db' from mirror.archlinuxarm.org : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed to update extra (download library error)
error: failed retrieving file 'community.db' from mirror.archlinuxarm.org : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed to update community (download library error)
error: failed retrieving file 'alarm.db' from mirror.archlinuxarm.org : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed to update alarm (download library error)
error: failed retrieving file 'aur.db' from mirror.archlinuxarm.org : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed to update aur (download library error)
error: failed to synchronize any databases
error: failed to init transaction (download library error)')

For the record - as advised, ipv6 is disabled in my /boot/cmdline.txt:
$this->bbcode_second_pass_code('', 'ipv6.disable=1 selinux=0 plymouth.enable=0 smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p5 rootfstype=ext4 elevator=noop rootwait')

I know how to fix this, and the fix is very quick and easy:
$this->bbcode_second_pass_code('', '[root@alarmpi ~]$ nano /etc/pacman.d/mirrorlist
=> Add a "#" to comment the line Server = http://mirror.archlinuxarm.org/armv6h/$repo
=> Remove the "#" to uncomment the line corresponding to the mirror you'd like to use, in my case Server = http://nl.mirror.archlinuxarm.org/armv6h/$repo
[root@alarmpi ~]$ pacman -Syy
You're good to go')

However, isn't there a possibility to improve the mechanism that's supposed to select the best server for you ?
I don't think it unfeasible nor very hard, I'm sure newbies who just got their Raspberry Pi (for example ) would appreciate it, just as much as experienced users who won't loose some precious seconds to fix this on every fresh install they do or manage.

Hoping to get some feedback from you

David,

[WarheadsSE]

Would you happen to have a suggestion for improving the IP based selection?

[Flekain]

Well, I am not familiar with how the current system works, but since I don't remember running into this problem with any other distributions that I used (X86_64: Arch, Debian testing, Fedora, Ubuntu - ARM: Raspbmc, Raspbian), I just thought there would be room for improvement on this matter.

I do love archlinuxarm, and I am willing to do everything that I can to help improve it (for example adding a mirror hosted on my own future dedicated server, in France), even though my Linux/coding skills are quite low.

Isn't mirror.archlinuxarm.org the main server from whom all the other official mirrors sync ? If that's the case I'm thinking its priority would be to sync other mirrors, and to be used as less as possible by final alarm users. To ensure that, the Geo-IP mechanism could for example not use it as default, discourage its use (if not already done) by setting up measures the way archlinux (desktop) did, and manage to somehow lower its priority by the mechanism which is supposed to calculate which is the best mirror for you.

That's a few thoughts on my head right now, as I said, I don't intend to fully understand this thing, I am just eager to try to make alarm better

[WarheadsSE]

Actually, no, mirror.archlinuxarm.org is a round-robin Geo-IP based dns name.

- us.mirror.archlinuxarm.org is the primary
- Everything else goes gets rsync-push'd to from there
- several countries have multiples, so for example *.us.mirror.archlinuxarm.org and de*.mirror.archlinuxarm.org
- there are a few Tier 2 mirrors that sync from one of them.

We use the standard publicly used and agreed as quality Geo-IP cache from MaxMind.

As the mirrors are being updated, we disable them in dns, and when they are up to date, we re-enable them.

[kmihelich]

You don't have to use the default mirror, that is just a way to help load balance among mirrors. Feel free to choose your own mirror to use, like you do in any other distribution. What it really looks like you're asking for is for us to somehow magically figure out, for you, what the best mirror is. We're not here to administrate your installation.

[Flekain]

@WarheadsSE: Thanks for all this very interesting infos

@kmihelich: Magically, really ? While I don't have any illusions about my (very limited) Linux skills, even I know that the kind of thing I had in my head could certainly be done.
I get it that the main purpose of the GEO-IP system is to balance the charge between mirrors, but I think it is definitely doable to take the user's location into account in order to make it select the best mirror. If automation can be safely used, I can't find a reason not to use it, as long as it doesn't affect negatively some users, and in this case, I don't know how it could.

[kmihelich]

The potential for issues due to the added complexity simply outweighs any gain to be had, especially when it's such an isolated issue. I don't see what your problem is with simply configuring your mirrorlist to your needs like every other Arch user.

[WarheadsSE]

Thats the point of the GeoIP selection. It sends you the the most likely known mirror. It doesn't really "balance" the load.

[Flekain]

@kmihelich: I guess it could be called laziness - I don't really like programming, but I do envjoy shell scripting because it gives me the feeling that I save time, and do thing more cleverly than repeating the same tasks manually over and over again (okay in this particular example, manually configuring the mirro I want to use is probably a one time thing only, but still )

@WarheadsSE: My xDSL ISP (which is also a well known hosting company in Europe - OVH) offers me a voucher for every year that I spend with them. Since I can't use this voucher on anything that would be really useful for me - I thought I could rent a dedicated server in order to add a mirror for alarm repos, and possibly other Linux distributions (if there's enough ressources) that have my support and love.
It could be a way to bypass this "problem" (yeah I get it, not really a problem) I was talking about, by adding the first mirror AFAIK in my country - France, and benefits me but possibly others users as well. What do you (WarheadsSE & kmihelich) think ?

I've never done something like this before, but I don't think it's something I can't learn.
In case you want to give me your first thoughts about whether it would be possible or not, here are the specs of the server:
$this->bbcode_second_pass_quote('', 'I')ntel Atom CPU (Diamondville Cedarview - 1 Core - 2 Threads - 1.6+ Ghz - 64 bits)
2 Go of RAM
500 Go of HDD (SATA2)
100 Mbps Connectivity

[kmihelich]

I'm always happy to add new, stable mirror hosts. Just requires you to set up rsyncd and a couple vhosts. We used to have a mirror in France, but the server went down and the admin was unreachable.

DNS round-robin is also problematic because a lot of DNS servers don't properly cycle the records. They grab the list of A records, then always return them in the same order they got them in (which is randomized on our end). So depending on who or what you're using for DNS, the benefits of the round-robin can be killed if your ISP has a poor or broken route to the one at the top of the list. Google's DNS servers don't cycle the records, for instance.