Mirror copy

Problems with packages? Post here, using [tags] of the package name.

Mirror copy

Postby xaoc » Wed Aug 02, 2017 2:57 pm

Hi,
We are using archlinuxarm as an operating system for our IoT devices. I would like to frozen specific version of mirror.

We have two ways of strategy:
1. Automatic update (mirror archlinux repo -> test -> update devices).
-- so we sync our mirror with you at some time (rarely i think once for 3month) and then update our devices from our mirror after tests. This allow us add script that protect us from some dangerous changes.
2. Frozen version. After sometime this devices will be probably frozen. It would be nice to can install package without braking dependency let's say after 3 years (just in case of emergency this should never happen).

Is possible to somehow create such mirror?
xaoc
 
Posts: 1
Joined: Fri Feb 03, 2017 3:03 pm

Re: Mirror copy

Postby WarheadsSE » Wed Aug 02, 2017 2:58 pm

That is possible, and I suggest reviewing the Arch Rollback Machine documentation, as well as looking into the repositories tools such as `repo-add`

This will take some oversight from you, of course.
Core Developer
Remember: Arch Linux ARM is entirely community donation supported!
WarheadsSE
Developer
 
Posts: 6807
Joined: Mon Oct 18, 2010 2:12 pm

Re: Mirror copy

Postby brokebit » Mon Aug 28, 2017 3:25 pm

Hello,

We have the same IoT use case as xaoc. We need to freeze the mirror because we build several of our own packages that we maintain in a separate repo. Periodically as the main repos update the dependencies for our packages get broken. Consequently, we created a local copy of the mirror that we refresh quarterly. We refresh, test our packages, and make updates as needed to to fix dependency issues that arose from the refresh.

We created our local mirror using lftp, however, there seems to be some strange issues. For example, if we try to install opencv using pacman -S opencv it cannot resolve all the dependencies of opencv (xproto is example dependency).

Mirror Script:
#!/usr/bin/lftp -f
lcd /var/arch/mirror
open http://mirror.archlinuxarm.org
mirror -cve


Attempt to install opencv:

[root@b827eba5df87 ~]# pacman -S opencv
resolving dependencies...
warning: cannot resolve "xproto", a dependency of "libxdmcp"
warning: cannot resolve "libxdmcp", a dependency of "libxcb"
warning: cannot resolve "xproto>=7.0.18", a dependency of "libxau"
warning: cannot resolve "libxau", a dependency of "libxcb"
warning: cannot resolve "libxcb", a dependency of "libx11"
warning: cannot resolve "xproto", a dependency of "libx11"
warning: cannot resolve "libx11", a dependency of "libxext"
warning: cannot resolve "libxext", a dependency of "libxv"
warning: cannot resolve "libxv", a dependency of "libxvmc"
warning: cannot resolve "libxvmc", a dependency of "xine-lib"
warning: cannot resolve "zita-alsa-pcmi", a dependency of "jack"
warning: cannot resolve "zita-resampler", a dependency of "jack"
warning: cannot resolve "jack", a dependency of "ffmpeg"
warning: cannot resolve "xproto", a dependency of "libxdmcp"
warning: cannot resolve "libxdmcp", a dependency of "libxcb"
warning: cannot resolve "xproto>=7.0.18", a dependency of "libxau"
warning: cannot resolve "libxau", a dependency of "libxcb"
warning: cannot resolve "libxcb", a dependency of "libx11"
warning: cannot resolve "xproto", a dependency of "libx11"
warning: cannot resolve "libx11", a dependency of "libxext"
warning: cannot resolve "libxext", a dependency of "libxtst"
warning: cannot resolve "xproto", a dependency of "libxdmcp"
warning: cannot resolve "libxdmcp", a dependency of "libxcb"
warning: cannot resolve "xproto>=7.0.18", a dependency of "libxau"
warning: cannot resolve "libxau", a dependency of "libxcb"
warning: cannot resolve "libxcb", a dependency of "libx11"
warning: cannot resolve "xproto", a dependency of "libx11"
warning: cannot resolve "libx11", a dependency of "libxext"
warning: cannot resolve "libxext", a dependency of "libxi"
warning: cannot resolve "libxi", a dependency of "libxtst"
warning: cannot resolve "xproto", a dependency of "libxdmcp"
warning: cannot resolve "libxdmcp", a dependency of "libxcb"
warning: cannot resolve "xproto>=7.0.18", a dependency of "libxau"
warning: cannot resolve "libxau", a dependency of "libxcb"
warning: cannot resolve "libxcb", a dependency of "libx11"
warning: cannot resolve "xproto", a dependency of "libx11"
warning: cannot resolve "libx11", a dependency of "libxfixes"
warning: cannot resolve "xproto", a dependency of "fixesproto"
warning: cannot resolve "fixesproto>=5.0", a dependency of "libxfixes"
warning: cannot resolve "libxfixes", a dependency of "libxtst"
warning: cannot resolve "libxtst", a dependency of "libpulse"
warning: cannot resolve "xproto>=7.0.18", a dependency of "libice"
warning: cannot resolve "libice", a dependency of "libsm"
warning: cannot resolve "libsm", a dependency of "libpulse"
warning: cannot resolve "libpulse", a dependency of "ffmpeg"
:: There are 5 providers available for libgl:
:: Repository extra
1) libglvnd
:: Repository alarm
2) imx-gpu-viv-dfb 3) imx-gpu-viv-fb 4) imx-gpu-viv-wl 5) imx-gpu-viv-x11

Enter a number (default=1): ^C

- Select option 1 does nothing and you can stuck in a dependency loop.

However, if we browse through the files on our local mirror all the dependencies are there. In fact, I download the dependencies via curl and installed them using pacman -U. I can resolve all dependencies that way and then install opencv using pacman -S opencv.

It seems as if the *.db files are out of sync with the files we sync'd from the mirror possibly. I also downloaded some of *.db.tar.gz files and untar'd them. From what I can see, they do have references to the dependencies (xproto) in there. So, again, I'm confused why pacman cannot resolve the dependency.

Any thoughts why pacman isn't finding the dependencies in our local mirror even though they appear to be there?

Thanks,
Eric
brokebit
 
Posts: 3
Joined: Mon Aug 28, 2017 2:28 pm

Re: Mirror copy

Postby WarheadsSE » Tue Aug 29, 2017 12:11 am

I'll point out, as did for xaoc, to be familiar with repository tools such as `repo-add`.
Core Developer
Remember: Arch Linux ARM is entirely community donation supported!
WarheadsSE
Developer
 
Posts: 6807
Joined: Mon Oct 18, 2010 2:12 pm

Re: Mirror copy

Postby brokebit » Wed Aug 30, 2017 12:19 pm

Hi WarheadsSE,

I am familiar with repo-add and we're using it to maintain a custom repo of our packages. However, I'm not sure how it could help me solve the problem with our copy of the mirror. As I said in my previous post, I do think xproto is in our mirror and in the extra.db.tar.gz file and yet pacman can't seem to find it. So, I decided to use repo-add to try and re-add it to the extra.db.tar.gz database:

[root@dev extra]# repo-add extra.db.tar.gz xproto-7.0.31-1-any.pkg.tar.xz
==> Extracting database to a temporary location...
==> Extracting database to a temporary location...
==> Adding package 'xproto-7.0.31-1-any.pkg.tar.xz'
==> WARNING: An entry for 'xproto-7.0.31-1' already existed
-> Adding package signature...
-> Computing checksums...
-> Removing existing entry 'xproto-7.0.31-1'...
-> Creating 'desc' db entry...
-> Creating 'files' db entry...
==> Creating updated database file 'extra.db.tar.gz'

You can see above that repo-add is warning me that xproto is already part of the database. However, what is interesting is that I did an md5sum of extra.db.tar.gz before running repo-add and after running it. I'm getting a different md5 sum. So repo-add did change something. I'm going to do a pacman -Syy, pacman -Syu and a pacman -S xproto against the mirror and test shortly. I'll let you know how it goes.

Eric
brokebit
 
Posts: 3
Joined: Mon Aug 28, 2017 2:28 pm

Re: Mirror copy

Postby brokebit » Wed Aug 30, 2017 6:14 pm

Hey Everyone,

FYI - So, we ended up going into the "extra" folder and executing a repo-add extra.db.tar.gz *.xz to rebuild the entire package database even though xproto and other dependencies were already in the database. After that, we ran a pacman -Syy, pacman -Syu and were able to resolve all dependencies. So it appears that something was corrupt in the copy of the mirror db files we sync'd down. We going to add rebuilding the db files to our mirror update process going forward.

Eric
brokebit
 
Posts: 3
Joined: Mon Aug 28, 2017 2:28 pm

Re: Mirror copy

Postby WarheadsSE » Wed Aug 30, 2017 6:43 pm

8-)
Core Developer
Remember: Arch Linux ARM is entirely community donation supported!
WarheadsSE
Developer
 
Posts: 6807
Joined: Mon Oct 18, 2010 2:12 pm


Return to Packages

Who is online

Users browsing this forum: No registered users and 12 guests