[SOLVED] Updating the latest aarch64 archive breaks on RPI

This forum is for topics dealing with problems with software specifically in the AArch64 repo.

[SOLVED] Updating the latest aarch64 archive breaks on RPI

Postby Antiz » Sun Jun 05, 2022 5:07 pm

Hi everyone,

I get a weird behavior with the latest RPI aarch64 archive (http://os.archlinuxarm.org/os/ArchLinuxARM-rpi-aarch64-latest.tar.gz), the one that uses the 5.16.3 kernel (at the time I'm writing those lines).

Indeed, performing an update from it (pacman -Syy && pacman -Syu) seems to "break it". Rebooting after performing the update makes the Raspberry Pi tries to boot from some PXE images, failing at each one indefinitly.
The loop looks something like this :
[...]
Filename pxeconfig/XXXXX
LoadAdress 0X20000
Loading TTTTTTT


I can interrupt the process by pressing ctrl+c which gives me the following prompt :
U-Boot>

If you don't perform an update after the `pacman-key --init && pacman-key --populate archlinuxarm`, everything works well. I can reboot without any problems. So the update (`pacman -Syy && pacman -Syu`) really seems to be the cause of breakage that makes the raspberry falls down to the U-Boot> prompt after reboot.

By the way, I have multiple raspberry PIs already running on Arch Linux ARM for a while and I update them every week without any issue.
This is why I guess this problem is only about the latest RPI aarch64 archive.

I don't have anything else than Raspberry PIs that run ARM CPUs so I cannot try on other hardware, but I tried several fresh installs on them and got the same behavior each times.

Step to reproduce :
- Performing a fresh aarch64 install (https://archlinuxarm.org/platforms/armv8/broadcom/raspberry-pi-3)
- Check that everything works, reboot included
- Performing an update (`pacman -Syy && pacman -Syu`) and reboot
- The raspberry PI will not boot the OS but instead try to load some PXE images. Interrupting the process (ctrl+c) gives you the U-Boot> prompt

As I said earlier, it looks to only happen with a fresh install using the latest aarch64 archive, as my other raspberry PIs already running arch ARM didn't had this issue after being updated.

Is anyone having the same issue or is able to reproduce this ?

Thanks in advance for you help :)
Last edited by Antiz on Sun Jul 24, 2022 7:28 pm, edited 1 time in total.
Antiz
 
Posts: 7
Joined: Sat Jun 04, 2022 1:00 pm
Location: Rouen, France

Re: [RPI] Updating the latest aarch64 archive breaks the sys

Postby robg » Sun Jun 05, 2022 11:07 pm

Just some initial questions:

Are all of your RPIs the same model? Are they all running the generic aarch64 kernel (instead of linux-rpi)? Is the firmware up-to-date on all of your RPIs?

Does your RPI boot properly if you upgrade everything except uboot?
robg
 
Posts: 186
Joined: Tue Jan 05, 2021 8:22 am

Re: [RPI] Updating the latest aarch64 archive breaks the sys

Postby Antiz » Mon Jun 06, 2022 2:37 pm

$this->bbcode_second_pass_quote('robg', 'J')ust some initial questions:

Are all of your RPIs the same model? Are they all running the generic aarch64 kernel (instead of linux-rpi)? Is the firmware up-to-date on all of your RPIs?

Does your RPI boot properly if you upgrade everything except uboot?


Hi,

Are all of your RPIs the same model? --> I've got one PI 3B+ and one PI 4B, both running Arch ARM. I'm currently facing this issue on the 3B+ when trying to install Arch ARM on another sd card that the one currently running. Unfortunately it is a bit hard for me to try to reproduce it on my PI 4 as it running in "production" (it is my DHCP and DNS server, among others...). While not being impossible, it is a bit inconvenient to shut it down, even temporarily, but I can consider it if needed ! However, I don't think that this issue is hardware related as this PI 3B+ has no issue with its current SD that already has Arch ARM installed for a while. This is why I guessed the issue was about the latest aarch64-rpi archive, but I may be wrong.

Are they all running the generic aarch64 kernel (instead of linux-rpi)? --> Indeed, they all seems to run the generic aarch64 kernel.
[user@PI ~]$ uname -r
5.18.1-1-aarch64-ARCH

Is the firmware up-to-date on all of your RPIs? --> It is, I update all my PIs once a week.
[user@PI ~]$ pacman -Q | grep firmware-raspberrypi
firmware-raspberrypi 20220514-1
https://archlinuxarm.org/packages/any/firmware-raspberrypi

Does your RPI boot properly if you upgrade everything except uboot? --> It looks like uboot is already in the latest version with the latest RPI aarch64 archive, so it is not part of the update. Below command ran just after a fresh install :
[root@alarm ~]# pacman -Q | grep uboot
uboot-raspberrypi 2021.04-1
https://archlinuxarm.org/packages/aarch64/uboot-raspberrypi

By the way, this is the list of package updated by pacman when performing `pacman -Syu` after a fresh install :
Packages (75) audit-3.0.8-1 binutils-2.38-5.1 brotli-1.0.9-8 ca-certificates-mozilla-3.79-1 coreutils-9.1-1 crda-4.14-4 [removal] replaced by core/wireless-regdb curl-7.83.1-1 device-mapper-2.03.16-1 dialog-1:1.3_20220414-1 expat-2.4.8-1 firmware-raspberrypi-20220514-1 gcc-libs-12.1.0-2.1 gdbm-1.23-1 glib2-2.72.2-1 glibc-2.35-5.1 gmp-6.2.1-2 gnupg-2.2.35-2 gnutls-3.7.6-1 gzip-1.12-1 hwdata-0.359-1 iana-etc-20220603-1 icu-71.1-1 iproute2-5.18.0-1 iptables-1:1.8.8-1 iw-5.19-1 json-c-0.16-1 kbd-2.5.0-1 kmod-29-3 krb5-1.19.3-1 libarchive-3.6.1-1 libbpf-0.8.0-1 libcap-2.64-1 libcap-ng-0.8.3-1 libelf-0.187-1 libffi-3.4.2-5 libgcrypt-1.10.1-1 libgpg-error-1.45-1 libldap-2.6.2-2 libmnl-1.0.5-1 libnetfilter_conntrack-1.0.9-1 libnfnetlink-1.0.2-1 libnl-3.6.0-3 libsasl-2.1.28-1 libseccomp-2.5.4-1 libsecret-0.20.5-2 libsysprof-capture-3.44.0-1 libunistring-1.0-1 libxml2-2.9.14-1 linux-arch64-5.18.1-1 linux-api-headers-5.17.5-2 linux-firmware-20220509.b19cbdc-1 linux-firmware-whence-20220509.b19cbdc-1 mkinitcpio-31-2.1 nano-6.3-1 ncurses-6.3-3 netctl-1.28-1 openssh-9.0p1-1 openssl-1.1.1.o-1 pacman-6.0.1-5 pacman-mirrorlist-20220329-1 pciutils-3.8.0-1 pcre2-10.40-1 psmisc-23.5-1 raspberrypi-bootloader-20220601-1 sqlite-3.38.5-1 systemd-251.2-1 systemd-libs-251.2-1 systemd-sysvcompat-251.2-1 tzdata-2022a-1 util-linux-2.38-1 util-linux-libs-2.38-1 wireless-regdb-22.04.08-1 wpa_supplicant-2:2.10-4 xz-5.2.5-3 zlib-1:1.2.12-2

There should be one (or some) of these updates causing the "breakage" I guess, but as the system won't even boot after performing the update and rebooting, it is a bit hard to analyze...
I tried to mount the SD card on my PC in order to search for logs but there again, as the OS don't even boot after the update, logs fatally stops right after the reboot command so I couldn't find something relevant.

I hope those details helps ! :)
Antiz
 
Posts: 7
Joined: Sat Jun 04, 2022 1:00 pm
Location: Rouen, France

Re: [RPI] Updating the latest aarch64 archive breaks the sys

Postby ajsmiller » Mon Jun 06, 2022 3:42 pm

I have a similar experience. This was my first install of archlinux arm, but not my first archlinux install, nor my first rpi install.

* sha256sum: B6911be93c4b8a7b46bb734e58c4787c3386b175d271a24b1f73e0de498ee4bdf ArchLinuxARM-rpi-aarch64-latest.tar.gz
* Raspberry Pi 3B in a Pi-Top (The green laptop)

I followed the instructions at https://archlinuxarm.org/platforms/armv8/broadcom/raspberry-pi-3 with exceptions:

* I mounted the root device on an Intel PC: $this->bbcode_second_pass_code('', 'mount /dev/sde2 /media/flash')
* I mounted the boot device on the same PC: $this->bbcode_second_pass_code('', 'mkdir /media/flash/boot; mount /dev/sde1 /media/flash/boot')
* I therefore did not need the step to copy the contents of /boot to the right place

The system booted into an emergency shell. Root cause: The instructions did not require labeling the partitions, but the boot loader required a labeled partition.

The Error message was enough to realise I just needed to mount /dev/sd2 on /new_root and exit the emergency shell, after which the system comes up normally.

Not knowing the label, and knowing that the device mapping would not change, I decided to change the configuration in /boot/boot.txt to use /dev/mmcblk0p2 to at least move along in the process (this may be an error on my part, I have spent far too many hours fighting with grub because a uuid or label is used for finding disks after something has gone wrong,)

/boot/boot.txt warns thus $this->bbcode_second_pass_code('', '# After modifying, run ./mkscr') which fails with the error message
$this->bbcode_second_pass_code('', 'mkimage not found. Please install uboot-tools:
pacman -S uboot-tools')

The system has no network set up yet, so I set up USB Tethering to my mobile and continue to
$this->bbcode_second_pass_code('', 'pacman-key --init
pacman-key --populate archlinuxarm
pacman -Syu
pacman -S uboot-tools')

and run ./mkscr as instructed. If I had a serial console connected, or a working network, I might be able to see the same loop as above. Neither is the case - so I came here and then found this thread.

If the instructions had mentioned the labels, and uboot-tools where in the image tar delivered I might not have encountered this problem. Where can I report such documentation bugs, or even better, help fix them?
ajsmiller
 
Posts: 2
Joined: Mon Jun 06, 2022 2:59 pm

Re: [RPI] Updating the latest aarch64 archive breaks the sys

Postby Antiz » Mon Jun 06, 2022 5:26 pm

$this->bbcode_second_pass_quote('ajsmiller', 'I') have a similar experience. This was my first install of archlinux arm, but not my first archlinux install, nor my first rpi install.


Hi,

I'm not really sure our respective problems are similar actually.
From my understanding, you seems to have a problem with the installation process while I have a problem after performing an update on a fresh (and working) install.

Also, my other raspberry PIs running Arch ARM doesn't have the "uboot-tools" package installed and they work perfectly. According to that, I don't think this package has to be included in the image tar, as it doesn't seems necessary.
Maybe you needed it because of the tinkering you did with the installation process ? I mean, the trick you did to prevent yourself to copy the content of /boot to the right place seems okay on paper (if you extracted the archive to /media/flash at least) but I personally always followed the installation instructions as they were written and never had such issues.

Anyway, to report your issue you can create a dedicated topic on the forum. Responding to mine would have been okay if our issue were actually related but I think they're not (at least from my understanding, but I may be wrong).
Also, if you're sure your issue is related to a bug (and not from actions you did wrong), you can report it in the "Packages" sub-forum (https://archlinuxarm.org/forum/viewforum.php?f=15) according to the following thread : https://archlinuxarm.org/forum/viewtopic.php?f=3&t=2675.
Antiz
 
Posts: 7
Joined: Sat Jun 04, 2022 1:00 pm
Location: Rouen, France

Re: [RPI] Updating the latest aarch64 archive breaks the sys

Postby ajsmiller » Tue Jun 07, 2022 7:01 am

Thank you for the links. I hope these are also the right place for Documentation reports.

I understand your concerns about the differences between our situations:

/boot has less metadata than / and its other children, because it has a FAT file system and not ext4fs, you loose the ownership and permission bits and ACLs as soon as either tar unpacks to FAT or cp writes to fat. /boot does not need to be mounted for normal operation, only when the bootware needs changing in some way. Empirically the last 10 Intel Gentoo/Funtoo/Archlinux installations were all done exactly by mounting /boot before unpacking the stage1/2/3 or packstrapping, and the only problems where ever grub not finding the disk to boot from properly, which grub just made harder than need be to fix.

The Pi booted successfully and, after a manual intervention in the boot process, worked with the installed software until an update was needed (ie the "tinkering with the boot process"). I guess if the boot process had worked flawlessly, then my next steps would have been to install the payload software, via pacman, and the next reboot would have been the surprise. uboot-tools are, just like /boot not needed in normal operations, just when the bootware needs changing... After the updates the screen was blank! I *guess* that the helpful output was going to a device not attached - no VT commands - my guess could be wrong and something else is broken. Adding uboot-tools to the tarbal would make recovering from this and similar mishaps easier.

Probably there should be 2 reports: Clarification of documentation for the install procedure and suggestion to include uboot-tools on default install media.
ajsmiller
 
Posts: 2
Joined: Mon Jun 06, 2022 2:59 pm

Re: [RPI] Updating the latest aarch64 archive breaks the sys

Postby robg » Fri Jun 10, 2022 10:05 pm

@Antiz: Thanks for detailed reply. Alas, this is a weird situation: You have the same RPI models running (most likely) the same software. At this point my suspicion lies on the uboot config. Can you verify if said configs differs between the faulty and the working installs?
robg
 
Posts: 186
Joined: Tue Jan 05, 2021 8:22 am

Re: [RPI] Updating the latest aarch64 archive breaks the sys

Postby Antiz » Sun Jul 03, 2022 11:40 am

Hi @robg,

Sorry for the late answer... I've been busy those days :?

I'm not sure to know what you're referencing as "uboot config". Are those the "dtb" files ?

If so, here's the result :
/boot/dtbs/broadcom/bcm2837-rpi-3-b-plus.dtb --> differs between the faulty and the working installs
/boot/bcm2710-rpi-3-b-plus.dtb --> Are the same between the faulty and the working installs

If you didn't referencing those "dtb" files, what should I look for ?
Antiz
 
Posts: 7
Joined: Sat Jun 04, 2022 1:00 pm
Location: Rouen, France

Re: [RPI] Updating the latest aarch64 archive breaks the sys

Postby Sylvan » Mon Jul 04, 2022 4:59 am

In the past I had tens of similar problems. Often a reboot after an update failed - especially with updates of the kernel or bootloader.

So I switched to using the rpi kernel and bootloader and since then I have peace of mind - no more such problems.
To make this switch, basically all you need is the following command:

$this->bbcode_second_pass_code('', 'pacman -S --needed linux-rpi raspberrypi-bootloader raspberrypi-firmware')

And when making adjustments to the kernel parameters for the boot loader, you must now adjust cmdline.txt instead of boot.txt.
Sylvan
 
Posts: 7
Joined: Sun May 12, 2019 8:55 am

Re: [RPI] Updating the latest aarch64 archive breaks the sys

Postby Antiz » Mon Jul 04, 2022 5:45 pm

$this->bbcode_second_pass_quote('Sylvan', 'I')n the past I had tens of similar problems [...] so I switched to using the rpi kernel and bootloader and since then I have peace of mind - no more such problems.


That might be a potential workaround, indeed.
But that still doesn't explain why I encounter this weird problem. I'm just following the official install guide (https://archlinuxarm.org/platforms/armv ... berry-pi-3) without any extra step. Also, all my other RPIs all work with the generic aarch64 kernel without such issue.

My goal here is not specifically to get a working installation but more to understand why I have this issue, how to correct it and who should correct it (is it a problem on my side or on Arch ARM side ?).

Thanks anyway for your answer. That may help those that have a similar problem and just want things to work :)
Antiz
 
Posts: 7
Joined: Sat Jun 04, 2022 1:00 pm
Location: Rouen, France
Top

Next

Return to ARMv8

Who is online

Users browsing this forum: No registered users and 10 guests