[solved] raspberrypi-bootloader-20240122 breaks boot on rpi4

[Musikolo]

Hi guys,

I just upgraded my rpi4 using "pacman -Syu" as I usually do, and I found it failed rebuilding the kernel images. It produced the the error "WARNING: errors were encountered during the build. The image may not be complete.", as you can see below:

[code][musikolo@my-rpi4 ~]$ sudo pacman -Syu

:: Synchronizing package databases...
core 234.8 KiB 397 KiB/s 00:01 [#############################################################] 100%
extra 9.1 MiB 3.30 MiB/s 00:03 [#############################################################] 100%
alarm 57.3 KiB 140 KiB/s 00:00 [#############################################################] 100%
aur is up to date
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...

Packages (33) alsa-lib-1.2.11-1 alsa-ucm-conf-1.2.11-1 curl-8.6.0-3 expat-2.6.0-1 ffmpeg-2:6.1.1-5 fftw-3.3.10-5 hwdata-0.379-1 libavif-1.0.4-1 libelf-0.190-2
libgit2-1:1.7.2-1 libjxl-0.9.2-1 libnghttp3-1.1.0-2 libpciaccess-0.18-1 libsecret-0.21.3-1 libsysprof-capture-45.2-1 libusb-1.0.27-1 libxext-1.3.6-1
libxml2-2.12.5-1 licenses-20240206-1 linux-api-headers-6.7-1 mpfr-4.2.1-2 openmpi-5.0.1-2 pacman-6.0.2-9 prrte-3.0.3-2 python-pydantic-2.6.1-1
python-pydantic-core-1:2.16.2-1 python-pytz-2024.1-1 raspberrypi-bootloader-20240208-1 raspberrypi-utils-20240204-1 sdl2-2.30.0-1 tzdata-2024a-1
vapoursynth-R65-1 wireless-regdb-2024.01.23-1

Total Download Size: 32.02 MiB
Total Installed Size: 137.05 MiB
Net Upgrade Size: 3.61 MiB

:: Proceed with installation? [Y/n]
:: Retrieving packages...
fftw-3.3.10-5-aarch64 1533.5 KiB 3.15 MiB/s 00:00 [#############################################################] 100%
openmpi-5.0.1-2-aarch64 2.8 MiB 5.04 MiB/s 00:01 [#############################################################] 100%
libjxl-0.9.2-1-aarch64 1296.4 KiB 8.67 MiB/s 00:00 [#############################################################] 100%
python-pydantic-core-1:2.16.2-1-aarch64 1260.5 KiB 8.10 MiB/s 00:00 [#############################################################] 100%
linux-api-headers-6.7-1-any 1200.5 KiB 8.50 MiB/s 00:00 [#############################################################] 100%
curl-8.6.0-3-aarch64 1046.2 KiB 9.29 MiB/s 00:00 [#############################################################] 100%
sdl2-2.30.0-1-aarch64 889.2 KiB 8.60 MiB/s 00:00 [#############################################################] 100%
hwdata-0.379-1-any 1540.0 KiB 1467 KiB/s 00:01 [#############################################################] 100%
pacman-6.0.2-9-aarch64 831.1 KiB 7.52 MiB/s 00:00 [#############################################################] 100%
libxml2-2.12.5-1-aarch64 734.5 KiB 6.40 MiB/s 00:00 [#############################################################] 100%
python-pydantic-2.6.1-1-any 693.0 KiB 6.84 MiB/s 00:00 [#############################################################] 100%
vapoursynth-R65-1-aarch64 602.3 KiB 10.1 MiB/s 00:00 [#############################################################] 100%
prrte-3.0.3-2-aarch64 483.8 KiB 9.26 MiB/s 00:00 [#############################################################] 100%
libelf-0.190-2-aarch64 447.7 KiB 7.67 MiB/s 00:00 [#############################################################] 100%
libgit2-1:1.7.2-1-aarch64 722.3 KiB 2.36 MiB/s 00:00 [#############################################################] 100%
tzdata-2024a-1-aarch64 367.8 KiB 6.91 MiB/s 00:00 [#############################################################] 100%
mpfr-4.2.1-2-aarch64 318.5 KiB 5.02 MiB/s 00:00 [#############################################################] 100%
alsa-lib-1.2.11-1-aarch64 440.0 KiB 2.81 MiB/s 00:00 [#############################################################] 100%
libavif-1.0.4-1-aarch64 132.0 KiB 2.30 MiB/s 00:00 [#############################################################] 100%
libsecret-0.21.3-1-aarch64 171.0 KiB 2.74 MiB/s 00:00 [#############################################################] 100%
raspberrypi-utils-20240204-1-aarch64 120.0 KiB 2.02 MiB/s 00:00 [#############################################################] 100%
expat-2.6.0-1-aarch64 103.1 KiB 1842 KiB/s 00:00 [#############################################################] 100%
licenses-20240206-1-any 102.2 KiB 1793 KiB/s 00:00 [#############################################################] 100%
libxext-1.3.6-1-aarch64 102.1 KiB 1760 KiB/s 00:00 [#############################################################] 100%
alsa-ucm-conf-1.2.11-1-any 87.7 KiB 1950 KiB/s 00:00 [#############################################################] 100%
libnghttp3-1.1.0-2-aarch64 70.9 KiB 1126 KiB/s 00:00 [#############################################################] 100%
libusb-1.0.27-1-aarch64 68.9 KiB 1148 KiB/s 00:00 [#############################################################] 100%
raspberrypi-bootloader-20240208-1-any 4.3 MiB 2.28 MiB/s 00:02 [#############################################################] 100%
libsysprof-capture-45.2-1-aarch64 46.1 KiB 782 KiB/s 00:00 [#############################################################] 100%
python-pytz-2024.1-1-any 43.7 KiB 824 KiB/s 00:00 [#############################################################] 100%
libpciaccess-0.18-1-aarch64 20.1 KiB 373 KiB/s 00:00 [#############################################################] 100%
wireless-regdb-2024.01.23-1-any 10.2 KiB 178 KiB/s 00:00 [#############################################################] 100%
ffmpeg-2:6.1.1-5-aarch64 9.7 MiB 1314 KiB/s 00:08 [#############################################################] 100%
Total (33/33) 32.0 MiB 4.19 MiB/s 00:08 [#############################################################] 100%
(33/33) checking keys in keyring [#############################################################] 100%
(33/33) checking package integrity [#############################################################] 100%
(33/33) loading package files [#############################################################] 100%
(33/33) checking for file conflicts [#############################################################] 100%
(33/33) checking available disk space [#############################################################] 100%
:: Processing package changes...
( 1/33) upgrading alsa-ucm-conf [#############################################################] 100%
( 2/33) upgrading linux-api-headers [#############################################################] 100%
( 3/33) upgrading tzdata [#############################################################] 100%
( 4/33) upgrading alsa-lib [#############################################################] 100%
( 5/33) installing libnghttp3 [#############################################################] 100%
( 6/33) upgrading curl [#############################################################] 100%
( 7/33) upgrading expat [#############################################################] 100%
( 8/33) upgrading libjxl [#############################################################] 100%
( 9/33) upgrading libxext [#############################################################] 100%
(10/33) upgrading libxml2 [#############################################################] 100%
(11/33) upgrading libusb [#############################################################] 100%
(12/33) upgrading libelf [#############################################################] 100%
(13/33) upgrading libpciaccess [#############################################################] 100%
(14/33) upgrading sdl2 [#############################################################] 100%
(15/33) installing vapoursynth [#############################################################] 100%
(16/33) upgrading libsysprof-capture [#############################################################] 100%
(17/33) upgrading hwdata [#############################################################] 100%
(18/33) installing prrte [#############################################################] 100%
Optional dependencies for prrte
prrte-docs: for documentation
(19/33) upgrading openmpi [#############################################################] 100%
(20/33) upgrading fftw [#############################################################] 100%
(21/33) upgrading ffmpeg [#############################################################] 100%
(22/33) upgrading libavif [#############################################################] 100%
(23/33) upgrading libgit2 [#############################################################] 100%
(24/33) upgrading libsecret [#############################################################] 100%
(25/33) upgrading licenses [#############################################################] 100%
(26/33) upgrading mpfr [#############################################################] 100%
(27/33) upgrading pacman [#############################################################] 100%
(28/33) upgrading python-pydantic-core [#############################################################] 100%
(29/33) upgrading python-pydantic [#############################################################] 100%
(30/33) upgrading python-pytz [#############################################################] 100%
(31/33) upgrading raspberrypi-bootloader [#############################################################] 100%
(32/33) upgrading raspberrypi-utils [#############################################################] 100%
(33/33) upgrading wireless-regdb [#############################################################] 100%
:: Running post-transaction hooks...
(1/7) Reloading system manager configuration...
(2/7) Reloading device manager configuration...
(3/7) Arming ConditionNeedsUpdate...
(4/7) Updating the MIME type database...
(5/7) Updating linux initcpios...
==> Building image from preset: /etc/mkinitcpio.d/linux-aarch64.preset: 'default'
==> Using configuration file: '/etc/mkinitcpio.conf'
-> -k 6.2.10-1-aarch64-ARCH -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: '6.2.10-1-aarch64-ARCH'
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [autodetect]
-> Running build hook: [modconf]
-> Running build hook: [kms]
-> Running build hook: [keyboard]
-> Running build hook: [keymap]
-> Running build hook: [consolefont]
==> WARNING: consolefont: no font found in configuration
-> Running build hook: [block]
-> Running build hook: [filesystems]
-> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: '/boot/initramfs-linux.img'
==> WARNING: errors were encountered during the build. The image may not be complete.
==> Building image from preset: /etc/mkinitcpio.d/linux-aarch64.preset: 'fallback'
==> Using configuration file: '/etc/mkinitcpio.conf'
-> -k 6.2.10-1-aarch64-ARCH -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
==> Starting build: '6.2.10-1-aarch64-ARCH'
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [modconf]
-> Running build hook: [kms]
==> WARNING: Possibly missing firmware for module: 'ast'
==> WARNING: Possibly missing firmware for module: 'msm'
-> Running build hook: [keyboard]
-> Running build hook: [keymap]
-> Running build hook: [consolefont]
==> WARNING: consolefont: no font found in configuration
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: 'qla1280'
==> WARNING: Possibly missing firmware for module: 'qla2xxx'
==> WARNING: Possibly missing firmware for module: 'wd719x'
-> Running build hook: [filesystems]
-> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: '/boot/initramfs-linux-fallback.img'
==> WARNING: errors were encountered during the build. The image may not be complete.
error: command failed to execute correctly
(6/7) Probing GDK-Pixbuf loader modules...
(7/7) Updating the info directory file...
[musikolo@my-rpi4 ~]$
[/code]

In an attempt to understand the root cause, I rebuild all kernet images in debug mode using "mkinitcpio -v -P", but unfortunately I didn't find any errors as you can see atmake_clickable_callback(MAGIC_URL_FULL, ' ', 'https://pastebin.com/g3QzRiwg', '', ' class="postlink"') The problem I have now is that my rpi4 is no longer booting. I'm now unsure on how to fix because I don't understand what the root cause was. If anyone can point me out to sort it out that would be much appreciated.

Thanks.

[NeoSeppl]

Same here...

[Musikolo]

Hi,

I found a way to restore the damaged files of my rpi4 and booting again. Since my device is configured to boot from an external USB drive, I plugged it in and mounted it on my PC in a directory I'll call $RPI4_MOUNT_POINT. Then, I extracted the content of the base installation filemake_clickable_callback(MAGIC_URL_FULL, ' ', 'http://os.archlinuxarm.org/os/ArchLinuxARM-rpi-aarch64-latest.tar.gz', '', ' class="postlink"') in a directory I'll call $ArchLinuxARM-rpi-aarch64-latest. From there I copied the *.elf, *.dat and *.bin on the rpi4 /boot directory. These files belong to the raspberrypi-bootloader package. I also needed to restore the initramfs-linux.img and initramfs-linux-fallback.img files from the base installation.

You can do all that with the following 2 commands:
[code]
cp $ArchLinuxARM-rpi-aarch64-latest/boot/initramfs-linux*.img $RPI4_MOUNT_POINT/boot/
cp $ArchLinuxARM-rpi-aarch64-latest/boot/*.{elf,dat,bin} $RPI4_MOUNT_POINT/boot/
[/code]

After rebooting, I found my rpi4 up and running again. However, I didn't have time to fully understand what's wrong yet. The only thing I could confirm is that with the lastest raspberrypi-bootloader package (20240204-1), my device doesn't boot anymore. If I plug it in a monitor I get a pallete of colors printed on the screen and nothing else. Using the previous raspberrypi-bootloader package (20240122-1) it works well though.

On top of that, I see mkinitcpio continues to fail. Since I don't have much room left on my /boot partition (although it should be enough), I tried rebuilding the inital ramdisk on a temporary location that's plenty of room, but I still see the same error:
[code]
[root@my-rpi4 test2]# mkinitcpio -g /tmp/test2/file.img
==> Starting build: '6.2.10-1-aarch64-ARCH'
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [autodetect]
-> Running build hook: [modconf]
-> Running build hook: [kms]
-> Running build hook: [keyboard]
-> Running build hook: [keymap]
-> Running build hook: [consolefont]
==> WARNING: consolefont: no font found in configuration
-> Running build hook: [block]
-> Running build hook: [filesystems]
-> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: '/tmp/test2/file.img'
==> WARNING: errors were encountered during the build. The image may not be complete.
[root@my-rpi4 test2]#
[/code]

The generated image file is corrupt and break the boot system. I'm still investigating a way to to build a healthy initial ramdisk on my device again.

I hope that helps some of you in the while the root cause if identified and corrected.

Thanks.

[gontzalm]

Same here... Thanks for the temporal fix @Musikolo, it worked for me .

[KwaXi]

Some issue here, downgraded the raspberrypi-bootloader package, found the old package in the wayback archive linked

[url]https://archlinuxarm.org/forum/viewtopic.php?f=8&t=15698[/url]

[Musikolo]

Hi KwaXi,

I appreciate your comments because I think you completely nailed it down. I just rerun mkinitcpio and, despite the errors it showed, I rebooted my device and it worked well.

You see what packages I have installed and what I did below:
[code]
[root@my-rpi4 ~]# pacman -Su
:: Starting full system upgrade...
warning: raspberrypi-bootloader: ignoring package upgrade (20240122-1 => 20240208-1)
there is nothing to do
[root@my-rpi4 ~]#
[root@my-rpi4 ~]#
[root@my-rpi4 ~]# pacman -Qs raspberrypi
local/firmware-raspberrypi 20231022-1
Additional firmware for Raspberry Pi
local/raspberrypi-bootloader 20240122-1
Bootloader files for Raspberry Pi
local/raspberrypi-utils 20240204-1
Legacy scripts and simple applications for Raspberry Pi
local/uboot-raspberrypi 2021.04-1
U-Boot for Raspberry Pi
[root@my-rpi4 ~]#
[root@my-rpi4 ~]#
[root@my-rpi4 ~]# mkinitcpio -P
==> Building image from preset: /etc/mkinitcpio.d/linux-aarch64.preset: 'default'
==> Using configuration file: '/etc/mkinitcpio.conf'
-> -k 6.2.10-1-aarch64-ARCH -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: '6.2.10-1-aarch64-ARCH'
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [autodetect]
-> Running build hook: [modconf]
-> Running build hook: [kms]
-> Running build hook: [keyboard]
-> Running build hook: [keymap]
-> Running build hook: [consolefont]
==> WARNING: consolefont: no font found in configuration
-> Running build hook: [block]
-> Running build hook: [filesystems]
-> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: '/boot/initramfs-linux.img'
==> WARNING: errors were encountered during the build. The image may not be complete.
==> Building image from preset: /etc/mkinitcpio.d/linux-aarch64.preset: 'fallback'
==> Using configuration file: '/etc/mkinitcpio.conf'
-> -k 6.2.10-1-aarch64-ARCH -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
==> Starting build: '6.2.10-1-aarch64-ARCH'
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [modconf]
-> Running build hook: [kms]
==> WARNING: Possibly missing firmware for module: 'ast'
==> WARNING: Possibly missing firmware for module: 'msm'
-> Running build hook: [keyboard]
-> Running build hook: [keymap]
-> Running build hook: [consolefont]
==> WARNING: consolefont: no font found in configuration
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: 'qla1280'
==> WARNING: Possibly missing firmware for module: 'qla2xxx'
==> WARNING: Possibly missing firmware for module: 'wd719x'
-> Running build hook: [filesystems]
-> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: '/boot/initramfs-linux-fallback.img'
==> WARNING: errors were encountered during the build. The image may not be complete.
[elduro@my-rpi4 ~]$
[/code]

So you're right, this confirms the only thing breaking boot on my RPi4 is the latest raspberrypi-bootloader package. I hope a fix is found soon!

Thank you so much for helping clarify and unravel the issue.

Regards.

[jeancf]

I have a raspberry Pi 3+ running 6.2.10-1-aarch64-ARCH. After the update last week it failed to boot and remained stuck on the multi-color screen. I figured that it was related to the bootloader and this update in particular:

[ALPM] upgraded raspberrypi-bootloader (20240122-1 -> 20240208-1)

I reverted to 20240122-1 and the device booted again.

[flooklab]

I have the same problem with my Pi 3 B.
Reverting solved it for now.

Edit:
See also here [url]https://archlinuxarm.org/forum/viewtopic.php?f=15&t=16753[/url]

[jeancf]

Thanks for the link flooklab.

For the record, to restore the bootloader I copied the files inside the rapberrypi-bootloader-20240122-1 package found under /var/cache/pacman/pkg/ to the /boot partition and rebooted. After that, I aligned the pacman database with the files on disk with pacman -U rapberrypi-bootloader-20240122-1 (i.e. installing the same files but 'officially' this time) and blacklisted rapberrypi-bootloader in /etc/pacman.conf for now to avoid further issues.

[darksky]

Just an observation being on the outside looking in. The 6.2.10 kernel is pretty old. Maybe the devices's .dtb is not compatible with the new .dat/.elf files.

I compiled the latest 6.8-rc4 upstream kernel today to test and I had no issues booting it with the latest raspberrypi-bootloader packages.

[ray@kde ~]$ pacman -Ss raspberrypi-bootloader
core/raspberrypi-bootloader 20240208-1 [installed]
Bootloader files for Raspberry Pi
core/raspberrypi-bootloader-x 20240208-1 [installed]
Bootloader with extra codecs for Raspberry Pi

[ray@kde ~]$ inxi -Fxz
System:
Kernel: 6.8.0-rc4-1-MANJARO-ARM-RC arch: aarch64 bits: 64 compiler: gcc
v: 12.1.0 Desktop: KDE Plasma v: 5.27.10 Distro: Manjaro ARM
base: Arch Linux
Machine:
Type: ARM System: Raspberry Pi 4 Model B Rev 1.4 details: N/A
serial: <filter>
CPU:
Info: quad core model: N/A variant: cortex-a72 bits: 64 type: MCP
arch: ARMv8 rev: 3 cache: L1: 320 KiB L2: 1024 KiB
Speed (MHz): avg: 600 min/max: 600/2200 cores: 1: 600 2: 600 3: 600 4: 600
bogomips: 432
Features: Use -f option to see features
Graphics:
Device-1: bcm2711-hdmi0 driver: vc4_hdmi v: N/A bus-ID: N/A
Device-2: bcm2711-hdmi1 driver: vc4_hdmi v: N/A bus-ID: N/A
Device-3: 2711-v3d driver: v3d v: kernel bus-ID: N/A
Device-4: bcm2711-vc5 driver: vc4_drm v: N/A bus-ID: N/A
Display: wayland server: X.org v: 1.21.1.11 with: Xwayland v: 23.2.4
compositor: kwin_wayland driver: X: loaded: modesetting dri: vc4
gpu: vc4_hdmi,vc4_hdmi,v3d,vc4_drm resolution: 1440x900
API: EGL v: 1.4,1.5 drivers: swrast,v3d,vc4 platforms:
active: gbm,wayland,x11,surfaceless,device inactive: N/A
API: OpenGL v: 4.5 compat-v: 3.1 vendor: broadcom mesa v: 23.3.2-arch.2
glx-v: 1.4 direct-render: yes renderer: V3D 4.2
API: Vulkan v: 1.3.276 drivers: v3dv,llvmpipe surfaces: xcb,xlib,wayland
devices: 2
...
...