Regularly hangs when updating

This forum is for topics dealing with problems with software specifically in the AArch64 repo.

Regularly hangs when updating

Postby justinkb » Thu Mar 17, 2022 7:53 pm

First I thought it was just a fluke, but I've now had this happen twice in eight days.

My pi completely and instantly locks up when updating / having updated sometimes. This happened twice now when updating the bootloader and/or kernel files, so afterwards the pi wouldn't come back up since the needed files for proper booting were 0 bytes, since the filesystem hasn't had time to flush buffers properly before the issue occurs.

Putting this here so maybe others can chime in whether this has been an issue for them recently.

My boot partition is vfat on an SSD. Is it possible the 5.15 vfat driver has issues? This never happened on the 5.10 branch.
justinkb
 
Posts: 15
Joined: Tue Mar 16, 2021 11:06 am

Re: Regularly hangs when updating

Postby jakethedog » Fri Mar 18, 2022 4:05 am

I have observed the same behaviour I'm also booting from an USB SSD.
The kernel crashes started recently and I first accredited this to power instabilities, but removed now everything except the USB SSD and it is still happening.
It happens some times sporadically but mostly when performing updates via pacman often leaving 0 byte files.
The kernel crash shows something related to UAS (the interface used to the usb drive).
Attachments
54FEE454-4B5D-4C6D-B2E9-7616DADA607B.jpeg
Crash log
54FEE454-4B5D-4C6D-B2E9-7616DADA607B.jpeg (94.17 KiB) Viewed 3137 times
jakethedog
 
Posts: 11
Joined: Sat Jul 19, 2014 11:33 am

Re: Regularly hangs when updating

Postby jakethedog » Fri Mar 18, 2022 10:08 am

Could be related to the following issues:

Raspberry Pi 4 : USB3 SSD Connected via USB3 Hub : BOOT Files Corrupted Following Reboot
https://github.com/raspberrypi/linux/issues/4844

Kernels 5.15.26/5.16.12 64bit usb ssd issues:
https://github.com/raspberrypi/linux/issues/4930
jakethedog
 
Posts: 11
Joined: Sat Jul 19, 2014 11:33 am

Re: Regularly hangs when updating

Postby justinkb » Fri Mar 18, 2022 10:56 am

thanks for that photograph, I had no way to get the kernel panic stacktrace since I had no physical access to the screen output of the device
justinkb
 
Posts: 15
Joined: Tue Mar 16, 2021 11:06 am

Re: Regularly hangs when updating

Postby idovitz » Fri Mar 18, 2022 12:38 pm

I confirm it, write to /boot vfat freezes system.

ARM archlinux
TP-Link USB 3.1 Gen1 Hub - Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
NVME to usb3.1 - Bus 002 Device 004: ID 152d:0562 JMicron Technology Corp. / JMicron USA Technology Corp. JMS567 SATA 6Gb/s bridge

For now before an update, I umount boot and mount --bind a ext4 copied directory of /boot. Then i shutdown and sync on my laptop the ext4 boot dir with the vfat boot partition.
Sadly RPI wont boot from ext4 boot partition, tried that.
idovitz
 
Posts: 2
Joined: Thu Dec 02, 2021 10:51 pm

Re: Regularly hangs when updating

Postby darksky » Fri Mar 18, 2022 1:37 pm

I believe you will find that this is the issue on all of the newer 5.15/5.16/5.17 kernels.

https://github.com/raspberrypi/linux/issues/4930

Once booted into these kernels after installing them I will get this issue every time. The short term fix is to recover the lost data. I just cherry pick the /boot/ and /usr/lib/modules/ from the new kernel package and copy them over to the usb drive on my x86 and put total_mem=3072 in config.txt before booting up. I have made a patch to revert their commits on all of these kernels until they get it fixed so the memory does not have to be limited in config.txt.

https://gitlab.manjaro.org/manjaro-arm/packages/core/linux-rpi4/-/blob/master/xhci-revert.diff

Side note: I do not seem to have this issue when booting into vanilla upstream AARCH64 kernel.
darksky
 
Posts: 35
Joined: Thu Oct 15, 2020 5:57 pm

Re: Regularly hangs when updating

Postby justinkb » Fri Mar 18, 2022 8:13 pm

not sure about that memory limiting thing, I got this bug while my pi 4 has only 2GB mem
justinkb
 
Posts: 15
Joined: Tue Mar 16, 2021 11:06 am

Re: Regularly hangs when updating

Postby darksky » Fri Mar 18, 2022 9:06 pm

Reading over what the RPi people told me about my 8G board I got to wondering if the memory limits are even less with your 2G board.
darksky
 
Posts: 35
Joined: Thu Oct 15, 2020 5:57 pm

Re: Regularly hangs when updating

Postby justinkb » Fri Mar 18, 2022 9:11 pm

ah, so for your 8G B0, 5 out of 8 gb would be inaccessible to the usb chipset, and for mine, for example 1280MB out of 2GB, makes sense

ps. devmem returned the same for me as for you
justinkb
 
Posts: 15
Joined: Tue Mar 16, 2021 11:06 am

Re: Regularly hangs when updating

Postby jakethedog » Tue Apr 05, 2022 5:45 am

I just manually updated to the latest kernel (5.15.32-3-rpi-ARCH armv7l) using an external system and it seems to be stable now.
Still keeping an eye on it but already many thanks to darksky for investigating and working on it.
jakethedog
 
Posts: 11
Joined: Sat Jul 19, 2014 11:33 am


Return to ARMv8

Who is online

Users browsing this forum: No registered users and 6 guests