Arch Linux ARM v7 kernel/bootcode update on Raspberry Pi 3 i

[aroberts]

Yes just the standard kernel.

$this->bbcode_second_pass_code('', '
pacman -Qs kernel
local/iptables 1.6.1-1
Linux kernel packet control tool
local/kmod 24-1
Linux kernel module management tools and library
local/libdrm 2.4.81-1
Userspace interface to kernel DRM services
local/linux-api-headers 4.10.1-1
Kernel headers sanitized for use in userspace
local/linux-raspberrypi 4.9.38-1
The Linux Kernel and modules - Raspberry Pi
local/mtdev 1.1.5-1
A stand-alone library which transforms all variants of kernel MT events to
the slotted type B protocol
')

I've done some major rebooting and can reproduce the issue still.

1=ok, x=fail

No HDMI Monitor: 11x1x
HDMI Monitor: 111111
No HDMI Monitor: 111x
All affected PI's have Piface Digital cards attached (hence SPI in config.txt)
I removed this in case is was causing issue.
No HDMI Monitor: 1x (so still happens without Piface, but SPI still enabled in config.txt)
HDMI Monitor: 11111
No HDMI Monitor: x111xxx
Monitor: 11

This seems worse than last time on 7th Jul. Looks like it always boots ok with monitor attached.
When ethernet doesn't come up, neither does the USB (this machine has a USB audio card with a light on it).
But without monitor its kind of hard to see what's happening...

Regards

Andrew

[graysky]

Let's minimize the variables.

1) Take a basic RPi3 that is currently causing these problems for you but remove special hardware so it's just a stock board. Connect ethernet. No HDMI, no USB anything so it's completely headless.
2) Install a fresh Arm ARM image for RPi2 (not aarch64). Use a spare sd card so you don't lost a live system.
3) Boot into it, update to the latest packages and repeat your string of reboots.

I have several Pis here and will do the same except my Pis have HDMI connected to a powered off monitor and are not stripped down (they have systemd services enabled to run). I will post the results shortly... conducting now.

[graysky]

All of these are on the linux-raspberrypi kernel package (4.9.38-1-ARCH currently):

$this->bbcode_second_pass_code('', '1=ok, x=fail
RPi3: 11111111
RPi2: 11111111
RPi2: 11111x11')

The 6th time on one of my boxes, the reboot did not come back up.

EDIT: seems to be kernel panic. I will post a pic and investigate further.

[aroberts]

OK will do, this will take a while.
I did also try commenting out the enabling of SPI in config.txt and removing all hardware.
That didn't help.

Andrew

[graysky]

Click for a larger view:

[aroberts]

Ok I have created a fresh 32bit Pi image with latest updates, on same PI as yesterday.
This includes an updated kernel from that posted yesterday:
4.9.39-1

I'm still seeing the issues booting headless: (1=ok, x=failure)
HDMI+USB Keyboard: 11111 11111
Headless: x11x1 1xxxx
USB Keyboard only: 11x
Keyboard+HDMI: 11111 11111

No other hardware attached.

I also rebooted my other 5 headless 32 bit pi's due to kernel update, and one failed to come back up (again a PI 3).

Having HDMI attached even if the monitor is displaying another (display port) input seems to work in all cases. I've not been
able to reproduce a kernel crash on screen.

When the system fails to boot there is no journal log to look at either.

Regards

Andrew

[graysky]

What about booting it like I have with HDMI connected, but monitor off? Are you able to reproduce the panic I posted? The nice thing about HDMI connected/monitor off is that you can switch the monitor on and see the output.

EDIT: the kernel panic I got might be unrelated since you said the network just goes dead after some time. I too can confirm the random network breakage.

[aroberts]

Ok,

I tried with a different monitor (not shared with another machine). With the monitor off, I can reproduce the crash. But when the monitor is switched back on, there is no signal.

I then had the bright idea of using a serial console to capture what was happening.

I edited /boot/config.txt and added:
enable_uart=1
and
/boot/cmdline.txt and inserted
console=ttyS0,115200 kgdboc=ttyS0,115200
before console=tty1

In this configuration it didn't boot. I powered off and on with a monitor attached and got a kernel crash dump
https://postimg.org/image/f6v9puz7x/
It turns out this crash is related to:
kgdboc=ttyS0,115200

with this removed but the console=ttyS0,115200 and enable_uart=1 it boots fine. This isn't related to the issue I was having, but something is not right here as this used to work.

With the serial console running, I was not able to reproduce the crash I was having, which again seems to show small changes in the hardware from the kernel config/timing allow things to work ok.

[aroberts]

A further update. After messing about with the serial console, I can't reproduce the problem on that SD card at the momemt. Despite having disabled the serial console and the uart.
Switching back to a clean card on the same PI, the problem reappears immediately.

So this would suggest there is some persistent state relating to the card. Device tree or /dev or something?
This might also explain why kernel/boot code updates trigger it initially if they cause this state to be altered.

Regards

Andrew

[bulletmark]

I'm watching this thread with interest. I am running a RPi2 with a PiFace board (i.e. SPI enabled) which I log in to via ssh every few days, update, and then reboot. It serves as my home alarm system and has been running for a few years. On Friday, I watched it update with new linux-raspberrypi and raspberrypi-bootloader packages (and little else). I rebooted immediately after the update and it never came back up. A power off reset does not recover it and neither does booting without the network cable. This could of course just be a sudden SD card/hardware failure but seems coincidental given this thread.