Help Wanted Solving Kernel Panic

[robg]

Please consider merging future updates on your issue in the first post—people [i]will[/i] notice, and likely will be more willing to help than when seeing a thread bumped multiple times.

Now, as for your issue: Glancing over the screenshot you linked to, it appears that the issue occurs when writing to a swap file/partition on the NVME. It's worth to confirm this suspicion (supported further by the fact that you didn't find anything wrong with it in the hardware tests.) Pick a memory test suite from [url=https://wiki.archlinux.org/title/Stress_testing]this selection[/url] and try it with swap on 1) a USB-connected SSD and 2) a PCI-connected NVME. Report back if you experience any issues with 2).

[ecod00m]

[quote="robg"]Please consider merging future updates on your issue in the first post—people [i]will[/i] notice, and likely will be more willing to help than when seeing a thread bumped multiple times.[/quote]

Have adjusted. I guess it is up to mod to merge/delete if desired.

[quote="robg"]Now, as for your issue: Glancing over the screenshot you linked to, it appears that the issue occurs when writing to a swap file/partition on the NVME. It's worth to confirm this suspicion (supported further by the fact that you didn't find anything wrong with it in the hardware tests.) Pick a memory test suite from [url=https://wiki.archlinux.org/title/Stress_testing]this selection[/url] and try it with swap on 1) a USB-connected SSD and 2) a PCI-connected NVME. Report back if you experience any issues with 2).[/quote]

This is a raspberry pi 4. It does not have an option 2. What is it that you hope to observe by these tests? In addition, note that the issue occurs at the same time as a USB disconnect, which is more likely the problem root. These problems were non-existent last month, so I have rolled back to the prior kernel.

[robg]

Thank you for merging the information, it's definitely more accessible now.

Returning to your issue: You are right, of course—option 2) on a RPI4 is no option at all. My apologies. I was hoping to narrow down your issue a little more, but as is practically confirmed by the working rolled-back kernel, it appears to be a regression in the latter. I believe you have gathered enough evidence to raise your issue here:make_clickable_callback(MAGIC_URL_FULL, ' ', 'https://github.com/raspberrypi/linux/issues', '', ' class="postlink"') (mention that you are using the RPI kernel, despite running ALARM.)

Note that various IO issues were reported as recently as October:
make_clickable_callback(MAGIC_URL_FULL, '
', 'https://github.com/raspberrypi/linux/issues/6413', '', ' class="postlink"')make_clickable_callback(MAGIC_URL_FULL, '
', 'https://github.com/raspberrypi/linux/issues/6351', '', ' class="postlink"')make_clickable_callback(MAGIC_URL_FULL, '
', 'https://github.com/raspberrypi/linux/issues/6349', '', ' class="postlink"')

[ecod00m]

Thank you for the leads. It is most appreciated. Seeing the sheer volume of open and potentially related bugs has me reconsidering the role of the hardware in the network system as a whole. While it has been a reliable and robust cornerstone for years, issues stretching back as far as July for this particular configuration (external rootfs), has me considering other options. While I could (and will, for the time being) revert to a MMC rootfs, and suffer the performance penalty, I am looking into the possibility of upgrading hardware for the role to an entry-level NUC. A shame, really, as I know how capable and reliable the hardware can be, but in this role, stability and robustness of all aspects are crucial - and that extends to the kernel, its developers, and support. Plus, I am far too comfortable with Arch Linux and how much work I've put in to migrate *everything* to it, to use anything else.

Here's another interesting clue to the puzzle: https://github.com/raspberrypi/linux/issues/6260

(Issue is still occurring despite rolled back Kernel and I am reluctant to roll it back further. I've been battling this bug for the better part of a month and it is standing in the way of many other projects and tasks)

[robg]

I'm glad that I could at least offer some helpful pointers. For the benefit of all RPI users, I would suggest that you report the issue upstream (maybe even try replicating it on Raspberry Pi OS). With a little luck, you may even avoid purchasing that NUC...