rpi3b lock up over ssh on intensive I/O ?

[zebul666]

hi.
Lately, I am encountering problem with one of my rpi3b+ on aarch64. It has a SSD on USB. Boots from sd-card.

It just locks up, and becomes unresponsive. I can still ping it, but can't ssh to it anymore, and the current ssh session is dead/unresponsive.

I can't find anything in the log. A simple command like `pacman -F libcares` seems to trigger the problem.

I have tried the SSD from a systemd-nspawn from a x86_64, and did not see the lockup. So I am, for the moment ruling out the SSD.

I have to turn it off and on again, each time.

[koogas]

I'm experiencing similar stuff on a Pi 4B.

Boot takes a LONG time and displays some errors (kworker blocked for more than 120 seconds).

Just a simple system upgrade takes a long time as well. The system becomes unusable as well.

$this->bbcode_second_pass_code('', '
jun 04 15:06:28 alarmpi kernel: INFO: task kworker/2:3:275 blocked for more than 241 seconds.
jun 04 15:06:28 alarmpi kernel: Tainted: G C 5.15.40-3-rpi-ARCH #1
jun 04 15:06:28 alarmpi kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
jun 04 15:06:28 alarmpi kernel: task:kworker/2:3 state:D stack: 0 pid: 275 ppid: 2 flags:0x00000208
jun 04 15:06:28 alarmpi kernel: Workqueue: events_freezable mmc_rescan
jun 04 15:06:28 alarmpi kernel: Call trace:
jun 04 15:06:28 alarmpi kernel: __switch_to+0x110/0x164
jun 04 15:06:28 alarmpi kernel: __schedule+0x2f4/0x914
jun 04 15:06:28 alarmpi kernel: schedule+0x44/0xfc
jun 04 15:06:28 alarmpi kernel: __mmc_claim_host+0xb4/0x1e0
jun 04 15:06:28 alarmpi kernel: mmc_get_card+0x34/0x44
jun 04 15:06:28 alarmpi kernel: mmc_sd_detect+0x20/0x90
jun 04 15:06:28 alarmpi kernel: mmc_rescan+0x8c/0x2ec
jun 04 15:06:28 alarmpi kernel: process_one_work+0x1fc/0x4b0
jun 04 15:06:28 alarmpi kernel: worker_thread+0x13c/0x470
jun 04 15:06:28 alarmpi kernel: kthread+0x138/0x150
jun 04 15:06:28 alarmpi kernel: ret_from_fork+0x10/0x20
')

[krabat]

Running the "mainline" (kernel.org) kernel as provided by package linux-aarch64 on a Raspberry Pi 3B (no plus) I'm facing similar issues.
The SD card is slow to a degree making the whole system unusable, even simple tasks like package updates are tainting the kernel.
It is definitely a regression in 5.18.n, though.

Maybe you can try and verify this on your systems?
In case you should do so with a monitor attached please note there's another problem in 5.18.x which is affecting module vc4 and may make screens turn black (workaround is blacklisting the module).

koogas:
You seem to be running the RPi Foundation kernel. Maybe you can check which version of it was affected first and/or whether the latest version is still affected (even if updating the kernel tends to be a bit nerv wrecking right now)?

[lategoodbye]

I'm interested in analyzing this lock up, but i need a scenario which doesn't depend on Arch Linux parts.

Do i need a SSD or is it reproducible with SD card?
Is it possible to trigger with common Linux tools like dd or fio or etc?
How long does it take to lock up?

[krabat]

SD cards are definitely affected by the problem. As I understand the other posts in this thread, storage devices connected in other ways like SSDs are not.

Interestingly, writing large amounts of data in one go does not trigger the problem. E. g. $this->bbcode_second_pass_code('', '$ dd bs=1M count=300 if=/dev/urandom of=dummy.bin') will finish in a couple of seconds without any issue.
But the problem can be triggered reliably by writing a large number of objects. Downloading the latest package linux-aarch64 and unpacking by $this->bbcode_second_pass_code('', '$ bsdtar -xpf linux-aarch64-5.18.1-1-aarch64.pkg.tar.xz') should make it manifest at your side, too.

[lategoodbye]

Yes, i also identified the kernel unpacking scenario, but "hanging" isn't completely unavoidable.
The SDHCI is a small bottleneck to the SD card and a filesystem like ext4 make the situation even worse.

Is it correct that the swap is enabled and also located on the SD card?

But a hung task warning is something which should not happen. @Krabat this warning doesn't occur in your case?

[krabat]

Assuming that by "hanging" you mean "takes pretty long" or so, it is not something that comes as surprise on SD cards or I for one would worry about. This even applies to those "hung task" messages, too.
But while running appropriate operations like unpacking a complex archive file the running kernel starts getting tainted again and again after a while. And this is something I do care about.

Yes, I do have a swap partition on the SD card.
Tbh. I'm not even sure whether it makes any sense on a rather slow device like that. I once established it mostly "just in case" in lights of the device's restricted amount of memory.

Still, what ever the findings are considered as, there's definitely a massive regression in 5.18.x.
Before, things wouldn't run at lightning speed either, but neither were hung task messages an issue nor was the kernel tainted.
I do remember that similar issues did exist under the "mainline" kernel quite a while ago. But I never bothered looking into these as much more complex tasks than currently were needed to trigger the problem and as it ceased some kernel version anyway.

[lategoodbye]

Okay, i tested with a Raspbian Rootfs and Mainline kernel/DTB (multi_v7_defconfig) on a Raspberry Pi 3 Plus. But i wasn't able to trigger these hung task messages. I will try to look if arm64 on RPi3 without B makes any difference.

I tested Linux 5.17.14 and 5.18, both of them show this ugly hanging. Are you connected via SSH, HDMI or UART?

What is the latest kernel version which didn't have the lock up?

[krabat]

I first ran into this in an SSH session via WLAN. Further tests were made after attaching keyboard and monitor, the latter either DVI with HDMI-DVI adapter or HDMI.
All the same with regards to the problem we're discussing here.

The last version to work well on Arch Linux ARM was 5.17.1. This was the last point release of the 5.17.x series before the distro switched to 5.18.x.

[lategoodbye]

tbh there is a regression fix which has been applied last night, i don't know how long it takes until it goes into the 5.18.y tree:
https://git.kernel.org/pub/scm/linux/ke ... 1b680d1cd3

But i think it's unlikely to fix your problem.