ODROID-C2 crashes on heavy EMMC activity

[Crazyachmed]

Hi guys!

My odroid-c2 shits itself every time there is some heavy disk activity. It doesn't even take very long, some two or three minutes at most.

I managed to pull the dmesg output this one time, because I still had second screen window open that somewhat continued to work. I could not restart it using "reboot -f" and had to pull the powercord.

This time I tried to backup the system using tar directly to a mounted CIFS share. I already had disabled the gzip compression for the tar, because it actually has better throughput. The CPU usage was minimal, so I think it has to do with disk, not CPU.

Any thoughts?

$this->bbcode_second_pass_code('', '[35189.978601] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected by 2, t=2102 jiffies, g=10000, c=9999, q=92)
[35189.983799] Task dump for CPU 3:
[35189.986989] kthread_hdcp R running task 0 81 2 0x00000002
[35189.993975] Call trace:
[35189.996399] [<ffffffc0010855b4>] __switch_to+0x6c/0x88
[35253.027841] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected by 2, t=8407 jiffies, g=10000, c=9999, q=126)
[35253.033127] Task dump for CPU 3:
[35253.036317] kthread_hdcp R running task 0 81 2 0x00000002
[35253.043303] Call trace:
[35253.045727] [<ffffffc0010855b4>] __switch_to+0x6c/0x88
[35316.077055] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected by 1, t=14712 jiffies, g=10000, c=9999, q=316)
[35316.082428] Task dump for CPU 3:
[35316.085618] kthread_hdcp R running task 0 81 2 0x00000002
[35316.092605] Call trace:
[35316.095029] [<ffffffc0010855b4>] __switch_to+0x6c/0x88
[35379.126245] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected by 2, t=21017 jiffies, g=10000, c=9999, q=332)
[35379.131617] Task dump for CPU 3:
[35379.134808] kthread_hdcp R running task 0 81 2 0x00000002
[35379.141794] Call trace:
[35379.144218] [<ffffffc0010855b4>] __switch_to+0x6c/0x88
[35383.476261] systemd[1]: systemd-logind.service: State 'stop-sigabrt' timed out. Terminating.
[35383.479664] systemd[1]: systemd-journald.service: State 'stop-sigabrt' timed out. Terminating.
[35442.175418] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected by 0, t=27322 jiffies, g=10000, c=9999, q=350)
[35442.180790] Task dump for CPU 3:
[35442.183981] kthread_hdcp R running task 0 81 2 0x00000002
[35442.190966] Call trace:
[35442.193391] [<ffffffc0010855b4>] __switch_to+0x6c/0x88
[35473.735059] systemd[1]: systemd-logind.service: State 'stop-sigterm' timed out. Killing.
[35473.738057] systemd[1]: systemd-journald.service: State 'stop-sigterm' timed out. Killing.
[35473.747125] systemd[1]: systemd-logind.service: Main process exited, code=killed, status=9/KILL
[35473.755573] systemd[1]: systemd-logind.service: Unit entered failed state.
[35473.761144] systemd[1]: systemd-logind.service: Failed with result 'signal'.
[35473.768893] systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.
[35473.778541] systemd[1]: Stopped Login Service.
[35473.783353] systemd[1]: Starting Login Service...
[35473.792877] systemd[1]: Started Login Service.
[35494.774711] INFO: rcu_sched detected stalls on CPUs/tasks: { 3} (detected by 2, t=2102 jiffies, g=18446744073709551474,

c=18446744073709551473, q=5)
[35494.782319] Task dump for CPU 3:
[35494.785509] kthread_hdcp R running task 0 81 2 0x00000002
[35494.792495] Call trace:
[35494.794919] [<ffffffc0010855b4>] __switch_to+0x6c/0x88
[35505.224567] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected by 2, t=33627 jiffies, g=10000, c=9999, q=964)
[35505.229928] Task dump for CPU 3:
[35505.233118] kthread_hdcp R running task 0 81 2 0x00000002
[35505.240103] Call trace:
[35505.242522] [<ffffffc0010855b4>] __switch_to+0x6c/0x88
[35557.823851] INFO: rcu_sched detected stalls on CPUs/tasks: { 3} (detected by 1, t=8407 jiffies, g=18446744073709551474,

c=18446744073709551473, q=5)
[35557.831466] Task dump for CPU 3:
[35557.834655] kthread_hdcp R running task 0 81 2 0x00000002
[35557.841642] Call trace:
[35557.844065] [<ffffffc0010855b4>] __switch_to+0x6c/0x88
[35563.973833] systemd[1]: systemd-journald.service: Processes still around after SIGKILL. Ignoring.
[35568.273707] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected by 2, t=39932 jiffies, g=10000, c=9999, q=986)
[35568.279074] Task dump for CPU 3:
[35568.282265] kthread_hdcp R running task 0 81 2 0x00000002
[35568.289251] Call trace:
[35568.291675] [<ffffffc0010855b4>] __switch_to+0x6c/0x88
[35593.473428] systemd[1]: systemd-coredump@0-6615-0.service: Service reached runtime time limit. Stopping.
[35620.872977] INFO: rcu_sched detected stalls on CPUs/tasks: { 3} (detected by 1, t=14712 jiffies, g=18446744073709551474,

c=18446744073709551473, q=5)
[35620.880672] Task dump for CPU 3:
[35620.883863] kthread_hdcp R running task 0 81 2 0x00000002
[35620.890849] Call trace:
[35620.893273] [<ffffffc0010855b4>] __switch_to+0x6c/0x88
')

[AndyOfLinux]

I am seeing same problem. During filesystem writes (eMMC or USB HD) the C2 hangs. I have confirmed this on all 4 on my C2s. Tried different power supplies, etc. I believe this started in the last week. I remember doing 2 full system updates via paceman, 1 of which included a new kernel? So, somethings changed, lately

My MongoDB log shows a file allocation as last operation before crash:

2016-06-06T10:14:14.579-0700 I STORAGE [FileAllocator] allocating new datafile /mnt/USBdrive/mongodb/bldata.4, filling with zeroes...
2016-06-06T10:14:14.609-0700 I STORAGE [FileAllocator] done allocating datafile /mnt/USBdrive/mongodb/bldata.4, size: 1024MB, took 0.03 secs

- Andy

[AndyOfLinux]

OK, my issue was solved (worked-around) by reverting from headless mode. Setting "nographics 0" in boot.ini did the trick. I had to connect a monitor to the C2s to get them to boot reliably but once set back to enabling graphics, things became stable again -- even without a monitor connected. I have seen this ODROID C2 Forum thread and it looks like progress is being made on this issue - http://forum.odroid.com/viewtopic.php?f=136&t=21520

[Crazyachmed]

Yeah, it even works in headless mode now with a modified boot.ini posted there. I integrated the changes into the default one, because it didn't work out of the box.

I'm now using http://pastebin.com/SzvUPUPK and it gives me 1974MB of RAM.
(please note that I changed the root-partiton to "/dev/mmcblk0p2" because of my setup, replace it with "/dev/mmcblk0p1")

[amiga23]

Hi,

I do also get crashes with heavy eMMC usage. But I am not using the headless mode and the dmesg output is different.

It is the same issue with kernel version 3.14.65-17 and 3.14.77-1

I've reproduced this several times now.

It always happens when installing a lot of packages.

The filesystem is corrupt afterwards and the odroid does not even boot anymore, with blue LED flashing, black screen.

[ 462.816667] systemd-sysuser[5446]: syscall 285
[ 462.821874] Code: aa0503e4 aa0603e5 aa0703e6 d4000001 (b13ffc1f)
[ 462.827253] CPU: 2 PID: 5446 Comm: systemd-sysuser Not tainted 3.14.65-17-ARCH #1
[ 462.834693] task: ffffffc05bd38000 ti: ffffffc05bc04000 task.ti: ffffffc05bc04000
[ 462.842103] PC is at 0x7fa2d773e4
[ 462.846991] LR is at 0x7fa2ecaf9c
[ 462.851866] pc : [<0000007fa2d773e4>] lr : [<0000007fa2ecaf9c>] pstate: 60000000
[ 462.858556] sp : 0000007fe634a720
[ 462.863523] x29: 0000007fe634a720 x28: ffffffffffffffff
[ 462.871825] x27: 0000000000000001 x26: 0000000000000001
[ 462.877075] x25: 0000000000000001 x24: 00000000ffffffff
[ 462.882306] x23: 7fffffffffffffff x22: 0000000000000010
[ 462.887542] x21: 000000000000000f x20: 0000000000004000
[ 462.892792] x19: 0000005555581000 x18: 0000000000000000
[ 462.898012] x17: 0000007fa2d773c0 x16: 0000007fa3029420
[ 462.903264] x15: 000000002b169a40 x14: 000000000158b4d2
[ 462.908502] x13: 00000000537bcaf9 x12: 00000000000016f4
[ 462.913747] x11: 000000000000002c x10: 0000007fe634a6b0
[ 462.918979] x9 : 0000000000000000 x8 : 000000000000011d
[ 462.924234] x7 : 0000005555589410 x6 : 0000005555589410
[ 462.929479] x5 : 0000000000000000 x4 : 7fffffffffffffff
[ 462.934750] x3 : 0000000000000000 x2 : 0000000000000010
[ 462.939991] x1 : 0000000000000000 x0 : 000000000000000f

[ 464.569926] systemd-sysuser[5489]: syscall 285
[ 464.575116] Code: aa0503e4 aa0603e5 aa0703e6 d4000001 (b13ffc1f)
[ 464.580738] CPU: 2 PID: 5489 Comm: systemd-sysuser Not tainted 3.14.65-17-ARCH #1
[ 464.588052] task: ffffffc05bcaac00 ti: ffffffc05bc04000 task.ti: ffffffc05bc04000
[ 464.595554] PC is at 0x7fa62243e4
[ 464.600453] LR is at 0x7fa6377f9c
[ 464.605314] pc : [<0000007fa62243e4>] lr : [<0000007fa6377f9c>] pstate: 60000000
[ 464.611996] sp : 0000007fc63e2870
[ 464.616908] x29: 0000007fc63e2870 x28: ffffffffffffffff
[ 464.625259] x27: 0000000000000001 x26: 0000000000000001
[ 464.630447] x25: 0000000000000001 x24: 00000000ffffffff
[ 464.635565] x23: 7fffffffffffffff x22: 0000000000000010
[ 464.640854] x21: 000000000000000f x20: 0000000000004000
[ 464.646117] x19: 0000005580096000 x18: 0000000000000000
[ 464.651247] x17: 0000007fa62243c0 x16: 0000007fa64d6420
[ 464.656491] x15: 000000000a31a200 x14: 0000000000518d10
[ 464.661638] x13: 0000000013c029f0 x12: 000000000000056e
[ 464.666891] x11: 0000000000000004 x10: 0000007fc63e2800
[ 464.672179] x9 : 0000000000000000 x8 : 000000000000011d
[ 464.677297] x7 : 000000558009e9f0 x6 : 000000558009e9f0
[ 464.682551] x5 : 0000000000000000 x4 : 7fffffffffffffff
[ 464.687688] x3 : 0000000000000000 x2 : 0000000000000010
[ 464.692828] x1 : 0000000000000000 x0 : 000000000000000f

UPDATE:
in headless mode it is the same issue.
Steps to reproduce:
1. Install ArchLinuxArm on eMMC by following official guide.
2. Boot with monitor and keyboard attached
3. pacman -Syu
4. systemctl reboot
5. pacman -S base-devel git mariab apache postfix dovecot php-apache

Headless:
1. Install ArchLinuxArm on eMMC by following official guide.
2. Boot with monitor and keyboard attached
3. pacman -Syu
4. systemctl reboot
5. systemctl enable sshd
6. vi /boot/boot.ini
7. change mesontimer to 0 and nographics to 1
8. systemctl poweroff
9. unplug power, monitor and keyboard
10. plugin power and network cable
11. ssh to odroid (user alarm as root is permitted)
12. su -
13. pacman -S base-devel git mariab apache postfix dovecot php-apache
14. dmesg