Kernel panic with high network(wifi) speeds + usb storage

[WarheadsSE]

Hi,

I've had my pi for a couple of weeks and in that time I've noticed it does crash on a fairly regular basis. The causes so far seem to be: VNC, qbittorrent.

I started doing some more serious investigation yesterday and found this thread which seems to be similar to my problem. The main difference is that I'm not writing anything to USB storage. I'm doing either "network access" (VNC) or "network access + sd card i/o" (qbittorrent).

Some configuration details:

OS:

root@pi:/var/log# uname -a
Linux pi 3.1.9+ #90 Wed Apr 18 18:23:05 BST 2012 armv6l GNU/Linux

I tried increasing vm.min_free_kbytes to 32768, that made no difference.

I noticed this message in kern.log:
$this->bbcode_second_pass_code('', '
Jun 15 19:20:48 pi kernel: ------------[ cut here ]------------
Jun 15 19:20:48 pi kernel: WARNING: at net/sched/sch_generic.c:255 0xc0280ec4()
Jun 15 19:20:48 pi kernel: NETDEV WATCHDOG: eth0 (smsc95xx): transmit queue 0 timed out
Jun 15 19:20:48 pi kernel: Modules linked in: fuse
Jun 15 19:20:48 pi kernel: Backtrace:
Jun 15 19:20:48 pi kernel: Function entered at [<c0011a7c>] from [<c02fe678>]
Jun 15 19:20:48 pi kernel: r6:000000ff r5:c0280ec4 r4:c0399e28 r3:c039efa4
Jun 15 19:20:48 pi kernel: Function entered at [<c02fe660>] from [<c001ed28>]
Jun 15 19:20:48 pi kernel: Function entered at [<c001ecd4>] from [<c001ede4>]
Jun 15 19:20:48 pi kernel: r8:00000100 r7:c03fb948 r6:c039fcd0 r5:00000000 r4:cb9d6800
Jun 15 19:20:48 pi kernel: r3:00000009
Jun 15 19:20:48 pi kernel: Function entered at [<c001edac>] from [<c0280ec4>]
Jun 15 19:20:48 pi kernel: r3:cb9d6800 r2:c0353c10
Jun 15 19:20:48 pi kernel: Function entered at [<c0280c70>] from [<c0029f8c>]
Jun 15 19:20:48 pi kernel: r7:c0398000 r6:c039fcd0 r5:c03d6b00 r4:c0399e80
Jun 15 19:20:48 pi kernel: Function entered at [<c0029e98>] from [<c00240cc>]
Jun 15 19:20:48 pi kernel: Function entered at [<c0024034>] from [<c0024520>]
Jun 15 19:20:48 pi kernel: Function entered at [<c0024498>] from [<c000f0e0>]
Jun 15 19:20:48 pi kernel: r4:c03a84d8 r3:c0050084
Jun 15 19:20:48 pi kernel: Function entered at [<c000f0a4>] from [<c0008190>]
Jun 15 19:20:48 pi kernel: r6:f200b200 r5:60000013 r4:c000f234 r3:c000f230
Jun 15 19:20:48 pi kernel: Function entered at [<c0008180>] from [<c000ddd4>]
Jun 15 19:20:48 pi kernel: Exception stack(0xc0399f40 to 0xc0399f88)
Jun 15 19:20:48 pi kernel: 9f40: ffffffed 00000000 c0399f88 00000000 c0398000 c03b5dc4 c039cfdc c039cfd4
Jun 15 19:20:48 pi kernel: 9f60: 00004008 410fb767 003919b0 c0399f94 c0399f98 c0399f88 c000f230 c000f234
Jun 15 19:20:48 pi kernel: 9f80: 60000013 ffffffff
Jun 15 19:20:48 pi kernel: Function entered at [<c000f208>] from [<c000f444>]
Jun 15 19:20:48 pi kernel: Function entered at [<c000f3a4>] from [<c02fcf10>]
Jun 15 19:20:48 pi kernel: r6:c0582960 r5:c0392718 r4:c039a0ac r3:c039c050
Jun 15 19:20:48 pi kernel: Function entered at [<c02fceb0>] from [<c037b72c>]
Jun 15 19:20:48 pi kernel: Function entered at [<c037b500>] from [<0000803c>]
Jun 15 19:20:48 pi kernel: ---[ end trace f134530073e80423 ]---
Jun 15 19:21:03 pi kernel: TCP: Possible SYN flooding on port 6881. Sending cookies. Check SNMP counters.
')
This is just before the crash (I think...)

Some trawling around on the internet reveals this has been a major source of problems on various versions of linux for many years. I can't quite figure out what the solution is. The most common advice is to set pci=noacpi as kernel boot parameter, but that is (I think) x86 specific. Is there an equivalent for ARM?

Cheers,

Gary

[WarheadsSE]

I don't believe the pi has PCI, thus it's not worth bothering.

[pepedog]

I think this stems back to when I got my Pi last October, it had a different problem.
Google "dwc_otg interrupt storm"
In fixing this, another problem has been introduced

[garypen]

Hi,

Thanks for the pointers. I've so far tried:

increasing vm.min_free_kbytes 8192 -> 32768 (No Effect)
increasing net.ipv4.tcp_max_syn_backlog 128 -> 1024 (Delayed onset of crash, but still crashes)
setting smsc95xx.turbo_mode=N (No Effect)

I get the impression that these various measures may delay the onset of the crash but, depending on your work profile, are effective for different reasons for different folks. I haven't tried them all in combination yet (so far I'm doing it scientifically, one at a time), so I'll try them all in combination now. Any more suggestions/ideas would be welcome.

Thanks,

Gary

[guisacouto]

Any news on this?

Best regards

[guisacouto]

Don't know if this is relevant, while watching everything.log I noticed that the last message before the panic was:
$this->bbcode_second_pass_code('', '
Jun 20 20:19:29 berry ntpd[351]: skew change -5.053 exceeds limit
')

The curious part is that I know that it crashed some seconds later. Could this be related? For what I can understand what ntpd is telling is that the clock is somehow getting out of synch, which I know can cause some unpredictable behavior in filesystems, logs, etc. Don't know if it is the case, but if it is, it might make some sense to users with more knowledge then I.

[pepedog]

I posted a message here, it never appeared.
Does the lan chip get hot under load?
Do you have iptables running?
I pm'd my email, can you send me the torrents you have so I can see if it does it for me.

[guisacouto]

It just crashed while downloading the new debian beta that is on the raspberrypi.org website. I don't have iptables, and the lan chip doesn't get hot because I'm using wifi.. so temperature is not the problem since using ethernet or wifi causes the same reaction.

[guisacouto]

Not sure if this helps anything. While I was testing now, I tailed the kernel.log and got to see some output I hadn't been able before.

$this->bbcode_second_pass_code('', '
Jan 1 01:16:20 berry kernel: [ 985.188140] usb 1-1.3: usb_sg_cancel, unlink --> -43
Jan 1 01:16:21 berry kernel: [ 985.281154] usb 1-1.3: reset high speed USB device number 5 using dwc_otg
')

[pepedog]

Different device, when I had resets it turned out to be the drive
http://www.solid-run.com/phpbb/viewtopic.php?f=6&t=246