kernel panic with latest kirkwood 3.9 update

[akr]

I am getting the same kernel panic after updating the kernel on my SheevaPlug. Sadly I made the mistake of clearing my pacman cache before rebooting so now I am unable to rollback to the old kernel. Could somebody who had better foresight than myself upload the previous kirkwood package until the source of this problem can be found? Thanks

Update #1: I am now trying to rollback to the 3.6.10 kernel from my last full system backup
Update #2: Rollback to 3.6.10 was successful and the SheevaPlug is up and running again for now

[replabrobin]

akr, I have the 3.8.12 alarm linux-kirkwood if that's of use, not sure where it should be uploaded though.

[replabrobin]

OK Moonman, I won't try that experiment. If you are right then the problem is coming from some where up-stream. Is there something else that can be done with the current working kernel or the failing one to narrow down the problem space?

[replabrobin]

Moonman, for what it's worth here is the kmsg from a boot on the sheeva with 3.9.1

$this->bbcode_second_pass_code('', '<6>[ 0.000000] Booting Linux on physical CPU 0x0
<6>[ 0.000000] Initializing cgroup subsys cpuset
<6>[ 0.000000] Initializing cgroup subsys cpu
<5>[ 0.000000] Linux version 3.9.1-1-ARCH (nobody@panda1) (gcc version 4.7.2 (GCC) ) #1 PREEMPT Fri May 10 00:01:24 UTC 2013
<4>[ 0.000000] CPU: Feroceon 88FR131 [56251311] revision 1 (ARMv5TE), cr=00053977
<4>[ 0.000000] CPU: VIVT data cache, VIVT instruction cache
<4>[ 0.000000] Machine: Marvell SheevaPlug Reference Board
<4>[ 0.000000] Memory policy: ECC disabled, Data cache writeback
<7>[ 0.000000] On node 0 totalpages: 131072
<7>[ 0.000000] free_area_init_node: node 0, pgdat c0675284, node_mem_map c0747000
<7>[ 0.000000] Normal zone: 1024 pages used for memmap
<7>[ 0.000000] Normal zone: 0 pages reserved
<7>[ 0.000000] Normal zone: 131072 pages, LIFO batch:31
<7>[ 0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
<7>[ 0.000000] pcpu-alloc: [0] 0
<4>[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 130048
<5>[ 0.000000] Kernel command line: console=ttyS0,115200
<6>[ 0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes)
<6>[ 0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
<6>[ 0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
<5>[ 0.000000] __ex_table already sorted, skipping sort
<6>[ 0.000000] Memory: 512MB = 512MB total
<5>[ 0.000000] Memory: 509844k/509844k available, 14444k reserved, 0K highmem
<5>[ 0.000000] Virtual kernel memory layout:
<5>[ 0.000000] vector : 0xffff0000 - 0xffff1000 ( 4 kB)
<5>[ 0.000000] fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB)
<5>[ 0.000000] vmalloc : 0xe0800000 - 0xff000000 ( 488 MB)
<5>[ 0.000000] lowmem : 0xc0000000 - 0xe0000000 ( 512 MB)
<5>[ 0.000000] modules : 0xbf000000 - 0xc0000000 ( 16 MB)
<5>[ 0.000000] .text : 0xc0008000 - 0xc0602e80 (6124 kB)
<5>[ 0.000000] .init : 0xc0603000 - 0xc062fcb4 ( 180 kB)
<5>[ 0.000000] .data : 0xc0630000 - 0xc0679800 ( 294 kB)
<5>[ 0.000000] .bss : 0xc0679800 - 0xc074634c ( 819 kB)
<6>[ 0.000000] SLUB: Genslabs=13, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
<6>[ 0.000000] NR_IRQS:114
<6>[ 0.000000] sched_clock: 32 bits at 200MHz, resolution 5ns, wraps every 21474ms
<6>[ 0.000000] Console: colour dummy device 80x30
<6>[ 2.592697] Calibrating delay loop... 1191.11 BogoMIPS (lpj=5955584)
<6>[ 2.682578] pid_max: default: 32768 minimum: 301
<6>[ 2.682690] Security Framework initialized
<6>[ 2.682716] AppArmor: AppArmor disabled by boot time parameter
<6>[ 2.682783] Mount-cache hash table entries: 512
<6>[ 2.683354] Initializing cgroup subsys cpuacct
<6>[ 2.683374] Initializing cgroup subsys devices
<6>[ 2.683386] Initializing cgroup subsys freezer
<6>[ 2.683397] Initializing cgroup subsys net_cls
<6>[ 2.683406] Initializing cgroup subsys blkio
<6>[ 2.683415] Initializing cgroup subsys perf_event
<6>[ 2.683435] Initializing cgroup subsys bfqio
<6>[ 2.683533] CPU: Testing write buffer coherency: ok
<6>[ 2.683882] Setting up static identity map for 0xc04711e0 - 0xc047121c
<6>[ 2.685220] devtmpfs: initialized
<6>[ 2.687123] pinctrl core: initialized pinctrl subsystem
<6>[ 2.687432] NET: Registered protocol family 16
<6>[ 2.688517] DMA: preallocated 1024 KiB pool for atomic coherent allocations
<6>[ 2.689076] Kirkwood: MV88F6281-A1, TCLK=200000000.
<6>[ 2.689097] Feroceon L2: Enabling L2
<6>[ 2.689134] Feroceon L2: Cache support initialised.
<7>[ 2.689567] initial MPP regs: 01111111 11113322 00001111 00100000 00000000 00000000 00000000
<7>[ 2.689594] final MPP regs: 01111111 11113322 00001111 00000000 00000000 00000000 00000000
<6>[ 2.694638] bio: create slab <bio-0> at 0
<6>[ 2.695013] vgaarb: loaded
<5>[ 2.695258] SCSI subsystem initialized
<7>[ 2.695403] libata version 3.00 loaded.
<6>[ 2.695601] usbcore: registered new interface driver usbfs
<6>[ 2.695661] usbcore: registered new interface driver hub
<6>[ 2.695762] usbcore: registered new device driver usb
<6>[ 2.695870] pps_core: LinuxPPS API ver. 1 registered
<6>[ 2.695880] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
<6>[ 2.695904] PTP clock support registered
<6>[ 2.696461] Switching to clocksource orion_clocksource
<5>[ 2.708758] FS-Cache: Loaded
<6>[ 2.717885] NET: Registered protocol family 2
<6>[ 2.718479] TCP established hash table entries: 4096 (order: 3, 32768 bytes)
<6>[ 2.718579] TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
<6>[ 2.718632] TCP: Hash tables configured (established 4096 bind 4096)
<6>[ 2.718690] TCP: reno registered
<6>[ 2.718704] UDP hash table entries: 256 (order: 0, 4096 bytes)
<6>[ 2.718726] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
<6>[ 2.718928] NET: Registered protocol family 1
<7>[ 2.718963] PCI: CLS 0 bytes, default 32
<6>[ 2.719146] Trying to unpack rootfs image as initramfs...
<6>[ 2.920690] Freeing initrd memory: 2452K
<6>[ 2.921510] audit: initializing netlink socket (disabled)
<5>[ 2.921551] type=2000 audit(0.320:1): initialized
<5>[ 3.014876] VFS: Disk quotas dquot_6.5.2
<4>[ 3.015087] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
<6>[ 3.017211] jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
<6>[ 3.018101] aufs 3.9-20130506
<6>[ 3.018123] msgmni has been set to 1000
<6>[ 3.019609] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
<6>[ 3.019623] io scheduler noop registered
<6>[ 3.019630] io scheduler deadline registered
<6>[ 3.019918] io scheduler cfq registered
<6>[ 3.019950] io scheduler bfq registered (default)
<5>[ 3.020069] mv_xor mv_xor.0: Marvell shared XOR driver
<6>[ 3.056550] mv_xor mv_xor.0: Marvell XOR: ( xor cpy )
<6>[ 3.096556] mv_xor mv_xor.0: Marvell XOR: ( xor fill cpy )
<5>[ 3.096695] mv_xor mv_xor.1: Marvell shared XOR driver
<6>[ 3.136547] mv_xor mv_xor.1: Marvell XOR: ( xor cpy )
<6>[ 3.176546] mv_xor mv_xor.1: Marvell XOR: ( xor fill cpy )
<6>[ 3.180588] Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled
<6>[ 3.201257] serial8250.0: ttyS0 at MMIO 0xf1012000 (irq = 33) is a 16550A
<6>[ 3.201605] console [ttyS0] enabled
<6>[ 3.207759] brd: module loaded
<6>[ 3.210803] loop: module loaded
<6>[ 3.211657] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit), 512MiB, page size: 2048, OOB size: 64
<6>[ 3.211683] Scanning device for bad blocks
<5>[ 3.523596] Creating 3 MTD partitions on "orion_nand":
<5>[ 3.523618] 0x000000000000-0x000000100000 : "u-boot"
<5>[ 3.523983] 0x000000100000-0x000000500000 : "uImage"
<5>[ 3.524305] 0x000000500000-0x000020000000 : "root"
<5>[ 3.525451] mv643xx_eth: MV-643xx 10/100/1000 ethernet driver version 1.4
<6>[ 3.525549] libphy: mv643xx_eth smi: probed
<5>[ 3.531489] mv643xx_eth_port mv643xx_eth_port.0 eth0: port 0 with MAC address f0:ad:4e:00:03:93
<6>[ 3.531538] pch_gbe: EG20T PCH Gigabit Ethernet Driver - version 1.01
<6>[ 3.531604] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
<6>[ 3.531666] orion-ehci orion-ehci.0: Marvell Orion EHCI
<6>[ 3.531695] orion-ehci orion-ehci.0: new USB bus registered, assigned bus number 1
<6>[ 3.531816] orion-ehci orion-ehci.0: irq 19, io mem 0xf1050000
<6>[ 3.546511] orion-ehci orion-ehci.0: USB 2.0 started, EHCI 1.00
<6>[ 3.547067] hub 1-0:1.0: USB hub found
<6>[ 3.547099] hub 1-0:1.0: 1 port detected
<6>[ 3.547385] ehci-pci: EHCI PCI platform driver
<6>[ 3.547491] Initializing USB Mass Storage driver...
<6>[ 3.547593] usbcore: registered new interface driver usb-storage
<6>[ 3.547601] USB Mass Storage support registered.
<6>[ 3.547902] mousedev: PS/2 mouse device common for all mice
<6>[ 3.548169] rtc-mv rtc-mv: rtc core: registered rtc-mv as rtc0
<6>[ 3.548577] orion_wdt: Initial timeout 21 sec
<6>[ 3.548619] cpuidle: using governor ladder
<6>[ 3.548628] cpuidle: using governor menu
<6>[ 3.548846] sdhci: Secure Digital Host Controller Interface driver
<6>[ 3.548854] sdhci: Copyright(c) Pierre Ossman
<4>[ 3.548917] mvsdio mvsdio: no pins associated
<7>[ 3.548955] orion_gpio_is_valid: invalid GPIO 0
<7>[ 3.548964] orion_gpio_is_valid: invalid GPIO 0
<4>[ 3.548998] mvsdio: probe of mvsdio failed with error -22
<6>[ 3.549046] sdhci-pltfm: SDHCI platform and OF driver helper
<4>[ 3.549091] leds-gpio leds-gpio: pins are not configured from the driver
<6>[ 3.549567] usbcore: registered new interface driver r8712u
<6>[ 3.549910] gre: GRE over IPv4 demultiplexor driver
<6>[ 3.550037] TCP: cubic registered
<6>[ 3.550048] NET: Registered protocol family 17
<5>[ 3.550151] Key type dns_resolver registered
<6>[ 3.550634] registered taskstats version 1
<6>[ 3.551594] console [netcon0] enabled
<6>[ 3.551603] netconsole: network logging started
<6>[ 3.551694] rtc-mv rtc-mv: setting system clock to 2000-01-01 00:00:00 UTC (946684800)
<6>[ 3.552513] Freeing init memory: 176K
<30>[ 3.607070] systemd-udevd[45]: starting version 195[')

[replabrobin]

I think the error must be related to the lines

3.8.12
$this->bbcode_second_pass_code('', '
sdhci: Secure Digital Host Controller Interface driver
sdhci: Copyright(c) Pierre Ossman
mmc0: mvsdio driver initialized, lacking card detect (fall back to polling)
')

3.9.1
$this->bbcode_second_pass_code('', '
sdhci: Secure Digital Host Controller Interface driver
sdhci: Copyright(c) Pierre Ossman
mvsdio mvsdio: no pins associated
orion_gpio_is_valid: invalid GPIO 0
orion_gpio_is_valid: invalid GPIO 0
mvsdio: probe of mvsdio failed with error -22
')

[moonman]

That's a start. I'll see if I can find something

[moonman]

Found a patch that may affect this and it was merged into the main tree. https://patchwork.kernel.org/patch/2324621/ I'll reverse it and post the package here for testing. Hopefully this is the one to blame.'

There's also one here: http://lists.infradead.org/pipermail/li ... 57157.html (for future reference if the problem is not fixed)

[moonman]

Alright here are the kernels. Please test these and report which works if any Be sure to have a backup.

1 patch reverted
$this->bbcode_second_pass_code('', 'http://dl.dropboxusercontent.com/u/15043728/ArchLinuxArm/kernels/testing/fixgpio-reverted/linux-kirkwood-3.9.2-1-arm.pkg.tar.xz')

1 patch revrted + 1 applied
$this->bbcode_second_pass_code('', 'http://dl.dropboxusercontent.com/u/15043728/ArchLinuxArm/kernels/testing/fixgpio-reverted%2Bapplied/linux-kirkwood-3.9.2-1-arm.pkg.tar.xz')

1 patch applied
$this->bbcode_second_pass_code('', 'http://dl.dropboxusercontent.com/u/15043728/ArchLinuxArm/kernels/testing/fixgpio-applied/linux-kirkwood-3.9.2-1-arm.pkg.tar.xz')

[replabrobin]

Moonman, thanks for swift work. Here are the results of booting the various kernels with the uInitrd that you recommended.

kernel with patch reverted - mmc devices in /dev NO

kernel with patch reverted + 1 applied - mmc devices in /dev YES

kernel with patch applied - mmc devices in /dev YES

I haven't yet tried a full boot with the associated 3.9.2 modules. I can do that and run further tests if you deem it necessary.

[moonman]

it would be nice if you could try full boot. I want to make sure it actually works and doesn't mess anything up before I include it in the official