Failed boot with initrd after 4.3.0-1 kernel update

[Sergeanter]

Just in case somebody else had problems, I had 2 out of 3 Pogoplug Mobile that failed to boot after latest 4.3.0-1 kernel update.
Both of them could not chroot to ROOTFS and could not fsck it with a message to the effect of PARTUUID-0000... not found.
I have looked at the MMC cards I use for boot and two of them somehow were showing LABEL and UUID but not PARTUUID. I could not find how to assign PARTUUID to a partition , so I have backed up contents of the boot cards and re-formatted them per installation instructions . PARTUUID has shown up and after restore both machines booted OK.
My guess is previous version of the initramfs image used UUID to pass to kernel , but the newer one requires actual PARTUUID.
Correct me if I am wrong.

[moonman]

Previously there was no initramfs at all. It is there now to fack root partition on boot. Kernel knows what partition's PARTUUID is without the help of initrd (which is not the case with UUID or LABEL) it is in fact U-Boot that passes PARTUUID to the kernel. You can remove initramfs-linux.img and it should still boot. My guess is that you had errors on the partition which could not be fixed without user interaction, but since there is no screen you just did not know about it.

[Sergeanter]

I had initrd boot set on all three plugs for well over a year now because I used to boot from USB stick and constantly had filesystem corrupted after power outage. Then I put a uboot from Bodhi and moved everything to MMC. So I had initramfs before this last update that included uboot 2015.10, systemd 228 and kernel 4.3.1. In fact , the machine that did not fail also had hooks for RAID set up in initrd.
I have serial connected on all three and boot looked normal up to the point when it needs to fsck rootfs and chroot.
Cards were clean in all three, and I force fscked them both on the pogo that stayed up and on a stand alone PC.
blkid was showing PARTUUID on the working card and no PARTUUID on those that failed to boot. My only guess is maybe I orginally formatted the other 2 cards on a PC with obsolete linux ( I have a few ancient laptops running various flavors of linux here and there).Or maybe there was something different in the way Arch set UUID and PARTUUID long ago.
But until this last update all three were booting just fine. One of the non-booting machines had uboot 2015.04 RC1 and I thought that might be a problem , but I flashed the new one and problem remained until I re-formatted the cards.

[moonman]

It still does not make sense to me fully. If you have U-Boot from us, then it passes PARTUUID (and only PARTUUID, there's no fallback to /dev/sdXY) to the kernel's root= arg since version 2015.04. My understanding is that you were on >=2015.04 U-Boot before and at the time of kernel 4.3.0 release in our repo. So this means that PARTUUID was there before when it worked, and at 4.3 release and the actual problem was something else (i.e. not PARTUUID).

Then if you were on 2014.04 U-Boot (which passes root=/dev/sdXY to the kernel) It would not have been a problem at all.

I don't know bodhi's U-Boot environemnt setup, so can't comment on that one.

Somebody else mentioned that the system would boot up with 4.3 kernel if initramfs-linux.img was removed. I was unable to reproduce this myself though as I could never get my system not to boot up with initrd.

You suggestion to reformat is good, but I still don't understand what causes certain devices not to boot up.

[Sergeanter]

All this PARTUUID business seems to be quite fuzzy. I could not find a way to set it and it looks like in some cases UUID and PARTUUID are used interchangeably such as in this recent uboot bug report
https://lists.debian.org/debian-kernel/2015/10/msg00125.html
It looks like they use long UUID as PARTUUID for gpt for example. Is it possible that until recently UUID was passed as PARTUUID and now it requires short PARTUUID which was not set in my case ?
Again, I am quite certain that two cards that failed to boot had no filesystem or sector errors and were showing no PARTUUID.
Interestingly , the "good" card that has a PARTUUID corresponding to mmcblk0p1 does not show any PARTUUID when placed in cardreader attached via USB. It appears as /dev/sde1 in my case and blkid shows label and uuid but not PARTUUID.

[wobble]

I ran into a similar problem and thanks to your findings so far I was able to come up with a solution.

Apparently what blkid reports for MBR PTs as PARTUUID (which - as far as I understand it - only really exists in GPTs) is the concatenation of what fdisk calls the "Disk identifier", a dash and two digits for the partition number, for example "5f5dca3e-01" for the first partition of a device with disc identifier 0x5f5dca3e. Turns out my device which didn't boot with an initramfs had an "disc identifier" of 0x00000000, what seems to make blkid report no PARTUUID at all.

With the help of this post I was able to change my zero identifier in fdisk to some random 8 hex digits and voila! Boots with initramfs.

I know it won't help you as you already went through the hassle of reinstalling, but I hope it will help others...

[wobble]

FWIW I've just checked and my device still doesn't boot with an initramfs after downgrading the kernel to 4.2.6-1 if the disc identifier is set to 0x00000000. So in my case the update to 4.3.0-1 only triggered the issue insofar that before there was no initramfs.

[Sergeanter]

Thanks. I concur
Somehow "bad" cards had disk identifier set to 0. Not sure how plugs with initramfs were booting with them until latest update.