pacman Suy last night almost bricked my aarch64 odroid C2s

Problems with packages? Post here, using [tags] of the package name.

pacman Suy last night almost bricked my aarch64 odroid C2s

Postby EkriirkE » Thu Mar 23, 2023 11:42 am

After running pacman -Suy last night to stay up to date, mkinitcpio was called and failing 99% completely: Every single file it attempted to copy resulted as such:

$this->bbcode_second_pass_code('', '[ALPM-SCRIPTLET] cp: failed to clone '/tmp/mkinitcpio.D1LhVr/root/bin/busybox' from '/usr/lib/initcpio/busybox': Inappropriate ioctl for device')

and left me with a 4KB /boot/initramfs-linux.img. A pausing the middle and examination of the temp working directory shows that du /tmp/mkinitcpio.XXXX was 0 bytes! but all the files were there.

Manually trying to cp A B in the same folder resulted in the same error, so it was not some linking-across-devices problem (eg initcpio bins/libs to /tmp). Something broke cp (and install), therefore breaking the initcpio

This happened on a remote system and a local one. I rebooted the local one and, of course, it was bricked. No kernel! I also did not have a backup initramfs img...

Here is what I did to cobble the remote to stay up:

I tried aliasing cp with cat>,progress, and various other alternative copy methods with no success. I did not have busybox installed but luckily initcpio includes it so I backed up (renamed) /bin/cp and /bin/install, then linked busybox to those commands:

$this->bbcode_second_pass_code('', 'ln -s /usr/lib/initcpio/busybox /bin/cp
ln -s /usr/lib/initcpio/busybox /bin/install')

mkinitcpio almost started working, but passes --preserve=mode,ownership to cp which that version of busybox does not support, so editing /usr/lib/initcpio/functions to find that at line 701 to just be -p worked!

$this->bbcode_second_pass_code('', 'command cp --remove-destination --preserve=mode,ownership "$src" "${BUILDROOT}${dest}"
to
command cp --remove-destination -p "$src" "${BUILDROOT}${dest}"')

I ran mkinitcpio -g /boot/initramfs-linux.img without issue and got a nice 7MB initramfs. I copied this to the SD card of my local one, then it booted right back up! Made the same modifications to override cp and install with busybox and things are smooth again so far.

Updating/reinstalling mkinitcpio breaks this and requires editing the functions file again. Other updates that trigger running mkinitcpio work fine.

I've attached the logs and an strace of cp failing for any insight into the failure. "native" cp and install are still broken after full update.
Attachments
logs.zip
(9.76 KiB) Downloaded 70 times
EkriirkE
 
Posts: 3
Joined: Sat Feb 11, 2023 11:40 am

Re: pacman Suy last night almost bricked my aarch64 odroid C

Postby loki279 » Fri Mar 31, 2023 3:11 pm

I had the same problem and was able to fix it with your thread. Toyday there was annother update to mkinitcpio (mkinitcpio 35.2-1.1), resultion in the same trouble again.
Still the cp command fails with the message:
$this->bbcode_second_pass_code('', 'Inappropriate ioctl for device')

So if you have a C2, do not update!
loki279
 
Posts: 54
Joined: Tue Mar 07, 2017 7:59 am

Re: pacman Suy last night almost bricked my aarch64 odroid C

Postby zw3n » Sat Apr 01, 2023 12:18 am

The problem is not with mkinitcpio but cp, part of coreutils, downgrading to any release before 9.2 solves the problem.

Also discussed here
zw3n
 
Posts: 3
Joined: Fri Mar 31, 2023 11:52 pm

Re: pacman Suy last night almost bricked my aarch64 odroid C

Postby EkriirkE » Sat Apr 01, 2023 1:06 pm

Yes as I wrote I narrowed the issue down to cp and install, with a slight patch to make mkinitcpio work with busybox
EkriirkE
 
Posts: 3
Joined: Sat Feb 11, 2023 11:40 am

Re: pacman Suy last night almost bricked my aarch64 odroid C

Postby EkriirkE » Sat Apr 01, 2023 7:53 pm

The conflict seems to be from HardKernel's unmaintained kernel (linux-odroid-c2) which is stuck on 3.16.85-1 for years now

Another option instead of preventing coreutils updating or my above hacks is to update to mainline (matches current releases -- as of today this is 6.2.8-1). I did this with success, even with current coreutils post reboot:

install mainline:

$this->bbcode_second_pass_code('', 'pacman -S linux-aarch64 uboot-odroid-c2-mainline dkms-mali-utgard-meson mali-utgard-meson-libgl-x11 xf86-video-armsoc-meson')

which should remove hardkernel specifics linux-odroid-c2 uboot-odroid-c2 odroid-c2-libgl-x11. Note that the new uboot must be written for the new kernel(s) to boot. These will ignore your old boot.ini file, and some kernel parameters will disappear or be renamed

Make note of your old kernel parameters via cat /proc/cmdline

Two that I would like to persist are fsck.repair=yes net.ifnames=0

You need to migrate these to the new boot.txt file, e.g. for me this line changes

$this->bbcode_second_pass_code('', 'setenv bootargs "console=ttyAML0,115200n8 root=PARTUUID=${uuid} rw rootwait earlycon fsck.repair=yes net.ifnames=0"')

Any changes to this file require you to run ./mkscr in the /boot directors to take effect

Since I run a headless server from here I wanted to disable HDMI detection where previously in the old boot.ini I would uncomment setenv hpd "true" and setenv m "1080p60hz", these translated to adding drm.edid_firmware=edid/1920x1080.bin video=HDMI-A-1:1920x1080@60e to the bootargs of the boot.txt


Some things may not be functional as they were with mainline drivers, so either test with a new boot card, or you can undo it by re-installing pacman -S linux-odroid-c2 uboot-odroid-c2 odroid-c2-libgl-x11 and ensuring the old uboot is restored (sdfusing.sh was run)
EkriirkE
 
Posts: 3
Joined: Sat Feb 11, 2023 11:40 am

Re: pacman Suy last night almost bricked my aarch64 odroid C

Postby loki279 » Wed Apr 05, 2023 6:54 am

with the current version of coreutils (9.2-3) cp and install are working again...
loki279
 
Posts: 54
Joined: Tue Mar 07, 2017 7:59 am


Return to Packages

Who is online

Users browsing this forum: No registered users and 5 guests