Odroid C2 does not boot after system update

This is for ARMv8 based devices

Odroid C2 does not boot after system update

Postby loki279 » Fri Mar 24, 2023 1:10 pm

My odroid does not boot after running pacman -Syu and updating the system.
I do not get a signal on a monitor, sceen stays black. Also I cannot ping or ssh the C2. The blue LED flashes few times after power is supplied and goes dark, the red stays on. The system is oin a EMMC module which seems to work alright, pluged to a linux PC I can mount, see and edit files on it.
Unfornunatly I do not know which packages were updated by pacman.

Does anyone have an idea how to debug?
loki279
 
Posts: 54
Joined: Tue Mar 07, 2017 7:59 am

Re: Odroid C2 does not boot after system update

Postby karog » Fri Mar 24, 2023 1:31 pm

If your emmc contains your root dir and you can mount and access it, look in var/log/pacman.log within that root dir. This contains a log of upgrades so look at the end of it for the most recent changes.

Also the best way to understand boot failures is to have console (serial tty) access to see booting from its earliest stages. I have odroid XU4, N2, and N2+ but not C2 so I don't know what kind of console access it has. But for my odroids I got a USB UART from ameridroid.com and it plugs into all of my odroids.
karog
 
Posts: 300
Joined: Thu Jan 05, 2012 7:55 pm

Re: Odroid C2 does not boot after system update

Postby loki279 » Fri Mar 24, 2023 5:44 pm

good hint witch the pacman logs. On the first look it seems that somethink went wrong updating mkinitcpio.
The logs show a lot of errors for different files looking like this:
$this->bbcode_second_pass_code('', '
[2023-03-24T11:17:18+0000] [ALPM-SCRIPTLET] cp: failed to clone '/tmp/mkinitcpio.r2tkMx/root/bin/busybox' from '/usr/lib/initcpio/busybox': Inappropriate ioctl for device
')
and
$this->bbcode_second_pass_code('', '
[2023-03-24T11:17:44+0000] [ALPM-SCRIPTLET] depmod: ERROR: failed to load symbols from /tmp/mkinitcpio.r2tkMx/root/lib/modules/3.16.85-1-ARCH/kernel/drivers/spi/spi-gpio.ko.gz: Invalid argument
')
resulting in
$this->bbcode_second_pass_code('', '
[2023-03-24T11:17:47+0000] [ALPM-SCRIPTLET] depmod: WARNING: could not open modules.builtin.modinfo at /tmp/mkinitcpio.QNcA4e/root/lib/modules/3.16.85-1-ARCH: No such file or directory
[2023-03-24T11:17:47+0000] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: '/boot/initramfs-linux.img'
[2023-03-24T11:17:48+0000] [ALPM-SCRIPTLET] ==> WARNING: errors were encountered during the build. The image may not be complete.
')

So my first guess would be the kernel image is corrupted. I will try to chroot into the emmc later, maybe I can generate a working image again...
loki279
 
Posts: 54
Joined: Tue Mar 07, 2017 7:59 am

Re: Odroid C2 does not boot after system update

Postby loki279 » Fri Mar 24, 2023 5:47 pm

Chrooting is difficult though as I don't have another arm device. I guess I cannot chroot from a x86 machine?
loki279
 
Posts: 54
Joined: Tue Mar 07, 2017 7:59 am

Re: Odroid C2 does not boot after system update

Postby karog » Fri Mar 24, 2023 8:54 pm

if you searched the forum for mkinitcpio you would find from yesterday this post:

viewtopic.php?f=15&t=16432&p=71012&hilit=mkinitcpio#p71012
karog
 
Posts: 300
Joined: Thu Jan 05, 2012 7:55 pm

Re: Odroid C2 does not boot after system update

Postby loki279 » Sat Mar 25, 2023 6:46 am

Thanks you, that should help me to fix this!
loki279
 
Posts: 54
Joined: Tue Mar 07, 2017 7:59 am

Re: Odroid C2 does not boot after system update

Postby loki279 » Sat Mar 25, 2023 1:28 pm

I did chroot into the arm EMMC today and luckly found an update to mkinitcpio which I could install after following the steps from your post. System is up and running again. Thanks for your help!
loki279
 
Posts: 54
Joined: Tue Mar 07, 2017 7:59 am

Re: Odroid C2 does not boot after system update

Postby loki279 » Fri Mar 31, 2023 2:59 pm

Today I run the next update and the problem is still there with mkinitcpio 35.2-1.1
every attempt to use cp results in an error:
$this->bbcode_second_pass_code('', 'Inappropriate ioctl for device')
If you have a C2, do not update!
loki279
 
Posts: 54
Joined: Tue Mar 07, 2017 7:59 am

Re: Odroid C2 does not boot after system update

Postby zw3n » Sat Apr 01, 2023 12:15 am

I ran into this a couple of hours ago and traced the problem to a change regarding the cp option --reflinks in coreutils.

TLDR: Downgrading coreutils to 9.1-3 worked for me.

I had already rebooted with a broken initcpio and had to go through the hassle of a qemu chroot.
If you're in the same situation the steps are (inside chroot):
$this->bbcode_list('1')
  • Downgrade coreutils to 9.1-3 (any below 9.2 should do): $this->bbcode_second_pass_code('', 'pacman -U /var/cache/pacman/pkg/coreutils-9.1-3-aarch64.pkg.tar.xz')
  • $this->bbcode_list('1')
  • Rebuild initcpio: $this->bbcode_second_pass_code('', 'mkinitcpio -p linux-odroid-c2')
  • A little brief but hope it helps!
    zw3n
     
    Posts: 3
    Joined: Fri Mar 31, 2023 11:52 pm

    Re: Odroid C2 does not boot after system update

    Postby zw3n » Sat Apr 01, 2023 12:33 am

    Just jotting down my findings of the night before crashing at 3 am after this marathon sysadmin session:

    This stackexchange thread pointed me in the right direction by explaining the different error messages from cp with different filesystems and --reflog parameters.

    Scrolling through the NEWS file of the current coreutils release gave me this:
    $this->bbcode_second_pass_quote('', '
    ')** Changes in behavior

    cp and install now default to copy-on-write (COW) if available.
    I.e., cp now uses --reflink=auto mode by default.


    and voila! Downgrading coreutils solved the problem.
    zw3n
     
    Posts: 3
    Joined: Fri Mar 31, 2023 11:52 pm

    Next

    Return to ARMv8 Devices

    Who is online

    Users browsing this forum: No registered users and 2 guests