[HOW-TO ARMv5-Kirkwood] NAND rescue system

[moonman]

There are times when rootfs gets corrupted on whatever storage device you are using and the device just won't boot. This is exactly what this rescue system is for. It is installed into internal NAND storage on the device and will boot up when there are no rootfs found on usb or sata.

By installing the rescue image you will erase everything that was previously in NAND (stock OS image).

Requirements:
-working ArchLinux ARM installation
-mtd-utils and uboot-tools packages
-A kirkwood device (should work on any device for which we have a uboot package)
-ArchLinuxARM 2014.04 uboot or 2016.05 with unmodified environment (not strictly required, but you will have to figure out how to boot kernel + dtb from nand with ubi and your uboot).

Installation:

Code: Select all

ubiformat /dev/mtd1
ubiattach -p /dev/mtd1

Make ubi volume

Code: Select all

ubimkvol /dev/ubi0 -m -N rootfs

Download rescue image:

Code: Select all

cd ~
wget http://os.archlinuxarm.org/os/armv5te/rescue1.4.img.ubifs

Flash rescue image:

Code: Select all

ubiupdatevol /dev/ubi0_0 rescue1.4.img.ubifs

Setting U-Boot environment (do NOT do this if you have updated uboot to >= 2015.04):
U-BOOT >=2015.04 already has environment setup to boot rescue system.

Now rootfs is in NAND, but we still need to tell u-boot how to boot it:

Code: Select all

fw_setenv mountubi 'ubi part rootfs; ubifsmount ubi0:rootfs'

fw_setenv loadubizimage 'ubifsload ${loadaddr} ${zimage}'

fw_setenv loadubifdt 'ubifsload ${fdt_addr} ${fdt_file}'

fw_setenv bootzubi 'echo Booting from nand ...; run setargsubi; bootz ${loadaddr} - ${fdt_addr};'

fw_setenv setargsubi 'setenv bootargs console=${console},${baudrate} ${optargs} ubi.mtd=1 root=ubi0:rootfs ro rootfstype=ubifs  rootwait ${mtdparts}'

fw_setenv loadubi 'echo Trying to boot from NAND ...;if run mountubi; then run loadubizimage;run loadubifdt;ubifsumount;run bootzubi;fi'

Now below you need to be careful: bootcmd is different on different devices. Below are some examples. Essentially, you just need to add "run loadubi" at the end of the string.

Pogoplug V2/Dockstar:

Code: Select all

fw_setenv bootcmd 'usb start; setenv letter 9;for type in usb; do for disk in 0 1 2 3; do if ${type} part ${disk};then setexpr letter $letter + 1;run load;fi;done;done;run loadubi'

GoFlexHome:

Code: Select all

fw_setenv bootcmd 'ide reset; usb start; setenv letter 9;for type in ide usb; do for disk in 0; do if ${type} part ${disk};then setexpr letter $letter + 1; run load;fi;done;done;run loadubi'

GoFlexNet:

Code: Select all

fw_setenv bootcmd 'ide reset; usb start; setenv letter 9;for type in ide usb; do for disk in 0 1; do if ${type} part ${disk};then setexpr letter $letter + 1;run load;fi;done;done;run loadubi'

Done.

Recommendations:
Boot into rescue system by removing all external storage devices.
Regenerate openssh host keys (be careful: do not end ssh session before host keys are generated):
By default these get generated automatically the first time you connect, but the filesystem is mounted ro by default so I kept them in the image for the first session. It is a good idea to regenerate them.

Code: Select all

mount -o remount,rw /
rm /etc/ssh/ssh_host*
ssh-keygen -A

and change root password:

Code: Select all

passwd

remount read only (or reboot)

Code: Select all

mount -o remount,ro /

You don't want too many writes on the NAND so rootfs is mounted read-only by default. Journald logging has been disabled and whole /var directory was relocated to ram (tmpfs).

Avialble tools (outside of base system):
-mc (Midnight Commander)
-btrfs,xfs,f2fs,ntfs,vfat,exfat filesystems
-fdisk, gdisk, parted utilitites
-fw_setenv,fw_printenv
-mtd-utils
-wget
-arch-chroot,pacstrap
-+all the tools that come with base system.

[lauri4n]

Hi,

I tried to upgrade my old archlinux-arm install, but I did something in wrong order and my goflex didn't boot anymore. I've now followed these instructions:
viewtopic.php?f=53&t=762#p4447

I can now ssh into goflex, and I got this:

root@host-001's password:
/etc/ubit/images:
/etc/ubit/ubimounts:
/etc/ubit/extramounts:

U-Boot ethaddr value is: 02:50:43:48:7f:a9
There was no current cached ethaddr value.
No change has been made
UBIT v0.6 by peaslaker 2010.11-00125-ga21bcca

What should I do next (or which guide I should follow) to update ubit and the rest of the system to the latest versions?

Should I install this NAND rescue system?


Thanks in advance,

Lauri

[moonman]

Please make a new thread in ARMv5 forum about your issue as it is unrelated to Rescue system and post output Post output of fw_printenv..
You can't boot rescue system with your current Ubit bootloader as it doesn't support FDT kernel, you can install it after you update everything.

[woody]

I've installed the bodhi 2014.07.tld-2 u-boot on a Pogo E02. I found out that the rescue system I ended up with is old and outdated and doesn't work correctly. I'd like to install this Kirkwood rescue system. Here are some issues I'm having:

1. Looks like these instructions assume that the rootfs of the rescue system should reside in mtd1, but on my installation, it appears on mtd2. Should I adapt these instructions for mtd2?

2. I tried to attach the existing mtd2/rootfs using ubiattach -p , but this produces an error. I was able to mount it using mknod + mount -t jffs2 -o rw /dev/mtdblock2 /mnt . Is there some reason that ubiattach does not work as discussed in this thread?

3. Assuming that I should put the updated rescue system on mtd2, what's the best way to do it?

[moonman]

1. You appear to have stock mtd partitioning scheme. You won't have enough space to do it with mtd2 still there. E02 has ~128MB flash. With 1MB for uboot and the rest for rescue system I have 2MB free on mtd1.
2. ubi and jffs2 are different. ubi is more modern and faster replacement for jffs2. You need to format mtd partition before you can attach it as ubi.
3. Install Arch u-boot. You will only have mtd0 and mtd1 with uboot environment setup the right way. If you want to keep bodhi's uboot you need to figure out too many things yourself.

[PogoDancer]

Install Arch u-boot. You will only have mtd0 and mtd1 with uboot environment setup the right way.

While the idea of an embedded Emergency Arch is intriguing and I understand you want Arch's USB & uBoot MTDs set up to make Rescue easy to install, defining the stock OS out of existence by setting a partitioning different from what's on the NAND seems far from ideal, in particular since Arch's uBoot install & instructions do little to alert & enable to revert (http://myownhomeserver.com/2012/11/how- ... omment-202 seems unlikely to work on E02), cf. Pogoplug E02: tweak recent U-Boot to re-allow NAND fallback? - while there is an existing internal fallback that it just won't boot anymore (and may even have damaged on mount - jury's still out on this one).

If you want to keep bodhi's uboot you need to figure out too many things yourself.

Then again for the majority without JTAGs or low-voltage serial cables, especially on the all-too-easily brickable Kirkwood family, besides backup Arch install would have to prompt for properly setting up a preboot netconsole at least, and probably users should be told how to launch the stock Pogo OS and bodhi's FDTs through Arch's uBoot as well, if not how to start Arch from his.

Mainline U-Boot ... will not boot into stock with our U-Boot. Using the stock firmware is not the goal of this distribution.

viewtopic.php?f=53&t=7290#p40106 is hidden way too deep for anyone to find before running a http://archlinuxarm.org/os/armv5te/boot ... ogo_e02.sh of no return, and too much of a surprise decision, just like viewtopic.php?f=53&t=7290&start=20#p42034 without giving sufficient thought (let alone disclosure) to the fact that Stock itself might be a valuable (and the only) fallback if USB boot fails in the first place:

The only thing that changes is the partitioning, so that one could make use of the full NAND space for an installation if so desired. The stock partitioning scheme is more restrictive on space and pointless as the goal is to transition into this decade, not stay in the last one.

[moonman]

New uboot cannot chainload stock uboot because it is too different. I believe it is possible to chainload dockstar uboot to boot stock but it also involves flashing and you have to have a bootable os to flash.

If you want stock just stay with stock, if not - rescue system is here.

TTL adapter is 5 bucks on ebay.

[PogoDancer]

New uboot cannot chainload stock uboot because it is too different. I believe it is possible to chainload dockstar uboot to boot stock

Arch's uBoot lacking fsload, one would have to see whether there's a raw read from NAND or ext2load from USB (though the latter would be a cumbersome kind of rescue system) to do the trick; otherwise bodhi's uBoot should be able to boot them all.

If you want stock just stay with stock, if not - rescue system is here.

Can't stay with what's left of stock since for my machine too it did get damaged by mount from Arch (probably rw due to bug at viewtopic.php?f=18&t=3490&start=10#p20032) with mismatched partitioning (and after http://archlinuxarm.org/os/armv5te/boot ... ogo_e02.sh failed to copy /dev/mtd0 to a file on what was /dev/mtd2 in stock at install, or at least to USB).
So I might as well try out your rescue system as I will have to reflash anyway at 0x000000100000-0x000008000000 (why do ARM kernels always show one byte more than actual end address of partition; shouldn't that be 0x7ffffff ), and possibly even "dedicate that particular Pogo to Arch" (rather than restore stock as a failsafe, though I resent how Arch tried to force my hand).

Is the following supposed to happen and be answered with y, or is there any benefit/need to flash_erase in advance?

Code: Select all

ubiformat /dev/mtd1
ubiformat: mtd1 (nand), size 133169152 bytes (127.0 MiB), 1016 eraseblocks of 131072 bytes (128.0 KiB), min. I/O size 2048 bytes
libscan: scanning eraseblock 1015 -- 100 % complete 
ubiformat: 936 eraseblocks are supposedly empty
ubiformat: warning!: 80 of 1016 eraseblocks contain non-UBI data
ubiformat: continue? (y/N)

Just would have appreciated a clearer approach to preserve stock in the first place.

will erase everything that was previously in NAND

This statement is hopefully just oversimplified in omitting a first 8 Mbits of difference, as /dev/mtd0 shouldn't be touched.

TTL adapter is 5 bucks on ebay

...and mostly a clumsier alternative to uBoot netconsole on E02 which can't boot from serial anyway, i.e. where anything but JTAG is useless when "a brick is a brick is a brick".
Might as well say "Bus Pirate is 30 bucks at seeed Studio", but why invest in solutions if the problem could be avoided so easily in the first place?

[moonman]

Yes i is safe to say 'y' to erase.

[PogoDancer]

Yes i is safe to say 'y' to erase.

There is more to it and I have taken the time to write up on further issues I found with the initial instructions, documenting the expected outputs and interactions as well (IP address subject to each user's setup of course).

As actually there's no wget in Arch by default, and Internet not necessarily up&available all the time, you'll want the following first to avoid not getting the downloader or the download half way through rewriting NAND:

[root@alarm ~]# cd ~
[root@alarm ~]# wget https://dl.dropboxusercontent.com/u/15043728/ArchLinuxArm/kirkwood-rescue/rescue1.4.img.ubifs
-bash: wget: command not found
[root@alarm ~]# pacman -S wget
resolving dependencies...
looking for conflicting packages...

Packages (1) wget-1.16.3-1

Total Download Size: 0.45 MiB
Total Installed Size: 2.29 MiB

:: Proceed with installation? [Y/n]
:: Retrieving packages ...
wget-1.16.3-1-arm 458.7 KiB 156K/s 00:03 [#########################] 100%
(1/1) checking keys in keyring [#########################] 100%
(1/1) checking package integrity [#########################] 100%
(1/1) loading package files [#########################] 100%
(1/1) checking for file conflicts [#########################] 100%
(1/1) checking available disk space [#########################] 100%
(1/1) installing wget [#########################] 100%
Optional dependencies for wget
ca-certificates: HTTPS downloads [installed]
[root@alarm ~]# wget https://dl.dropboxusercontent.com/u/15043728/ArchLinuxArm/kirkwood-rescue/rescue1.4.img.ubifs
--2015-03-21 07:27:50-- https://dl.dropboxusercontent.com/u/15043728/ArchLinuxArm/kirkwood-rescue/rescue1.4.img.ubifs
Resolving dl.dropboxusercontent.com (dl.dropboxusercontent.com)... 54.243.46.82, 54.243.255.237, 107.20.165.93, ...
Connecting to dl.dropboxusercontent.com (dl.dropboxusercontent.com)|54.243.46.82|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 120121344 (115M) [application/octet-stream]
Saving to: 'rescue1.4.img.ubifs'

rescue1.4.img.ubifs 100%[=======================>] 114.56M 1.14MB/s in 1m 41s s

2015-03-21 07:29:34 (1.14 MB/s) - 'rescue1.4.img.ubifs' saved [120121344/120121344]

There should be some sanity checks about addressing the proper devices&geometry too...

[root@alarm ~]# dmesg | grep 0x00
[ 1.119689] 0x000000000000-0x000000100000 : "u-boot"
[ 1.125169] 0x000000100000-0x000008000000 : "rootfs"
[root@alarm ~]# fw_printenv mtdparts
mtdparts=mtdparts=orion_nand:1M(u-boot),-(rootfs)

...before proceeding to and all along the way of...

[root@alarm ~]# ubiformat /dev/mtd1
ubiformat: mtd1 (nand), size 133169152 bytes (127.0 MiB), 1016 eraseblocks of 131072 bytes (128.0 KiB), min. I/O size 2048 bytes
libscan: scanning eraseblock 1015 -- 100 % complete
ubiformat: 936 eraseblocks are supposedly empty
ubiformat: warning!: 80 of 1016 eraseblocks contain non-UBI data
ubiformat: continue? (y/N) y
ubiformat: warning!: only 0 of 1016 eraseblocks have valid erase counter
ubiformat: erase counter 0 will be used for all eraseblocks
ubiformat: note, arbitrary erase counter value may be specified using -e option
ubiformat: continue? (y/N) y
ubiformat: use erase counter 0 for all eraseblocks
ubiformat: formatting eraseblock 1015 -- 100 % complete
[root@alarm ~]# ls -l /dev/ubi*
crw------- 1 root root 248, 0 Dec 10 22:16 /dev/ubi0
crw------- 1 root root 248, 1 Dec 10 22:16 /dev/ubi0_0
crw------- 1 root root 10, 58 Dec 10 22:16 /dev/ubi_ctrl
[root@alarm ~]# ubiattach -p /dev/mtd1
UBI device number 0, total 1016 LEBs (131088384 bytes, 125.0 MiB), available 992 LEBs (127991808 bytes, 122.1 MiB), LEB size 129024 bytes (126.0 KiB)
[root@alarm ~]# ubimkvol /dev/ubi0 -m -N rootfs
Set volume size to 127991808
Volume ID 0, size 992 LEBs (127991808 bytes, 122.1 MiB), LEB size 129024 bytes (126.0 KiB), dynamic, name "rootfs", alignment 1
[root@alarm ~]# ubiupdatevol /dev/ubi0_0 rescue1.4.img.ubifs
[root@alarm ~]# shutdown -h now
Connection to 192.168.2.94 closed by remote host.
Connection to 192.168.2.94 closed.

One will probably want to perform this clean shutdown at least (no sync and/or ubidetach BTW?) before considering it all...

Done.

Recommendations:
Boot into rescue system by removing all external storage devices.

Presumably one's "production Arch" USB stick holding the active system shouldn't be pulled just like this, but rather considered safe to remove only after shutdown and preferably powering off.

Install Arch u-boot. You will only have mtd0 and mtd1 with uboot environment setup the right way. If you want to keep bodhi's uboot you need to figure out too many things yourself.

Actually most readers won't anymore (to the extent that now I did), as per netconsole on...

Hit any key to stop autoboot: 10
0
PogoE02> printenv

...shows what's missing from the more versatile bodhi uBoot's environment, and provides a chance to test the lines below before committing the environment to flash (and potentially brick if they were in error):

setenv fdt_addr 0x800000
setenv fdt_file /boot/dtbs/kirkwood-pogo_e02.dtb
setenv loadaddr 0x810000
setenv zimage /boot/zImage
setenv mountubi 'ubi part rootfs; ubifsmount ubi0:rootfs'
setenv loadubizimage 'ubifsload ${loadaddr} ${zimage}'
setenv loadubifdt 'ubifsload ${fdt_addr} ${fdt_file}'
setenv bootzubi 'echo Booting rescue Arch from NAND ...; run setargsubi; bootz ${loadaddr} - ${fdt_addr};'
setenv setargsubi 'setenv bootargs console=${console},${baudrate} ${optargs} ubi.mtd=1 root=ubi0:rootfs ro rootfstype=ubifs rootwait ${mtdparts}'
setenv loadubi 'echo Trying to boot rescue Arch from NAND ...;if run mountubi; then run loadubizimage;run loadubifdt;ubifsumount;run bootzubi;fi'
setenv bootcmd 'run bootcmd_uenv; run bootcmd_usb; run bootcmd_mmc; run bootcmd_sata; echo Trying to fall back to stock from NAND ...; fsload uboot-original-mtd0.kwb && go 0x800200; echo Assuming Arch NAND partitioning...; setenv mtdparts mtdparts=orion_nand:1M(u-boot),-(rootfs); setenv partition nand0,1; usb start; setenv letter 9;for type in usb; do for disk in 0 1 2 3; do if ${type} part ${disk};then setexpr letter $letter + 1;run load;fi;done;done;run loadubi; reset'
boot

With all USB media unplugged, then one can...

b o o t
(Re)start USB...
USB0: USB EHCI 1.00
scanning bus 0 for devices... 2 USB Device(s) found
scanning usb for storage devices... 0 Storage Device(s) found
Unknown command 'mmc' - try 'help'
Unknown command 'ide' - try 'help'

no USB devices available
Unknown command 'mmc' - try 'help'
Unknown command 'ide' - try 'help'
(Re)start USB...
USB0: USB EHCI 1.00
scanning bus 0 for devices... 2 USB Device(s) found
scanning usb for storage devices... 0 Storage Device(s) found
** Bad device usb 0 **
** Bad device usb 0 **
Wrong Image Format for bootm command
ERROR: can't get kernel image!
Unknown command 'mmc' - try 'help'
** Bad device mmc 0 **
** Bad device mmc 0 **
Wrong Image Format for bootm command
ERROR: can't get kernel image!
Unknown command 'ide' - try 'help'
** Bad device ide 0 **
** Bad device ide 0 **
Wrong Image Format for bootm command
ERROR: can't get kernel image!
Trying to fall back to stock from NAND ...
### JFFS2 loading 'uboot-original-mtd0.kwb' to 0x810000
Scanning JFFS2 FS: get_fl_mem: unknown device type, using raw offset!
get_fl_mem: unknown device type, using raw offset!
... keeps repeating through mtd2 all NAND long ...
get_fl_mem: unknown device type, using raw offset!
done.
find_inode failed for name=uboot-original-mtd0.kwb
load: Failed to find inode
### JFFS2 LOAD ERROR<0> for uboot-original-mtd0.kwb!
Assuming Arch NAND partitioning...
(Re)start USB...
USB0: USB EHCI 1.00
scanning bus 0 for devices... 2 USB Device(s) found
scanning usb for storage devices... 0 Storage Device(s) found

no USB devices available

no USB devices available

no USB devices available

no USB devices available
Trying to boot rescue Arch from NAND ...
UBI: attaching mtd1 to ubi0
UBI: physical eraseblock size: 131072 bytes (128 KiB)
UBI: logical eraseblock size: 129024 bytes
UBI: smallest flash I/O unit: 2048
UBI: sub-page size: 512
UBI: VID header offset: 512 (aligned 512)
UBI: data offset: 2048
UBI: attached mtd1 to ubi0
UBI: MTD device name: "mtd=1"
UBI: MTD device size: 127 MiB
UBI: number of good PEBs: 1016
UBI: number of bad PEBs: 0
UBI: max. allowed volumes: 128
UBI: wear-leveling threshold: 4096
UBI: number of internal volumes: 1
UBI: number of user volumes: 1
UBI: available PEBs: 10
UBI: total number of reserved PEBs: 1006
UBI: number of PEBs reserved for bad PEB handling: 10
UBI: max/mean erase counter: 1/0
UBIFS: mounted UBI device 0, volume 0, name "rootfs"
UBIFS: mounted read-only
UBIFS: file system size: 126572544 bytes (123606 KiB, 120 MiB, 981 LEBs)
UBIFS: journal size: 9033728 bytes (8822 KiB, 8 MiB, 71 LEBs)
UBIFS: media format: w4/r0 (latest is w4/r0)
UBIFS: default compressor: zlib
UBIFS: reserved for root: 0 bytes (0 KiB)
Loading file '/boot/zImage' to addr 0x00810000 with size 4130248 (0x003f05c8)...
Done
Loading file '/boot/dtbs/kirkwood-pogo_e02.dtb' to addr 0x00800000 with size 10228 (0x000027f4)...
Done
Unmounting UBIFS volume rootfs!
Booting rescue Arch from NAND ...
Kernel image @ 0x810000 [ 0x000000 - 0x3f05c8 ]
## Flattened Device Tree blob at 00800000
Booting using the fdt blob at 0x800000


Starting kernel ...

Which looks good but the chainload should still be tested on a machine with conventional 4-partition layout and working stock Pogo OS preserved.

Then that would seem to handle fallbacks to traditional and Arch NAND gracefully, i.e. could be committed to the NAND environment at 0xc0000 by saveenv in bodhi's uBoot or by prepending the above setenv lines with fw_ each in Arch.

SSH security having been tightened, e.g. Ubuntu won't let you connect to the embedded rescue system, hence you'll need to make it either ignore the changed fingerprint...

Code: Select all

ssh -o UserKnownHostsFile=/dev/null root@192.168.2.94

...or delete the known one (which will change again due to the procedure below) when switching between rescue and regular Arch:

Code: Select all

ssh-keygen -f "/home/user/.ssh/known_hosts" -R 192.168.2.94
ssh -o 'StrictHostKeyChecking=no' root@192.168.2.94

Then the rescue system setup can be completed (optionally including the lines for fw_setenv as above after further testing):

Code: Select all

        .
       /#\                     _      _ _                   _
      /###\      __ _ _ __ ___| |__  | (_)_ __  _   ___  __| |  _   ___ __  __
     /#####\    / _` | '__/ __| '_ \ | | | '_ \| | | \ \/ /| | / \ | _ \  \/  |
    /##.-.##\  | (_| | | | (__| | | || | | | | | |_| |>  < | |/ ^ \|   / |\/| |
   /##(   )##\  \__,_|_|  \___|_| |_||_|_|_| |_|\__,_/_/\_\| /_/ \_\_|_\_|  |_|
  /#.--   --.#\                                            |_|                 
 /'           '\

(rescue)[root@alarm ~]# mount -o remount,rw /
(rescue)[root@alarm ~]# rm /etc/ssh/ssh_host*
(rescue)[root@alarm ~]# ssh-keygen -A
ssh-keygen: generating new host keys: RSA1 RSA DSA ECDSA ED25519
(rescue)[root@alarm ~]# passwd
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
(rescue)[root@alarm ~]# mount -o remount,ro /
(rescue)[root@alarm ~]# shutdown -h now

Thanks again to moonman for providing this really nice & powerful embedded rescue system!

Hopefully the above should help to keep all other OS options open as well:
Everyone please feel free to enhance, extend & refine - as well as most importantly integrate flexibility & backups with installers.