Arch Linux ARM v7 kernel/bootcode update on Raspberry Pi 3 i

[aroberts]

Arch Linux ARM v7 kernel/bootcode update on Raspberry Pi 3 is breaking Networking.
I have 7 Pi's (5 Pi 3's, 2 originals), running arch 24/7. The last kernel/boot code update stopped
the networking starting on 2 of the 5 v3's.

In all cases, installed latest updates (pacman -Syu)
and then rebooted after kernel updates had installed. I was logged on to all machines remotely via ssh (they are all headless).
On the 2 machines in question the networking lights didn't come on to indicate a link. Tried powering off/on, still no joy.

I found I had to power on with network cable UNPLUGGED and then plug it in later. This allowed network to start and remote login via ssh.

Both Pi's are plugged into different Netgear gigabit routers.

I had the same issue around 8th July when there were 3 systemd updates in 1 day, but there may also have been kernel/bootcode updates as well. In that case one of the same Pi's was affected, and a different one also (but all PI 3's).

After the network starts up successfully further reboots seem to be ok, I've rebooted ok many times since 8th July, until today.

The machines aren't running out of storage as I purge the old packages regularly etc. I've been running these machines 24/7
since they were released (about a year or so), and the networking issues have only occurred twice, both in the last month. All are configured identically.

uname -a
Linux alarmpi 4.9.38-1-ARCH #1 SMP Sat Jul 22 02:48:38 UTC 2017 armv7l GNU/Linux

tail /proc/cpuinfo
processor : 3
model name : ARMv7 Processor rev 4 (v7l)
BogoMIPS : 38.40
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4

Hardware : BCM2835
Revision : a02082
Serial : Redacted

I'm running sshd (obviously) with an override.conf file (so it starts even if network is slow to start, why is this not the default!!!)
[Unit]
After=ntpd.service

[Service]
Restart=always
RestartSec=5

ntpd is also running with an override.conf: (to ensure time syncs early from local ntpd server)
[Unit]
After=network.target
[Service]
ExecStartPre=/usr/bin/ntpd -gq

I probably can't reproduce this now I've got them running again, but given its happened to 4 machines on 2 occasions, it needs noting.

Regards

Andrew

[aroberts]

Additional info:
cat /boot/config.txt
# See /boot/overlays/README for all available options

device_tree_param=audio=on
device_tree_param=spi=on
gpu_mem=64

initramfs initramfs-linux.img followkernel

cat /boot/cmdline.txt
root=/dev/mmcblk0p2 rw rootwait console=tty1 selinux=0 plymouth.enable=0 smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 elevator=noop

[graysky]

First of all please use code tags. Secondly, I don't know you're using some override file for sshd... it works out-of-the-box. Thirdly, why are you posting your ntpd service? Finally, how is your networking configured???

Post the output of the following using code tags:
$this->bbcode_second_pass_code('', 'find /etc/systemd/system -mindepth 1 -type d | sed '/getty.target/d' | xargs ls -gG')

[aroberts]

I'm using an override for sshd as the defaults (certainly in the past) didn't auto restart it with a sensible delay.
If networking was slow to start up sshd would fail to start, and you were back to hunting for a keyboard and monitor.
I posted ntpd for completeness, as these were the services I had altered from default.

Here is the output:
$this->bbcode_second_pass_code('', '
find /etc/systemd/system -mindepth 1 -type d | sed '/getty.target/d' | xargs ls -gG
/etc/systemd/system/multi-user.target.wants:
total 0
lrwxrwxrwx 1 39 Aug 2 2016 haveged.service -> /usr/lib/systemd/system/haveged.service
lrwxrwxrwx 1 40 Aug 2 2016 remote-fs.target -> /usr/lib/systemd/system/remote-fs.target
lrwxrwxrwx 1 36 Aug 2 2016 sshd.service -> /usr/lib/systemd/system/sshd.service
lrwxrwxrwx 1 48 Aug 2 2016 systemd-networkd.service -> /usr/lib/systemd/system/systemd-networkd.service
lrwxrwxrwx 1 48 Jul 7 03:13 systemd-resolved.service -> /usr/lib/systemd/system/systemd-resolved.service

/etc/systemd/system/ntpd.service.d:
total 4
-rw-r--r-- 1 70 Aug 29 2016 override.conf
/etc/systemd/system/sockets.target.wants:
total 0
lrwxrwxrwx 1 47 Aug 2 2016 systemd-networkd.socket -> /usr/lib/systemd/system/systemd-networkd.socket

/etc/systemd/system/sshd.service.d:
total 4
-rw-r--r-- 1 66 Aug 22 2016 override.conf

/etc/systemd/system/sysinit.target.wants:
total 0
lrwxrwxrwx 1 49 Aug 2 2016 systemd-timesyncd.service -> /usr/lib/systemd/system/systemd-timesyncd.service
')

Regards

Andrew

[graysky]

So you're using systemd-networkd. Post the network config file/files.

[aroberts]

These haven't been altered from default:

$this->bbcode_second_pass_code('', '
cat systemd-networkd.service
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.

[Unit]
Description=Network Service
Documentation=man:systemd-networkd.service(8)
ConditionCapability=CAP_NET_ADMIN
DefaultDependencies=no
# systemd-udevd.service can be dropped once tuntap is moved to netlink
After=systemd-udevd.service network-pre.target systemd-sysusers.service systemd-sysctl.service
Before=network.target multi-user.target shutdown.target
Conflicts=shutdown.target
Wants=network.target

# On kdbus systems we pull in the busname explicitly, because it
# carries policy that allows the daemon to acquire its name.
Wants=org.freedesktop.network1.busname
After=org.freedesktop.network1.busname

[Service]
Type=notify
Restart=on-failure
RestartSec=0
ExecStart=/usr/lib/systemd/systemd-networkd
WatchdogSec=3min
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_BROADCAST CAP_NET_RAW CAP_SETUID CAP_SETGID CAP_SETPCAP CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER
ProtectSystem=full
ProtectHome=yes
ProtectControlGroups=yes
MemoryDenyWriteExecute=yes
RestrictRealtime=yes
RestrictAddressFamilies=AF_UNIX AF_NETLINK AF_INET AF_INET6 AF_PACKET
SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module @mount @obsolete @raw-io

[Install]
WantedBy=multi-user.target
Also=systemd-networkd.socket

# We want to enable systemd-networkd-wait-online.service whenever this service
# is enabled. systemd-networkd-wait-online.service has
# WantedBy=network-online.target, so enabling it only has an effect if
# network-online.target itself is enabled or pulled in by some other unit.
Also=systemd-networkd-wait-online.service
')

And
$this->bbcode_second_pass_code('', '
cat systemd-networkd.socket
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.

[Unit]
Description=Network Service Netlink Socket
Documentation=man:systemd-networkd.service(8) man:rtnetlink(7)
ConditionCapability=CAP_NET_ADMIN
DefaultDependencies=no
Before=sockets.target

[Socket]
ReceiveBuffer=8M
ListenNetlink=route 1361
PassCredentials=yes

[Install]
WantedBy=sockets.target
')

[graysky]

No, I meant the any config files in /etc/systemd/network

[aroberts]

Ok, again these haven't been altered from default:

$this->bbcode_second_pass_code('', '
cat eth0.network
[Match]
Name=eth0

[Network]
DHCP=yes
')

DHCP is reserved IP by MAC address on a netgear router. Again not had any issues prior to updates this month.
Neither have I seen issues (yet) on a Pi3 running Arch ARM aarch64
$this->bbcode_second_pass_code('', '
uname -a
Linux alarm 4.12.3-1-ARCH #1 SMP Fri Jul 21 21:02:07 MDT 2017 aarch64 GNU/Linux
')

Or an ODroid C2 running Arch ARM aarch64
$this->bbcode_second_pass_code('', '
uname -a
Linux alarm 3.14.79-26-ARCH #1 SMP PREEMPT Tue Jul 4 20:58:48 MDT 2017 aarch64 GNU/Linux
')

Also both original Pi B's (v6l) haven't seen any issues.

Everything is configured identically using DHCP to same router, with IP's reserved by MAC address.

Regards

Andrew

[aroberts]

Last time this happened (Jul 7th), I moved the PI to another switch and connected keyboard/monitor.
It booted ok and I could login on console (I'm not booting to desktop on any machines).But I suspect
network still wasn't working.

When I moved it back to the original switch without keyboard/monitor, networking was still not working. Until I booted without
ethernet cable plugged in. I then plugged it back in and it worked fine, over many reboots. 2nd PI I simply powered off and
booted without cable, and plugged it in after boot, and again everything was fine.

Booting without cable in seems to be key to fixing it. Once you do this once, it then seems ok until whatever triggers it in kernel/boot updates. Not every kernel/boot update causes the issue. I only reboot every time there is a kernel update, I guess there have been a few betweem 7th Jul and 22nd Jul.

I'll try repeatedly rebooting 1 PI to see if I can trigger it without updates.

Andrew

[graysky]

$this->bbcode_second_pass_quote('aroberts', '
')I have 7 Pi's (5 Pi 3's, 2 originals), running arch 24/7. The last kernel/boot code update stopped
the networking starting on 2 of the 5 v3's.