Arch Linux ARM v7 kernel/bootcode update on Raspberry Pi 3 i

Problems with packages? Post here, using [tags] of the package name.

Arch Linux ARM v7 kernel/bootcode update on Raspberry Pi 3 i

Postby aroberts » Sat Jul 22, 2017 11:11 am

Arch Linux ARM v7 kernel/bootcode update on Raspberry Pi 3 is breaking Networking.
I have 7 Pi's (5 Pi 3's, 2 originals), running arch 24/7. The last kernel/boot code update stopped
the networking starting on 2 of the 5 v3's.

In all cases, installed latest updates (pacman -Syu)
and then rebooted after kernel updates had installed. I was logged on to all machines remotely via ssh (they are all headless).
On the 2 machines in question the networking lights didn't come on to indicate a link. Tried powering off/on, still no joy.

I found I had to power on with network cable UNPLUGGED and then plug it in later. This allowed network to start and remote login via ssh.

Both Pi's are plugged into different Netgear gigabit routers.

I had the same issue around 8th July when there were 3 systemd updates in 1 day, but there may also have been kernel/bootcode updates as well. In that case one of the same Pi's was affected, and a different one also (but all PI 3's).

After the network starts up successfully further reboots seem to be ok, I've rebooted ok many times since 8th July, until today.

The machines aren't running out of storage as I purge the old packages regularly etc. I've been running these machines 24/7
since they were released (about a year or so), and the networking issues have only occurred twice, both in the last month. All are configured identically.

uname -a
Linux alarmpi 4.9.38-1-ARCH #1 SMP Sat Jul 22 02:48:38 UTC 2017 armv7l GNU/Linux

tail /proc/cpuinfo
processor : 3
model name : ARMv7 Processor rev 4 (v7l)
BogoMIPS : 38.40
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4

Hardware : BCM2835
Revision : a02082
Serial : Redacted

I'm running sshd (obviously) with an override.conf file (so it starts even if network is slow to start, why is this not the default!!!)
[Unit]
After=ntpd.service

[Service]
Restart=always
RestartSec=5

ntpd is also running with an override.conf: (to ensure time syncs early from local ntpd server)
[Unit]
After=network.target
[Service]
ExecStartPre=/usr/bin/ntpd -gq

I probably can't reproduce this now I've got them running again, but given its happened to 4 machines on 2 occasions, it needs noting.

Regards

Andrew
aroberts
 
Posts: 42
Joined: Tue Mar 15, 2016 4:32 am

Re: Arch Linux ARM v7 kernel/bootcode update on Raspberry Pi

Postby aroberts » Sat Jul 22, 2017 11:14 am

Additional info:
cat /boot/config.txt
# See /boot/overlays/README for all available options

device_tree_param=audio=on
device_tree_param=spi=on
gpu_mem=64

initramfs initramfs-linux.img followkernel

cat /boot/cmdline.txt
root=/dev/mmcblk0p2 rw rootwait console=tty1 selinux=0 plymouth.enable=0 smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 elevator=noop
aroberts
 
Posts: 42
Joined: Tue Mar 15, 2016 4:32 am

Re: Arch Linux ARM v7 kernel/bootcode update on Raspberry Pi

Postby graysky » Sat Jul 22, 2017 11:49 am

First of all please use code tags. Secondly, I don't know you're using some override file for sshd... it works out-of-the-box. Thirdly, why are you posting your ntpd service? Finally, how is your networking configured???

Post the output of the following using code tags:
Code: Select all
find /etc/systemd/system -mindepth 1 -type d | sed '/getty.target/d' | xargs ls -gG
graysky
 
Posts: 355
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

Re: Arch Linux ARM v7 kernel/bootcode update on Raspberry Pi

Postby aroberts » Sat Jul 22, 2017 12:21 pm

I'm using an override for sshd as the defaults (certainly in the past) didn't auto restart it with a sensible delay.
If networking was slow to start up sshd would fail to start, and you were back to hunting for a keyboard and monitor.
I posted ntpd for completeness, as these were the services I had altered from default.

Here is the output:
Code: Select all
find /etc/systemd/system -mindepth 1 -type d | sed '/getty.target/d' | xargs ls -gG
/etc/systemd/system/multi-user.target.wants:
total 0
lrwxrwxrwx 1 39 Aug  2  2016 haveged.service -> /usr/lib/systemd/system/haveged.service
lrwxrwxrwx 1 40 Aug  2  2016 remote-fs.target -> /usr/lib/systemd/system/remote-fs.target
lrwxrwxrwx 1 36 Aug  2  2016 sshd.service -> /usr/lib/systemd/system/sshd.service
lrwxrwxrwx 1 48 Aug  2  2016 systemd-networkd.service -> /usr/lib/systemd/system/systemd-networkd.service
lrwxrwxrwx 1 48 Jul  7 03:13 systemd-resolved.service -> /usr/lib/systemd/system/systemd-resolved.service

/etc/systemd/system/ntpd.service.d:
total 4
-rw-r--r-- 1 70 Aug 29  2016 override.conf
/etc/systemd/system/sockets.target.wants:
total 0
lrwxrwxrwx 1 47 Aug  2  2016 systemd-networkd.socket -> /usr/lib/systemd/system/systemd-networkd.socket

/etc/systemd/system/sshd.service.d:
total 4
-rw-r--r-- 1 66 Aug 22  2016 override.conf

/etc/systemd/system/sysinit.target.wants:
total 0
lrwxrwxrwx 1 49 Aug  2  2016 systemd-timesyncd.service -> /usr/lib/systemd/system/systemd-timesyncd.service


Regards

Andrew
aroberts
 
Posts: 42
Joined: Tue Mar 15, 2016 4:32 am

Re: Arch Linux ARM v7 kernel/bootcode update on Raspberry Pi

Postby graysky » Sat Jul 22, 2017 12:31 pm

So you're using systemd-networkd. Post the network config file/files.
graysky
 
Posts: 355
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

Re: Arch Linux ARM v7 kernel/bootcode update on Raspberry Pi

Postby aroberts » Sat Jul 22, 2017 12:39 pm

These haven't been altered from default:

Code: Select all
cat systemd-networkd.service
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Network Service
Documentation=man:systemd-networkd.service(8)
ConditionCapability=CAP_NET_ADMIN
DefaultDependencies=no
# systemd-udevd.service can be dropped once tuntap is moved to netlink
After=systemd-udevd.service network-pre.target systemd-sysusers.service systemd-sysctl.service
Before=network.target multi-user.target shutdown.target
Conflicts=shutdown.target
Wants=network.target

# On kdbus systems we pull in the busname explicitly, because it
# carries policy that allows the daemon to acquire its name.
Wants=org.freedesktop.network1.busname
After=org.freedesktop.network1.busname

[Service]
Type=notify
Restart=on-failure
RestartSec=0
ExecStart=/usr/lib/systemd/systemd-networkd
WatchdogSec=3min
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_BROADCAST CAP_NET_RAW CAP_SETUID CAP_SETGID CAP_SETPCAP CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER
ProtectSystem=full
ProtectHome=yes
ProtectControlGroups=yes
MemoryDenyWriteExecute=yes
RestrictRealtime=yes
RestrictAddressFamilies=AF_UNIX AF_NETLINK AF_INET AF_INET6 AF_PACKET
SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module @mount @obsolete @raw-io

[Install]
WantedBy=multi-user.target
Also=systemd-networkd.socket

# We want to enable systemd-networkd-wait-online.service whenever this service
# is enabled. systemd-networkd-wait-online.service has
# WantedBy=network-online.target, so enabling it only has an effect if
# network-online.target itself is enabled or pulled in by some other unit.
Also=systemd-networkd-wait-online.service


And
Code: Select all
cat systemd-networkd.socket
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Network Service Netlink Socket
Documentation=man:systemd-networkd.service(8) man:rtnetlink(7)
ConditionCapability=CAP_NET_ADMIN
DefaultDependencies=no
Before=sockets.target

[Socket]
ReceiveBuffer=8M
ListenNetlink=route 1361
PassCredentials=yes

[Install]
WantedBy=sockets.target
aroberts
 
Posts: 42
Joined: Tue Mar 15, 2016 4:32 am

Re: Arch Linux ARM v7 kernel/bootcode update on Raspberry Pi

Postby graysky » Sat Jul 22, 2017 12:48 pm

No, I meant the any config files in /etc/systemd/network
graysky
 
Posts: 355
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

Re: Arch Linux ARM v7 kernel/bootcode update on Raspberry Pi

Postby aroberts » Sat Jul 22, 2017 1:02 pm

Ok, again these haven't been altered from default:

Code: Select all
cat eth0.network
[Match]
Name=eth0

[Network]
DHCP=yes


DHCP is reserved IP by MAC address on a netgear router. Again not had any issues prior to updates this month.
Neither have I seen issues (yet) on a Pi3 running Arch ARM aarch64
Code: Select all
uname -a
Linux alarm 4.12.3-1-ARCH #1 SMP Fri Jul 21 21:02:07 MDT 2017 aarch64 GNU/Linux


Or an ODroid C2 running Arch ARM aarch64
Code: Select all
uname -a
Linux alarm 3.14.79-26-ARCH #1 SMP PREEMPT Tue Jul 4 20:58:48 MDT 2017 aarch64 GNU/Linux


Also both original Pi B's (v6l) haven't seen any issues.

Everything is configured identically using DHCP to same router, with IP's reserved by MAC address.

Regards

Andrew
aroberts
 
Posts: 42
Joined: Tue Mar 15, 2016 4:32 am

Re: Arch Linux ARM v7 kernel/bootcode update on Raspberry Pi

Postby aroberts » Sat Jul 22, 2017 1:15 pm

Last time this happened (Jul 7th), I moved the PI to another switch and connected keyboard/monitor.
It booted ok and I could login on console (I'm not booting to desktop on any machines).But I suspect
network still wasn't working.

When I moved it back to the original switch without keyboard/monitor, networking was still not working. Until I booted without
ethernet cable plugged in. I then plugged it back in and it worked fine, over many reboots. 2nd PI I simply powered off and
booted without cable, and plugged it in after boot, and again everything was fine.

Booting without cable in seems to be key to fixing it. Once you do this once, it then seems ok until whatever triggers it in kernel/boot updates. Not every kernel/boot update causes the issue. I only reboot every time there is a kernel update, I guess there have been a few betweem 7th Jul and 22nd Jul.

I'll try repeatedly rebooting 1 PI to see if I can trigger it without updates.

Andrew
aroberts
 
Posts: 42
Joined: Tue Mar 15, 2016 4:32 am

Re: Arch Linux ARM v7 kernel/bootcode update on Raspberry Pi

Postby graysky » Sat Jul 22, 2017 1:25 pm

aroberts wrote:I have 7 Pi's (5 Pi 3's, 2 originals), running arch 24/7. The last kernel/boot code update stopped
the networking starting on 2 of the 5 v3's.


So RPi3 + which kernel package? Only linux-rasberrypi?
graysky
 
Posts: 355
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

Next

Return to Packages

Who is online

Users browsing this forum: No registered users and 1 guest