RPi4 AArch64 Network interface error after reboot

This is for ARMv8 based devices

RPi4 AArch64 Network interface error after reboot

Postby tristess » Fri Oct 09, 2020 9:11 pm

Hello,

I am having an issue on my Raspberry Pi 4B using the aarch64 version of ArchLinux.
My setup is up to date and I am facing the issue since I have installed Arch a few weeks ago. I was using Ubuntu 20.04 (kernel 5.4) before and never had this issue so I am raising it here.

The issue is the following:
Most of the time when I am rebooting my Raspberry, the ethernet connection (eth0) stays down. I can see that the ethernet leds gets turned on after the initial boot sequence, but they are not blinking.
When it is working, the ethernet leds are turned off for a few seconds, and when they are back, it's working correctly.

For reference, I am on:
Code: Select all
Linux pi 5.8.9-2-ARCH #1 SMP Tue Sep 15 00:48:59 UTC 2020 aarch64 GNU/Linux

After more investigation, I managed to determine the following:
- When the issue occurs, the interface is down (link down). Manually setting it up afterwards
Code: Select all
ip link set up eth0
works but unfortunately this is something that you cannot do via SSH since you cannot connect to the machine in the first place.
- systemd-networkd reports an issue telling that "there is no such device" as eth0
- I had a look at dmesg and found the following relevant logs:
Code: Select all
bcmgenet fd580000.ethernet: GENET 5.0 EPHY: 0x0000
libphy: bcmgenet MII bus: probed
could not attach to PHY
bcmgenet fd580000.ethernet eth0: failed to connect to PHY

After some research, it seems that I am facing the same issue as reported here:
https://www.raspberrypi.org/forums/viewtopic.php?p=1725123

What I have also determined is the following:
- Rebooting via command line will most of the time trigger the issue
- Unplugging and plugging back the power cable just after will also trigger the issue
- However, if I unplug the power cable, wait for a few seconds and plug it back, the connection will set up fine and the issue won't happen
- Unplugging and plugging back the ethernet cable has no effect

This is as far as I managed to go.

I have read here and there potential solutions - delaying the ethernet connection on boot, write a script to turn it up manually after a delay etc..., but I was wondering if anybody else got a similar issue? Did anyone find a solution/workaround for this?

Could it be something to be fixed on Arch side as I did not have this issue on Ubuntu nor Raspbian?
tristess
 
Posts: 4
Joined: Wed Sep 16, 2020 7:30 am

Re: RPi4 AArch64 Network interface error after reboot

Postby slashdolt » Fri Oct 16, 2020 3:17 am

Bumping for the same issue on an RPI4 8gb. The showstopper issues for aarch64 on this pi were no USB with 8gb RAM and this intermittent ethernet. Ethernet connection needs to be reliable on reboots.

USB now works with 8gb so I thought I was good to go. See: viewtopic.php?f=65&t=14734&start=10#p64769

But this Ethernet issue persists. Sometimes it works after reboot but frequently it fails.

Also running:
Code: Select all
Linux rpi4 5.8.9-2-ARCH #1 SMP Tue Sep 15 00:48:59 UTC 2020 aarch64 GNU/Linux
slashdolt
 
Posts: 8
Joined: Mon Sep 21, 2020 2:01 am

Re: RPi4 AArch64 Network interface error after reboot

Postby kernelpanic » Tue Oct 20, 2020 10:39 am

I had the same issues and there is a workaround to add a small delay to the systemd-networkd.service.

Code: Select all
systemctl edit systemd-networkd


Code: Select all
[Service]
ExecStartPre=/bin/sleep 5
kernelpanic
 
Posts: 4
Joined: Fri Sep 25, 2020 11:09 pm

Re: RPi4 AArch64 Network interface error after reboot

Postby tristess » Mon Oct 26, 2020 12:16 pm

Hi,

Thanks for you replay. Indeed this is a workaround that I saw posted in other threads.

However, I think that this is still something worth investigating because:
- The proposed workaround is still a workaround. We still don’t understand why delaying the activation of eth0 seems to solve the issue. What is 5 seconds might not be enough in some cases?
- I did not face such issue on Ubuntu. So this seems to be Arch-related, and possibly because of the use of mainline kernel. This is why I raised the issue here. I’d be happy to provide more help/logs on it, however I do not know how to do more than I did.
If someone tells me how to provide more detailed info, I’d be happy to help.
tristess
 
Posts: 4
Joined: Wed Sep 16, 2020 7:30 am

Re: RPi4 AArch64 Network interface error after reboot

Postby affeltranger » Mon Nov 02, 2020 9:01 am

Hi

I had the same issue on my pi and delaying the start of systemd-networkd made it better but didn't solve the issue for me.
My current workaround is to use netctl instead of systemd-networkd. Seems to work fine so far (survived about 10 reboots).

https://wiki.archlinux.org/index.php/Netctl

Something like:
Code: Select all
systemctl disable systemd-networkd
systemctl stop systemd-networkd

cp /etc/netctl/examples/ethernet-dhcp /etc/netctl/
systemctl enable netctl
systemctl start netctl
netctl enable ethernet-dhcp

reboot

should do the trick
affeltranger
 
Posts: 1
Joined: Mon Nov 02, 2020 8:32 am

Re: RPi4 AArch64 Network interface error after reboot

Postby graysky » Mon Nov 02, 2020 11:40 am

Confirmed on linux-aarch64-5.9.2. Seems to be a systemd-networkd + aarch64 thing since none of my armv7h boxes do this. Has anyone scanned bug reports against systemd?
EDIT: I did a quick search and found nothing so opened Issue#17509

@affeltranger - Can you confirm that using netctl solves this? How many reboot cycles have you been through with it?

tristess wrote:- When the issue occurs, the interface is down (link down). Manually setting it up afterwards
Code: Select all
ip link set up eth0
works but unfortunately this is something that you cannot do via SSH since you cannot connect to the machine in the first place.


As an aside, you can just restart systemd-network from the terminal rather than running that. I understand ssh access is preferred as many (including myself) run these headless and without an attached keyboard.

EDIT2: https://github.com/systemd/systemd/issues/17509

Based on Lennart's comment about the network driver/PHY disconnect, I did come searching and happened upon https://github.com/raspberrypi/linux/issues/3108
Related?

EDIT3: See, https://github.com/raspberrypi/linux/is ... -721030032

I am trying to build linux-aarch64 now using the workaround patch called out in that ticket. Seems promising from the comments in the patch itself.
graysky
 
Posts: 663
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

Re: RPi4 AArch64 Network interface error after reboot

Postby graysky » Sun Nov 08, 2020 5:04 pm

I think a work-around to the network not coming up on aarch64, is to compile into the image several modules. You can do this easily by:

1. Editing /etc/mkinitcpio.conf and adding the following:
Code: Select all
MODULES=(bcm_phy_lib broadcom mdio_bcm_unimac genet)

2. Regenerate your kernel image
Code: Select all
# mkinitcpio -p linux-aarch64


Credit to ialexiad here: https://github.com/raspberrypi/linux/is ... -723550749
graysky
 
Posts: 663
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

Re: RPi4 AArch64 Network interface error after reboot

Postby faddat » Mon Nov 09, 2020 12:25 pm

I will try this and see if it happens to address my strange issues as well. I sometimes get the solid ethernet port light condition that others are describing here.
faddat
 
Posts: 2
Joined: Tue May 24, 2016 3:31 am

Re: RPi4 AArch64 Network interface error after reboot

Postby tristess » Mon Nov 16, 2020 12:38 pm

Hello,

I tried this workaround, and this is so far so good.

Indeed, the interface seems to be probed earlier on boot. I looked quickly at dmesg and the interface is now probed and mounted early during boot.

I’ll try some more tests and report.
tristess
 
Posts: 4
Joined: Wed Sep 16, 2020 7:30 am

Re: RPi4 AArch64 Network interface error after reboot

Postby graysky » Wed Nov 25, 2020 12:26 am

Glad the information is helpful.
graysky
 
Posts: 663
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

Next

Return to ARMv8 Devices

Who is online

Users browsing this forum: No registered users and 2 guests