Kernel 6.18.20-1 breaks networking

[graysky]

Can one of you guys experiencing this problem try the config I posted above with systemd-networkd? I cannot reproduce this.

[Rapty]

@graysky
I did. Please see my previous message.

It's probably linked to another specific config. When I have time I will try to start a new SD from scratch and I will try to found out which component produce this issue.

[graysky]

I missed that reply. We have one report that linux-rpi-6.18.18-3 does not cause this. If you don't have a copy of that, you can grab it from https:// pkgmirror.sametimetomorrow.net/aarch64/packages/l/linux-rpi/

If downgrading the kernel restores functionality, it feels like an upstream issue that got introduced between those two versions.

[df8oe]

I think, too it is an upstream issue. Because my rpi4 is a production machine which controls my home automation and regenerative energy management I cannot simply leave NetworkManager and step to systemd-networkd because that would result in many configuration changes. I only can state that the problem does not exist with 6.18.18-3 and pops up with 6.18.20-1. Exact report is:
when booting network devices get ip-addresses and network is managed correctly by NetworkManager. After a short time (max. 1 minute) all network devices go offline without anything in logs what can be the cause. NetworkManager.service is running continuously and there is absolute nothing different than running kernel 6.18.18-3. I tried to restart all services that are related to network but nothing restores networking. Only reboot gives network for a couple of seconds. Is that machine-related (firmware package incompatible with new kernel in a small part)? All who have issues use rpi4?

[Rapty]

Just the same, except systemd-network instead of Network Manager.

RPi4
EEPROM up to date

@graysky: we are actually two, saying downgrading kernel solves the issue.

[graysky]

> If downgrading the kernel restores functionality, it feels like an upstream issue that got introduced between those two versions.

You can reference a last-known-good commit of 95f09046ad083afdd431bf3ea075c891e571db36 (our 6.18.18-3)
And a first-bad commit of 6ad963aa83d44281c028ca03537157aa654374ed (our 6.28.20-1)

If this were me, I would perform a git bisect using these two as anchors. Since I can't trigger the bug, It doesn't make any sense for me to do this. Does anyone on this thread experiencing the bug have the ability to do the bisect? In the meantime, someone needs to open an issue against the Raspberry Pi Foundation kernel so they are aware.

[unifai]

Same issue here, upgrading the kernel 6.18.16-3 => 6.18.20-1 broke the ethernet. seems to be working fine at boot, but then after a few minutes download rate collapses. upload seems to be working, but slow.

using NetworkManager too.
using linux-rpi aarch64 on RPi 4 8GB.

downgrading to 6.18.16-3 fixes it.

here is a ping with 6.18.20-1:

64 bytes from 192.168.1.13: icmp_seq=384 ttl=64 time=2056 ms
64 bytes from 192.168.1.13: icmp_seq=385 ttl=64 time=1032 ms
64 bytes from 192.168.1.13: icmp_seq=386 ttl=64 time=8.02 ms
64 bytes from 192.168.1.13: icmp_seq=388 ttl=64 time=2.93 ms
64 bytes from 192.168.1.13: icmp_seq=389 ttl=64 time=3.07 ms
64 bytes from 192.168.1.13: icmp_seq=390 ttl=64 time=2.14 ms
64 bytes from 192.168.1.13: icmp_seq=391 ttl=64 time=6.38 ms
64 bytes from 192.168.1.13: icmp_seq=392 ttl=64 time=289 ms
64 bytes from 192.168.1.13: icmp_seq=393 ttl=64 time=51.8 ms
64 bytes from 192.168.1.13: icmp_seq=394 ttl=64 time=924 ms
64 bytes from 192.168.1.13: icmp_seq=395 ttl=64 time=929 ms
64 bytes from 192.168.1.13: icmp_seq=396 ttl=64 time=27.5 ms
64 bytes from 192.168.1.13: icmp_seq=397 ttl=64 time=18.9 ms
64 bytes from 192.168.1.13: icmp_seq=398 ttl=64 time=498 ms
64 bytes from 192.168.1.13: icmp_seq=399 ttl=64 time=3.80 ms
64 bytes from 192.168.1.13: icmp_seq=400 ttl=64 time=3.17 ms
64 bytes from 192.168.1.13: icmp_seq=401 ttl=64 time=395 ms
64 bytes from 192.168.1.13: icmp_seq=402 ttl=64 time=492 ms
64 bytes from 192.168.1.13: icmp_seq=403 ttl=64 time=491 ms
...
after some time it becomes unreachable.


ping with 6.18.16-3:

64 bytes from 192.168.1.13: icmp_seq=736 ttl=64 time=2.23 ms
64 bytes from 192.168.1.13: icmp_seq=737 ttl=64 time=2.04 ms
64 bytes from 192.168.1.13: icmp_seq=738 ttl=64 time=3.04 ms
64 bytes from 192.168.1.13: icmp_seq=739 ttl=64 time=3.25 ms
64 bytes from 192.168.1.13: icmp_seq=740 ttl=64 time=1.69 ms
64 bytes from 192.168.1.13: icmp_seq=741 ttl=64 time=2.13 ms
64 bytes from 192.168.1.13: icmp_seq=742 ttl=64 time=1.81 ms
64 bytes from 192.168.1.13: icmp_seq=743 ttl=64 time=2.48 ms
64 bytes from 192.168.1.13: icmp_seq=744 ttl=64 time=1.70 ms
64 bytes from 192.168.1.13: icmp_seq=745 ttl=64 time=6.92 ms
64 bytes from 192.168.1.13: icmp_seq=746 ttl=64 time=1.97 ms
64 bytes from 192.168.1.13: icmp_seq=747 ttl=64 time=2.92 ms
64 bytes from 192.168.1.13: icmp_seq=748 ttl=64 time=1.91 ms
64 bytes from 192.168.1.13: icmp_seq=749 ttl=64 time=1.95 ms
64 bytes from 192.168.1.13: icmp_seq=750 ttl=64 time=2.52 ms
64 bytes from 192.168.1.13: icmp_seq=751 ttl=64 time=2.81 ms
64 bytes from 192.168.1.13: icmp_seq=752 ttl=64 time=2.92 ms
64 bytes from 192.168.1.13: icmp_seq=753 ttl=64 time=2.43 ms
64 bytes from 192.168.1.13: icmp_seq=754 ttl=64 time=2.17 ms
64 bytes from 192.168.1.13: icmp_seq=755 ttl=64 time=2.52 ms
64 bytes from 192.168.1.13: icmp_seq=756 ttl=64 time=67.1 ms
64 bytes from 192.168.1.13: icmp_seq=757 ttl=64 time=1.89 ms

[graysky]

Someone needs to open an issue upstream https:// github.com/raspberrypi/linux/issues

[Rapty]

Thank you, @amphiprion-symb !

https:// github.com/raspberrypi/linux/issues/7304

[df8oe]

Of course the bug is also present in 6.18.21-2. It is definitely a "bad commit".