Espressobin not forwarding packets correctly with mainline

This is for ARMv8 based devices

Espressobin not forwarding packets correctly with mainline

Postby tonipatroni » Sat Dec 04, 2021 6:02 pm

Hi,

I got quite a weird problem, short backstory: Using an espressobin as my main router since a while, noticed that linux-espressobin seemed to be stuck on 5.10 for to long but as I read here and there that mainline support should work out nowadays I go for it and install linux-aarch64, it initially fails but after I'm re-understanding u-boot again (it has been a while) I get it to boot. Now, seemed to be working great initially, the espressobin can connect to the network, my wireguard setup also still works, but then I notice that my whole LAN has no internet, well besides the one raspi zero w that solely uses the espressobin's wiregurad, fishy as I changed nothing on how I do masquerading with my quite simple nftables setup.

I switch to my ISP's router for now to get internet to my people and plug the espressobin for testing between that ISP router as new WAN and my desktop as test client, so that part of the network looks currently about like:
Untitled-2021-12-04-1843(2).png
Untitled-2021-12-04-1843(2).png (52.75 KiB) Viewed 2153 times


So I connect to my Raspberry Pi Zero W, add a route for the espressobin LAN like:
$this->bbcode_second_pass_code('', 'ip route add 192.168.1.0/24 via 192.168.0.115')
And sure enough I can ping the Espressobin also on 192.168.1.1, but when I try my desktop while running tcpdump I get the following weird result:
$this->bbcode_second_pass_code('', '20:21:13.482962 IP 192.168.0.80 > desktop: ICMP echo request, id 4, seq 11, length 64
20:21:13.483370 IP desktop > 192.168.0.80: ICMP echo reply, id 4, seq 11, length 64
20:21:14.537007 IP 192.168.0.80 > desktop: ICMP echo request, id 4, seq 12, length 64
20:21:14.537420 IP desktop > 192.168.0.80: ICMP echo reply, id 4, seq 12, length 64
20:21:15.563434 IP 192.168.0.80 > desktop: ICMP echo request, id 4, seq 13, length 64
20:21:15.563851 IP desktop > 192.168.0.80: ICMP echo reply, id 4, seq 13, length 64')

Iow., I see both request and also the reply so my desktop gets the ICMP request, but the RPi Zero doesn't get that reply anymore, so its lost, I have a suspicion regarding the kernels forwarding chain, or the network stack itself, possible in the Forwarding DB of the nic/bridge or the like..

And if I then boot back to the 5.10 espressobin setup all works out great again, ping and also other connections from the LAN behind of the espressobin to the WAN in front.

Is this a kernel bug, do newer kernel require something else? Albeit I checked on my x86_64 work setup and 5.15.5, masquerading worked there flawlessly without special care required...
tonipatroni
 
Posts: 6
Joined: Thu Sep 03, 2020 7:31 am

Re: Espressobin not forwarding packets correctly with mainli

Postby summers » Sat Dec 04, 2021 7:45 pm

Have you set up forwarding on the espressobin? Does the desktop know how to route, and does the epressobin? Are you just using numbers, or also dns lookup? All of these can go wrong. I've a set up like yours and it works fine, although the router not updated in some time as root partition is only 1gb, and that makes it painful. Anyway if any of above wrong, I should be able to check my set up and give commands.
summers
 
Posts: 984
Joined: Sat Sep 06, 2014 12:56 pm

Re: Espressobin not forwarding packets correctly with mainli

Postby summers » Sun Dec 05, 2021 10:11 am

OK on my desktop now - so can look at commands I used.

On the expressobin in the various /etc/system/network/* files you'll want:
$this->bbcode_second_pass_code('', '[network]
IPForward=true')
Check it works via:
$this->bbcode_second_pass_code('', 'sysctl -a | grep forward')
On the expressonbin interface for the desk top you also want to include:
$this->bbcode_second_pass_code('', '[Network]
DHCPServer=true')
Then think about what services you want to export.

Then probably worth doing an "ip r" on all machines. Most should be set up automatically - you have already done the one necessary one. It will be worth checking the espressobin default route - and check its correct.
summers
 
Posts: 984
Joined: Sat Sep 06, 2014 12:56 pm

Re: Espressobin not forwarding packets correctly with mainli

Postby tonipatroni » Tue Dec 07, 2021 2:24 pm

Thanks for your reply.

Yes, I triple-triple checked that I got forwarding (and sysctl actually shows it), and the espressobin itself has network which it gets over DHCP, so that part works too, and yes routes are all there, e.g.:

$this->bbcode_second_pass_code('', 'default via 192.168.0.1 dev wan proto dhcp src 192.168.0.115 metric 1024
10.11.11.0/24 dev wg0 scope link
192.168.0.0/24 dev wan proto kernel scope link src 192.168.0.115 metric 1024
192.168.0.1 dev wan proto dhcp scope link src 192.168.0.115 metric 1024
192.168.1.0/24 dev br0 proto kernel scope link src 192.168.1.1')

I really just need to boot into linux-espressobin to make it work and boot into linux-aarch64 to break it, forwarding, routes, firewall, ... all stays the same 1:1
tonipatroni
 
Posts: 6
Joined: Thu Sep 03, 2020 7:31 am

Re: Espressobin not forwarding packets correctly with mainli

Postby tonipatroni » Wed Dec 08, 2021 3:56 pm

@summers can you confirm that you use the mainline kernel and that forwarding/routing of egress traffic including replying back coming from "behind" the espressobin works for you with that kernel? Or do you use the linux-espressobin one?
tonipatroni
 
Posts: 6
Joined: Thu Sep 03, 2020 7:31 am

Re: Espressobin not forwarding packets correctly with mainli

Postby tonipatroni » Wed Dec 08, 2021 4:36 pm

Just posting more info as I find it, maybe it helps someone else in the future.

Seems like it *maybe* could be a regression in the phylink part of the kernel that came in with 5.15, at least the following recent thread has similar switch chip using the same kernel module and reports issues that could explain my behaviour, not sure yet though, it seems I need to setup an arch cross-compile setup to test that..

https://lore.kernel.org/netdev/b98043f6 ... abora.com/
tonipatroni
 
Posts: 6
Joined: Thu Sep 03, 2020 7:31 am


Return to ARMv8 Devices

Who is online

Users browsing this forum: No registered users and 7 guests