Can't get internet working because of systemd-resolved

[Sourav]

Hi, I have a very weird issue along with many others. The problem is that:

I am unable to ping to a website.
I am unable to access a website from the browser.

I have two more computers (all running Arch Linux) connected to the internet, where I can ping and use the internet. Also, the /etc/resolv.conf is same in the other computers:

$this->bbcode_second_pass_code('', '
nameserver 10.230.252.252
nameserver 203.147.88.2
nameserver 8.8.8.8
search domain.name
')

I can use VNC. I can also ping to 8.8.8.8. When trying to access duckduckgo on chromium I get:

$this->bbcode_second_pass_code('', '
This site can’t be reached
duckduckgo.com’s server IP address could not be found.
DNS_PROBE_FINISHED_NXDOMAIN
')

I have an active internet connection.

I can solve it by stopping $this->bbcode_second_pass_code('', 'systemd-resolved'). Sometimes restarting $this->bbcode_second_pass_code('', 'systemd-resolved') works as well.

The question is asked on unix.stackexchange
https://unix.stackexchange.com/question ... -not-known

And others commented that they have this problem too!

Is this a normal behaviour of systemd-resolved?

[summers]

In the last month or so, this problem has popped up regularly, often on RPi machines. If you look at threads here, you see it has often come up. Problem sounds like it a name lookup problem, but many say its a network problem.

I don't know if we are sure of what is causing the problem. Two things seemed involved:
. Multiple network managers running, and getting conflicts between them
. DNSSEC in systemd-resolved

So lets see if we can get more information this time.

So can you post what the following give:
$this->bbcode_second_pass_code('', 'networkctl status')
$this->bbcode_second_pass_code('', 'resolvectl status')
$this->bbcode_second_pass_code('', 'systemctl --all list-units | egrep "net|resol"')
And what is probably fair to say is make sure you are running only one network manager, and if upstream your DNS server has a broken DNSSEC then look at disabling it.

Oh yes - if its broken DNS, you should still be able to ssh into the box from that LAN using the IP address - this will make it easier to run the commands, and post the results.

[Minding]

I have the same issue and setup, this is what my system prints:

$this->bbcode_second_pass_code('', '
[alarm@alarm ~]$ networkctl status
* State: routable
Address: 192.168.2.109 on wlan0
2003:eb:6f06:1e74:ba27:ebff:fef3:56a6 on wlan0
fe80::ba27:ebff:fef3:56a6 on wlan0
Gateway: 192.168.2.1 (HUAWEI TECHNOLOGIES CO.,LTD) on wlan0
fe80::1 (HUAWEI TECHNOLOGIES CO.,LTD) on wlan0
[alarm@alarm ~]$ resolvectl status
Failed to get global data: Unit dbus-org.freedesktop.resolve1.service not found.
[alarm@alarm ~]$ systemctl --all list-units | egrep "net|resol"
sys-devices-platform-soc-3f300000.sdhci-mmc_host-mmc1-mmc1:0001-mmc1:0001:1-net-wlan0.device loaded active plugged /sys/devices/platform/soc/3f300000.sdhci/mmc_host/mmc1/mmc1:0001/mmc1:0001:1/net/wlan0
sys-devices-platform-soc-3f980000.usb-usb1-1\x2d1-1\x2d1.1-1\x2d1.1:1.0-net-eth0.device loaded active plugged SMSC9512/9514 Fast Ethernet Adapter
sys-subsystem-net-devices-eth0.device loaded active plugged SMSC9512/9514 Fast Ethernet Adapter
sys-subsystem-net-devices-wlan0.device loaded active plugged /sys/subsystem/net/devices/wlan0
netctl.service loaded inactive dead (Re)store the netctl profile state
netctl@wlan0\x2dHomeWLAN.service loaded active exited Automatically generated profile by wifi-menu
systemd-networkd.service loaded active running Network Service
system-netctl.slice loaded active active system-netctl.slice
systemd-networkd.socket loaded active running Network Service Netlink Socket
network-pre.target loaded inactive dead Network (Pre)
network.target loaded active active Network
[alarm@alarm ~]$ systemctl status systemd-resolved
* systemd-resolved.service - Network Name Resolution
Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:systemd-resolved.service(8)
https://www.freedesktop.org/wiki/Software/systemd/resolved
https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
[alarm@alarm ~]$ ping google.de
ping: google.de: Temporary failure in name resolution
[alarm@alarm ~]$
')

I previously disabled systemd-resolved.
Each time I boot I need to systemctl start systemd-resolved and systemctl stop systemd-resolved in order to get DNS working.

[summers]

OK can see you are running two network managers, both netctl:
$this->bbcode_second_pass_code('', ' netctl.service loaded inactive dead (Re)store the netctl profile state
netctl@wlan0\x2dHomeWLAN.service loaded active exited Automatically generated profile by wifi-menu
')
and systemd-networkd:
$this->bbcode_second_pass_code('', ' systemd-networkd.service loaded active running Network Service
')
And systemd-resolved isn't runnning - which is why resolvectl doesn't work, a pity as it would tell us what it was trying to do.

So first off, switch to either netctl alone:
$this->bbcode_second_pass_code('', 'systemctl disable systemd-networkd.service')
Or to systemd-networkd:
$this->bbcode_second_pass_code('', 'systemctl disable netctl.service
systemctl disable netctl@wlan0\\x2dHomeWLAN.service
systemctl enable systemd-resolved.service
systemctl start systemd-resolved.service')
Note the netctl@ command has a "\\" as you need to escape it from the shell .... enabling systemd-resolved will replace /etc/resolv.conf with a link.

Either netctl or systemd-networkd should work, but I'd prefer systemd-networkd as then we can ask resolved what it is doing, and see what is going wrong. Anyway make your choice, and do one or the other - then we can dig more into what DNS is doing.

So if you flip to just systemd-networkd, can you run the networkctl and resolvectl commands again, it should give more information.

Edit: And an idle thought after some browsing this lunch time. Is the clock set correctly on the machine, as this would invalidate DNSSEC lookup of *.pool.ntp.org, which could then break all lookups? If so adding the ntp hosts to /etc/hosts may help ...

[Minding]

I stopped systemd-networkd, because I'm connected via SSH over wlan and don't know how to set this up using systemd.

Here is the output:

$this->bbcode_second_pass_code('', '
[root@alarm alarm]# echo "Current date: 16:22 02.01.2020 (UTC+1)"
Current date: 16:22 02.01.2020 (UTC+1)
[root@alarm alarm]# date
Thu Jan 2 02:07:16 UTC 2020
[root@alarm alarm]# networkctl status
WARNING: systemd-networkd is not running, output will be incomplete.

* State: n/a
Address: 192.168.2.109 on wlan0
2003:eb:6f06:1e36:ba27:ebff:fef3:56a6 on wlan0
fe80::ba27:ebff:fef3:56a6 on wlan0
Gateway: 192.168.2.1 (HUAWEI TECHNOLOGIES CO.,LTD) on wlan0
fe80::1 (HUAWEI TECHNOLOGIES CO.,LTD) on wlan0
[root@alarm alarm]# resolvectl status
Failed to get global data: Unit dbus-org.freedesktop.resolve1.service not found.
[root@alarm alarm]# systemctl --all list-units | egrep "net|resol"
sys-devices-platform-soc-3f300000.sdhci-mmc_host-mmc1-mmc1:0001-mmc1:0001:1-net-wlan0.device loaded active plugged /sys/devices/platform/soc/3f300000.sdhci/mmc_host/mmc1/mmc1:0001/mmc1:0001:1/net/wlan0
sys-devices-platform-soc-3f980000.usb-usb1-1\x2d1-1\x2d1.1-1\x2d1.1:1.0-net-eth0.device loaded active plugged SMSC9512/9514 Fast Ethernet Adapter
sys-subsystem-net-devices-eth0.device loaded active plugged SMSC9512/9514 Fast Ethernet Adapter
sys-subsystem-net-devices-wlan0.device loaded active plugged /sys/subsystem/net/devices/wlan0
netctl.service loaded inactive dead (Re)store the netctl profile state
netctl@wlan0\x2dHomeWLAN.service loaded active exited Automatically generated profile by wifi-menu
system-netctl.slice loaded active active system-netctl.slice
network-pre.target loaded inactive dead Network (Pre)
network.target loaded active active Network
[root@alarm alarm]# ping google.de
ping: google.de: Temporary failure in name resolution
[root@alarm alarm]#
')

I did see DNSSEC errors before, but when I checked the time it was set correctly. But now the time is off.

If you could tell me how to setup wlan using systemd, I would send you the info using netctl.

[summers]

netctl does DNS via resolvconf, these days thats provided by openresolv, so make sure thats installed. It should then generate a correct /etc/resolv.conf.

Wifi, if you have a reasonably secure wifi, you'll need wpa_supplicant to authenticate. Details on : https://wiki.archlinux.org/index.php/WPA_supplicant so in systemctl you'll need to enable something like wpa_supplicant.service - I'll check when I get home whats usual - as there it is tied to the interface name ...

For the authentication, I add it to a file in /etc/wpa_supplicant format is on the arch page, but I post an example when I get home.

Apparently time is used in the DNSSEC calculation, so if that is off, you can't do DNS lookups. Thats a problem when time is done via ntp, as it can't connect to the network to set the time. So if you use systemd-timesyncd.service to set the time $this->bbcode_second_pass_code('', 'timedatectl status') should tell you if its working, and it would confirm that it isn't the time which is giving DNSSEC and DNS lookup the problem.

So in the directory /etc/wpa_supplicant/ create the file wpa_supplicant-wlp3s0.conf where replace wlp3s0 with your wifi name ("ip a" will give it). You'll want something in it like:
$this->bbcode_second_pass_code('', 'ctrl_interface=/var/run/wpa_supplicant
network={
ssid="PlusnetWirelessBE0514"
scan_ssid=1
key_mgmt=WPA-PSK
psk="password"
}')
in systemctl you should have something like "wpa_supplicant@.service" enabled.

Oh yes for the systemd ntp time client (systemd-timesyncd.service), you want a file like /etc/systemd/timesyncd.conf
$this->bbcode_second_pass_code('', '# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See timesyncd.conf(5) for details.

[Time]
NTP=ntp.xxxxxx.net
FallbackNTP=0.arch.pool.ntp.org 1.arch.pool.ntp.org 2.arch.pool.ntp.org 3.arch.pool.ntp.org
')
And suggest you put the ip of the main ntp site in /etc/hosts
$this->bbcode_second_pass_code('', 'echo -e 43.72.11.45\\tntp.xxxxxx.net >> /etc/hosts')
you'll get the ip address using "drill" - you should change for your local ntp source.

[Minding]

openresolv and wpa_supplicant are installed already. Here is my output (real date was ~21:40 02.01.2020):

$this->bbcode_second_pass_code('', '
[root@alarm alarm]# timedatectl status
Local time: Thu 2020-01-02 07:19:13 UTC
Universal time: Thu 2020-01-02 07:19:13 UTC
RTC time: n/a
Time zone: UTC (UTC, +0000)
System clock synchronized: no
NTP service: active
RTC in local TZ: no
')

Why does Arch Linux ARM ship with two network managers? I thought it is supposed to be very minimal.

[summers]

Good. Yes the ntp active means time should be accurate, and good enough for DNSSEC. Though it was worth checking, as it seems so strange that this only really affects RPi machines, and the lack of a RTC, could be an explaination. Oh wait clock is way out - so although ntp is active, it can't have syncronised. That probable causes DNSSEC to fail had time so could explain the dns problem. Adding the ntp host to hosts though means it can be looked up before dns is running.

Arch probably has two network managers due to history. Five years ago systemd came along. Before then separate programs were needed for many things that started running at boot. Since then systemd has grown many of these things internally, so networkd, resolved, timesyncd, etc. Arch though still has the old system for those that want it. And I guess there are still users of netctl.

[Sourav]

Hi, I didn't post for a while because the problem occurs rarely on my Pi after a recent installation.
Anyway, I have deliberately turned on both netctl and systemd-networkd and the internet works just fine.

In my case, the output is:
$this->bbcode_second_pass_code('', '
┌┄┄[sourav::archlinux-arm]┈[~]
└──╼⮚ systemctl --all list-units | egrep "net|resol"
sys-devices-platform-soc-3f300000.mmcnr-mmc_host-mmc1-mmc1:0001-mmc1:0001:1-net-wlan0.device loaded active plugged /sys/devices/platform/soc/3f300000.mmcnr/mmc_host/mmc1/mmc1:0001/mmc1:0001:1/net/wlan0
sys-devices-platform-soc-3f980000.usb-usb1-1\x2d1-1\x2d1.1-1\x2d1.1:1.0-net-eth0.device loaded active plugged SMSC9512/9514 Fast Ethernet Adapter
sys-subsystem-net-devices-eth0.device loaded active plugged SMSC9512/9514 Fast Ethernet Adapter
sys-subsystem-net-devices-wlan0.device loaded active plugged /sys/subsystem/net/devices/wlan0
netctl.service loaded active exited (Re)store the netctl profile state
systemd-networkd.service loaded active running Network Service
systemd-resolved.service loaded active running Network Name Resolution
systemd-networkd.socket loaded active running Network Service Netlink Socket
network-pre.target loaded inactive dead Network (Pre)
network.target loaded active active Network
')

I am using NetworkManager to connect to the internet. Although the internet is working now, but it sometimes doesn't work at all, and that's happens at randomly after boots.

[summers]

Sourav. The problem that you sometimes get running both netctl and networkd is oddcasional race condition, e.g. at boot where one or other grabs the interface first, and the other then just free wheels trying to work out whats going on. There is a similar conflict between openresolv and systemd-resolved, the former tends to write to files in /etc whilst the latter maintains a virtual file space in /proc; and then interact with things like dhcp different, e.g. dhcpcd directly calls openresolv; but the systemd dhcp I think goes via its own internal interface/dbus to systemd-resolved. Anyway many of these programs do try and co-exist with others, e.g. systemd-resolved think if it notices something else operating on /etc/resolv.conf; then it leaves it there and works with it.

Good though that your system works. I guess $this->bbcode_second_pass_code('', 'timedatectl status') shows both NTP as active, and the system clock as syncronised? If you network fails to come up again, can you try timedatectl status again, and see if this happens when syncronisation is lost? This will help establishing exactly what the problem is ....