Can't get internet working because of systemd-resolved

Problems with packages? Post here, using [tags] of the package name.

Can't get internet working because of systemd-resolved

Postby Sourav » Wed Jan 01, 2020 6:32 am

Hi, I have a very weird issue along with many others. The problem is that:

I am unable to ping to a website.
I am unable to access a website from the browser.

I have two more computers (all running Arch Linux) connected to the internet, where I can ping and use the internet. Also, the /etc/resolv.conf is same in the other computers:

Code: Select all
nameserver 10.230.252.252
nameserver 203.147.88.2
nameserver 8.8.8.8
search domain.name


I can use VNC. I can also ping to 8.8.8.8. When trying to access duckduckgo on chromium I get:

Code: Select all
This site can’t be reached
duckduckgo.com’s server IP address could not be found.
DNS_PROBE_FINISHED_NXDOMAIN


I have an active internet connection.

I can solve it by stopping
Code: Select all
systemd-resolved
. Sometimes restarting
Code: Select all
systemd-resolved
works as well.

The question is asked on unix.stackexchange
https://unix.stackexchange.com/question ... -not-known

And others commented that they have this problem too!

Is this a normal behaviour of systemd-resolved?
Sourav
 
Posts: 9
Joined: Mon Sep 09, 2019 8:38 am

Re: Can't get internet working because of systemd-resolved

Postby summers » Wed Jan 01, 2020 11:46 am

In the last month or so, this problem has popped up regularly, often on RPi machines. If you look at threads here, you see it has often come up. Problem sounds like it a name lookup problem, but many say its a network problem.

I don't know if we are sure of what is causing the problem. Two things seemed involved:
. Multiple network managers running, and getting conflicts between them
. DNSSEC in systemd-resolved

So lets see if we can get more information this time.

So can you post what the following give:
Code: Select all
networkctl status

Code: Select all
resolvectl status

Code: Select all
systemctl --all list-units | egrep "net|resol"

And what is probably fair to say is make sure you are running only one network manager, and if upstream your DNS server has a broken DNSSEC then look at disabling it.

Oh yes - if its broken DNS, you should still be able to ssh into the box from that LAN using the IP address - this will make it easier to run the commands, and post the results.
summers
 
Posts: 950
Joined: Sat Sep 06, 2014 12:56 pm

Re: Can't get internet working because of systemd-resolved

Postby Minding » Wed Jan 01, 2020 10:10 pm

I have the same issue and setup, this is what my system prints:

Code: Select all
[alarm@alarm ~]$ networkctl status
*   State: routable
  Address: 192.168.2.109 on wlan0
           2003:eb:6f06:1e74:ba27:ebff:fef3:56a6 on wlan0
           fe80::ba27:ebff:fef3:56a6 on wlan0
  Gateway: 192.168.2.1 (HUAWEI TECHNOLOGIES CO.,LTD) on wlan0
           fe80::1 (HUAWEI TECHNOLOGIES CO.,LTD) on wlan0
[alarm@alarm ~]$ resolvectl status
Failed to get global data: Unit dbus-org.freedesktop.resolve1.service not found.
[alarm@alarm ~]$ systemctl --all list-units | egrep "net|resol"
  sys-devices-platform-soc-3f300000.sdhci-mmc_host-mmc1-mmc1:0001-mmc1:0001:1-net-wlan0.device loaded    active   plugged   /sys/devices/platform/soc/3f300000.sdhci/mmc_host/mmc1/mmc1:0001/mmc1:0001:1/net/wlan0
  sys-devices-platform-soc-3f980000.usb-usb1-1\x2d1-1\x2d1.1-1\x2d1.1:1.0-net-eth0.device      loaded    active   plugged   SMSC9512/9514 Fast Ethernet Adapter
  sys-subsystem-net-devices-eth0.device                                                        loaded    active   plugged   SMSC9512/9514 Fast Ethernet Adapter
  sys-subsystem-net-devices-wlan0.device                                                       loaded    active   plugged   /sys/subsystem/net/devices/wlan0
  netctl.service                                                                               loaded    inactive dead      (Re)store the netctl profile state
  netctl@wlan0\x2dHomeWLAN.service                                                             loaded    active   exited    Automatically generated profile by wifi-menu
  systemd-networkd.service                                                                     loaded    active   running   Network Service
  system-netctl.slice                                                                          loaded    active   active    system-netctl.slice
  systemd-networkd.socket                                                                      loaded    active   running   Network Service Netlink Socket
  network-pre.target                                                                           loaded    inactive dead      Network (Pre)
  network.target                                                                               loaded    active   active    Network
[alarm@alarm ~]$ systemctl status systemd-resolved
* systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: man:systemd-resolved.service(8)
             https://www.freedesktop.org/wiki/Software/systemd/resolved
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
[alarm@alarm ~]$ ping google.de
ping: google.de: Temporary failure in name resolution
[alarm@alarm ~]$


I previously disabled systemd-resolved.
Each time I boot I need to systemctl start systemd-resolved and systemctl stop systemd-resolved in order to get DNS working.
Minding
 
Posts: 4
Joined: Wed Jan 01, 2020 10:06 pm

Re: Can't get internet working because of systemd-resolved

Postby summers » Thu Jan 02, 2020 8:57 am

OK can see you are running two network managers, both netctl:
Code: Select all
  netctl.service                                                                               loaded    inactive dead      (Re)store the netctl profile state
  netctl@wlan0\x2dHomeWLAN.service                                                             loaded    active   exited    Automatically generated profile by wifi-menu

and systemd-networkd:
Code: Select all
  systemd-networkd.service                                                                     loaded    active   running   Network Service

And systemd-resolved isn't runnning - which is why resolvectl doesn't work, a pity as it would tell us what it was trying to do.

So first off, switch to either netctl alone:
Code: Select all
systemctl disable systemd-networkd.service

Or to systemd-networkd:
Code: Select all
systemctl disable netctl.service
systemctl disable netctl@wlan0\\x2dHomeWLAN.service
systemctl enable systemd-resolved.service
systemctl start systemd-resolved.service

Note the netctl@ command has a "\\" as you need to escape it from the shell .... enabling systemd-resolved will replace /etc/resolv.conf with a link.

Either netctl or systemd-networkd should work, but I'd prefer systemd-networkd as then we can ask resolved what it is doing, and see what is going wrong. Anyway make your choice, and do one or the other - then we can dig more into what DNS is doing.

So if you flip to just systemd-networkd, can you run the networkctl and resolvectl commands again, it should give more information.

Edit: And an idle thought after some browsing this lunch time. Is the clock set correctly on the machine, as this would invalidate DNSSEC lookup of *.pool.ntp.org, which could then break all lookups? If so adding the ntp hosts to /etc/hosts may help ...
summers
 
Posts: 950
Joined: Sat Sep 06, 2014 12:56 pm

Re: Can't get internet working because of systemd-resolved

Postby Minding » Thu Jan 02, 2020 3:27 pm

I stopped systemd-networkd, because I'm connected via SSH over wlan and don't know how to set this up using systemd.

Here is the output:

Code: Select all
[root@alarm alarm]# echo "Current date: 16:22 02.01.2020 (UTC+1)"
Current date: 16:22 02.01.2020 (UTC+1)
[root@alarm alarm]# date
Thu Jan  2 02:07:16 UTC 2020
[root@alarm alarm]# networkctl status
WARNING: systemd-networkd is not running, output will be incomplete.

*   State: n/a
  Address: 192.168.2.109 on wlan0
           2003:eb:6f06:1e36:ba27:ebff:fef3:56a6 on wlan0
           fe80::ba27:ebff:fef3:56a6 on wlan0
  Gateway: 192.168.2.1 (HUAWEI TECHNOLOGIES CO.,LTD) on wlan0
           fe80::1 (HUAWEI TECHNOLOGIES CO.,LTD) on wlan0
[root@alarm alarm]# resolvectl status
Failed to get global data: Unit dbus-org.freedesktop.resolve1.service not found.
[root@alarm alarm]# systemctl --all list-units | egrep "net|resol"
  sys-devices-platform-soc-3f300000.sdhci-mmc_host-mmc1-mmc1:0001-mmc1:0001:1-net-wlan0.device loaded    active   plugged   /sys/devices/platform/soc/3f300000.sdhci/mmc_host/mmc1/mmc1:0001/mmc1:0001:1/net/wlan0
  sys-devices-platform-soc-3f980000.usb-usb1-1\x2d1-1\x2d1.1-1\x2d1.1:1.0-net-eth0.device      loaded    active   plugged   SMSC9512/9514 Fast Ethernet Adapter
  sys-subsystem-net-devices-eth0.device                                                        loaded    active   plugged   SMSC9512/9514 Fast Ethernet Adapter
  sys-subsystem-net-devices-wlan0.device                                                       loaded    active   plugged   /sys/subsystem/net/devices/wlan0
  netctl.service                                                                               loaded    inactive dead      (Re)store the netctl profile state
  netctl@wlan0\x2dHomeWLAN.service                                                             loaded    active   exited    Automatically generated profile by wifi-menu
  system-netctl.slice                                                                          loaded    active   active    system-netctl.slice             
  network-pre.target                                                                           loaded    inactive dead      Network (Pre)                   
  network.target                                                                               loaded    active   active    Network                         
[root@alarm alarm]# ping google.de
ping: google.de: Temporary failure in name resolution
[root@alarm alarm]#


I did see DNSSEC errors before, but when I checked the time it was set correctly. But now the time is off.

If you could tell me how to setup wlan using systemd, I would send you the info using netctl.
Minding
 
Posts: 4
Joined: Wed Jan 01, 2020 10:06 pm

Re: Can't get internet working because of systemd-resolved

Postby summers » Thu Jan 02, 2020 4:07 pm

netctl does DNS via resolvconf, these days thats provided by openresolv, so make sure thats installed. It should then generate a correct /etc/resolv.conf.

Wifi, if you have a reasonably secure wifi, you'll need wpa_supplicant to authenticate. Details on : https://wiki.archlinux.org/index.php/WPA_supplicant so in systemctl you'll need to enable something like wpa_supplicant.service - I'll check when I get home whats usual - as there it is tied to the interface name ...

For the authentication, I add it to a file in /etc/wpa_supplicant format is on the arch page, but I post an example when I get home.

Apparently time is used in the DNSSEC calculation, so if that is off, you can't do DNS lookups. Thats a problem when time is done via ntp, as it can't connect to the network to set the time. So if you use systemd-timesyncd.service to set the time
Code: Select all
timedatectl status
should tell you if its working, and it would confirm that it isn't the time which is giving DNSSEC and DNS lookup the problem.

So in the directory /etc/wpa_supplicant/ create the file wpa_supplicant-wlp3s0.conf where replace wlp3s0 with your wifi name ("ip a" will give it). You'll want something in it like:
Code: Select all
ctrl_interface=/var/run/wpa_supplicant
network={
   ssid="PlusnetWirelessBE0514"
   scan_ssid=1
   key_mgmt=WPA-PSK
   psk="password"
}

in systemctl you should have something like "wpa_supplicant@.service" enabled.

Oh yes for the systemd ntp time client (systemd-timesyncd.service), you want a file like /etc/systemd/timesyncd.conf
Code: Select all
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See timesyncd.conf(5) for details.

[Time]
NTP=ntp.xxxxxx.net
FallbackNTP=0.arch.pool.ntp.org 1.arch.pool.ntp.org 2.arch.pool.ntp.org 3.arch.pool.ntp.org

And suggest you put the ip of the main ntp site in /etc/hosts
Code: Select all
echo -e 43.72.11.45\\tntp.xxxxxx.net >> /etc/hosts

you'll get the ip address using "drill" - you should change for your local ntp source.
Last edited by summers on Fri Jan 03, 2020 1:52 pm, edited 1 time in total.
summers
 
Posts: 950
Joined: Sat Sep 06, 2014 12:56 pm

Re: Can't get internet working because of systemd-resolved

Postby Minding » Thu Jan 02, 2020 8:40 pm

openresolv and wpa_supplicant are installed already. Here is my output (real date was ~21:40 02.01.2020):

Code: Select all
[root@alarm alarm]# timedatectl status
               Local time: Thu 2020-01-02 07:19:13 UTC
           Universal time: Thu 2020-01-02 07:19:13 UTC
                 RTC time: n/a
                Time zone: UTC (UTC, +0000)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no


Why does Arch Linux ARM ship with two network managers? I thought it is supposed to be very minimal.
Minding
 
Posts: 4
Joined: Wed Jan 01, 2020 10:06 pm

Re: Can't get internet working because of systemd-resolved

Postby summers » Thu Jan 02, 2020 10:49 pm

Good. Yes the ntp active means time should be accurate, and good enough for DNSSEC. Though it was worth checking, as it seems so strange that this only really affects RPi machines, and the lack of a RTC, could be an explaination. Oh wait clock is way out - so although ntp is active, it can't have syncronised. That probable causes DNSSEC to fail had time so could explain the dns problem. Adding the ntp host to hosts though means it can be looked up before dns is running.

Arch probably has two network managers due to history. Five years ago systemd came along. Before then separate programs were needed for many things that started running at boot. Since then systemd has grown many of these things internally, so networkd, resolved, timesyncd, etc. Arch though still has the old system for those that want it. And I guess there are still users of netctl.
summers
 
Posts: 950
Joined: Sat Sep 06, 2014 12:56 pm

Re: Can't get internet working because of systemd-resolved

Postby Sourav » Fri Jan 03, 2020 7:36 am

Hi, I didn't post for a while because the problem occurs rarely on my Pi after a recent installation.
Anyway, I have deliberately turned on both netctl and systemd-networkd and the internet works just fine.

In my case, the output is:
Code: Select all
┌┄┄[sourav::archlinux-arm]┈[~]
└──╼⮚ systemctl --all list-units | egrep "net|resol"
  sys-devices-platform-soc-3f300000.mmcnr-mmc_host-mmc1-mmc1:0001-mmc1:0001:1-net-wlan0.device loaded    active   plugged   /sys/devices/platform/soc/3f300000.mmcnr/mmc_host/mmc1/mmc1:0001/mmc1:0001:1/net/wlan0
  sys-devices-platform-soc-3f980000.usb-usb1-1\x2d1-1\x2d1.1-1\x2d1.1:1.0-net-eth0.device      loaded    active   plugged   SMSC9512/9514 Fast Ethernet Adapter                                                   
  sys-subsystem-net-devices-eth0.device                                                        loaded    active   plugged   SMSC9512/9514 Fast Ethernet Adapter                                                   
  sys-subsystem-net-devices-wlan0.device                                                       loaded    active   plugged   /sys/subsystem/net/devices/wlan0                                                     
  netctl.service                                                                               loaded    active   exited    (Re)store the netctl profile state                                                   
  systemd-networkd.service                                                                     loaded    active   running   Network Service                                                                       
  systemd-resolved.service                                                                     loaded    active   running   Network Name Resolution                                                               
  systemd-networkd.socket                                                                      loaded    active   running   Network Service Netlink Socket                                                       
  network-pre.target                                                                           loaded    inactive dead      Network (Pre)                                                                         
  network.target                                                                               loaded    active   active    Network


I am using NetworkManager to connect to the internet. Although the internet is working now, but it sometimes doesn't work at all, and that's happens at randomly after boots.
Sourav
 
Posts: 9
Joined: Mon Sep 09, 2019 8:38 am

Re: Can't get internet working because of systemd-resolved

Postby summers » Fri Jan 03, 2020 11:38 am

Sourav. The problem that you sometimes get running both netctl and networkd is oddcasional race condition, e.g. at boot where one or other grabs the interface first, and the other then just free wheels trying to work out whats going on. There is a similar conflict between openresolv and systemd-resolved, the former tends to write to files in /etc whilst the latter maintains a virtual file space in /proc; and then interact with things like dhcp different, e.g. dhcpcd directly calls openresolv; but the systemd dhcp I think goes via its own internal interface/dbus to systemd-resolved. Anyway many of these programs do try and co-exist with others, e.g. systemd-resolved think if it notices something else operating on /etc/resolv.conf; then it leaves it there and works with it.

Good though that your system works. I guess
Code: Select all
timedatectl status
shows both NTP as active, and the system clock as syncronised? If you network fails to come up again, can you try timedatectl status again, and see if this happens when syncronisation is lost? This will help establishing exactly what the problem is ....
summers
 
Posts: 950
Joined: Sat Sep 06, 2014 12:56 pm

Next

Return to Packages

Who is online

Users browsing this forum: No registered users and 3 guests