[SOLVED] DNS not working after pacman -Syu

Ask questions about Arch Linux ARM. Please search before making a new topic.

[SOLVED] DNS not working after pacman -Syu

Postby McFox » Fri May 03, 2019 7:54 am

Hi!

I did a full system update on my Raspberry Pi 2 yesterday. Everything went fine, but after rebooting it can't resolve hostnames anymore. Any idea what could've changed? I tried editing resolvconf.conf with various different DNS's (8.8.8.8 and 8.8.8.4, my ISP's own ones, and my router address, which are usually the ones that work) and doing resolvconf -u , but that didn't change anything. The configuration files seemed fine to begin with too, so I don't think that's the issue anyway.

(I decided to post this here instead of Armv7 because it doesn't seem a like RPi specific thing to me)

Could the dhcp stuff happening during boot have changed somehow? I've disabled dhcpcd, because I'm booting off of an nfs share and if dhcpcd starts, the connection drops for a while resulting in a kernel panic. Not running dhcpcd caused some similar problems before, but I don't remember how I fixed it...

Thanks in advance for any help anyone can provide! I'm not at home at the moment so I can't provide any version numbers etc. right now (since the Pi can't resolve addresses, it can't connect to my VPN either so I can't access it "outside"), but I'll get back to it in the evening.

By the way, earlier the Pi wouldn't boot and when I connected it to a monitor it said "vers=3" is unknown - that's an nfs parameter, and it worked again after removing that from the cmdline.txt. Weird, because it always worked before. I don't even recall updating anything between...
Last edited by McFox on Sat May 04, 2019 11:20 am, edited 1 time in total.
McFox
 
Posts: 20
Joined: Fri Mar 21, 2014 12:57 pm

Re: DNS not working after pacman -Syu

Postby DanSman » Fri May 03, 2019 9:53 am

Do you use unbound/bind/pihole etc.?

Anyhow, you can try to stop
Code: Select all
sudo systemctl stop systemd-resolved.service

If it's working again.
Code: Select all
sudo systemctl disable systemd-resolved.service


Check pacnews as well.

Good luck
DanSman
 
Posts: 9
Joined: Wed Jan 30, 2019 7:09 pm

Re: DNS not working after pacman -Syu

Postby McFox » Fri May 03, 2019 10:17 am

DanSman wrote:Do you use unbound/bind/pihole etc.?


Nope, I don't.

DanSman wrote:systemd-resolved.service


Hmmm... I tried start/restart on that. I haven't enabled it myself, so I thought it wasn't enabled in the first place. I'll check that out when I get home.

Oh and yeah, I totally forgot about pacnew-files possibly being created. I don't update that particular system very often, so when I do, it updates basically everything and it'd be a huge operation to go through every change.
McFox
 
Posts: 20
Joined: Fri Mar 21, 2014 12:57 pm

Re: DNS not working after pacman -Syu

Postby McFox » Fri May 03, 2019 7:23 pm

Hey, that worked! Thanks a lot :)
McFox
 
Posts: 20
Joined: Fri Mar 21, 2014 12:57 pm

Re: DNS not working after pacman -Syu

Postby graysky » Fri May 03, 2019 8:47 pm

I found the same issue: system time was unset, could not resolve host names (ie ping http://www.google.com failed).

Code: Select all
% journalctl -u systemd-resolved
Apr 30 08:51:57 basement systemd-resolved[484]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in->
Apr 30 08:51:57 basement systemd-resolved[484]: Using system hostname 'basement'.
Apr 30 08:51:57 basement systemd[1]: Started Network Name Resolution.
Apr 30 08:51:57 basement systemd-resolved[484]: DNSSEC validation failed for question . IN DNSKEY: signature-expired
Apr 30 08:51:57 basement systemd-resolved[484]: DNSSEC validation failed for question com IN DS: signature-expired
Apr 30 08:51:57 basement systemd-resolved[484]: DNSSEC validation failed for question com IN DNSKEY: signature-expired
Apr 30 08:51:57 basement systemd-resolved[484]: DNSSEC validation failed for question yahoo.com IN DS: signature-expired
Apr 30 08:51:57 basement systemd-resolved[484]: DNSSEC validation failed for question yahoo.com IN SOA: signature-expired
Apr 30 08:51:57 basement systemd-resolved[484]: DNSSEC validation failed for question www.yahoo.com IN A: signature-expired
Apr 30 08:51:58 basement systemd-resolved[484]: DNSSEC validation failed for question . IN DNSKEY: signature-expired
Apr 30 08:51:58 basement systemd-resolved[484]: DNSSEC validation failed for question org IN DS: signature-expired
Apr 30 08:51:58 basement systemd-resolved[484]: DNSSEC validation failed for question org IN DNSKEY: signature-expired
...
Apr 30 08:52:14 basement systemd-resolved[484]: DNSSEC validation failed for question 0.arch.pool.ntp.org IN DS: signature-expired
Apr 30 08:52:14 basement systemd-resolved[484]: DNSSEC validation failed for question 0.arch.pool.ntp.org IN SOA: signature-expired
Apr 30 08:52:14 basement systemd-resolved[484]: DNSSEC validation failed for question 0.arch.pool.ntp.org IN A: signature-expired
Apr 30 08:52:14 basement systemd-resolved[484]: DNSSEC validation failed for question . IN DNSKEY: signature-expired
Apr 30 08:52:14 basement systemd-resolved[484]: DNSSEC validation failed for question org IN DS: signature-expired
Apr 30 08:52:14 basement systemd-resolved[484]: DNSSEC validation failed for question org IN DNSKEY: signature-expired
Apr 30 08:52:14 basement systemd-resolved[484]: DNSSEC validation failed for question ntp.org IN DS: signature-expired
Apr 30 08:52:14 basement systemd-resolved[484]: DNSSEC validation failed for question ntp.org IN SOA: signature-expired
Apr 30 08:52:14 basement systemd-resolved[484]: DNSSEC validation failed for question pool.ntp.org IN DS: signature-expired
Apr 30 08:52:14 basement systemd-resolved[484]: DNSSEC validation failed for question 1.arch.pool.ntp.org IN SOA: signature-expired
Apr 30 08:52:14 basement systemd-resolved[484]: DNSSEC validation failed for question 1.arch.pool.ntp.org IN A: signature-expired


Stopping systemd-resolved, and manually defining /etc/resolv.conf fixed the issue:

Code: Select all
# cat /etc/resolv.conf
nameserver 192.168.1.1


What I find odd is that only 1 of 3 RPi boxes had this problem.

EDIT: https://bbs.archlinux.org/viewtopic.php?id=240427
graysky
 
Posts: 940
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

Re: [SOLVED] DNS not working after pacman -Syu

Postby MattM » Sun May 05, 2019 11:13 am

The issues with systemd-resolved and DNSSEC are not limited to the failed validation when the system clock is incorrect.
When the clock was set correctly and google.com was reachable, it was still not possible on one of my RPis to access mirror.archlinuxarm.org.
Code: Select all
[user@alarmpi ~]$ sudo pacman -Syu
:: Synchronizing package databases...
error: failed retrieving file 'core.db' from mirror.archlinuxarm.org : Could not resolve host: mirror.archlinuxarm.org
error: failed to update core (invalid url for server)
error: failed retrieving file 'extra.db' from mirror.archlinuxarm.org : Could not resolve host: mirror.archlinuxarm.org
error: failed to update extra (invalid url for server)
error: failed retrieving file 'community.db' from mirror.archlinuxarm.org : Could not resolve host: mirror.archlinuxarm.org
error: failed to update community (invalid url for server)
error: failed retrieving file 'alarm.db' from mirror.archlinuxarm.org : Could not resolve host: mirror.archlinuxarm.org
error: failed to update alarm (invalid url for server)
error: failed retrieving file 'aur.db' from mirror.archlinuxarm.org : Could not resolve host: mirror.archlinuxarm.org
error: failed to update aur (invalid url for server)
error: failed to synchronize all databases

[user@alarmpi ~]$ ping mirror.archlinuxarm.org
ping: mirror.archlinuxarm.org: Name or service not known

journalctl showed failures:
Code: Select all
May 04 21:08:15 alarmpi systemd-resolved[12389]: DNSSEC validation failed for question isc.org IN DS: failed-auxiliary
May 04 21:08:15 alarmpi systemd-resolved[12389]: DNSSEC validation failed for question isc.org IN DNSKEY: failed-auxiliary
May 04 21:08:15 alarmpi systemd-resolved[12389]: DNSSEC validation failed for question isc.org IN A: failed-auxiliary
May 04 21:11:08 alarmpi systemd-resolved[12389]: DNSSEC validation failed for question archlinuxarm.org IN DS: failed-auxiliary
May 04 21:11:08 alarmpi systemd-resolved[12389]: DNSSEC validation failed for question mirror.archlinuxarm.org IN DS: failed-auxiliary
May 04 21:11:08 alarmpi systemd-resolved[12389]: DNSSEC validation failed for question mirror.archlinuxarm.org IN SOA: failed-auxiliary
May 04 21:11:08 alarmpi systemd-resolved[12389]: DNSSEC validation failed for question mirror.archlinuxarm.org IN A: failed-auxiliary

It seems that systemd-resolved was trying to get and validate the DNSKEY of the .org TLD and it was failing somehow.
I was able to reproduce the issue; note the rcode: SERVFAIL and lack of proper answer.
Code: Select all
[user@alarmpi ~]$ drill -D mirror.archlinuxarm.org @127.0.0.53
;; ->>HEADER<<- opcode: QUERY, rcode: SERVFAIL, id: 56013
;; flags: qr rd ra ; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; mirror.archlinuxarm.org.   IN   A

;; ANSWER SECTION:

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 6 msec
;; EDNS: version 0; flags: do ; udp: 65494
;; Data: \# 23 000500070507080a0d0e0f000600030102040007000101
;; SERVER: 127.0.0.53
;; WHEN: Sat May  4 20:00:14 2019
;; MSG SIZE  rcvd: 75

The same query send directly to the home router.
Code: Select all
[user@alarmpi ~]$ drill -D mirror.archlinuxarm.org @192.168.8.1
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 34625
;; flags: qr rd ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; QUESTION SECTION:
;; mirror.archlinuxarm.org.   IN   A

;; ANSWER SECTION:
mirror.archlinuxarm.org.   80023   IN   A   50.116.36.110

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:
.   0   CLASS0   TYPE0   \# 0

;; Query time: 7 msec
;; SERVER: 192.168.8.1
;; WHEN: Sat May  4 20:04:37 2019
;; MSG SIZE  rcvd: 68

Nonetheless, everything worked fine on another RPi with Arch in another location, connected to a different home router.
I believe the bug is triggered by combination of two factors:
- a way systemd-resolved creates the DNS queries (its queries contain additional optional fields), and
- the DNS server in the crappy Huawei home router, which might have issues with large DNS responses.

Anyway, the workaround was simply:
Code: Select all
sudo systemctl disable --now systemd-resolved
MattM
 
Posts: 7
Joined: Sun May 05, 2019 10:33 am

Re: [SOLVED] DNS not working after pacman -Syu

Postby slackline » Sun May 26, 2019 4:05 pm

Just a heads up that I've been experiencing this on one (and only one) of three Raspberry Pi's on my home network, albeit intermittently.

I've two Pi2 B+'s (armv71) and one Pi Zero W (armv61) and its one of the armv71 that this happens under.

That its only one of three and DNS on my home router (a Linksys WRT 1900ACS running OpenWRT 18.06.2) makes me suspicious that its down to a crappy home router as MattM suggests.

Copying MattM I have the following output...

Code: Select all

# pacman -Syu
:: Synchronising package databases...
error: failed retrieving file 'core.db' from mirror.archlinuxarm.org : Could not resolve host: mirror.archlinuxarm.org
error: failed to update core (invalid url for server)
error: failed retrieving file 'extra.db' from mirror.archlinuxarm.org : Could not resolve host: mirror.archlinuxarm.org
error: failed to update extra (invalid url for server)
error: failed retrieving file 'community.db' from mirror.archlinuxarm.org : Could not resolve host: mirror.archlinuxarm.org                                   
error: failed to update community (invalid url for server)
error: failed retrieving file 'alarm.db' from mirror.archlinuxarm.org : Could not resolve host: mirror.archlinuxarm.org
error: failed to update alarm (invalid url for server)
aur is up to date
error: failed to synchronize all databases


Check systemctl...

Code: Select all
# systemctl status systemd-resolved.service
 systemd-resolved.service - Network Name Resolution
  Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
  Active: active (running) since Sun 2019-05-26 15:27:40 UTC; 24s ago
    Docs: man:systemd-resolved.service(8)
          https://www.freedesktop.org/wiki/Software/systemd/resolved
          https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
          https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
Main PID: 1030 (systemd-resolve)
  Status: "Processing requests..."
   Tasks: 1 (limit: 2219)
  Memory: 4.1M
  CGroup: /system.slice/systemd-resolved.service
          └─1030 /usr/lib/systemd/systemd-resolved
 
May 26 15:27:56 alarmpi2 systemd-resolved[1030]: DNSSEC validation failed for question mirror.archlinuxarm.org IN AAAA: missing-key
May 26 15:27:56 alarmpi2 systemd-resolved[1030]: DNSSEC validation failed for question . IN DNSKEY: missing-key   
May 26 15:27:56 alarmpi2 systemd-resolved[1030]: DNSSEC validation failed for question org IN DS: missing-key
May 26 15:27:56 alarmpi2 systemd-resolved[1030]: DNSSEC validation failed for question org IN DNSKEY: missing-key
May 26 15:27:56 alarmpi2 systemd-resolved[1030]: DNSSEC validation failed for question archlinuxarm.org IN DS: missing-key
May 26 15:27:56 alarmpi2 systemd-resolved[1030]: DNSSEC validation failed for question archlinuxarm.org IN SOA: missing-key
May 26 15:27:56 alarmpi2 systemd-resolved[1030]: DNSSEC validation failed for question mirror.archlinuxarm.org IN DS: missing-key
May 26 15:27:56 alarmpi2 systemd-resolved[1030]: DNSSEC validation failed for question mirror.archlinuxarm.org IN SOA: missing-key
May 26 15:27:56 alarmpi2 systemd-resolved[1030]: DNSSEC validation failed for question mirror.archlinuxarm.org IN AAAA: missing-key
May 26 15:27:56 alarmpi2 systemd-resolved[1030]: DNSSEC validation failed for question mirror.archlinuxarm.org IN A: missing-key


I then ran the following and it resolved the issue and I could sync from mirror.archlinuxarm.org again....

Code: Select all
# drill -D mirror.archlinuxarm.org @192.168.1.1

;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 40357
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; mirror.archlinuxarm.org.     IN      A

;; ANSWER SECTION:
mirror.archlinuxarm.org.        9567    IN      A       50.116.36.110

;; AUTHORITY SECTION:
 
;; ADDITIONAL SECTION:

;; Query time: 33 msec
;; EDNS: version 0; flags: do ; udp: 1452
;; SERVER: 192.168.1.1
;; WHEN: Sun May 26 15:33:30 2019
;; MSG SIZE  rcvd: 68

# drill -D mirror.archlinuxarm.org @127.0.0.53
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 759
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; mirror.archlinuxarm.org.     IN      A

;; ANSWER SECTION:
mirror.archlinuxarm.org.        6810    IN      A       50.116.36.110

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 1 msec
;; EDNS: version 0; flags: do ; udp: 65494
;; Data: \# 23 000500070507080a0d0e0f000600030102040007000101
;; SERVER: 127.0.0.53
;; WHEN: Sun May 26 15:34:30 2019
;; MSG SIZE  rcvd: 91

alarmpi2 etc # pacman -Syu

:: Synchronising package databases...                                                                             core is up to date
extra                                                                   2.2 MiB  3.53M/s 00:01 [########################################################] 100%
ommunity                                                               5.1 MiB  3.84M/s 00:01 [########################################################] 100%
alarm is up to date
aur is up to date
:: Starting full system upgrade...
 aur is up to datethere is nothing to do
slackline
 
Posts: 15
Joined: Wed May 11, 2016 8:18 pm

Re: [SOLVED] DNS not working after pacman -Syu

Postby cedricmc » Sun Jun 09, 2019 7:47 pm

Why do you tag it as [SOLVED]? To disable a system service as systemd-resolved is not a solution but a mere workaround.

Also, to have
Code: Select all
DNSEC=no
in my .network file seems not to prevent it.
cedricmc
 
Posts: 25
Joined: Sun Dec 01, 2013 8:55 pm

Re: [SOLVED] DNS not working after pacman -Syu

Postby MattM » Mon Jun 10, 2019 12:39 pm

cedricmc wrote:to have
Code: Select all
DNSEC=no
in my .network file seems not to prevent it.

It's DNSSEC, not DNSEC. See man systemd.network.
MattM
 
Posts: 7
Joined: Sun May 05, 2019 10:33 am


Return to User Questions

Who is online

Users browsing this forum: No registered users and 6 guests