Hi,
So I've been debugging this issue for many hours again. I don't even bother posting here if i haven't done my own extensive troubleshooting.
My ARM (an odroid-xu4) is the NFS server in this story and it's setup to only handle NFS v4.x (not 3.x). Some command outputs to help debugging this:
The /etc/exports file looks like this:
$this->bbcode_second_pass_code('', '/mnt/media 10.0.3.0/24(rw,sync,insecure,no_root_squash,no_subtree_check)
')
Confusingly, `exportfs -v` looks to have a lot more settings applied that i didn't set in the exports file. I'm just going to assume those are defaults hidden supremely well somewhere and shown in the exportfs command...
$this->bbcode_second_pass_code('', '/mnt/media 10.0.3.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,insecure,no_root_squash,no_all_squash)')
$this->bbcode_second_pass_code('', '└─> # rpcinfo -p | grep nfs
16: 100003 3 tcp 2049 nfs
17: 100003 4 tcp 2049 nfs
18: 100227 3 tcp 2049 nfs_acl')
$this->bbcode_second_pass_code('', '└─> # ps -aux | grep nfs
61:root 67 0.0 0.0 0 0 ? I< 01:09 0:00 [nfsiod]
114:root 274 0.0 0.1 4200 2216 ? Ss 01:09 0:00 /usr/sbin/nfsdcld
141:root 394 0.0 0.0 0 0 ? S 01:09 0:00 [nfsd]
142:root 395 0.0 0.0 0 0 ? S 01:09 0:00 [nfsd]
143:root 396 0.0 0.0 0 0 ? S 01:09 0:00 [nfsd]
144:root 397 0.0 0.0 0 0 ? S 01:09 0:00 [nfsd]
145:root 398 0.0 0.0 0 0 ? S 01:09 0:00 [nfsd]
146:root 399 0.0 0.0 0 0 ? S 01:09 0:00 [nfsd]
147:root 400 0.0 0.0 0 0 ? S 01:09 0:00 [nfsd]
148:root 401 0.0 0.0 0 0 ? S 01:09 0:00 [nfsd]')
$this->bbcode_second_pass_code('', '└─> # uname -a
Linux media-server 4.14.180-3-ARCH #1 SMP PREEMPT Sat Jun 5 22:29:47 UTC 2021 armv7l GNU/Linux')
The server doesn't run a firewall! Although, i must admit that with the maze of firewalls in linux land it's really difficult to even know if you're running one or not.
iptables is installed but disabled.
The export line you see up above at forst didn't have the insecure part in it. With that option out, mounting gave me these errors:
$this->bbcode_second_pass_code('', '❯ sudo mount 10.0.3.22:/ media -vvv
mount.nfs: timeout set for Sun Sep 12 01:11:29 2021
mount.nfs: trying text-based options 'vers=4.2,addr=10.0.3.22,clientaddr=10.0.3.26'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'vers=4,minorversion=1,addr=10.0.3.22,clientaddr=10.0.3.26'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'vers=4,addr=10.0.3.22,clientaddr=10.0.3.26'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=10.0.3.22'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.0.3.22 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 10.0.3.22 prog 100005 vers 3 prot UDP port 20048
mount.nfs: portmap query retrying: RPC: Unable to receive - Connection refused
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 10.0.3.22 prog 100005 vers 3 prot TCP port 20048
mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
mount.nfs: access denied by server while mounting 10.0.3.22:/')
However, if i add the insecure option it does this and stays hanging there:
$this->bbcode_second_pass_code('', '❯ sudo mount 10.0.3.22:/ media -vvv
mount.nfs: timeout set for Sun Sep 12 11:34:42 2021
mount.nfs: trying text-based options 'vers=4.2,addr=10.0.3.22,clientaddr=10.0.3.26'
')
Using showmount (later i found out through the fine arch wiki that it shouldn't even work for nfsv4 mounts) outputs this slightly more helpful error:
$this->bbcode_second_pass_code('', '❯ showmount -e 10.0.3.22
rpc mount export: RPC: Unable to receive; errno = Connection refused')
Interestingly, and that's why i said it in the title, running this same command on the server itself (so with the 10.0.3.22 ip, but also with 127.0.0.1) give me the exact same permission denied error.
After googling this error a lot. Like REALLY a LOT! I literally set the results to the last year and went over the first 100 results and tried all possible solutions! None of it is working sadly.
I did search way further back up to somewhere in 2005 from fedora with the same error. Suffice to say, not applicable anymore
The no.1 hit from the past year is from the manjaro forums with the same error triggered by the same cause (updating the system). There it was enough to upgrade again which pulled in a new kernel.
I cannot do that on ARM. The kernel updates don't go as fast there and the kernel that you see way up in the command outputs is the latest kernel for that arm platform.
Now i'm really thoroughly stuck in this issue. I don't know any possible solution for it anymore.
Also, it seems stupendously difficult to figure out what exactly is causing the permission denied error! I can disable rpcbind which gives me another init error kinda indicating that it needs rpcbind. But i don't know if the error is from rpcbind itself of from the thing it's "binding" to (rpc.nfsd?). Setting debug enabled in the nfs.conf file for the nfsd section doesn't seem to help at all. Debugging from nfsd seems to be nowhere. All i see in journalctl -u nfs-server is the startup process and that's it.
I hope you folks could provide some help to get to the root of this issue and squash it.
Any help is very much appreciated!
Best regards,
Mark
p.s. Yes, i posted this first on the Archlinux forums. They politely directed me to this forum instead: https://bbs.archlinux.org/viewtopic.php?pid=1992472