NFS mount gives permission denied, even on localhost

Problems with packages? Post here, using [tags] of the package name.

NFS mount gives permission denied, even on localhost

Postby markg85 » Sun Sep 12, 2021 9:45 am

Hi,

So I've been debugging this issue for many hours again. I don't even bother posting here if i haven't done my own extensive troubleshooting.
My ARM (an odroid-xu4) is the NFS server in this story and it's setup to only handle NFS v4.x (not 3.x). Some command outputs to help debugging this:

The /etc/exports file looks like this:
$this->bbcode_second_pass_code('', '/mnt/media 10.0.3.0/24(rw,sync,insecure,no_root_squash,no_subtree_check)
')

Confusingly, `exportfs -v` looks to have a lot more settings applied that i didn't set in the exports file. I'm just going to assume those are defaults hidden supremely well somewhere and shown in the exportfs command...
$this->bbcode_second_pass_code('', '/mnt/media 10.0.3.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,insecure,no_root_squash,no_all_squash)')

$this->bbcode_second_pass_code('', '└─> # rpcinfo -p | grep nfs
16: 100003 3 tcp 2049 nfs
17: 100003 4 tcp 2049 nfs
18: 100227 3 tcp 2049 nfs_acl')

$this->bbcode_second_pass_code('', '└─> # ps -aux | grep nfs
61:root 67 0.0 0.0 0 0 ? I< 01:09 0:00 [nfsiod]
114:root 274 0.0 0.1 4200 2216 ? Ss 01:09 0:00 /usr/sbin/nfsdcld
141:root 394 0.0 0.0 0 0 ? S 01:09 0:00 [nfsd]
142:root 395 0.0 0.0 0 0 ? S 01:09 0:00 [nfsd]
143:root 396 0.0 0.0 0 0 ? S 01:09 0:00 [nfsd]
144:root 397 0.0 0.0 0 0 ? S 01:09 0:00 [nfsd]
145:root 398 0.0 0.0 0 0 ? S 01:09 0:00 [nfsd]
146:root 399 0.0 0.0 0 0 ? S 01:09 0:00 [nfsd]
147:root 400 0.0 0.0 0 0 ? S 01:09 0:00 [nfsd]
148:root 401 0.0 0.0 0 0 ? S 01:09 0:00 [nfsd]')

$this->bbcode_second_pass_code('', '└─> # uname -a
Linux media-server 4.14.180-3-ARCH #1 SMP PREEMPT Sat Jun 5 22:29:47 UTC 2021 armv7l GNU/Linux')

The server doesn't run a firewall! Although, i must admit that with the maze of firewalls in linux land it's really difficult to even know if you're running one or not.
iptables is installed but disabled.

The export line you see up above at forst didn't have the insecure part in it. With that option out, mounting gave me these errors:
$this->bbcode_second_pass_code('', '❯ sudo mount 10.0.3.22:/ media -vvv
mount.nfs: timeout set for Sun Sep 12 01:11:29 2021
mount.nfs: trying text-based options 'vers=4.2,addr=10.0.3.22,clientaddr=10.0.3.26'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'vers=4,minorversion=1,addr=10.0.3.22,clientaddr=10.0.3.26'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'vers=4,addr=10.0.3.22,clientaddr=10.0.3.26'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=10.0.3.22'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.0.3.22 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 10.0.3.22 prog 100005 vers 3 prot UDP port 20048
mount.nfs: portmap query retrying: RPC: Unable to receive - Connection refused
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 10.0.3.22 prog 100005 vers 3 prot TCP port 20048
mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
mount.nfs: access denied by server while mounting 10.0.3.22:/')

However, if i add the insecure option it does this and stays hanging there:
$this->bbcode_second_pass_code('', '❯ sudo mount 10.0.3.22:/ media -vvv
mount.nfs: timeout set for Sun Sep 12 11:34:42 2021
mount.nfs: trying text-based options 'vers=4.2,addr=10.0.3.22,clientaddr=10.0.3.26'
')

Using showmount (later i found out through the fine arch wiki that it shouldn't even work for nfsv4 mounts) outputs this slightly more helpful error:
$this->bbcode_second_pass_code('', '❯ showmount -e 10.0.3.22
rpc mount export: RPC: Unable to receive; errno = Connection refused')

Interestingly, and that's why i said it in the title, running this same command on the server itself (so with the 10.0.3.22 ip, but also with 127.0.0.1) give me the exact same permission denied error.

After googling this error a lot. Like REALLY a LOT! I literally set the results to the last year and went over the first 100 results and tried all possible solutions! None of it is working sadly.
I did search way further back up to somewhere in 2005 from fedora with the same error. Suffice to say, not applicable anymore ;)
The no.1 hit from the past year is from the manjaro forums with the same error triggered by the same cause (updating the system). There it was enough to upgrade again which pulled in a new kernel.
I cannot do that on ARM. The kernel updates don't go as fast there and the kernel that you see way up in the command outputs is the latest kernel for that arm platform.

Now i'm really thoroughly stuck in this issue. I don't know any possible solution for it anymore.
Also, it seems stupendously difficult to figure out what exactly is causing the permission denied error! I can disable rpcbind which gives me another init error kinda indicating that it needs rpcbind. But i don't know if the error is from rpcbind itself of from the thing it's "binding" to (rpc.nfsd?). Setting debug enabled in the nfs.conf file for the nfsd section doesn't seem to help at all. Debugging from nfsd seems to be nowhere. All i see in journalctl -u nfs-server is the startup process and that's it.

I hope you folks could provide some help to get to the root of this issue and squash it.
Any help is very much appreciated!

Best regards,
Mark

p.s. Yes, i posted this first on the Archlinux forums. They politely directed me to this forum instead: https://bbs.archlinux.org/viewtopic.php?pid=1992472
markg85
 
Posts: 4
Joined: Sun Sep 12, 2021 9:24 am

Re: NFS mount gives permission denied, even on localhost

Postby graysky » Sun Sep 12, 2021 9:54 am

Do you have an x86_64 box? If so, recommend you set up NFS there with your config and see if it works. I suspect it's something with your setup and/or firewall settings, not Arch ARM packages. I suggest you do it on x86_64/Arch so you can get support on the Arch BBS where there is a more active user base.

My setup that works fine is:
$this->bbcode_second_pass_code('', '# cat /etc/exports
/srv/nfs 10.1.8.0/24(ro,no_subtree_check,async,fsid=0)
/srv/nfs/music 10.1.8.0/24(rw,no_subtree_check,async,no_root_squash)
')

I'm using the bind mount suggestion on the NFS wiki.
graysky
Developer
 
Posts: 1728
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

Re: NFS mount gives permission denied, even on localhost

Postby markg85 » Sun Sep 12, 2021 4:43 pm

$this->bbcode_second_pass_quote('', 'D')o you have an x86_64 box? If so, recommend you set up NFS there with your config and see if it works.


I'm sorry, but definitely not!
Yes, i do have that. But that pc is not the one that should be the server. It doesn't even have the HDD to serve.

I'm frankly really surprised that you even make such a suggestion. To exaggerate it sounds like "Ohh, doesn't work on your arm, try on a windows pc and see if it does there". Sure, you said "arch x84" but the idea is the same. It makes no sense to me as the issue i'm trying to resolve has 0 to do with that platform.

This broken NFS started after updating. And, as i mentioned and i'm now repeating, i did find someone from manjaro that had the exact same issue but where the next update (which pulled in an updated kernel) fixed it.

We should try to find what's wrong, not put it on other machines.

Edit
For the record, i did post this on the Arch forum but was directed here: https://bbs.archlinux.org/viewtopic.php?id=269572
markg85
 
Posts: 4
Joined: Sun Sep 12, 2021 9:24 am

Re: NFS mount gives permission denied, even on localhost

Postby markg85 » Tue Sep 14, 2021 12:05 pm

I give up!

After days of fiddling with this and just being unable to figure out what causes the permission denied crap, i've just about had it with the never ending debugging.

I've fallen back to SMB.. Which is super sad because now i'm using a reverse engineered protocol just because the linux native (NFS) is crap. Oh well, SMB works.
markg85
 
Posts: 4
Joined: Sun Sep 12, 2021 9:24 am

Re: NFS mount gives permission denied, even on localhost

Postby graysky » Tue Sep 14, 2021 12:35 pm

See my recommendation above. Unless you are hit with some Arch ARM specific bug, NFS is NFS is NFS.
graysky
Developer
 
Posts: 1728
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

Re: NFS mount gives permission denied, even on localhost

Postby moonman » Tue Sep 14, 2021 5:45 pm

$this->bbcode_second_pass_quote('markg85', 'I') give up!

After days of fiddling with this and just being unable to figure out what causes the permission denied crap, i've just about had it with the never ending debugging.

I've fallen back to SMB.. Which is super sad because now i'm using a reverse engineered protocol just because the linux native (NFS) is crap. Oh well, SMB works.


graysky is correct NFS is NFS regardless of architecture, and ArchLinux does have more users to help out. However, I think it may be the combination of that odroid older kernel and new nfs-utils. See viewtopic.php?f=15&t=15396&hilit=nfs+utils Hardkernel does NOT have a newer official kernel.

You can try linux-armv7 kernel (I'm running it on my XU4, but I'm running it headless) if you don't need video output (which may or may not work with mainline kernel - i haven't tested).
Pogoplug V4 | GoFlex Home | Raspberry Pi 4 4GB | CuBox-i4 Pro | ClearFog | BeagleBone Black | Odroid U2 | Odroid C1 | Odroid XU4
-----------------------------------------------------------------------------------------------------------------------
[armv5] Updated U-Boot | [armv5] NAND Rescue System
moonman
Developer
 
Posts: 3387
Joined: Sat Jan 15, 2011 3:36 am

Re: NFS mount gives permission denied, even on localhost

Postby graysky » Tue Sep 14, 2021 5:53 pm

Good point, moonman. I also think having the OP replicate a setup on his x86_64 box will serve as a control for the experiment. If it works on x86_64 but not on his ARM box, we know it's not the setup but the ARM box.
graysky
Developer
 
Posts: 1728
Joined: Sun Jun 26, 2011 6:56 am
Location: /run/user/1000

Re: NFS mount gives permission denied, even on localhost

Postby moonman » Tue Sep 14, 2021 5:57 pm

$this->bbcode_second_pass_quote('graysky', 'G')ood point, moonman. I also think having the OP replicate a setup on his x86_64 box will serve as a control for the experiment. If it works on x86_64 but not on his ARM box, we know it's not the setup but the ARM box.


Agreed, that would be my first troubleshooting step too if it wasn't for that thread (which I only knew about because I helped in it).
Pogoplug V4 | GoFlex Home | Raspberry Pi 4 4GB | CuBox-i4 Pro | ClearFog | BeagleBone Black | Odroid U2 | Odroid C1 | Odroid XU4
-----------------------------------------------------------------------------------------------------------------------
[armv5] Updated U-Boot | [armv5] NAND Rescue System
moonman
Developer
 
Posts: 3387
Joined: Sat Jan 15, 2011 3:36 am
Top

Re: NFS mount gives permission denied, even on localhost

Postby markg85 » Wed Sep 15, 2021 10:01 pm

Sorry, but no. I won't be trying to debug this any further.
Here are my reasons. And i definitely did try!

During my last update cycle in archlinux arm i was hit with, in hindsight, at least 2 issues in my network that cost me DAYS to figure out. Really, days! Per each of those bugs, about a week in total.

1. Archlinuxarm updated the mosquitto mqtt package. Now that on it's on is just fine. However, they - since version 2.0.0 - have changed the default to only work on the local loopback device. Meanwhile, i was using that very MQTT to drive my lights! That single update caused me to sit in the dark - yes really, no joke here - for at least 2 days. It wasn't immediately clear, like at all, that MQTT was the issue as i was thrown off by other issues that turned out to be a non issue. It took about 3 days, updating firmware in my lights and much debugging to finally figure out that MQTT simply needed 2 new setting lines to make it all work again.

2. This very NFS bug took me a couple days to debug to no avail. What doesn't help, at all, is that searching for the nfs.conf manual doesn't give you anything you're actually searching for. You plain and simply cannot find (at least i couldn't) what the proper debug values needs to be for nfsd to freaking spit out debug information. Things are so freaking bad in NFS that i'm surprised the protocol still exists. Then again, perhaps it's just showing it's age (1989). It didn't age well and isn't used that much in the linux world. I'm done trying to use NFS in an attempt to use linux native tools. I might try something else later, but for now i'm stuck to SMB. Specially the fact that after days of debugging i still can't find proper ways to debug this damn thing. And the issue seems to be coming out of bloody nowhere. I'm out, no more NFS.

So there you have it. My reasons for not going to spend one more second in debugging NFS.
markg85
 
Posts: 4
Joined: Sun Sep 12, 2021 9:24 am

Re: NFS mount gives permission denied, even on localhost

Postby Tub » Mon Sep 27, 2021 10:27 am

I just wanted to add that I had similar trouble with NFS4 on my new Odroid N2+. I simply copied over a working configuration from a RPi4, and it stopped working. Same symptoms: absolutely zero errors server side, but mounting anything failed with a permission error.

It is indeed a problem with the ancient kernels supported by odroid. Installing an image from archdroid repositories (containing a patched 5.10 kernel and a matching bootloader) fixed the nfs problems (and a few other problems) without any configuration changes. I'm not happy about using 3rd party repositories, but at least it works.
Tub
 
Posts: 2
Joined: Fri Jul 12, 2019 6:02 pm


Return to Packages

Who is online

Users browsing this forum: No registered users and 5 guests