Slow/hanging ssh logins

[keithspg]

Very recently, the SSH logins on armv6 and armv7 are balky and slow. It used to be immediate: login, password, prompt. Now login and password are quick, but it delays a while for the prompt. Sometimes I start multiple attempts and see which window actually responds. When this initially started, I found a forum post saying I could backdate to an older pam/pambase and get the previous response. This worked for a while, but with a recent update, something else was updated and when I backdated pam/pambase, I was locked out. Even with the correct password, I could not log in. So, I am now fully up to date, but am now saddled with slow logins on PiZero/B,B+ on armv6 and Pi2/3/3+ on armv7.
Anyone else seeing this?
I looked for the article, but cannot seem to find it any more.

Keith

[solskogen]

Could it be that your DNS are not responding? Check your /etc/resolv.conf

[keithspg]

Thanks for the help!

It does not seem to be resolv.conf nor does it seem to be systemd-resolved as it is disabled. connman populates resolv.conf with the IP of my PiHole:
$this->bbcode_second_pass_code('', '# cat /etc/resolv.conf
# Generated by Connection Manager
search griders
nameserver 2601:241:4200:28d:ba27:ebff:fe0b:9be6
nameserver 192.168.2.3')
I do not see this on my x86 laptops running Arch nor my aarch64 RPi. It appeared this summer and was, initially, fixed by downgrading pam/pambase, but that no longer helps. It only appears on my 'up to date' armv6/v7 rpis. I have one Pi B+ that I have no t updated in a long time and it is much faster (doe snot hang and logs in first time). It is also configured with connman with the same info in /etc/resolv.conf.

The log shows this on armv6 and armv7 it hangs for a while:
$this->bbcode_second_pass_code('', 'Oct 06 07:56:24 kitchenrune sshd[3406]: Accepted password for root from 192.168.2.111 port 51643 ssh2
Oct 06 07:56:24 kitchenrune sshd[3406]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Oct 06 07:56:24 kitchenrune sshd[3406]: pam_systemd(sshd:session): Failed to create session: Input/output error')

The log shows this on aarch64 and is instant:
$this->bbcode_second_pass_code('', '
Oct 06 08:00:44 rune64 sshd[49604]: Accepted password for root from 192.168.2.111 port 51696 ssh2
Oct 06 08:00:44 rune64 sshd[49604]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Oct 06 08:00:44 rune64 systemd-logind[188]: New session c5 of user root.
Oct 06 08:00:44 rune64 systemd[1]: Started Session c5 of user root.
')

[keithspg]

A little more digging.

For some reason, there is no /run/user/0 directory for my root login on my armv6 or armv7 images but there is on the x86 and aarch64 images. How can this be? How can I fix this?

[keithspg]

And yet more digging.

This only happens on armv6 and armv7 and only if the login is root. I made a fresh image and updated and then logged in as alarm. It logs in immediately. If I try to log in as root, I get 1 out of 5, maybe, that actually eventually log in. I spam the login with many putty windows until it eventually logs in. These are headless audio players.
I have this in my /etc/ssh/sshd_config
$this->bbcode_second_pass_code('', 'PermitRootLogin yes')
This cannot be intended behavior and it does not behave like this on aarch64 which is set up identically.

[kiriakos]

Hi there,

I am seeing the same behavior.

I'm setting up a new Raspberry Pi 4 and I was experiencing intermittent connection issues over ssh.
Specifically after accepting keys, and setting up the connection, the ssh client would just hang from 30 seconds to infinity.

Looking on the RPI I could see an ssh process sshd: root [priv] ramping up to 100% and staying there and every time this was accompanied by an ssdh.service log
$this->bbcode_second_pass_code('', '
Oct 12 14:21:24 beato sshd[767]: Accepted publickey for root from 192.168.6.4 port 48727 ssh2: ED25519 SHA256:QFF83h>
Oct 12 14:21:24 beato sshd[767]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Oct 12 14:21:24 beato sshd[767]: pam_systemd(sshd:session): Failed to create session: Input/output error
')

After searching around I found some complaints that were similar in nature and related to pam_systemd.so in pam's session config.

looking at /etc/pam.d/system-login I see the systemd line

$this->bbcode_second_pass_code('', '
session optional pam_loginuid.so
session optional pam_keyinit.so force revoke
session include system-auth
session optional pam_motd.so motd=/etc/motd
session optional pam_mail.so dir=/var/spool/mail standard quiet
-session optional pam_systemd.so
session required pam_env.so user_readenv=1
')

Removing the pam_systemd.so line causes sshd to behave again.

Now the big question is: What is the purpose of that module there?

[keithspg]

I cam vouch for this change to /etc/pamd/system-login 'fixing' the slow/laggy logins on armv6/7. The ssh logins are responsive as before. What is also strange is that I believe the aarch64 version has this un-commented and it works 'just fine'. I cannot verify and will do so this evening.

Keith

[keithspg]

Just verified. The 64 bit aarch64 does not have that line commented out and it logs in quickly every time. Something strange is going on.

[Kabbone]

That's interesting, it is not only ssh related.
I can experience a similar behaviour on a rock64 with aarch64 inside a systemd container. The root login on the command line with machinectl login container was always very slow, but funnily only at one of two containers. After commenting out the pam_systemd.so line it is normal again.

[jonsnowd3n]

I can confirm this is still happening as of today on Arch Linux ARM on my Raspberry Pi 2b.
I installed from ArchLinuxARM-rpi-2-latest.tar.gz dated 8-9-2020.
The SSH server worked fine until doing a full "pacman -Syu" upgrade.
After the upgrade and rebooting, SSH was almost completely unusable.
I commented out the line "-session optional pam_systemd.so" from /etc/pam.d/system-login and rebooted, but the issue was not resolved.
I further commented out the lines in /etc/pam.d/system-auth : "-auth [success=1 default=ignore] pam_systemd_home.so" and "-account [success=1 default=ignore] pam_systemd_home.so" and also "#-password [success=1 default=ignore] pam_systemd_home.so"
After restarting the sshd service, I noticed an immediate improvement. SSH is usable again, but still less responsive than expected.
After some additional testing, I found the SSH was much more responsive after I switched from WiFi to a wired Ethernet connection on the Pi.

package: pam
version: 1.5.0-1
build date: Nov 16, 2020

package: openssh
version: 8.4p1-2
build date: Oct 29, 2020