openssl_cryptodev does not seem to work anymore

[Sergeanter]

FIXED: sshd cryptodev support began working again after kernel update to 4.8.1-1-ARCH
I have lost ssh access recently after updating all packages . It has been about two months since last update so I cannot attribute this to anything specific.
sshd -d drops out at the key exchange phase without any meaningful message. Replacing openssl-cryptodev with openssl fixes the problem.
lsmod shows cryptodev loaded and I cannot see anything in the log ( I only checked journalctl -xe so if there is anywhere else to look I would)

[moonman]

It's got to be something else as both of my kirkwood devices work fine:

$this->bbcode_second_pass_code('', '[root@goflex ~]# lsmod
Module Size Used by
blowfish_generic 3649 0
blowfish_common 6549 1 blowfish_generic
cfg80211 434806 0
rfkill 16148 2 cfg80211
marvell_cesa 25452 0
cryptodev 33671 2
ip_tables 10778 0
x_tables 12847 1 ip_tables
ipv6 330373 22

[root@goflex ~]# pacman -Qs openssl
local/openssl-cryptodev 1.0.2.h-1
The Open Source toolkit for Secure Sockets Layer and Transport Layer Security

[root@goflex ~]# uname -a
Linux goflex 4.7.0-2-ARCH #1 PREEMPT Thu Jul 28 22:42:49 MDT 2016 armv5tel GNU/Linux
[root@goflex ~]#
')

[Sergeanter]

I cant show the openssl-cryptodev version now for obvious reasons but it is same as yours.
I am on non dt kernel. Is this an issue? The module shows as mv_cesa, I remember they were saying at some point they would not support cryptodev on non dt. But it worked until recently
$this->bbcode_second_pass_code('', '
[root@alarm ~]# uname -a
Linux alarm 4.4.17-1-ARCH #1 PREEMPT Fri Aug 12 14:26:55 MDT 2016 armv5tel GNU/Linux
[root@alarm ~]# lsmod
Module Size Used by
m25p80 4691 0
spi_nor 12827 1 m25p80
mv_cesa 11350 0
cryptodev 33735 0
ip_tables 10675 0
x_tables 12425 1 ip_tables
ipv6 323673 24
raid456 95610 1
async_raid6_recov 5352 1 raid456
async_memcpy 1684 2 raid456,async_raid6_recov
async_pq 4898 2 raid456,async_raid6_recov
async_xor 3361 3 async_pq,raid456,async_raid6_recov
xor 4225 1 async_xor
async_tx 2300 5 async_pq,raid456,async_xor,async_memcpy,async_raid6_recov
raid6_pq 87146 3 async_pq,raid456,async_raid6_recov
md_mod 123580 2 raid456
')

[moonman]

mv_cesa is the old driver without dma support hence slower compared to marvell_cesa and you are right marvell_cesa does not work without dt (but mv_cesa does).

Still, with linux-kirkwood everything works fine for me
$this->bbcode_second_pass_code('', '[root@alarm-PogoplugV4 ~]# lsmod
Module Size Used by
blowfish_generic 3649 0
blowfish_common 6549 1 blowfish_generic
cfg80211 427510 0
rfkill 16171 2 cfg80211
mv_cesa 11350 0
m25p80 4691 0
spi_nor 12827 1 m25p80
cryptodev 33735 2
ip_tables 10675 0
x_tables 12425 1 ip_tables
ipv6 323673 18

[root@alarm-PogoplugV4 ~]# pacman -Qs openssl
local/openssl-cryptodev 1.0.2.h-1
The Open Source toolkit for Secure Sockets Layer and Transport Layer Security

[root@alarm-PogoplugV4 ~]# uname -a
Linux alarm-PogoplugV4 4.4.17-1-ARCH #1 PREEMPT Fri Aug 12 14:26:55 MDT 2016 armv5tel GNU/Linux')

[Sergeanter]

Do you enforce hardware accelerated ciphers in sshd_config? Could that be the difference? Can you check in putty or whatever you use to connect that negotiated algorithm is one of those supported AES and DES ? Once I get home , I will try it without Ciphers line in sshd_config and see what happens

[Sergeanter]

I have changed from openssl to openssl-cryptodev on one of them and checked openssl speed with aes128-cbc. Acceleration worked. Then I closed ssh session and tried to re-connect. No luck. See ssh response below.
$this->bbcode_second_pass_code('', 'debug1: Authenticating to 192.168.1.102:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes128-cbc MAC: umac-64-etm@openssh.com compression: none
debug1: kex: client->server cipher: aes128-cbc MAC: umac-64-etm@openssh.com compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
Connection closed by 192.168.1.102 port 22
')
Now on the second machine I did the same except I commented out Ciphers line. After reboot it re-connected fine with non -accelerated server->client cipher: chacha20-poly1305@openssh.com
I un-commented Ciphers line that reads
$this->bbcode_second_pass_code('', 'Ciphers aes128-cbc,aes256-cbc,3des-cbc')
Rebooted , and that was it : I cannot connect anymore.
So the problem seems to be not in openssl-cryptodev but somewhere in sshd where it calls accelerated encryption engine.
sshd works fine with non-accelerated openssl using the same ciphers (though I noticed I cannot tunnel over ssh anymore, but the console works).
Would you try to reproduce this?

[moonman]

Maybe something changed in the kernel so that it does not work anymore. linux-kirkwood is almost dead and non-dt support for kirkwood was removed a long time ago. It is really there to buy people time to finally upgrade to linux-kirkwood-dt, so I would suggest you look into updating

[Sergeanter]

I am sorry to tell you but accelerated sshd does not work with 4.7 dt kernel either. As soon as I force hardware accelerated ciphers in sshd_config, it cannot connect. It says connection unexpectedly terminated. If I comment out Ciphers line, it connects but uses one of the non-accelerated ciphers. Tunneling over ssh stopped working either ( I had a browser proxy over ssh set at work).
I guess the good news is I got a newer uboot and dt kernel now.
I will try and see if sftp still works tomorrow.
When openssl-cryptodev is installed , available ciphers are:
$this->bbcode_second_pass_code('', ' ssh -c aes128-cbc 192.168.1.102
Unable to negotiate with 192.168.1.102 port 22: no matching cipher found. Their offer: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm @openssh.com,aes256-gcm@openssh.com
')
This does not change whether I force cbc ciphers or not in shhd_config.
However when openssl is installed , you can make cbc ciphers available if you put proper Ciphers line in the config file.
My guess is these accelerated ciphers are considered weak /obsolete and sshd + openssl-cryptodev bundle has been configured in such a way that cbc ciphers are not registered in sshd no matter what even though openssl-cryptodev clearly supports them.

[moonman]

I will take a look. It works from the client side with limited cyphers. (i.e. edit ssh_config on pogoplug and ssh in to another device from this pogoplug). I will have to setup telnet or something so I can get back in without opening the device for serial or swapping storage in case it fails

[Sergeanter]

I see somebody made openssl-cryptodev dependent on linux-kirkwood-dt. This however leads to a missing crypto hook in initramfs because libcrypto.so is not available when initramfs is generated by pacman ( when you switch from openssl to openssl-crypto). You would need to re-generate initramfs when pacman is done.
Accelerated ciphers in sshd still do not work even after I downgraded openssh to 7.2p2-2 and openssl-cryptodev to 1.0.2.g-1

$this->bbcode_second_pass_code('', '[root@alarm ~]# ssh alarm2 -c aes128-cbc
Connection closed by 192.168.1.102 port 22
[root@alarm ~]# ssh alarm2 -c blowfish-cbc
Unable to negotiate with 192.168.1.102 port 22: no matching cipher found. Their offer: aes128-cbc,aes256-cbc,3des-cbc,aes128-ctr')
As you can see aes128-cbc is registered but cannot be used by sshd so the problem must be somewhere else