5.5x speed reduction from a dependency of asyncssh

[pete312]

We've observed a 5.5x reduction in performance between cryptography 41 and 45

with the env set to this, its about 1.34 second for 42 machines.
pip install cryptography==41.*

with this, it drops to about 7.5 seconds.
pip install cryptography==45.*

Can you verify these timing results, and what do you think is the next course of action ?

Many thanks

[ronf]

I'm not seeing any difference between cryptography 41 and 45. However, I am seeing a noticeable difference in single connection connect times between Python 3.12 and other Python versions when running the exact same versions of AsyncSSH and cryptography. The time increases from about 0.5 seconds on Python 3.12 to a little over 1.5 seconds for all the other Python versions I tried (from 3.8 to 3.14 beta). I've never noticed this before, so I don't really have an explanation for it. I'm guessing there's something different in the way the version of 3.12 I have was built, but I didn't do anything special for the 3.12 install vs. other versions from what I remember. In all cases, I just used whatever the latest MacPorts package was.

I don't have a multi-connect benchmark test handy to see how this translates when running multiple connections, but if you have a sample of the code you saw the results on, I could try and reproduce your issue here.

When moving to OpenSSL 3.x, there is also a known slowdown with RSA keys. AsyncSSH added an argument named unsafe_skip_rsa_key_validation you can pass into load_keypairs() to mitigate this, if you trust the source of the keys you are loading. Was your version of cryptography 41 perhaps linked against an older OpenSSL?

[pete312]

Absolutely, I just adapted one of the examples you have, and that reproduced the same delay on the loopback and on one remote box with the same timing. It definitely seems to be the connect logic side.

here is the setup:

python3.11 -m venv /tmp/311
source /tmp/311/bin/activate
pip install asyncssh

(311) $ pip list
Package           Version
----------------- -------
asyncssh          2.21.0
cffi              1.17.1
cryptography      45.0.5
pip               22.3.1
pycparser         2.22
setuptools        65.5.1
typing_extensions 4.14.1

import asyncio, asyncssh
from time import time
from typing import List

 
connect_args = {'client_host_keysign':True, 'host_based_auth':True, 'known_hosts':None}

async def run_client(host, command: str, connect_times = {}) -> asyncssh.SSHCompletedProcess:
    start = time()
    async with asyncssh.connect(host, **connect_args) as conn:
        connect_times.setdefault(host,[]).append(round(time() - start,3) )
        return await conn.run(command)

async def run_multiple_clients(hosts:List[str], connect_times) -> None:
    
    tasks = (run_client(host, 'echo ok', connect_times=connect_times) for host in hosts)
    results = await asyncio.gather(*tasks, return_exceptions=True)
    [print(result) for result in results]


#sample1 = ['127.0.0.1' for i in range(20)] 
sample1 = ['remote_box' for i in range(20)] 
connect_times = {}

asyncio.run(run_multiple_clients(sample1, connect_times))
print('no of hosts', len(sample1))

print(connect_times)

(311) $ time /tmp/simple_client_speed_test.py 
<snip>

no of hosts 20
{'remote_box': [3.424, 3.435, 3.43, 3.446, 3.45, 3.033, 3.365, 2.869, 3.468, 2.963, 3.301, 2.97, 2.798, 2.884, 3.477, 2.804, 3.399, 2.895, 2.811, 3.489]}

real    0m3.864s
user    0m3.490s
sys     0m0.081s

then switch to 41 for much faster runtime.

(311) $ pip install cryptography==41.*
(311) $ pip list
Package           Version
----------------- -------
asyncssh          2.21.0
cffi              1.17.1
cryptography      41.0.7
pip               22.3.1
pycparser         2.22
setuptools        65.5.1
typing_extensions 4.14.1


no of hosts 20
{'remote_box': [0.493, 0.494, 0.483, 0.475, 0.469, 0.488, 0.524, 0.551, 0.547, 0.559, 0.55, 0.56, 0.543, 0.558, 0.537, 0.545, 0.641, 0.658, 0.651, 0.673]}

real    0m0.999s
user    0m8.053s
sys     0m0.057s

if you use one remote box this will happen, interestingly if you use the loopback, it seems to lose connection more often, so you would see missing duration entries even though all tasks complete the echo ok

[ronf]

One thing to be careful about in a test like this is trying to open too many connections at once to the same server. OpenSSH server only allows something like 10 connections to a destination host performing authentication in parallel. If you attempt to exceed that, you'll start to see connection failures.

[pete312]

I see. Makes sense. Regarding the versions of python. Im seeing same thing on 3.11 and 3.8 and 3.12. Almost identical time differences. It occurs between these versions of cryptography.

cryptography-41.0.7
cryptography-42.0.0

i have done a cProfile but it does not make much sense right now.

[ronf]

Which version of OpenSSL are each of those linked with? You can find out by doing:

import ssl
ssl.OPENSSL_VERSION

[ronf]

Actually, sorry - that will tell the version of OpenSSL used by Python for its SSL module, but not for cryptography, so that may not be relevant for AsyncSSH. Let me see if there's a way to get the version used by cryptography.

[ronf]

Here we go:

import cryptography.hazmat.backends.openssl.backend
cryptography.hazmat.backends.openssl.backend.openssl_version_text()

[pete312]

'OpenSSL 3.1.4 24 Oct 2023'

[ronf]

Is that the same version for both cryptography 41 and 42?

Since it's version 3, the slowdown on RSA keys applies here, but if both are using version 3.x, they should both see the slowdown if that was the problem.

[pete312]

Ok, looked at a few of them.

v 41 has this
'OpenSSL 3.1.4 24 Oct 2023'

and v 42 has this
OpenSSL 3.2.2 4 Jun 2024

and v 45 has
'OpenSSL 3.5.1 1 Jul 2025'

and ssh is
$ ssh -V
OpenSSH_8.0p1, OpenSSL 1.1.1k FIPS 25 Mar 2021

So i looked for a box with 3.x and on this one i ran the same test with python3.11

$ ssh -V
OpenSSH_8.7p1, OpenSSL 3.0.7 1 Nov 2022

This showed that the v45 and v41 have the same connect performance at the fast rate.

[ronf]

Are you saying that cryptography 42 was slower, but cryptography 45 was back to the speed of 41?

I'm not aware of any changes that would explain that, but I must admit I haven't paid close attention to recent versions of the cryptography library. The specific set of things I need from it in AsyncSSH hasn't changed in a while. The last major change was probably the one I mentioned around performance on RSA keys, and that was back in 2023 (around the time of cryptography 39). That said, there could easily be performance differences in later cryptography versions which I haven't run across.

[pete312]

Im saying that the issue starts with machines built with openssl 1.1.1 (i.e oracle linux 8 group), where cryptography v42 and above are slower to connect by a factor of 5x or more if hitting a bunch of hosts with asyncssh. Hosts with openssl 3 (i.e oracle linux 9 group), have no issues across all version of cryptography.

At least for now using v41 seems to be the only option.

[ronf]

When you say "built with openssl 1.1.1", are you referring to what is reported by cryptography.hazmat.backends.openssl.backend.openssl_version_text()? I didn't see 1.1.1 show up in your examples. The version reported by "ssh -V" would only be applicable to the OpenSSH executable, and not AsyncSSH.

[pete312]

Yes thank you for making that clear. FYI updated the other issue tracker pyca/cryptography#13147 with my test findings. Findings are very odd.