ssl: add post-quantum cryptography (PQC) tests

[junaruga]

This PR is related to the #894. And this PR is the last PR of the issue ticket. This PR is working in progress. Because I have the following 2 things to fix or investigate.

• I see the test_pqc and test_pqc_group failing on Windows OpenSSL 3.5 cases. The CI log is here. But as I could find the failures of the Windows cases, I could prove the usefulness of these PQC tests. 😃
• You see the comments "Post-quantum cryptography is supported on OpenSSL 3.5 or later." Because I got errors when I ran the tests on OpenSSL 3.4 or earlier. But I want to check the details of the errors, and check if the PQC is really supported in OpenSSL 3.5 or later.

What do you think?

---

Note the key files were created by the following commands.

$ ${HOME}/.local/openssl-3.6.0-dev-fips-debug-8253b58d60/bin/openssl genpkey \
  -algorithm mldsa65 \
  -out mldsa65.pem
$ ${HOME}/.local/openssl-3.6.0-dev-fips-debug-8253b58d60/bin/openssl genpkey \
  -algorithm mldsa65 \
  -out mldsa-1.pem

[junaruga]

@MSP-Greg Do you have any ideas why the post-quantum cryptography (PQC) tests are failing on Windows OpenSSL 3.5 cases?

[rhenium]

@MSP-Greg Do you have any ideas why the post-quantum cryptography (PQC) tests are failing on Windows OpenSSL 3.5 cases?

This is a known issue with TLS 1.3 and has nothing to do with PQC. This actually happens on other platforms, too. The TCP connection is closed by the client before the server finishes SSL_accept(). The current workaround is to insert ssl.puts("abc"); ssl.gets in the server_connect block so that we can ensure nothing is left in the buffers. A related discussion at openssl/openssl: https://redirect.github.com/openssl/openssl/issues/25538

[rhenium]

Please move the new methods closer to other relevant test cases.

[junaruga]

@MSP-Greg Do you have any ideas why the post-quantum cryptography (PQC) tests are failing on Windows OpenSSL 3.5 cases?

This is a known issue with TLS 1.3 and has nothing to do with PQC. This actually happens on other platforms, too. The TCP connection is closed by the client before the server finishes SSL_accept(). The current workaround is to insert ssl.puts("abc"); ssl.gets in the server_connect block so that we can ensure nothing is left in the buffers. A related discussion at openssl/openssl: https://redirect.github.com/openssl/openssl/issues/25538

All right. Thanks for the info. I added the ssl.puts("abc"); ssl.gets to the tests. Then the CI failures on Windows OpenSSL 3.5 disappeared.

[junaruga]

Please move the new methods closer to other relevant test cases.

All right, I moved the PQC tests to the other relevant test cases. The test_pqc_sigalg (which was the test_pqc) is below the test_get_sigalg. The test_pqc_group is below the test_set_groups_tls13.

[junaruga]

Thank you for your review! I rebased the PR, fixing most things mentioned by you. Please keep in mind that there is a new change exposing the ca_exts and ee_exts in the test/openssl/utils.rb.

Now what do you think?

[junaruga]

* You see the comments "Post-quantum cryptography is supported on OpenSSL 3.5 or later." Because I got errors when I ran the tests on OpenSSL 3.4 or earlier. But I want to check the details of the errors, and check if the PQC is really supported in OpenSSL 3.5 or later.

I investigated the above thing with the help of my colleagues working for OpenSSL. The following OpenSSL 3.5 release note helped me to check if the PQC is really supported in OpenSSL 3.5 or later.

https://openssl-library.org/post/2025-04-08-openssl-35-final-release/

Support for PQC algorithms (ML-KEM, ML-DSA and SLH-DSA)

The following commits are adding the entries in the Changes between 3.4 and 3.5.0 [8 Apr 2025] section in the CHANGES.md in the openssl/openssl repository.

• ML-KEM (FIPS 203): openssl/openssl@6ab8772
• ML-DSA (FIPS 204): openssl/openssl@6184259

I updated the comment above the line return unless openssl?(3, 5, 0) in the test_pqc_sigalg and test_pqc_group. I also updated the commit message with the above OpenSSL 3.5 release note link.

I think I fixed everything mentioned by you in the review. Now what do you think?

[rhenium]

Thanks!

[junaruga]

Thanks for your review!

By the way, for someone wondering why the PQC algorithm SLH-DSA test doesn't exist in this PR. The reason is below. It's not good time to add the SLH-DSA test for now.

#894 (comment)

However, I heard from my colleagues working for OpenSSL, that SLH-DSA in NIST or OpenSSL is not ready to be used. We can ignore SLH-DSA case.