openssl-master fips: Failure: test_s_generate_parameters(OpenSSL::TestPKey): [RuntimeError] exception expected, not #<OpenSSL::PKey::PKeyError: EVP_PKEY_paramgen>

[junaruga]

I am seeing the following error on the latest master branch a664af8 in my forked repository. This error only happens in the "openssl-master with fips provider" case in the CI. The error happened with the OpenSSL at the openssl/openssl@fd39d1c, and also at an older commit openssl/openssl@f98e49b (CI log)

https://github.com/junaruga/ruby-openssl/actions/runs/10302695081/job/28516950443#step:11:113

 2) Failure: test_s_generate_parameters(OpenSSL::TestPKey): [RuntimeError] exception expected, not #<OpenSSL::PKey::PKeyError: EVP_PKEY_paramgen>.
/home/runner/work/ruby-openssl/ruby-openssl/vendor/bundle/ruby/3.0.0/gems/test-unit-ruby-core-1.0.6/lib/core_assertions.rb:454:in `block in assert_raise'
/home/runner/work/ruby-openssl/ruby-openssl/vendor/bundle/ruby/3.0.0/gems/test-unit-ruby-core-1.0.6/lib/core_assertions.rb:540:in `assert'
/home/runner/work/ruby-openssl/ruby-openssl/vendor/bundle/ruby/3.0.0/gems/test-unit-ruby-core-1.0.6/lib/core_assertions.rb:453:in `rescue in assert_raise'
/home/runner/work/ruby-openssl/ruby-openssl/vendor/bundle/ruby/3.0.0/gems/test-unit-ruby-core-1.0.6/lib/core_assertions.rb:441:in `assert_raise'
/home/runner/work/ruby-openssl/ruby-openssl/test/openssl/test_pkey.rb:49:in `test_s_generate_parameters'
     46:       pend "unstable test on OpenSSL 3.0.[0-5]"
     47:     end
     48:     cb_called = []
  => 49:     assert_raise(RuntimeError) {
     50:       OpenSSL::PKey.generate_parameters("DSA") { |*args|
     51:         cb_called << args
     52:         raise "exit!" if cb_called.size == 3

The CI log succeeded in the case is with OpenSSL at openssl/openssl@f98e49b openssl/openssl@20bf3fe.
https://github.com/ruby/openssl/actions/runs/10261548837/job/28389492262#step:4:41

I am not sure where this issue comes from. It may be from OpenSSL or a Ruby library.

[junaruga]

I haven't debugged this issue by myself. However, I suspect the issue comes from the commit openssl/openssl@49a35f0 2 days ago.

[junaruga]

I haven't debugged this issue by myself. However, I suspect the issue comes from the commit openssl/openssl@49a35f0 2 days ago.

I executed the git bisect to find which commit causes this test failure. And the openssl/openssl@49a35f0 was the commit, as my assumption.

[junaruga]

Below is a minimal reproducer. This script from the unit test expects the cb_called.size == 3 to raise RuntimeError.

$ OPENSSL_CONF=$HOME/.local/openssl-3.4.0-dev-fips-debug-250a7adbea/ssl/openssl_fips.cnf \
  ruby -I./lib -ropenssl -e '
  cb_called = []
  OpenSSL::PKey.generate_parameters("DSA") { |*args|
    cb_called << args
    raise "exit!" if cb_called.size == 3
  }
'
-e:3:in 'OpenSSL::PKey.generate_parameters': EVP_PKEY_paramgen (OpenSSL::PKey::PKeyError)
	from -e:3:in '<main>'

[junaruga]

I haven't debugged this issue by myself. However, I suspect the issue comes from the commit openssl/openssl@49a35f0 2 days ago.

I executed the git bisect to find which commit causes this test failure. And the openssl/openssl@49a35f0 was the commit, as my assumption.

I am seeing the following part of the above commit causing the EVP_PKEY_paramgen (OpenSSL::PKey::PKeyError).

openssl/openssl@49a35f0#diff-605396c063194975af8ce31399d42690ab18186b422fb5012101cc9132660fe1R614

(gdb) f
#0  dsa_gen (genctx=0x964cb0, osslcb=0x7fffce237ee7 <ossl_callback_to_pkey_gencb>, cbarg=0x7ec8e0) at providers/implementations/keymgmt/dsa_kmgmt.c:614
614         return 0;

(gdb) l
609	     * need for DSA keygen either.
610	     */
611	    if (!OSSL_FIPS_IND_ON_UNAPPROVED(gctx, OSSL_FIPS_IND_SETTABLE0,
612	                                     gctx->libctx, "DSA", "Keygen",
613	                                     FIPS_dsa_sign_check))
614	        return 0;
615	#endif
616	
617	    dsa = ossl_dsa_new(gctx->libctx);
618	    if (dsa == NULL)

(gdb) bt
#0  dsa_gen (genctx=0x964cb0, osslcb=0x7fffce237ee7 <ossl_callback_to_pkey_gencb>, cbarg=0x7ec8e0) at providers/implementations/keymgmt/dsa_kmgmt.c:614
#1  0x00007fffce22a5fe in evp_keymgmt_gen (keymgmt=0x864340, genctx=0x964cb0, cb=0x7fffce237ee7 <ossl_callback_to_pkey_gencb>, cbarg=0x7ec8e0) at crypto/evp/keymgmt_meth.c:446
#2  0x00007fffce22921c in evp_keymgmt_util_gen (target=0x646fc0, keymgmt=0x864340, genctx=0x964cb0, cb=0x7fffce237ee7 <ossl_callback_to_pkey_gencb>, cbarg=0x7ec8e0)
    at crypto/evp/keymgmt_lib.c:518
#3  0x00007fffce238171 in EVP_PKEY_generate (ctx=0x7ec8e0, ppkey=0x7fffffffc458) at crypto/evp/pmeth_gn.c:189
#4  0x00007fffce2383d9 in EVP_PKEY_paramgen (ctx=0x7ec8e0, ppkey=0x7fffffffc458) at crypto/evp/pmeth_gn.c:265
#5  0x00007fffd06cc8be in pkey_blocking_gen (ptr=0x7fffffffc450) at ../../../../ext/openssl/ossl_pkey.c:357
#6  0x00007fffd06ccb7f in pkey_generate (argc=1, argv=0x7fffe9cff050, self=140737115274880, genparam=1) at ../../../../ext/openssl/ossl_pkey.c:431
#7  0x00007fffd06ccc38 in ossl_pkey_s_generate_parameters (argc=1, argv=0x7fffe9cff050, self=140737115274880) at ../../../../ext/openssl/ossl_pkey.c:472

[rhenium]

This was overlooked in #784, even while mentioning it in the commit message...

[rhenium]

Done by #786.