Fails to load with Rubygems 3.4

[davekaro]

I've installed ruby 3.2 on macOS Ventura (13.0.1) with M1 (arm64) using ruby-build. Launching irb and typing require "argon2" I see the following:

/Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/ffi-1.15.5/lib/ffi/library.rb:145:in `block in ffi_lib': Could not open library '/Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/ext/argon2_wrap/libargon2_wrap.bundle.dSYM/Contents/Resources/DWARF/libargon2_wrap.bundle': dlopen(/Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/ext/argon2_wrap/libargon2_wrap.bundle.dSYM/Contents/Resources/DWARF/libargon2_wrap.bundle, 0x0005): tried: '/Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/ext/argon2_wrap/libargon2_wrap.bundle.dSYM/Contents/Resources/DWARF/libargon2_wrap.bundle' (unloadable mach-o file type 10 '/Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/ext/argon2_wrap/libargon2_wrap.bundle.dSYM/Contents/Resources/DWARF/libargon2_wrap.bundle'), '/System/Volumes/Preboot/Cryptexes/OS/Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/ext/argon2_wrap/libargon2_wrap.bundle.dSYM/Contents/Resources/DWARF/libargon2_wrap.bundle' (no such file), '/Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/ext/argon2_wrap/libargon2_wrap.bundle.dSYM/Contents/Resources/DWARF/libargon2_wrap.bundle' (unloadable mach-o file type 10 '/Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/arg (LoadError)
	from /Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/ffi-1.15.5/lib/ffi/library.rb:99:in `map'
	from /Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/ffi-1.15.5/lib/ffi/library.rb:99:in `ffi_lib'
	from /Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/lib/argon2/ffi_engine.rb:10:in `<module:Ext>'
	from /Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/lib/argon2/ffi_engine.rb:8:in `<module:Argon2>'
	from /Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/lib/argon2/ffi_engine.rb:6:in `<top (required)>'
	from <internal:/Users/dave/.rbenv/versions/3.2.0/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from <internal:/Users/dave/.rbenv/versions/3.2.0/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from /Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/lib/argon2.rb:4:in `<top (required)>'
	from <internal:/Users/dave/.rbenv/versions/3.2.0/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:160:in `require'
	from <internal:/Users/dave/.rbenv/versions/3.2.0/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:160:in `rescue in require'
	from <internal:/Users/dave/.rbenv/versions/3.2.0/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:149:in `require'
	from (irb):2:in `<main>'
	from /Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/irb-1.6.2/exe/irb:11:in `<top (required)>'
	from /Users/dave/.rbenv/versions/3.2.0/bin/irb:25:in `load'
	from /Users/dave/.rbenv/versions/3.2.0/bin/irb:25:in `<main>'
<internal:/Users/dave/.rbenv/versions/3.2.0/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- argon2 (LoadError)
	from <internal:/Users/dave/.rbenv/versions/3.2.0/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from (irb):2:in `<main>'
	from /Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/irb-1.6.2/exe/irb:11:in `<top (required)>'
	from /Users/dave/.rbenv/versions/3.2.0/bin/irb:25:in `load'
	from /Users/dave/.rbenv/versions/3.2.0/bin/irb:25:in `<main>'

Running on CI with Github Actions and ruby 3.2 on Ubuntu 22.04.1 (x86_64 arch) I see a similar but different error:

LoadError: cannot find 'argon2_wrap' library
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/ffi-compiler-1.0.1/lib/ffi-compiler/loader.rb:21:in `find'
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/argon2-2.1.2/lib/argon2/ffi_engine.rb:10:in `<module:Ext>'
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/argon2-2.1.2/lib/argon2/ffi_engine.rb:8:in `<module:Argon2>'
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/argon2-2.1.2/lib/argon2/ffi_engine.rb:6:in `<main>'
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/argon2-2.1.2/lib/argon2.rb:4:in `<main>'
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/home/runner/work/my-project-name/my-project-name/config/application.rb:24:in `<top (required)>'
/home/runner/work/my-project-name/my-project-name/config/environment.rb:2:in `require_relative'
/home/runner/work/my-project-name/my-project-name/config/environment.rb:2:in `<top (required)>'
/home/runner/work/my-project-name/my-project-name/Rakefile:3:in `require_relative'
/home/runner/work/my-project-name/my-project-name/Rakefile:3:in `block in <top (required)>'
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/hostedtoolcache/Ruby/3.2.0/x64/bin/bundle:25:in `load'
/opt/hostedtoolcache/Ruby/3.2.0/x64/bin/bundle:25:in `<main>'
Tasks: TOP => assets:precompile => environment
(See full trace by running task with --trace)

I tried cloning the argon2 repo, and was able to run bin/setup, bin/test, and rake test all without any errors. I looked at https://github.com/technion/ruby-argon2/blob/master/test/test_helper.rb#L3 and found this helped. Steps to reproduce:

1. Open irb session in the ruby-argon2 directory
2. Type require "argon2" see failures (same as above for macOS)
3. Type $LOAD_PATH.unshift File.expand_path("lib")
4. Type require "argon2" and now it works.

Note the above only works after running bin/setup, even though I've installed argon2 already. This seems to be because when running bin/setup, the correct file in ext/argon2_wrap is created, specifically the libargon_wrap2 file:

On macOS, Ruby 3.1.3 (argon2 loads no issue)

~ ❯❯❯ ls -la .rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/argon2-2.1.2/ext/argon2_wrap/
total 496
drwxr-xr-x  9 dave  staff     288 Dec  5 16:58 .
drwxr-xr-x  4 dave  staff     128 Dec  5 16:58 ..
-rw-r--r--  1 dave  staff    1958 Dec  5 16:58 Makefile
-rw-r--r--  1 dave  staff    4504 Dec  5 16:58 argon_wrap.c
-rw-r--r--  1 dave  staff      98 Dec  5 16:58 extconf.rb
-rwxr-xr-x  1 dave  staff   76450 Dec  5 16:58 libargon2_wrap.bundle
drwxr-xr-x  3 dave  staff      96 Dec  5 16:58 libargon2_wrap.bundle.dSYM
-rwxr-xr-x  1 dave  staff  151552 Dec  5 16:58 libargon2_wrap.so
-rw-r--r--  1 dave  staff    4500 Dec  5 16:58 test.c

On macOS, Ruby 3.2.0 (missing the libargon2_wrap.bundle file, argon2 errors on require)

~ ❯❯❯ ls -la .rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/ext/argon2_wrap/
total 344
drwxr-xr-x  8 dave  staff     256 Dec 25 12:14 .
drwxr-xr-x  4 dave  staff     128 Dec 25 12:14 ..
-rw-r--r--  1 dave  staff    1958 Dec 25 12:14 Makefile
-rw-r--r--  1 dave  staff    4504 Dec 25 12:14 argon_wrap.c
-rw-r--r--  1 dave  staff      98 Dec 25 12:14 extconf.rb
drwxr-xr-x  3 dave  staff      96 Dec 25 12:14 libargon2_wrap.bundle.dSYM
-rwxr-xr-x  1 dave  staff  151552 Dec 25 12:14 libargon2_wrap.so
-rw-r--r--  1 dave  staff    4500 Dec 25 12:14 test.c

On Ubuntu, Ruby 3.1.3 (argon2 loads no issue)

$ ls -la .rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/argon2-2.1.2/ext/argon2_wrap/
total 256
drwxrwxr-x 2 ubuntu ubuntu   4096 Dec 25 18:10 .
drwxrwxr-x 4 ubuntu ubuntu   4096 Dec 25 18:10 ..
-rw-r--r-- 1 ubuntu ubuntu   1958 Dec 25 18:10 Makefile
-rw-r--r-- 1 ubuntu ubuntu   4504 Dec 25 18:10 argon_wrap.c
-rw-r--r-- 1 ubuntu ubuntu     98 Dec 25 18:10 extconf.rb
-rwxrwxr-x 1 ubuntu ubuntu 227056 Dec 25 18:10 libargon2_wrap.so
-rw-r--r-- 1 ubuntu ubuntu   4500 Dec 25 18:10 test.c

On Ubuntu, Ruby 3.2.0 (missing the libargon2_wrap.so file, argon2 errors on require)

$ ls -la .rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/ext/argon2_wrap/
total 32
drwxrwxr-x 2 ubuntu ubuntu 4096 Dec 25 17:49 .
drwxrwxr-x 4 ubuntu ubuntu 4096 Dec 25 17:49 ..
-rw-r--r-- 1 ubuntu ubuntu 1958 Dec 25 17:49 Makefile
-rw-r--r-- 1 ubuntu ubuntu 4504 Dec 25 17:49 argon_wrap.c
-rw-r--r-- 1 ubuntu ubuntu   98 Dec 25 17:49 extconf.rb
-rw-r--r-- 1 ubuntu ubuntu 4500 Dec 25 17:49 test.c

I'm not sure if this is an ffi issue, or the way argon2 is using ffi.

[scott-knight]

Found the same issue this morning when updating my app during deploy test to github. Ruby 3.2.0, Rails 7.0.4, Ubuntu 22.04 instance

[communiteq]

Same issue here. Ruby 2.7.6p219. Broke since this morning (5 AM UTC).

[technion]

Thanks for reporting this and the detail. Much like the work in this thread, I'm struggling to work out whether this is an FFI issue or just something involving the build scripts. I'm working with the Ruby 3.2-rc1 Docker image (which currently seems to be the latest) and I can replicate it there.

I'm going through open issues on FFI and nothing seems relevant.

I've found if you make sure the library is built and do an strace, it does load it even before complaining it's not built:

openat(AT_FDCWD, "/ruby-argon2/ext/argon2_wrap/libargon2_wrap.so", O_RDONLY|O_CLOEXEC) = 5

Whatever's going on I'm sure someone else's gem will have the same issue, which I'm hoping leads to a fix.

@communiteq Just to confirm are you saying you had this same break without upgrading to Ruby 3.2.0? I can't replicate that.

[technion]

@davekaro Regarding this step you supplied:

Type $LOAD_PATH.unshift File.expand_path("lib")

You may notice this is actually called in the Gemfile already:

https://github.com/technion/ruby-argon2/blob/master/argon2.gemspec#L5

I wonder if Ruby 3.2 changed the behaviour around recognising this?

[technion]

You know I went through the FFI changes and couldn't find anything that looked relevant, but I bumped from 1.14 to 1.15 and CI just gave an all pass with 3.2 now tested.

I will cut a new version as soon as I feed my cats.

[technion]

Done. Can anyone else check with version 2.1.3 and confirm?

[communiteq]

Yes, confirmed, tests suddenly started breaking yesterday. And worse, updating to 2.1.3 does not resolve the issue.
Let me know if there is anything you need to know.

ruby 2.7.6p219 (2022-04-12 revision c9c2245c0a) [x86_64-linux]
Ubuntu 22.04.1 LTS

Not happening on Ubuntu 18.04.6 LTS

The gem is loaded as part of a Discourse plugin. Something must have changed somewhere... (and it was not the plugin)

gem install ffi -v 1.15.5 -i /var/www/discourse/plugins/discourse-migratepassword/gems/2.7.6 --no-document --ignore-dependencies --no-user-install
Building native extensions. This could take a while...
Successfully installed ffi-1.15.5
1 gem installed
gem install ffi-compiler -v 1.0.1 -i /var/www/discourse/plugins/discourse-migratepassword/gems/2.7.6 --no-document --ignore-dependencies --no-user-install
Successfully installed ffi-compiler-1.0.1
1 gem installed
gem install argon2 -v 2.1.3 -i /var/www/discourse/plugins/discourse-migratepassword/gems/2.7.6 --no-document --ignore-dependencies --no-user-install
Building native extensions. This could take a while...
Successfully installed argon2-2.1.3
1 gem installed
rake aborted!
LoadError: cannot find 'argon2_wrap' library
/var/www/discourse/plugins/discourse-migratepassword/gems/2.7.6/gems/ffi-compiler-1.0.1/lib/ffi-compiler/loader.rb:21:in `find'

[technion]

@communiteq If you're running 2.7 I'm sure you have a different issue. I do know that I've walked over to multiple platforms, installed the 2.1.3 gem and then used it from irb without issue. I'm happy to see if there's anything I can do there but in the meantime, can either of the people who broke on 3.2 advise?

[communiteq]

Given the timing of "both" issues I do suspect that they have the same root cause. So yes, I absolutely understand that the issue is now resolved from your side. I am just hoping that someone ever finds out what has caused this in the first place, since that might give me a clue to working towards a resolution. It's surprising that bumping the requirements to higher versions worked for you since I was already on ffi-1.15.5 and ffi-compiler-1.0.1. Weird huh?

That said, the Argon2 gem is (somehow!) the only gem that has broken in this pretty complex install.

[technion]

I do see your point, I'm just stuck on the fact there haven't been any pushes to this repo for some time, so outside of Ruby 3.2 I can't see what would break your build. I do know that looking at the module you refer to:

https://github.com/communiteq/discourse-migratepassword/blob/master/plugin.rb

Use of 2.1.1 seems hardcoded there, so I don't believe it's going to matter what you happen to have installed.

[technion]

Just to confirm, it appears that although the CI works fine, there's something still broken here. I had a fair bit of trouble due to the lack of a current Ruby 3.2 docker container but I've built a more complete lab of this.

root@7e9fcf33fcaf:/# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"

# Install prereqs..

eval "$(/root/.rbenv/bin/rbenv init - bash)"

rbenv install 3.2.0
root@7e9fcf33fcaf:/# mkdir myapp
root@7e9fcf33fcaf:/# cd myapp
root@7e9fcf33fcaf:/myapp# bundle init
Writing new Gemfile to /myapp/Gemfile
root@7e9fcf33fcaf:/myapp# nano Gemfile
root@7e9fcf33fcaf:/myapp# cat Gemfile
# frozen_string_literal: true

source "https://rubygems.org"
gem "argon2", "2.1.3"
root@7e9fcf33fcaf:/myapp# bundle install
Don't run Bundler as root. Installing your bundle as root will break this application for all non-root users on this
machine.
Fetching gem metadata from https://rubygems.org/.
Using rake 13.0.6
Fetching ffi 1.15.5
Using bundler 2.4.1
Installing ffi 1.15.5 with native extensions
Fetching ffi-compiler 1.0.1
Installing ffi-compiler 1.0.1
Fetching argon2 2.1.3
Installing argon2 2.1.3 with native extensions
Bundle complete! 1 Gemfile dependency, 5 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
root@7e9fcf33fcaf:/myapp# bundle exec myapp.rb
bundler: failed to load command: myapp.rb (myapp.rb)
/root/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/ffi-compiler-1.0.1/lib/ffi-compiler/loader.rb:21:in `find': cannot find 'argon2_wrap' library (LoadError)

I'll continue investigating but I'm sure appreciative of any other suggestions. I do note that in this environment, it simply seems to never be running the actual build script.

openat(AT_FDCWD, "/root/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.3/ext/argon2_wrap", O_RDONLY|O_CLOEXEC|O_DIRECTORY) = 5
openat(AT_FDCWD, "/root/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.3/ext/argon2_wrap", O_RDONLY|O_CLOEXEC|O_DIRECTORY) = 5
write(2, "/root/.rbenv/versions/3.2.0/lib/"..., 2163/root/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/ffi-compiler-1.0.1/lib/ffi-compiler/loader.rb:21:in `find': cannot find 'argon2_wrap' library (LoadError)
root@7e9fcf33fcaf:/myapp# ls /root/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.3/ext/argon2_wrap
Makefile  argon_wrap.c  extconf.rb  test.c

This workaround takes care of it:

root@7e9fcf33fcaf:/myapp# cd /root/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.3/ext/argon2_wrap
root@7e9fcf33fcaf:~/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.3/ext/argon2_wrap# make
cc -pthread -O3 -Wall -g -I../phc-winner-argon2/include -I../phc-winner-argon2/src -shared -fPIC ../phc-winner-argon2/src/argon2.c ../phc-winner-argon2/src/core.c ../phc-winner-argon2/src/blake2/blake2b.c ../phc-winner-argon2/src/thread.c ../phc-winner-argon2/src/encoding.c argon_wrap.c ../phc-winner-argon2/src/ref.c -o libargon2_wrap.so
root@7e9fcf33fcaf:~/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.3/ext/argon2_wrap# cd
root@7e9fcf33fcaf:~# irb
irb(main):001:0> require 'argon2'
=> true
irb(main):002:0>

[communiteq]

Hmm why is extconf.rb empty?

4e70673#diff-cbcaa657275e9aedc9400e59a95de0559a89551fd82ef78ce5bc1aaf99f6276d

[technion]

@communiteq Because I had Makefile that took care of the external libraries and didn't need to learn another Ruby DSL to make it build one for me. They may well have broken that.

[technion]

It appears that is in fact what's going on here, it simply doesn't run the build. I've started with mkmf and it definitely tries to build then, but there's a rabbit hole of not finding the correct files and so on.

[davekaro]

it simply seems to never be running the actual build script.

yeah I think this is what I was seeing too, based on the missing files in the ext/argon2_wrap. The reason it works when building the gem locally and then adding lib to the $LOAD_PATH was because the locally built gem had the files. I don't think I'm saying anything you don't already know at this point. It is weird though because there is the output when installing the gem:

Fetching argon2-2.1.3.gem
Building native extensions. This could take a while...

and it takes the same amount of time as installing on 3.1...... so it does seem it's running the build script and doing something.