Use ctypes find_library to discover libcrypto better

[ryandub]

I ran into an issue trying to use PyChef in Docker containers:

  File "/.virtualenvs/sys/local/lib/python2.7/site-packages/chef/__init__.py", line 5, in <module>
    from chef.api import ChefAPI, autoconfigure
  File "/.virtualenvs/sys/local/lib/python2.7/site-packages/chef/api.py", line 18, in <module>
    from chef.rsa import Key
  File "/.virtualenvs/sys/local/lib/python2.7/site-packages/chef/rsa.py", line 9, in <module>
    _eay = CDLL('libcrypto.so')
  File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcrypto.so: cannot open shared object file: No such file or directory

Replicating the code in rsa.py that tries to load libcrypto.so I get the same error:

# python
Python 2.7.3 (default, Mar 13 2014, 11:03:55)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> ctypes.CDLL('libcrypto.so')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcrypto.so: cannot open shared object file: No such file or directory

Basically, there is no libcrypto.so library and there is no symlink for libcrypto.so.1.0.0 to libcrypto.so, so ctypes.CDLL('libcrypto.so') fails. Looking at the python executable being used, it is linked to libcrypto.so.1.0.0, not libcrypto.so which is probably why CDLL is failing:

# ldd `which python`
    linux-vdso.so.1 =>  (0x00007fff223f0000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f74b1a0d000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f74b1809000)
    libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f74b1605000)
    libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f74b13a7000)
    libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f74b0fcc000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f74b0db4000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f74b0ab8000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f74b06f8000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f74b04e1000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f74b1c30000)

Looking at the ctypes documentation there is a function called find_library that helps find linked libraries in a smart way. Using find_library cytpes is able to detect the libcrypto library linked with python and use it:

# python
Python 2.7.3 (default, Mar 13 2014, 11:03:55)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import util
>>> util.find_library('crypto')
'libcrypto.so.1.0.0'

Using this method allows rsa.py to find the correct linked version of libcrypto and does not require os-level symlinking shenanygans. :)

[ryandub]

The chef.tests.test_client.ClientTestCase appears to be failing off of master as well:

======================================================================
FAIL: test_get (chef.tests.test_client.ClientTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/projects/pychef/chef/tests/test_client.py", line 14, in test_get
    self.assertTrue(client.public_key)
AssertionError: None is not True

----------------------------------------------------------------------
Ran 72 tests in 35.723s

[coderanger]

Yeah, Opscode changed some APIs and so the tests no longer work. Its on my list to fix up when I get time. As for the find_library bit, the next release is hopefully going to ditch ctypes entirely in favor of cryptography, just waiting for RSA key loading to get in.

[gondoi]

@coderanger what is the expected timeframe on that? This seems like a pretty simple fix that can be used in the interim.

[ryandub]

@coderanger Do you have an ETA on the next release that ditches ctypes? If not, can this be incorporated in the interim? We have been running this code in production since September without issue, but I would like to use PyChef in other places where running a fork is less appropriate.

Thanks!

[viyh]

+1. I had to implement this in order for PyChef to work on AWS Lambda. Please merge this change.

[martinb3]

👍 I ran into this on Lambda as well today. I'd love to see this merged.

[ryandub]

@martinb3 @viyh You may want to look into using https://github.com/samstav/okchef - we have moved to using that for a lot of our projects.