Publish a platform-specific wheel for Apple M1 (macOS ARM64) on PyPI

[oliverl-21]

SUMMARY

add Package for macos arm64

ISSUE TYPE

• Feature Idea

ADDITIONAL INFORMATION

N/A

[webknjaz]

It's an interesting idea but until GitHub adds respective VMs to GHA, there's nothing we can do about this. So for now the users will have to install ansible-pylibssh from the sdist.

Refs:

• Support for VMs on Apple M1 actions/runner-images#2187
• Add ARM + MacOS target to be able to use on self-hosted runners actions/runner#805

[radokristof]

Well for me even building it locally does not work.
Before Ventura I was able to build it and install it locally. Now this fails with an error related to clang:

* Installing packages in isolated environment... (wheel)
* Building wheel...
[1/1] Cythonizing /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/pylibsshext/_libssh_version.pyx
running build_ext
building 'pylibsshext._libssh_version' extension
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/pylibsshext
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -DCYTHON_TRACE=1 -DCYTHON_TRACE_NOGIL=1 -I/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-env-yn3ws_5a/include -I/opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/pylibsshext/_libssh_version.c -o /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/pylibsshext/_libssh_version.o
creating build
creating build/lib.macosx-13-arm64-3.10
creating build/lib.macosx-13-arm64-3.10/pylibsshext
clang -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -lssh -DCYTHON_TRACE=1 -DCYTHON_TRACE_NOGIL=1 /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/pylibsshext/_libssh_version.o -o build/lib.macosx-13-arm64-3.10/pylibsshext/_libssh_version.cpython-310-darwin.so
ld: library not found for -lssh
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command '/usr/bin/clang' failed with exit code 1

ERROR Backend subprocess exited when trying to invoke build_wheel
ERROR: InvocationError for command /Users/radokristof/Documents/git/pylibssh/.tox/build-dists/bin/python -m build --outdir /Users/radokristof/Documents/git/pylibssh/dist/ . (exited with code 1)

[l3ender]

Seeing the same error after installing libssh (via brew) and soft linking libssh into command line tools (ref). I see the comment to "install ansible-pylibssh from the sdist"; can anyone provide detail on it?

I am only trying to install ansible-pylibssh because it a dependency of the ansible.netcommon collection. :(

[dmacduff]

@l3ender @radokristof I was able to build locally with the correct compiler flags exported, no symlink required. Procedure.

[webknjaz]

@dmacduff you shouldn't need to go through the contributor path (using tox etc.)
If you have all the build deps setting the env vars should be enough.
I'd expect CFLAGS="-I $(brew --prefix)/include -I ext -L $(brew --prefix)/lib -lssh" pip install ansible-pylibssh to just work. If you want to produce a wheel artifact, change the command to pip wheel or python -m build --wheel.

[dmacduff]

@webknjaz Brilliant! I spun up a new venv, and once I pip cache purged, your solution worked like a charm, no git clone, tox, or locally built wheels required. Much simpler. Now back to S.O. to update my answers.

[andresfrancopt]

Hi @webknjaz,
Will the solution you provided still work if I'm using a virtual environment?
Thanks in advance.

[webknjaz]

@andresfrancopt the person commenting before you said it worked.

[samdoran]

There is actually a universal wheel available, but the dynamic libraries in the universal wheel are x86_64 only.

> pip download ansible-pylibssh
> unzip ansible_pylibssh-1.1.0-cp311-cp311-macosx_10_9_universal2.whl
> file pylibsshext/.dylibs/*
pylibsshext/.dylibs/libcrypto.1.1.dylib: Mach-O 64-bit dynamically linked shared library x86_64
pylibsshext/.dylibs/libssh.4.9.4.dylib:  Mach-O 64-bit dynamically linked shared library x86_64

The .so files in the wheel are universal.

So long as libssh is compiled and in the include and and linker paths (see command above), the --no-binary flag will force a build. This is essentially the same as building a wheel from source and installing it.

pip install --no-binary :all: ansible-pylibssh

[samdoran]

This was the root cause of [WARNING]: ansible-pylibssh not installed, falling back to paramiko being displayed even though ansible-pylibssh is installed.

The network connection plugin checks for pylibssh by running three imports. The second fails:

> python -c 'from pylibsshext.errors import LibsshSCPException, LibsshSessionException'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: dlopen(/Users/sdoran/.pyenv/versions/debug/lib/python3.11/site-packages/pylibsshext/errors.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_ssh_get_error'

[NilashishC]

Hi @ryanmerolle. Does the steps shared by @webknjaz or @samdoran their comment work for you?

[ryanmerolle]

Sorry I missed this update. Here is the error I get on my m1 MacBook:

$ pip install --no-binary :all: ansible-pylibssh
WARNING: Skipping /opt/homebrew/lib/python3.11/site-packages/cffi-1.16.0.dist-info due to invalid metadata entry 'name'
Collecting ansible-pylibssh
  Downloading ansible-pylibssh-1.1.0.tar.gz (106 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 106.2/106.2 kB 4.0 MB/s eta 0:00:00
  Installing build dependencies ... /
done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: ansible-pylibssh
  Building wheel for ansible-pylibssh (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for ansible-pylibssh (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      [1/1] Cythonizing /private/var/folders/ct/kd8qq0sd7y1218smxjw9fwsh0000gn/T/pip-install-gmxpv3cu/ansible-pylibssh_35f12a538de049858f6ca36822265f2e/src/pylibsshext/_libssh_version.pyx
      /private/var/folders/ct/kd8qq0sd7y1218smxjw9fwsh0000gn/T/pip-install-gmxpv3cu/ansible-pylibssh_35f12a538de049858f6ca36822265f2e/src/pylibsshext/_libssh_version.c:1201:10: fatal error: 'libssh/libssh.h' file not found
      #include "libssh/libssh.h"
               ^~~~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for ansible-pylibssh
Failed to build ansible-pylibssh

[samdoran]

That looks like you are missing the path to libssh in your LDFLAGS and CPPFLAGS.