MacOS : Cannot allocate write+execute memory for ffi.callback()

[ealpha]

python3 play_file.py file.wav

MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks

[mgeier]

See #270.

I'm open to suggestions how to improve the situation.

[gormster]

I cloned the repo and hacked at it until it worked.

https://github.com/gormster/python-sounddevice/tree/api-mode-experimental

Note that this replaces ABI mode completely with API mode. I'm sure there's a graceful way to handle both.

also, I should probably be overriding build_py not install but I couldn't get that to work. I don't have a ton of experience with setuptools and the documentation is pretty spotty.

[mgeier]

Thanks @gormster, see also #91.

Note that this replaces ABI mode completely with API mode. I'm sure there's a graceful way to handle both.

This would be great, but I'm not so sure that's possible.

See also https://groups.google.com/g/python-cffi/c/oBMFw7R1sFI/m/LJDo6NPJCwAJ

[gormster]

Ok so… I've done a thing where it does (I think) handle this gracefully. (Well, sort of; it uses an environment variable at runtime to select which mode to use. That was the best I could think of without breaking the API.) However, in testing it, I realised the original problem is actually only occurring when using a conda environment and not in my base Python install.

So I tried using it under my conda install and… well, it also seems to work when I build it from source. I'm kind of a novice with conda, I usually use pipenv, but it seems like (especially for M1 Macs) conda is a lot more usable.

And here's the kicker: if I go into a fresh conda environment with numpy installed, then within that environment do python -m pip install sounddevice… that also works fine. Even though that should be identical to conda install. I mean… python -m pip install is literally exactly what conda install does according to the recipe for python-sounddevice. And yet… it doesn't work the same. I don't get it.

In short: I have no real way of testing that this patch fixes this bug at all.

Now, this all said, API mode is clearly superior in cases where a C compiler is available, or for which a wheel can be precompiled, and it sure seems like the way I've done it does switch between API and ABI mode seamlessly… but again, it's tough to test if that's actually what's happening.

[gormster]

@ealpha In case you missed it in that wall of text, here's a potential workaround:

1. Create a fresh conda environment. Do not install python-sounddevice. (You can install numpy or whatever else you want.)
2. Use conda activate to enter your new environment
3. Run python -m pip install sounddevice

[HaHeho]

Not contributing to the actual issue, but if I install via conda conda install python-sounddevice, I end up with the installation from conda-forge, not PyPI. This is due to my conda configuration also considering conda-forge.

Otherwise one would do conda install python-sounddevice -c conda-forge. But in that case one ends up with all packages from conda-forge instead of conda-main. For things like numpy this is not preferable however (strong performance differences depending on system architecture).

So an okay way around that would be (resulting in a version of portaudio from conda-forge, but same version number as conda-main):

conda create -n sd numpy
conda activate sd
conda install python-sounddevice -c conda-forge

[mgeier]

There are at least two versions of the PortAudio library availble for conda:

• https://anaconda.org/anaconda/portaudio seems to be not available for M1.
• https://anaconda.org/conda-forge/portaudio seems to support osx-arm64.

It is theoretically possible to install portaudio with conda and sounddevice with pip, but I don't think that makes sense.

However, it is possible to not install portaudio with conda and then install sounddevice with pip, as @gormster mentioned above. In this case, the bundled PortAudio library from https://github.com/spatialaudio/portaudio-binaries will be used.

In any case, you can check which library is actually used like this:

import sounddevice as sd
print(sd._libname)

In general, I would recommend conda-forge over the default channel. However, I wasn't aware of this:

For things like numpy this is not preferable however (strong performance differences depending on system architecture).

This seems unfortunate. Why is conda-forge not as good? Isn't that something that should be fixed?
I didn't find an issue for that: https://github.com/conda-forge/numpy-feedstock/issues

https://anaconda.org/anaconda/numpy
https://anaconda.org/conda-forge/numpy

[HaHeho]

In general, I would recommend conda-forge over the default channel. However, I wasn't aware of this:

For things like numpy this is not preferable however (strong performance differences depending on system architecture).

This seems unfortunate. Why is conda-forge not as good? Isn't that something that should be fixed? I didn't find an issue for that: https://github.com/conda-forge/numpy-feedstock/issues

https://anaconda.org/anaconda/numpy https://anaconda.org/conda-forge/numpy

FYI, the difference is what accelerated linear algebra library Numpy is built against, as documented here
https://numpy.org/install/#numpy-packages--accelerated-linear-algebra-libraries. I suppose the conclusion of "strong performance differences" should be taken with a grain of salt and seen in context of what one is intending to do, and on what system architecture.

[AjaniStewart]

@ealpha In case you missed it in that wall of text, here's a potential workaround:

1. Create a fresh conda environment. Do not install python-sounddevice. (You can install numpy or whatever else you want.)
2. Use conda activate to enter your new environment
3. Run python -m pip install sounddevice

This worked for me. Just a heads up for other people, if you conda install other packages that has cffi as a dependency (after python -m pip install sounddevice), it might muck things up. Better to use pip for those packages.

[ealpha]

I still have this problem in macOS 12.6

[mberz]

@ealpha You may want to make sure that you install cffi via pip as well as stated by @AjaniStewart. I can confirm that the workaround works on macOS 12.6 & arm64 if cffi and sounddevice are installed using pip.
You can use python -m pip install sounddevice --force-reinstall to do so, beware that this will do a force reinstall of all dependencies though.

[petriewong]

Hi, I had the same issue with my M1 mac mini, macos 13.0 & anaconda environment.

pip uninstall cffi
pip install cffi 

It works for me.

[davidjo]

I had the same issue on 15.6 using macports attempting to run the play_sine.py sample program (for python 3.13 - latest macports python).

I also found that installing using the system pip3 it worked (python 3.9).

I have pinned this down to the cffi python bindings (1.17.1).
There is a c code source _cffi_backend.c which is where the error message originates from.
Just before the error there is an ifdef based on CFFI_CHECK_FFI_CLOSURE_ALLOC_MAYBE which is defined near the top of the for various OSs - for macos it depends on APPLE and FFI_AVAILABLE_APPLE.
The error depends on FFI_AVAILABLE_APPLE not being defined which means cffi_closure_alloc is called instead of ffi_closure_alloc (which comes from libffi) which apparently returns a NULL variable which leads to the error.
Explicitly defining FFI_AVAILABLE_APPLE as a compiler flag in the setup.py for the cffi python bindings and play_sine.py works - if not it doesnt.
Not sure why FFI_AVAILABLE_APPLE isnt defined as macos ffi.h (SDK includes) does define it - but there is now a new definition FFI_AVAILABLE_APPLE_2019 for 10.15 which may be the issue.

Building the cffi bindings creates an .so file named like _cffi_backend.cpython-313-darwin.so.
I also noted that the pip installed cffi uses the system libffi for the .so file whereas macports uses its own libffi version.
However if FFI_AVAILABLE_APPLE is explicitly defined in the cffi build (1.17.1) then using either the system libffi or macports libffi both allow play_sine.py to work.
For a working .so file a nm on the .so file lists _ffi_closure_alloc as undefined symbol - non-working .so files do not have any _ffi_closure... definitions.