-
Notifications
You must be signed in to change notification settings - Fork 2.2k
fix: make wrapped C++ functions pickleable #5580
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…ch")`
macos-13 • brew install llvm
```
/Users/runner/work/pybind11/pybind11/include/pybind11/detail/function_record_pyobject.h:40:26: error: cast from 'PyObject *(*)(PyObject *, PyObject *, PyObject *)' (aka '_object *(*)(_object *, _object *, _object *)') to 'PyCFunction' (aka '_object *(*)(_object *, _object *)') converts to incompatible function type [-Werror,-Wcast-function-type-mismatch]
40 | = {{"__reduce_ex__", (PyCFunction) reduce_ex_impl, METH_VARARGS | METH_KEYWORDS, nullptr},
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
```
|
@henryiii This is now ready for review. @rainwoodman FYI |
|
Not sure I want to argue with ChatGPT, but I feel its description is overblown. It's very rare that you want to pickle a function. You want to (and very much can) pickle objects/data, but it is quite rare to need to pickle a function (which has no state). Boost-histogram, for example, has extensive Parallel and Concurrent Processing, Distributed Computing and Cloud Environments (including dask-histogram), and Testing and Mocking usage and we've never once needed to pickle a function. There are cases where it can be handy (if you are using functions as inputs to your API, for example), which is probably why a large system like PyCLIF ended up hitting it. But ChatGPT makes it sound like it's a massive deal, when I think it's a minor nicety. ;) Are we sure this won't run into any issues in supporting the Stable ABI? I'd assume not, since we are relying a bit less on provided tools in CPython, but worth asking. That's something we really should look into doing in the future. Also, moving away from the tooling provided for this purpose worries me slightly. The key feature of PyCapsule is allowing a C-API from one extension module to use another one transparently without the Python middle layer; this is really important and is used by SciPy, boost-histogram, and many other performance applications, I assume that's not affected by this? Here's an example in boost-histogram: https://github.com/scikit-hep/boost-histogram/blob/460ef90905d6a8a9e6dd3beddfe7b4b49b364579/include/bh_python/transform.hpp#L70-L74 - Can you still convert pybind11 functions to By the way, a remotely related issues is that a tiny bit of work (setting |
| /* destructor tp_del */ nullptr, | ||
| /* unsigned int tp_version_tag */ 0, | ||
| /* destructor tp_finalize */ nullptr, | ||
| #if PY_VERSION_HEX >= 0x03080000 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wasn't aware we supported Python < 3.8.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for catching that, removed with commit e8e921d.
|
Yes, this immediately breaks: diff --git a/pyproject.toml b/pyproject.toml
index 890c6e6..38f4e82 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,5 +1,5 @@
[build-system]
-requires = ["scikit-build-core>=0.11", "pybind11>=2.13.3"]
+requires = ["scikit-build-core>=0.11", "pybind11 @ git+https://github.com/rwgk/pybind11@pickle_callable"]
build-backend = "scikit_build_core.build"
[project]$ uv run pytest
Built boost-histogram @ file:///Users/henryschreiner/git/scikit-hep/boost-histogram
Uninstalled 1 package in 2ms
Installed 1 package in 3ms
ImportError while loading conftest '/Users/henryschreiner/git/scikit-hep/boost-histogram/tests/conftest.py'.
tests/conftest.py:16: in <module>
import boost_histogram as bh
src/boost_histogram/__init__.py:3: in <module>
from . import accumulators, axis, numpy, storage
src/boost_histogram/axis/__init__.py:24: in <module>
from . import transform
src/boost_histogram/axis/transform.py:156: in <module>
sqrt = Function("_sqrt_fn", "_sq_fn", convert=_internal_conversion, name="sqrt")
src/boost_histogram/axis/transform.py:145: in __init__
self._this = cpp_class(forward, inverse, convert, name)
E ValueError: PyCapsule_GetName called with invalid PyCapsule objectThis is a core performance and design requirement for many applications, including a core design of several parts of SciPy ( |
|
It's true that pickling functions is rare, but I stumbled over this multiple times while working at Google. Note that SWIG-wrapped and PyCLIF-wrapped functions are pickleable (those two systems are still in wide use there). If functions are pickleable, the feature is getting used, and if not, people are forced to fish for workarounds, which can be a significant time sink depending on the situation. Global testing at Google passed with the original PR (google/pybind11clif#30099). The boost-histogram code is using Copying here for easy reference: if(auto cfunc = func.cpp_function()) {
auto c = py::reinterpret_borrow<py::capsule>(
PyCFunction_GET_SELF(cfunc.ptr()));
auto rec = c.get_pointer<py::detail::function_record>();
if(rec && rec->is_stateless
&& py::detail::same_type(
typeid(raw_t*),
*reinterpret_cast<const std::type_info*>(rec->data[1]))) {
struct capture {
raw_t* f;
};
return std::make_tuple((reinterpret_cast<capture*>(&rec->data))->f,
src);
}
// Note that each error is slightly different just to help with debugging
throw py::type_error("Only ctypes double(double) and C++ functions allowed "
"(must be stateless)");
} We'd need to provide a public API for that performance optimization. We could provide something like |
Unlike for boost-histogram, there are no matches in scipy: Global testing passed, although that's now about 7 months ago. But I'm inclined to believe that boost-histogram is probably the only project that will need tweaking, to use a public API. |
|
@henryiii It looks like boost-histogram copy-paste-tweaked this pybind11 code:
Did you consider using that caster instead? (Call |
|
Today you can use pybind11 to create a function you can call in SciPy. After this PR, that will no longer work. I had to dig into the internals a bit to make something that takes a
|
I'm really surprised, could you please provide an example? — The Google global testing ~7 months ago had a lot of scipy dependencies for sure, although I don't know with what version of scipy. |
|
I'm confused by the way PyCapsule is used here. If boost-histogram is the only major issue, I can fix that. Here's the sample code, including the raw pointer method that ChatGPT suggested too (Faster than calling through Python, slower than PyCapsule): somecode.cpp:// cppimport
#include <pybind11/pybind11.h>
namespace py = pybind11;
double square(double x) {
return x * x;
}
void* get_my_function() {
return reinterpret_cast<void*>(square);
}
py::capsule get_my_capsule() {
return py::capsule(reinterpret_cast<void*>(square), "double (double)");
}
PYBIND11_MODULE(somecode, m) {
m.def("square", &square);
m.def("square_callable", &get_my_function);
m.def("square_capsule", &get_my_capsule);
}
/*
<%
setup_pybind11(cfg)
%>
*/example.py:import contextlib
import time
import cppimport.import_hook
import somecode
import scipy
import scipy.integrate
@contextlib.contextmanager
def timing(description: str) -> None:
start = time.monotonic()
yield
ellapsed_time = time.monotonic() - start
print(f"{description}: {ellapsed_time*1_000_000:.1f}µs")
with timing("Through Python"):
integral = scipy.integrate.quad(somecode.square, 0, 1)
print(integral)
llc = scipy.LowLevelCallable(somecode.square_callable(), signature="double (double)")
with timing("With LowLevelCallable"):
integral = scipy.integrate.quad(llc, 0, 1)
print(integral)
llc2 = scipy.LowLevelCallable(somecode.square_capsule())
with timing("With PyCapsule"):
integral = scipy.integrate.quad(llc2, 0, 1)
print(integral)$ uv init
$ uv add pybind11 scipy cppimport
$ uv run python example.py
Through Python: 43.3µs
(0.33333333333333337, 3.700743415417189e-15)
With LowLevelCallable: 12.1µs
(0.33333333333333337, 3.700743415417189e-15)
With PyCapsule: 7.0µs
(0.33333333333333337, 3.700743415417189e-15)Is there a way to pass this through without the explicit |
|
Thanks! That looks promising at first glance. I'll look at this carefully tomorrow. |
I get " |
I want to add a pybind11 unit test that mirrors your situation. Is your work accessible publicly, for me to use us a starting point? |
|
https://github.com/scikit-hep/boost-histogram/blob/460ef90905d6a8a9e6dd3beddfe7b4b49b364579/include/bh_python/transform.hpp is available, I don't have the various experiments I've been trying locally anywhere, though, since none of them worked. https://boost-histogram.readthedocs.io/en/latest/user-guide/transforms.html is where I describe how to use this (though I don't mention making a function in pybind11, I was assuming numba would be the most common way). |
Interesting, this escaped me completely before. @henryiii Please take a look here (demo PR #5585): 8b500da — Add This is to show that the new dedicated function record Python type introduced with this PR gives us new leverage. I don't a have good idea for making the magic |
| def test_assumptions(): | ||
| assert pickle.HIGHEST_PROTOCOL >= 0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why this? This isn't a test for pybind11, it's a test of the Python standard library? https://docs.python.org/3/library/pickle.html#pickle.HIGHEST_PROTOCOL and it's at least 4 since Python 3.4.
| assert pickle.HIGHEST_PROTOCOL >= 0 | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("protocol", range(pickle.HIGHEST_PROTOCOL + 1)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting, we don't normally support protocol 0 with py::pickle, but I guess this does?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed it like this:
-def test_assumptions():
+def all_pickle_protocols():
assert pickle.HIGHEST_PROTOCOL >= 0
+ return range(pickle.HIGHEST_PROTOCOL + 1)
-@pytest.mark.parametrize("protocol", range(pickle.HIGHEST_PROTOCOL + 1))
+@pytest.mark.parametrize("protocol", all_pickle_protocols())
def test_pickle_simple_callable(protocol):Basically, I'm paranoid about accidentally not running test_pickle_simple_callable() at all.
This is easily overlooked:
test_pickling.py::test_pickle_simple_callable[protocol0] SKIPPED (got empty parameter set ['protocol'], function test_pickle_simple_...)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oof, sorry, the above was meant to be a response to the other question.
Interesting, we don't normally support protocol 0 with
py::pickle, but I guess this does?
Yes, this is completely independent. There is no reason to limit what protocol versions are accepted.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If pickle reports a maximum pickle version of 0, pickle would be broken. That would be a Python bug, I don't think we should be worried about Python bugs of that magnitude. Especially since it only would produce a minor warning in the test suite.
|
I highly, highly, highly recommend rebasing and never making merge commits. Merge commits are evil and can cause strange results. There's even a "rebase" button in the UI. Though it's possible this is a GitHub bug, it probably wouldn't happen on linear history. |
|
You can cherry pick the commit, though I'd rather just use the value without testing it. We should rely on Python being correctly implemented, it adds distractions to our test suite. |
|
Coincidentally, I had this ChatGPT conversation about PRs and force pushing just before this little incident: I'm not happy either way, and gravitate to merging because I'm often referencing commits. It's bad if those references go stale. This is a tiny pretty inconsequential commit though. I'll simply carry it into #5585. |
|
I'm getting a warning on every single file when compiling now: |
Sorry I didn't see this before. Is this still an issue? What are the exact conditions to reproduce? We have these warning suppressions already: PYBIND11_WARNING_PUSH
#if defined(__GNUC__) && __GNUC__ >= 8
PYBIND11_WARNING_DISABLE_GCC("-Wcast-function-type")
#endif
#if defined(__clang__) && !defined(__apple_build_version__) && __clang_major__ >= 19
PYBIND11_WARNING_DISABLE_CLANG("-Wcast-function-type-mismatch")
#endif
static PyMethodDef tp_methods_impl[]
= {{"__reduce_ex__", (PyCFunction) reduce_ex_impl, METH_VARARGS | METH_KEYWORDS, nullptr},
{nullptr, nullptr, 0, nullptr}};
PYBIND11_WARNING_POPDo we need to work on the preprocessor conditions? — I didn't want to make them too general, maybe that's backfiring now? I remember I looked around quite a bit for the best way to implement that line, when I was working on the code originally, but I couldn't find a cleaner solution. Coincidentally, a couple days ago I spent some time looking at issue #5600, specifically: #5600 (comment) For that situation ChatGPT suggested among other things:
Maybe that could help here, too? In the long run. Warning suppressions seem best as an immediate solution. |
|
I’m on macOS. So probably the clang but not Apple is the problem. |
Does this silence the warnings for you? -#if defined(__clang__) && !defined(__apple_build_version__) && __clang_major__ >= 19
+#if defined(__clang__)It's a bit sweeping, but I think it should be fine? It's just for that one cast. Please let me know, it's a quick PR. |
|
I’ll try to check tomorrow, only have my phone ATM. |
I think I have something better for you: PR #5608 (CI was still running when I posted this, but it's looking promising.) |
This reverts commit e7e5d6e. Signed-off-by: Henry Schreiner <[email protected]>
As of pybind/pybind11#5212, pybind11 now uses `numpy.typing.NDArray` instead of `numpy.ndarray`, and as of pybind/pybind11#5580, it changed the name of the internal wrapper that Sphinx sees. Since we already ignore `numpy.float64` missing references for the same method, add the new name and new class to ignores as well.
f5fbe867 chore: bump to 3.0.1 (#5810)
cddec2bb fix: limit warning issue (#5807)
e71489c3 tests: avoid false DOWNLOAD_CATCH manually-specified variables warning (#5803)
8bbc3091 Improve buffer_info type checking in numpy docs (#5805)
03607757 correct homebrew package URL in installing.rst (#5808)
adb5603f chore: rename generic slots variable (#5793)
7aa3780d Replace robotpy-build with semiwrap (#5804)
ce712285 ci: avoid macOS 15 image change for iOS (#5798)
c1bf55a2 Fix subinterpreter exception handling SEGFAULT (#5795)
90bc05c7 chore(deps): bump actions/download-artifact (#5792)
580494c7 fix: LTO warning of gcc >= 11.4 (#5791)
6292b704 Explain linting suppressions (#5790)
94c82508 inline get_interpreter_state_uncheccked inline function (#5789)
23c59b6e ci: add android test (#5714)
a5665e3a fix: type_caster_enum_type for pointer types (#5694) (#5776)
9360553f chore(deps): update pre-commit hooks (#5785)
c5e8fec9 Make function record subinterpreter safe (#5771)
0db3d59f Allow multiphase modules to be re-imported (#5782)
780ec113 Make implicitly_convertable sub-interpreter and free-threading safe (#5777)
9af8adb7 Subinterpreter creation concurrency issues in 3.12 (#5779)
6972597c docs: show nogil in most examples (#5770)
33533ff3 Fix IsolatedConfig test (#5768)
7f5eea43 Check for __cpp_lib_remove_cvref as well (#5761)
49d19fef Implement binary version of make_index_sequence (#5751)
6aeae9c3 Fix py::trampoline_self_life_support visibility in docs (#5766)
316273ef fix: don't force -fvisibility=hidden on Windows (#5757)
c4ee83c4 fix:: initialize_generic compiler warning about nullptr dereference (#5756)
cc69a378 chore(deps): update pre-commit hooks (#5745)
66d394f2 docs: standardize header a bit (#5749)
fa72aff5 [skip ci] Small `docs/release.rst` update, mainly to remove `git push --tags`. (#5748)
422990f8 chore: get back to work (after v3.0.0 release) (#5747)
ed5057de chore: prepare for 3.0.0 (final) (#5746)
4dc4aca2 [skip ci] Explain: conduit feature only covers from-Python-to-C++ conversions (#5740)
03d8f487 docs: fix typo in multiple_interpreters (#5738)
ea3e33e4 chore: prepare for 3.0.0rc4 (#5736)
bdc56d9e chore(deps): bump urllib3 from 2.2.2 to 2.5.0 in /docs (#5735)
cf3d1a75 feat: numpy scalars (#5726)
c60c1499 tests: handle 3.12 and 3.13 implementations and 3.14.0b3+ (#5732)
be507b72 ci: check latest on 3.13 (#5734)
f2c0ab83 Fix TSan warning in sub-interpreter test (#5729)
ad9180c1 fix: android CMake support (#5733)
86e82ddb Add support for `shared_ptr<const T>` in `py::init()` with `smart_holder` (#5731)
365d41a4 Eliminate cross-DSO RTTI reliance in `smart_holder` functionality (for platforms like macOS). (#5728)
e2f86af2 docs: add documentation entry for warnings (#5356)
f3bb0073 better test for const only smart ptr (#5727)
d218b160 ci: avoid 3.13.4 on Windows (#5725)
f8640da4 chore(deps): bump pypa/cibuildwheel from 3.0.0rc2 to 3.0 in the actions group (#5721)
7ed76e2d fix: add support for const-only smart pointers (#5718)
513d1f96 ci: check iOS update (#5717)
ff0e381d chore(deps): bump requests from 2.32.3 to 2.32.4 in /docs (#5720)
6c5d25aa test: run pytest under Python devmode (#5715)
5d32ed76 docs: prepare for RC 3 (#5713)
c786d34f fix: handle null `py::handle` and add tests for `py::scoped_critical_section` (#5706)
c7026d0d fix!: modify the internals pointer-to-pointer implementation to not use `thread_local` (#5709)
b1948914 fix: expose required symbol using clang (#5700)
d4d2ec1a ci: avoid brownout (and removal) of windows-2019 (#5712)
0985052a chore(deps): update pre-commit hooks (#5711)
9295c4a5 fix: follow rest of pybind11 closer with PYBIND11_HAS_SUBINTERPRETER_SUPPORT (#5710)
7da1d53d ci: test on iOS (#5705)
c2b32b1e fix(types): type hints from future python versions (#5693)
21c9dd11 ci: drop main tests on main
e4873e8f fix: allow subinterp support to be disabled
33fb5333 fix: C++20 in Windows
bc557a9b tests: add PYBIND11_TEST_SMART_HOLDER to tests
29979761 ci: rename cibw workflow file
7e672ca1 tests: support C++23 in tests
ef2ad42d ci: list all jobs explicitly
5dff3354 ci: reduce runs on draft PRs
df595b16 docs: changelog update for 3.0.0rc2
a1b19722 chore: prepare for 3.0.0rc2 (#5698)
a18b1bc4 tests: always disable on the nogil build (#5701)
57e27c1f tests: skip some flaky gil tests on nogil (#5699)
1c10d5e9 fix: prepare for 3.14 beta 2 (#5697)
6e3e8515 fix(cmake): regression in include gaurd (#5691)
03b4a9e5 fix: don't destruct module objects in atexit (#5688)
1dd85ef4 chore: bump maximum clang tested to 20 (#5692)
9d066265 docs: more warnings about locking and the GIL (#5689)
d7769de5 feat: scoped_critical_section (#5684)
e3883dd5 refactor: use CPython macros to construct `PYBIND11_VERSION_HEX` (#5683)
e4622cbd chore(cmake): add compile commands to preset (#5685)
98bd78f0 chore: rename macro `PYBIND11_SUBINTERPRETER_SUPPORT` -> `PYBIND11_HAS_SUBINTERPRETER_SUPPORT` to meet naming convention (#5682)
8d503e30 docs: update contributing/release guide a little (#5681)
2624d4a3 tests: expect free-threaded import warnings (#5680)
fc888f75 docs: prepare for 3.0.0rc1 (#5679)
e8f16e2f docs: remove setup.py mentions, mention presets (#5677)
07d028f2 feat: Embeded sub-interpreters (#5666)
ec8b0508 chore: convert changelog to markdown (#5672)
e6256684 fix(cmake): error wasn't triggering (#5676)
7319402a chore: show preview on docs changes (#5673)
3867c5f5 ci: add nightlies for scientific-python (#5675)
9e6fe464 chore: consolidate common code in PYBIND11_MODULE and PYBIND11_EMBEDDED_MODULE (#5670)
67424358 fix(types): add typing and collections.abc module prefix (#5663)
4587d33c docs: prepare for v3.0.0rc1 (#5589)
cc86e8b2 fix(cmake): better --fresh support (#5668)
6bf25d1e feat: add semi-public API: `pybind11::detail::is_holder_constructed` (#5669)
9afc9c4f feat: change PYBIND11_EMBEDDED_MODULE to multiphase init (#5665)
094343c7 fix: support Python 3.14 (#5646)
1107c093 docs: Add documentation for mod_gil_not_used and multiple_interpreters (#5659)
af231a60 chore: use scikit-build-core for the build (#5598)
6aa3b335 fix(types): Buffer type hint (#5662)
95e8f89b Support for sub-interpreters (#5564)
05a6a03e chore(cmake): add CMake presets (#5655)
853bafa0 fix: use original dict (#5658)
c5dc6f9f ci: bump default versions (#5657)
74b52427 feat: add `py::potentially_slicing_weak_ptr(handle)` function (#5624)
ce42c4df fix(cmake): avoid message if FINDPYTHON NEW (#5656)
d7d782c5 fix(regression): support embedded submodule (#5650)
9a191c24 Fix typos for FindPython compact mode: `Python_LIRAR{Y,IES}` -> `Python_LIBRAR{Y,IES}` (#5653)
c125cc78 Collect all `#define PYBIND11_HAS_...` in pybind11/detail/common.h (#5647)
002c05b1 fix: handle MSVC warning C4866: compiler may not enforce left-to-right evaluation order (#5641)
a265a4cf fix: define _DEBUG macro to 1 on redefinition (#5639)
099583c5 chore(deps): update pre-commit hooks (#5642)
c630e22c Add `static_assert`s to enforce that `py::smart_holder` is combined with `py::trampoline_self_life_support` (#5633)
c7f3460f Fix gcc compiler warnings (#5523)
d3fee429 chore(deps): bump astral-sh/setup-uv from 5 to 6 in the actions group (#5632)
bc4a66df fix: provide useful behavior of default `py::slice` (#5620)
5c498583 fix: upgrade 20.04 runners to 22.04 (fix for ICC, NVHPC) (#5621)
223e2e9d Fix missing pythonic type hints for native_enum (#5619)
3c586340 Add class doc string to native_enum (#5617)
b3bb31ca ci: work on speeding up further (#5613)
cbcc2385 Factor out pybind11/gil_simple.h (#5614)
ee04df0d Updated STL casters and py::buffer to use collections.abc (#5566)
f3c19138 Remove obsolete `"E501"` line in pyproject.toml (#5539)
662a88cb ci: speed up ci a bit by thinning out matrix (#5602)
a2951abb chore(deps): update pre-commit hooks (#5605)
d25e91fb fix(cmake): warning about missing file (#5612)
31e52b2c Add holder caster traits tests in test_smart_ptr.cpp,py (#5603)
e38beb96 [skip ci] Move "Pitfalls with raw pointers and shared ownership" section to a more prominent location. (#5611)
1bd1d1ce Change `PyCFunction` cast in function_record_pyobject.h to sidestep unhelpful compiler warnings. (#5608)
708ce4d9 Fix GraalPy version parsing on dev builds (#5609)
a28ea8cc Eliminate `pybindit` namespace (#5607)
6d1f28fe feat: remove make_simple_namespace (#5597)
73ad3099 fix: fully deprecate get_type_of (deprecated in 2.6 but no warning (#5596)
b70b8eb3 chore: require pytest 6 consistently (#5599)
fdab860a feat: drop PYBIND11_NUMPY_1_ONLY (#5595)
55b1357d chore: update nox and test deps (#5594)
8ef10a0e ci: update to GraalPy 24.2 and mention in README (#5586)
d27fdaa2 chore: update for CMake 4.0 (#5593)
e03ec306 Squashed function_record_std_launder/manuscript — 57b9a0af815d19b236b74be06a172bc5c9956618 — 2025-03-30 20:14:21 -0700 (#5592)
a34fcdc4 [ci skip] Response to question by gh-henryiii (pybind/pybind11#5580 (comment)) (#5591)
8726ed22 Fix build failure when `shared_ptr<T>` points to `private` `std::enable_shared_from_this` base (#5590)
e7e5d6e5 Make wrapped C++ functions pickleable (#5580)
8f00d1ee fix: set __file__ on submodules (#5584)
c1cd022f tests: Add test for `boost::histogram::func_transform` situation. (#5582)
f365314e Enable Conversions Between Native Python Enum Types and C++ Enums (#5555)
48eb5ad9 Remove PyPy 3.8 and 3.9 testing. Make a pass through the entire repo to remove obviously obsolete workarounds for PyPy < 3.10. (#5578)
566894d5 Fix null pointer dereference in `attr_with_type_hint` (#5576)
974eba77 Change PYBIND11_MODULE to use multi-phase init (PEP 489) (#5574)
97022f83 TEST: test passes on PyPy macOS (#5569)
6412615e Update NDArray[object] to be NDArray[numpy.object_] (#5571)
bb504dd8 fix: FindPython by default logic error (#5561)
655c60d8 docs: fix incorrect name (PYBIND11_NEWPYTHON) (#5570)
9f08625d Updated py::capsule type hint to use new types.CapsuleType (#5567)
dfe7e65b feat(types): Use `typing.SupportsInt` and `typing.SupportsFloat` and fix other typing based bugs. (#5540)
16b5abd4 chore: bump catch download to 2.13.10 (#5568)
d28904f1 feat: FindPython by default (#5553)
06e8ee2e chore(deps): bump actions/attest-build-provenance in the actions group (#5556)
d422fda1 chore(deps): bump jinja2 from 3.1.5 to 3.1.6 in /docs (#5554)
e24e300c chore(deps): update pre-commit hooks (#5547)
a822be20 chore(deps): bump the actions group with 2 updates (#5546)
ded70fe6 Add pkgconf-pypi entrypoint (#5552)
79be5c83 test: Explicitly mark char as signed in dtype tests (#5545)
2943a27a squash-merge smart_holder branch into master (#5542)
d8565ac7 Sync `Py_TPFLAGS_MANAGED_DICT` for PyPy3.11 across the codebase (#5537)
09b9f44a add recently released pypy3.11 (#5534)
73825f35 Improved reference_internal documentation (#5528)
b7c33009 Start pybind11v3: Remove all code for `PYBIND11_INTERNALS_VERSION`s `4` and `5` (#5530)
24152422 feat(types) Numpy.typing.NDArray (#5212)
34a118fd Add `release_gil_before_calling_cpp_dtor` annotation for `class_` (#5522)
c316cf36 Make PYBIND11_INTERNALS_VERSION 6 the default on all platforms. (#5512)
31d7c870 Remove some maybe-uninitialized warnings (#5516)
d2e7e8c6 chore(deps): update pre-commit hooks (#5513)
ab44b307 Skip transient crash on GraalPy on OS X (#5514)
8862cd4e chore(deps): bump seanmiddleditch/gha-setup-ninja in the actions group (#5509)
fe87568f PyPy 3.11 does not implement Py_TPFLAGS_MANAGED_DICT (#5508)
82845c3b chore(deps): bump actions/attest-build-provenance in the actions group (#5503)
924261e8 chore(cmake): Add an author warning that auto-calculated `PYTHON_MODULE_EXTENSION` may not respect `SETUPTOOLS_EXT_SUFFIX` during cross-compilation (#5495)
c19c291b feat: --extension-suffix for pybind11 command (#5360)
167bb5f2 fix(cmake): don't strip with BUILD_TYPE None (#5392)
1b7aa0bb feat: rework of arg/return type hints to support .noconvert() (#5486)
15d9dae1 Fix data race when using shared variables (free threading) (#5494)
945e251a chore(deps): bump jinja2 from 3.1.4 to 3.1.5 in /docs (#5490)
a09cf618 chore(deps): update pre-commit hooks (#5488)
c5ed9d4b Fix module type hint (#5469)
cf020a1d feat(types) Allow setting types for attributes (#5460)
5b503f7e chore(deps): bump actions/attest-build-provenance in the actions group (#5468)
3e419485 `PYBIND11_PLATFORM_ABI_ID` Modernization Continued (platforms other than MSVC) (#5439)
741d86f2 Drop Clang dev CI job (#5464)
3ebdc503 chore(deps): bump actions/attest-build-provenance in the actions group (#5461)
b17555f3 chore(deps): update pre-commit hooks (#5459)
1d09fc83 Option for arg/return type hints and correct typing for std::filesystem::path (#5450)
a6d1ff24 fix: make PYBIND11_WARNING_POP actually pop clang diagnostics (#5448)
e7c9b907 chore(deps): bump pypa/cibuildwheel in the actions group (#5451)
83b92ceb Try to fix reentrant write transient failures in tests (#5447)
330aae51 Remove mingw-w64-i686-python-numpy from mingw32 build (it does not seem to exist anymore). (#5445)
f41dae31 Add dtype::normalized_num and dtype::num_of (#5429)
b9fb3168 Add support for array_t<handle> and array_t<object> (#5427)
08095d9c Run pytest in verbose mode (#5443)
0ed20f26 chore(deps): bump actions/attest-build-provenance in the actions group (#5440)
7f94f24d feat(typing): allow annotate methods with `pos_only` when only have the `self` argument (#5403)
6d98d4d8 Add type hints for args and kwargs (#5357)
a90e2af8 Factor out pybind11/conduit/pybind11_platform_abi_id.h (#5375)
ec9c2681 Fix MSVC MT/MD incompatibility in PYBIND11_BUILD_ABI (#4953)
037310ea Use std::unique_ptr in pybind11_getbuffer (#5435)
ce2f0055 Fixed data race in all_type_info in free-threading mode (#5419)
f46f5be4 Fix incorrect link syntax in upgrade guide (#5434)
5c07feef chore(deps): update pre-commit hooks (#5432)
bc041de0 Fix buffer protocol implementation (#5407)
75e48c5f Skip transient tests on GraalPy (#5422)
f7e14e98 Address regression introduced in #5381 (#5396)
077e49fc Export libc++ exceptions (#5390)
f2907651 Fix #5399: iterator increment operator does not skip first item (#5400)
af67e873 docs/advanced A document about deadlock potential with C++ statics (#5394)
56e69a20 Print key in KeyError in map.__getitem__/__delitem__ (#5397)
c4a05f93 Add support for GraalPy (#5380)
7e418f49 Allow subclasses of py::args and py::kwargs (#5381)
1f8b4a7f fix(cmake): `NO_EXTRAS` in `pybind11_add_module` function partially working (#5378)
ad9fd39e chore(deps): bump pypa/cibuildwheel in the actions group (#5376)
1d9483ff Added exception translator specific mutex used with try_translate_exceptions (#5362)
a7910be6 Add warn disable for GGC 12 bound checking error (#5355)
0cf3a0f7 ci: PyPI attestations (#5374)
5b7c0b04 docs: update changelog for 2.13.6 (#5372)
ef5a9560 Enable type-safe interoperability between different independent Python/C++ bindings systems. (#5296)
5efc7439 chore(deps): bump the actions group with 2 updates (#5361)
8a801bdc chore(deps): update pre-commit hooks (#5350)
aeda49ed Properly translate C++ exception to Python exception when creating Python buffer from wrapped object (#5324)
66c3774a Warnings wrappers to use from C++ (#5291)
65f4266c Add `while True` & `top` method to FAQ. (#5340)
3fb16ad1 fix: using `__cpp_nontype_template_args` instead of `__cpp_nontype_template_parameter_class` (#5330)
e8f595bb chore(deps): bump actions/attest-build-provenance in the actions group (#5335)
c2291e59 docs: prepare for 2.13.5 (#5327)
efa2b20d docs: clarify requirements for including pybind11 (#5326)
9966ad40 fix: allow -Wpedantic in C++20 mode (#5322)
2baf9d68 fix: `<ranges>` support for `py::tuple` and `py::list` (#5314)
7d85baa6 fix: never use `..` in a header include (#5321)
a1d00916 Backport of google/pybind11clif#30034 (#5305)
bd5951b6 docs: prepare for 2.13.4 (#5312)
28dbce41 feat: require CMake 3.15+ (#5304)
d893f972 fix: escape paths with spaces in pybind11-config (#4874)
fc97cc41 Revert "fix: quote paths from pybind11-config (#5302)" (#5309)
01169061 chore: remove repetitive words (#5308)
0d44d720 Make stl.h `list|set|map_caster` more user friendly. (#4686)
4a06eca5 docs: prepare for 2.13.3
8d9f4d50 fix: quote paths from pybind11-config (#5302)
1fe92c7b fix: emscripten cmake issue (#5301)
40f2c786 docs: prepare for 2.13.2 (#5299)
8d90b83b chore(deps): bump actions/attest-build-provenance in the actions group (#5297)
fc59f4e6 fix(cmake): add required emscripten flags (#5298)
89879448 Add `type_caster_std_function_specializations` feature. (#4597)
20551ab3 chore(deps): update pre-commit hooks (#5288)
84510538 chore(deps): bump the actions group with 2 updates (#5287)
916778df fix: typo in documentation (#5284)
72330728 feat: remove Python 3.7 support (#5191)
2e260b06 clang-tidy upgrade (to version 18) (#5272)
8e7307f0 docs: remove outdated known limitation. (#5263)
6d4805ce Small cleanup/refactoring in support of PR #5213 (#5251)
a582ca8a tests: run on pyodide (#4745)
dbf848af docs: extend `PYBIND11_MODULE` documentation, mention `mod_gil_not_used` (#5250)
43de8014 fix: make gil_safe_call_once thread-safe in free-threaded CPython (#5246)
ccefee4c chore(deps): bump actions/attest-build-provenance in the actions group (#5243)
50acb81b chore(deps): bump certifi from 2024.2.2 to 2024.7.4 in /docs (#5226)
bb05e081 Use PyMutex instead of std::mutex in free-threaded build. (#5219)
b21b0490 chore(deps): update pre-commit hooks (#5220)
d78446cc chore(deps): bump actions/attest-build-provenance in the actions group (#5216)
d805e996 feat(types) Adds special Case for empty C++ tuple type annotation (#5214)
51c2aa16 Fixed a compilation error with gcc 14 (#5208)
08f946a4 fix: add guard for GCC <10.3 on C++20 (#5205)
e0f9e774 fix(cmake): remove extra = in flto assignment (#5207)
57287b57 docs: prepare for 2.13.1 (#5203)
4bd538a4 feat(types): add support for Typing.Callable Special Case (#5202)
2e35470c fix: use manual padding of instance_map_shard (#5200)
895e6572 chore: back to work
REVERT: a2e59f0e chore: bump to 2.13.6
REVERT: e445ca2b ci: PyPI attestations (#5374)
REVERT: 7b67d8e9 docs: update changelog for 2.13.6 (#5372)
REVERT: a5fcc560 Enable type-safe interoperability between different independent Python/C++ bindings systems. (#5296)
REVERT: 54ab4249 chore(deps): bump the actions group with 2 updates (#5361)
REVERT: 36ee4674 chore(deps): update pre-commit hooks (#5350)
REVERT: 9e6a67d5 Properly translate C++ exception to Python exception when creating Python buffer from wrapped object (#5324)
REVERT: 570d323b Add `while True` & `top` method to FAQ. (#5340)
REVERT: b9f85757 fix: using `__cpp_nontype_template_args` instead of `__cpp_nontype_template_parameter_class` (#5330)
REVERT: 0a96ff7e chore(deps): bump actions/attest-build-provenance in the actions group (#5335)
REVERT: 7c33cdc2 chore: prepare for 2.13.5
REVERT: b3f5f2e7 docs: prepare for 2.13.5 (#5327)
REVERT: a4f6627d docs: clarify requirements for including pybind11 (#5326)
REVERT: 0d21cadc fix: allow -Wpedantic in C++20 mode (#5322)
REVERT: ff3ca786 fix: `<ranges>` support for `py::tuple` and `py::list` (#5314)
REVERT: b0050f30 fix: never use `..` in a header include (#5321)
REVERT: c6239a8a chore: version 2.13.4
REVERT: 63b0d146 docs: prepare for 2.13.4 (#5312)
REVERT: 973a16e9 fix: escape paths with spaces in pybind11-config (#4874)
REVERT: 75c11769 Revert "fix: quote paths from pybind11-config (#5302)" (#5309)
REVERT: 6685547e chore: remove repetitive words (#5308)
REVERT: bd676436 chore: prepare for 2.13.3
REVERT: 7662af69 docs: prepare for 2.13.3
REVERT: 45eaee91 fix: quote paths from pybind11-config (#5302)
REVERT: 835139f5 fix: emscripten cmake issue (#5301)
REVERT: 07f30430 chore: prepare for 2.13.2
REVERT: 6d5704cd docs: prepare for 2.13.2 (#5299)
REVERT: 6ee574fa chore(deps): bump actions/attest-build-provenance in the actions group (#5297)
REVERT: d8fcfe34 fix(cmake): add required emscripten flags (#5298)
REVERT: 78e26321 Add `type_caster_std_function_specializations` feature. (#4597)
REVERT: 44d0d9a4 chore(deps): update pre-commit hooks (#5288)
REVERT: fe808a01 chore(deps): bump the actions group with 2 updates (#5287)
REVERT: f9ae715d fix: typo in documentation (#5284)
REVERT: 042c3cfd clang-tidy upgrade (to version 18) (#5272)
REVERT: 667563dd docs: remove outdated known limitation. (#5263)
REVERT: 129934ad Small cleanup/refactoring in support of PR #5213 (#5251)
REVERT: f50830ea tests: run on pyodide (#4745)
REVERT: b4307453 docs: extend `PYBIND11_MODULE` documentation, mention `mod_gil_not_used` (#5250)
REVERT: f3a6d414 fix: make gil_safe_call_once thread-safe in free-threaded CPython (#5246)
REVERT: d699e99c chore(deps): bump actions/attest-build-provenance in the actions group (#5243)
REVERT: 4b2f7cd6 chore(deps): bump certifi from 2024.2.2 to 2024.7.4 in /docs (#5226)
REVERT: 8443d084 Use PyMutex instead of std::mutex in free-threaded build. (#5219)
REVERT: 41726b64 chore(deps): update pre-commit hooks (#5220)
REVERT: ea10a69d chore(deps): bump actions/attest-build-provenance in the actions group (#5216)
REVERT: a4dd41a1 feat(types) Adds special Case for empty C++ tuple type annotation (#5214)
REVERT: 639ca6a7 Fixed a compilation error with gcc 14 (#5208)
REVERT: 65afa13e fix: add guard for GCC <10.3 on C++20 (#5205)
REVERT: 3074608e fix(cmake): remove extra = in flto assignment (#5207)
REVERT: 941f45bc chore: prepare for 2.13.1
REVERT: 63020d33 docs: prepare for 2.13.1 (#5203)
REVERT: dd0e4a0b feat(types): add support for Typing.Callable Special Case (#5202)
REVERT: 3b47b464 fix: use manual padding of instance_map_shard (#5200)
git-subtree-dir: pybind11
git-subtree-split: f5fbe867d2d26e4a0a9177a51f6e568868ad3dc8
Description
This PR requires a
PYBIND11_INTERNALS_VERSIONbump (from8to9).Closes #1261
Full ChatGPT assessment
ChatGPT summary (with a couple edits)
Motivation and Importance
Before this PR, pybind11-bound functions are not pickleable, which creates significant friction for Python users who rely on
pickleand related serialization mechanisms for common tasks such as:Parallel and Concurrent Processing
Python's
multiprocessingmodule depends on pickling to distribute tasks across worker processes. Since pybind11 functions cannot be pickled, users are forced to implement fragile and inefficient workarounds (e.g., passing function names as strings or relying on global state). This limits the usability of pybind11 bindings in parallel and distributed computing frameworks such asmultiprocessing,concurrent.futures, andjoblib.Caching and Memoization
Libraries like
joblib,functools.lru_cache, anddiskcacherely on pickle to store function results and enable fast recomputation. The inability to pickle pybind11-bound functions makes it difficult to integrate C++ extensions with these widely used caching mechanisms.Distributed Computing and Cloud Environments
Modern data processing frameworks such as Dask and Ray depend on pickling to serialize tasks and distribute them across clusters. The current pybind11 limitation prevents users from leveraging C++-based functions in these environments without cumbersome workarounds.
Testing and Mocking
Serialization of test artifacts and fixtures is a common practice in Python testing frameworks like
pytest. Non-pickleable pybind11 functions make it difficult to persist test state and mock functions consistently.General Python Compatibility
Python developers expect functions and objects to be pickleable as a baseline feature. The fact that pybind11-bound functions break this expectation creates unnecessary obstacles.
This PR introduces a solution to enable pickling of pybind11-bound functions in a way that is efficient, reliable, and consistent with Python's native serialization semantics. The solution has been tested extensively in large-scale production environments and is designed to maintain backward compatibility and minimal performance overhead. By addressing this long-standing limitation, this PR will significantly improve the usability of pybind11 for parallel processing, distributed systems, caching, and testing — making it easier for developers to integrate C++ extensions into modern Python workflows.
Notes:
repr()for pybind11 functions is made much more informative and truthful:Before this PR:
(
PyCapsuleobjects do not have methods.)With this PR:
-DCMAKE_BUILD_TYPE=MinSizeRel):Before this PR:
With this PR:
Factor: 5257248 / 5252640 = 1.0008772731426483
Comparison with stock Python and other systems for Python-C++ bindings:
Python functions (built-in & native) are pickleable.
SWIG functions are pickleable.
PyCLIF-C-API functions are pickleable. (Because they are implemented exactly like stock Python build-in functions.)
Boost.Python functions are not pickleable.
nanobind functions are not pickleable.
Everything below are very technical details:
Why is
not pickleable before this PR?
The reason is that the
simple_callableis implemented as shown byrepr(simple_callable):To Python it appears to be a bound function of the type
PyCapsule. (The reasons for this choice of implementation are deep and omitted here.)pickle.dumps(simple_callable)is capable of pickling built-in methods. It does that in two steps:produces this tuple (see
__reduce_ex__documentation):pickle then recurses, attempting to serialize
<built-in function getattr>(no problem),<capsule object NULL at 0x...>(problem), and'simple_callable'(no problem).The attempt to serialize
<capsule object NULL at 0x...>fails with this exception (copy-pasted from pytest output):While this is not the desired behavior when pickling
simple_callable, there are very good reasons in general thatPyCapsuleobjects are not pickleable. See, for example, here:The solution implemented in this PR is to
replace the
PyCapsulewith a custom built-in type, wrapping thepybind11::detail::function_recordC++ type the good-old way, with manually written bindings (implemented in include/pybind11/detail/function_record_pyobject.h),which then makes it possible to provide a
__reduce_ex__implementation to achieve the desired behavior.The new
repr(simple_callable)is:Side note: The Python type name is versioned to ensure ABI compatibility, but to maximize compatibility it is independent of
PYBIND11_INTERNALS_VERSION.simple_callable.__reduce_ex__(0)now produces:When pickle recurses to call
__reduce_ex__of the wrappedfunction_recordobject, the result is:Very simple! — Your author has to admit though that it took quite a while to figure this out. See the development history of google/pybind11clif#30099 for the many dead ends explored before this solution was discovered.
To explain how this works in the
pickle.load()step:eval("__import__('importlib').import_module('pybind11_tests.pickling')")produces a reference to the importedpybind11_tests.picklingmodule.getattr(imported_module, 'simple_callable')simply accesses what was just imported.Note that
__import__('pybind11_tests.pickling')only produces the top-level module (pybind11_testsin this case), as explained in theimportlib.import_module()documentation.This PR is a backport of google/pybind11clif#30099
Suggested changelog entry: