Add py::module_local() attribute for module-local type bindings

This commit adds a `py::module_local` attribute that lets you confine a
registered type to the module (more technically, the shared object) in
which it is defined, by registering it with:

    py::class_<C>(m, "C", py::module_local())

This will allow the same C++ class `C` to be registered in different
modules with independent sets of class definitions.  On the Python side,
two such types will be completely distinct; on the C++ side, the C++
type resolves to a different Python type in each module.

This applies `py::module_local` automatically to `stl_bind.h` bindings
when the container value type looks like something global: i.e. when it
is a converting type (for example, when binding a `std::vector<int>`),
or when it is a registered type itself bound with `py::module_local`.
This should help resolve potential future conflicts (e.g. if two
completely unrelated modules both try to bind a `std::vector<int>`.
Users can override the automatic selection by adding a
`py::module_local()` or `py::module_local(false)`.

Note that this does mildly break backwards compatibility: bound stl
containers of basic types like `std::vector<int>` cannot be bound in one
module and returned in a different module.  (This can be re-enabled with
`py::module_local(false)` as described above, but with the potential for
eventual load conflicts).

Author: jagerman

docs/advanced/cast/stl.rst

@@ -150,10 +150,10 @@ the declaration
 before any binding code (e.g. invocations to ``class_::def()``, etc.). This
 macro must be specified at the top level (and outside of any namespaces), since
 it instantiates a partial template overload. If your binding code consists of
-multiple compilation units, it must be present in every file preceding any
-usage of ``std::vector<int>``. Opaque types must also have a corresponding
-``class_`` declaration to associate them with a name in Python, and to define a
-set of available operations, e.g.:
+multiple compilation units, it must be present in every file (typically via a
+common header) preceding any usage of ``std::vector<int>``. Opaque types must
+also have a corresponding ``class_`` declaration to associate them with a name
+in Python, and to define a set of available operations, e.g.:
 
 .. code-block:: cpp
 
@@ -167,6 +167,20 @@ set of available operations, e.g.:
         }, py::keep_alive<0, 1>()) /* Keep vector alive while iterator is used */
         // ....
 
+Please take a look at the :ref:`macro_notes` before using the
+``PYBIND11_MAKE_OPAQUE`` macro.
+
+.. seealso::
+
+    The file :file:`tests/test_opaque_types.cpp` contains a complete
+    example that demonstrates how to create and expose opaque types using
+    pybind11 in more detail.
+
+.. _stl_bind:
+
+Binding STL containers
+======================
+
 The ability to expose STL containers as native Python objects is a fairly
 common request, hence pybind11 also provides an optional header file named
 :file:`pybind11/stl_bind.h` that does exactly this. The mapped containers try
@@ -188,14 +202,34 @@ The following example showcases usage of :file:`pybind11/stl_bind.h`:
     py::bind_vector<std::vector<int>>(m, "VectorInt");
     py::bind_map<std::map<std::string, double>>(m, "MapStringDouble");
 
-Please take a look at the :ref:`macro_notes` before using the
-``PYBIND11_MAKE_OPAQUE`` macro.
+When binding STL containers pybind11 considers the types of the container's
+elements to decide whether the container should be confined to the local module
+(via the :ref:`module_local` feature).  If the container element types are
+anything other than already-bound custom types bound without
+``py::module_local()`` the container binding will have ``py::module_local()``
+applied.  This includes converting types such as numeric types, strings, Eigen
+types; and types that have not yet been bound at the time of the stl container
+binding.  This module-local binding is designed to avoid potential conflicts
+between module bindings (for example, from two separate modules each attempting
+to bind ``std::vector<int>`` as a python type).
+
+It is possible to override this behavior to force a definition to be either
+module-local or global.  To do so, you can pass the attributes
+``py::module_local()`` (to make the binding module-local) or
+``py::module_local(false)`` (to make the binding global) into the
+``py::bind_vector`` or ``py::bind_map`` arguments:
 
-.. seealso::
+.. code-block:: cpp
 
-    The file :file:`tests/test_opaque_types.cpp` contains a complete
-    example that demonstrates how to create and expose opaque types using
-    pybind11 in more detail.
+    py::bind_vector<std::vector<int>>(m, "VectorInt", py::module_local(false));
+
+Note, however, that such a global binding would make it impossible to load this
+module at the same time as any other pybind module that also attempts to bind
+the same container type (``std::vector<int>`` in the above example).
+
+See :ref:`module_local` for more details on module-local bindings.
+
+.. seealso::
 
     The file :file:`tests/test_stl_binders.cpp` shows how to use the
     convenience STL container wrappers.

docs/advanced/classes.rst

@@ -635,3 +635,139 @@ inheritance, which can lead to undefined behavior. In such cases, add the tag
 
 The tag is redundant and does not need to be specified when multiple base types
 are listed.
+
+.. _module_local:
+
+Module-local class bindings
+===========================
+
+When creating a binding for a class, pybind by default makes that binding
+"global" across modules.  What this means is that a type defined in one module
+can be passed to functions of other modules that expect the same C++ type.  For
+example, this allows the following:
+
+.. code-block:: cpp
+
+    // In the module1.cpp binding code for module1:
+    py::class_<Pet>(m, "Pet")
+        .def(py::init<std::string>());
+
+.. code-block:: cpp
+
+    // In the module2.cpp binding code for module2:
+    m.def("pet_name", [](Pet &p) { return p.name(); });
+
+.. code-block:: pycon
+
+    >>> from module1 import Pet
+    >>> from module2 import pet_name
+    >>> mypet = Pet("Kitty")
+    >>> pet_name(mypet)
+    'Kitty'
+
+When writing binding code for a library, this is usually desirable: this
+allows, for example, splitting up a complex library into multiple Python
+modules.
+
+In some cases, however, this can cause conflicts.  For example, suppose two
+unrelated modules make use of an external C++ library and each provide custom
+bindings for one of that library's classes.  This will result in an error when
+a Python program attempts to import both modules (directly or indirectly)
+because of conflicting definitions on the external type:
+
+.. code-block:: cpp
+
+    // dogs.cpp
+
+    // Binding for external library class:
+    py::class<pets::Pet>(m, "Pet")
+        .def("name", &pets::Pet::name);
+
+    // Binding for local extension class:
+    py::class<Dog, pets::Pet>(m, "Dog")
+        .def(py::init<std::string>());
+
+.. code-block:: cpp
+
+    // cats.cpp, in a completely separate project from the above dogs.cpp.
+
+    // Binding for external library class:
+    py::class<pets::Pet>(m, "Pet")
+        .def("get_name", &pets::Pet::name);
+
+    // Binding for local extending class:
+    py::class<Cat, pets::Pet>(m, "Cat")
+        .def(py::init<std::string>());
+
+.. code-block:: pycon
+
+    >>> import cats
+    >>> import dogs
+    Traceback (most recent call last):
+      File "<stdin>", line 1, in <module>
+    ImportError: generic_type: type "Pet" is already registered!
+
+To get around this, you can tell pybind11 to keep the external class binding
+localized to the module by passing the ``py::module_local()`` attribute into
+the ``py::class_`` constructor:
+
+.. code-block:: cpp
+
+    // Pet binding in dogs.cpp:
+    py::class<pets::Pet>(m, "Pet", py::module_local())
+        .def("name", &pets::Pet::name);
+
+.. code-block:: cpp
+
+    // Pet binding in cats.cpp:
+    py::class<pets::Pet>(m, "Pet", py::module_local())
+        .def("get_name", &pets::Pet::name);
+
+This makes the Python-side ``dogs.Pet`` and ``cats.Pet`` into distinct classes
+that can only be accepted as ``Pet`` arguments within those classes.  This
+avoids the conflict and allows both modules to be loaded.
+
+One limitation of this approach is that because ``py::module_local`` types are
+distinct on the Python side, it is not possible to pass such a module-local
+type as a C++ ``Pet``-taking function outside that module.  For example, if the
+above ``cats`` and ``dogs`` module are each extended with a function:
+
+.. code-block:: cpp
+
+    m.def("petname", [](pets::Pet &p) { return p.name(); });
+
+you will only be able to call the function with the local module's class:
+
+.. code-block:: pycon
+
+    >>> import cats, dogs  # No error because of the added py::module_local()
+    >>> mycat, mydog = cats.Cat("Fluffy"), dogs.Dog("Rover")
+    >>> (cats.petname(mycat), dogs.petname(mydog))
+    ('Fluffy', 'Rover')
+    >>> cats.petname(mydog)
+    Traceback (most recent call last):
+      File "<stdin>", line 1, in <module>
+    TypeError: petname(): incompatible function arguments. The following argument types are supported:
+        1. (arg0: cats.Pet) -> str
+
+    Invoked with: <dogs.Dog object at 0x123>
+
+.. note::
+
+    STL bindings (as provided via the optional :file:`pybind11/stl_bind.h`
+    header) apply ``py::module_local`` by default when the bound type might
+    conflict with other modules; see :ref:`stl_bind` for details.
+
+.. note::
+
+    The localization of the bound types is actually tied to the shared object
+    or binary generated by the compiler/linker.  For typical modules created
+    with ``PYBIND11_MODULE()``, this distinction is not significant.  It is
+    possible, however, when :ref:`embedding` to embed multiple modules in the
+    same binary (see :ref:`embedding_modules`).  In such a case, the
+    localization will apply across all embedded modules within the same binary.
+
+.. seealso::
+
+    The file :file:`tests/test_local_bindings.cpp` contains additional examples
+    that demonstrate how ``py::module_local()`` works.

docs/advanced/embedding.rst

@@ -1,3 +1,5 @@
+.. _embedding:
+
 Embedding the interpreter
 #########################
 
@@ -131,6 +133,7 @@ embedding the interpreter. This makes it easy to import local Python files:
     int n = result.cast<int>();
     assert(n == 3);
 
+.. _embedding_modules:
 
 Adding embedded modules
 =======================

include/pybind11/attr.h

@@ -64,6 +64,9 @@ struct metaclass {
     explicit metaclass(handle value) : value(value) { }
 };
 
+/// Annotation that marks a class as local to the module:
+struct module_local { const bool value; constexpr module_local(bool v = true) : value(v) { } };
+
 /// Annotation to mark enums as an arithmetic type
 struct arithmetic { };
 
@@ -196,7 +199,7 @@ struct function_record {
 /// Special data structure which (temporarily) holds metadata about a bound class
 struct type_record {
     PYBIND11_NOINLINE type_record()
-        : multiple_inheritance(false), dynamic_attr(false), buffer_protocol(false) { }
+        : multiple_inheritance(false), dynamic_attr(false), buffer_protocol(false), module_local(false) { }
 
     /// Handle to the parent scope
     handle scope;
@@ -243,6 +246,9 @@ struct type_record {
     /// Is the default (unique_ptr) holder type used?
     bool default_holder : 1;
 
+    /// Is the class definition local to the module shared object?
+    bool module_local : 1;
+
     PYBIND11_NOINLINE void add_base(const std::type_info &base, void *(*caster)(void *)) {
         auto base_info = detail::get_type_info(base, false);
         if (!base_info) {
@@ -408,6 +414,10 @@ struct process_attribute<metaclass> : process_attribute_default<metaclass> {
     static void init(const metaclass &m, type_record *r) { r->metaclass = m.value; }
 };
 
+template <>
+struct process_attribute<module_local> : process_attribute_default<module_local> {
+    static void init(const module_local &l, type_record *r) { r->module_local = l.value; }
+};
 
 /// Process an 'arithmetic' attribute for enums (does nothing here)
 template <>

include/pybind11/cast.h

@@ -58,6 +58,8 @@ struct type_info {
     bool simple_ancestors : 1;
     /* for base vs derived holder_type checks */
     bool default_holder : 1;
+    /* true if this is a type registered with py::module_local */
+    bool module_local : 1;
 };
 
 PYBIND11_UNSHARED_STATIC_LOCALS PYBIND11_NOINLINE inline internals *&get_internals_ptr() {
@@ -126,6 +128,12 @@ PYBIND11_NOINLINE inline internals &get_internals() {
     return *internals_ptr;
 }
 
+// Works like internals.registered_types_cpp, but for module-local registered types:
+PYBIND11_NOINLINE PYBIND11_UNSHARED_STATIC_LOCALS inline type_map<void *> &registered_local_types_cpp() {
+    static type_map<void *> locals{};
+    return locals;
+}
+
 /// A life support system for temporary objects created by `type_caster::load()`.
 /// Adding a patient will keep it alive up until the enclosing function returns.
 class loader_life_support {
@@ -213,7 +221,7 @@ PYBIND11_NOINLINE inline void all_type_info_populate(PyTypeObject *t, std::vecto
             // registered types
             if (i + 1 == check.size()) {
                 // When we're at the end, we can pop off the current element to avoid growing
-                // `check` when adding just one base (which is typical--.e. when there is no
+                // `check` when adding just one base (which is typical--i.e. when there is no
                 // multiple inheritance)
                 check.pop_back();
                 i--;
@@ -257,13 +265,18 @@ PYBIND11_NOINLINE inline detail::type_info* get_type_info(PyTypeObject *type) {
     return bases.front();
 }
 
-PYBIND11_NOINLINE inline detail::type_info *get_type_info(const std::type_info &tp,
+/// Return the type info for a given C++ type; on lookup failure can either throw or return nullptr.
+PYBIND11_NOINLINE inline detail::type_info *get_type_info(const std::type_index &tp,
                                                           bool throw_if_missing = false) {
+    std::type_index type_idx(tp);
     auto &types = get_internals().registered_types_cpp;
-
-    auto it = types.find(std::type_index(tp));
+    auto it = types.find(type_idx);
     if (it != types.end())
         return (detail::type_info *) it->second;
+    auto &locals = registered_local_types_cpp();
+    it = locals.find(type_idx);
+    if (it != locals.end())
+        return (detail::type_info *) it->second;
     if (throw_if_missing) {
         std::string tname = tp.name();
         detail::clean_type_id(tname);
@@ -719,10 +732,8 @@ class type_caster_generic {
     // with .second = nullptr.  (p.first = nullptr is not an error: it becomes None).
     PYBIND11_NOINLINE static std::pair<const void *, const type_info *> src_and_type(
             const void *src, const std::type_info &cast_type, const std::type_info *rtti_type = nullptr) {
-        auto &internals = get_internals();
-        auto it = internals.registered_types_cpp.find(std::type_index(cast_type));
-        if (it != internals.registered_types_cpp.end())
-            return {src, (const type_info *) it->second};
+        if (auto *tpi = get_type_info(cast_type))
+            return {src, const_cast<const type_info *>(tpi)};
 
         // Not found, set error:
         std::string tname = rtti_type ? rtti_type->name() : cast_type.name();
@@ -807,7 +818,6 @@ template <typename type> class type_caster_base : public type_caster_generic {
     template <typename T = itype, enable_if_t<std::is_polymorphic<T>::value, int> = 0>
     static std::pair<const void *, const type_info *> src_and_type(const itype *src) {
         const void *vsrc = src;
-        auto &internals = get_internals();
         auto &cast_type = typeid(itype);
         const std::type_info *instance_type = nullptr;
         if (vsrc) {
@@ -816,9 +826,8 @@ template <typename type> class type_caster_base : public type_caster_generic {
                 // This is a base pointer to a derived type; if it is a pybind11-registered type, we
                 // can get the correct derived pointer (which may be != base pointer) by a
                 // dynamic_cast to most derived type:
-                auto it = internals.registered_types_cpp.find(std::type_index(*instance_type));
-                if (it != internals.registered_types_cpp.end())
-                    return {dynamic_cast<const void *>(src), (const type_info *) it->second};
+                if (auto *tpi = get_type_info(*instance_type))
+                    return {dynamic_cast<const void *>(src), const_cast<const type_info *>(tpi)};
             }
         }
         // Otherwise we have either a nullptr, an `itype` pointer, or an unknown derived pointer, so