Mixing PyBind11 modules and Python modules

[r-barnes]

I am using PyBind11 to make a Python project.

My directory structure looks a like this:

./
  my_pkg/
    __init__.py
    func1.py
    func2.py

My C++ code looks like this:

int myfunc(){
  return 1;
}

PYBIND11_PLUGIN(cppmodule) {
  py::module m("cppmodule", "My cpp module");

  m.def("myfunc",&myfunc,"This does stuff");

  return m.ptr();
}

And my setup.py looks like this:

from setuptools import setup, Extension
import glob

ext_modules = [
  Extension(
    "cppmodule",
    glob.glob('src/*.cpp'),
    include_dirs       = ['lib/include', 'lib/pybind11/'],
    language           = 'c++',
    extra_compile_args = ['-std=c++17'],
    define_macros      = [('DOCTEST_CONFIG_DISABLE',None)]
  )
]

setup(name = 'bob',
  version      = '0.1',
  description  = 'A package about shrimp',
  url          = 'http://github.com/shrimp',
  author       = 'Bob',
  author_email = '',
  license      = 'MIT',
  ext_modules  = ext_modules
)

Now, if I runn

python setup.py install

everything compiles.

But here's the odd part, later, I can run import cppmodule but not import bob.

What I have not figured out how to do, but what I would like to do, is to have the C++ code incorporated into the bob module the same way func1 and func2 will be, so that I can type bob.myfunc() in Python.

How can I do this?

Put another way, my package contains some code that it makes sense to write in Python and other code that it makes sense to write in C++, and I'd like to include both in the same package

[dean0x7d]

A common naming/import pattern is the following:

Extension module: _mymodule.so
Python module or package: mymodule.py or mymodule/__init__.py

Where the __init__.py file contains:

from _mymodule import *

This way the main mymodule namespace has access to all of the function/classes from the extension module.

[r-barnes]

Thank you for your help!

That seems like a good pattern. I am now trying:

#setup.py
ext_modules = [
  Extension(
    "_mander",
    glob.glob('src/*.cpp') + glob.glob('lib/compactnesslib/*.cpp') + glob.glob('lib/compactnesslib/shapelib/*.c'),
    include_dirs = ['lib/compactnesslib', 'lib/pybind11/'],
    language = 'c++',
    extra_compile_args = ['-std=c++11'],
    define_macros = [('DOCTEST_CONFIG_DISABLE',None)]
  )
]

setup(name='mander',
  ext_modules  = ext_modules
)

and

#mycode.cpp
PYBIND11_PLUGIN(_mander) {
  py::module m("_mander", "Compactness calculator");
  return m.ptr()
}

Afterwards, I can use import _mander but not import mander.

To be clear, is it possible to include both the Python and C++ code in the same package, or are you suggesting, e.g., that I submit _mypackage and mypackage to pip?

[dean0x7d]

Both can be included in the same pip package/setup.py file. Python packages should be given to the packages argument of setup() which can be done alongside ext_modules.

[r-barnes]

That worked! I now have:

setuptools.setup(name='mander',
  packages     = setuptools.find_packages(),
  ext_modules  = ext_modules
)

And all is going well. Thanks so much :-)

[benmaier]

Using the method above, is there any way to include the docstrings from _mander in an auto-documentation generated with Sphinx? Let's say I have the package mander and in the __init__.py I use from _mander import *, is there any canonic way to include the docstrings from _mander?

[erickguan]

There is no way to point the docstring in _mander. Only if the user imports _mander and query that module.

[gagelarsen]

I know this issue is closed but i'm having an issue that might be related.

i have a package
_xmsconstraint

as part of that package i have the following module/class
_xmsconstraint.rectilinear.CoRectilinearGrid

in my __init__ as mentioned above i have:

from ._xmsconstraint import *

my base package is called
xmsconstraint

It will not allow me to import (it says it can't find the module xmsconstraint.rectilinear)

from xmsconstraint.rectilinear import CoRectilinearGrid

But i can do the following

import xmsconstraint
x = xmsconstraint.rectilinear.CoRectilinearGrid()

Any ideas why I can't import the class or what I could do to fix it?

[erickguan]

Basically, write everything out (what PyTorch did). The easiest way I found out is to come up with a new module name. Then you can also have docstring