Errors in build output of source distribution

[MathMagique]

I just played a little with the test project you kindly provided. I am currently facing the challenge of adapting the setup.py file for my project turbodbc to reflect my recent move from Boost.Python to pyodbc11 (thanks for that as well).

This example here (as well as my project) features pybind11 in the install_requires section. This seems to be sufficient when installing from git with pip install /path/to/source.

Now consider the situation where I want to create a source distribution with python setup.py sdist for later upload to pypi. Subsequent installation with pip install python-example.0.0.1.tar.gz yields error messages (output edited for brevity):

Processing ./python_example/dist/python_example-0.0.1.tar.gz
Collecting pybind11>=1.7 (from python-example==0.0.1)
...
Building wheels for collected packages: python-example
  Running setup.py bdist_wheel for python-example ... error
...
  building 'python_example' extension
  Traceback (most recent call last):
...
    File "/tmp/pip-0aU382-build/setup.py", line 20, in __str__
      import pybind11
  ImportError: No module named pybind11

  ----------------------------------------
  Failed building wheel for python-example
  Running setup.py clean for python-example
Failed to build python-example
Installing collected packages: pybind11, python-example
  Running setup.py install for python-example ... done
Successfully installed pybind11-1.8.1 python-example-0.0.1

Apparently, when building from a source distribution, pip install tries to build a wheel first. While building the wheel, the install requirements seem to be ignored. Even though this stage fails, pip install continues with a regular setup.py install afterwards that succeeds.

I have tried to resolve the issue by adding setup_requires=['pybind11>=1.7.0'] to setup.py. Indeed, the error message changes: Instead of a Python ImportError while building the wheel, I observe a compiler error instead, also while building the wheel:

src/main.cpp:1:31: fatal error: pybind11/pybind11.h: No such file or directory
  compilation terminated.
  error: command 'g++-5' failed with exit status 1

  ----------------------------------------
  Failed building wheel for python-example

Again, the overall installation completes afterwards.

It would be great if you had some more ideas or advice. In the interest of a clean build output, I would otherwise consider packing the pybind header files (and License, of course) in my source distribution and remove the dependency to the pybind package.

[SylvainCorlay]

Thank you for the detailed description. I will look into this.

In any case, I think that it would be preferable to avoid distributing pybind11 as part of your source distribution, especially if your package exposes headers. Indeed, this may result in conflicts when another package includes both pybind11 and your headers.... Let's try to find a solution instead.

This would also make packaging for other package managers (conda, apt-get) easier.

[SylvainCorlay]

Quick remark: If you also provide a wheel on pypi aside your sdist, this should fix the issue.

[SylvainCorlay]

Is there a branch of pyodbc11 that I can check out to reproduce?

[MathMagique]

Hi! Providing a wheel is difficult at the moment because right now I am targetting Linux only, and manylinux builds plague me with a very old (and very buggy) odbc library.

I would also prefer not to distribute pybind11 headers together with turbodbc. As for reproducing, checking out turbodbc is not necessary (and not easily possible given the still broken build on the branch), since the very same issue is present with this project, i.e., pybind11/python_example.

[dean0x7d]

The issue is also visible in the Travis logs for this repo. I guess this comes down to the fact that PyPI (unlike conda) does not differentiate between build-time and run-time requirements (related: pypa/pip#2381). The packages listed under install_requires are not guaranteed to be available for the build_ext stage. The process succeeds overall because build_ext is forced to run twice:

1. pip looks for requirements and downloads pybind11.
2. It runs bdist_wheel which calls build_ext. This fails because build_ext is unaware of the just-downloaded pybind11.
3. Now it runs install which again calls build_ext. This time it can see pybind11 and everything is fine.

[alexlenail]

Hi all,
What's the status of this bug? Would someone please post a workaround?

[wjakob]

I don't think it's clear how it could be fixed, the issue is potentially with PIP itself. Naturally you're free to investigate yourself and post a PR.

[SylvainCorlay]

There might be a means to fix this by overriding bdist_wheel and replace it with a new distutils command that performs some action before running the actual bdist_wheel.

I have had to overload standard distutils commands in the context of jupyter. I will look at this.

[lukeolson]

Adding

setup_requires=['pybind11'],

to setup() seems to fix the issue (this is similar to the numpy needs in other packages).

[marscher]

@lukeolson how did you obtain the include path then? I'm asking because when dragging in pybind11 via setup_requries, it will not be properly installed, eg the headers will not be unpacked. The headers are contained in an temporary unpacked egg, in the .eggs subdirectory next to your setup.py.

[marscher]

I came up with the following solution to make it work with direct invocation of setup.py and setup_requires=['pybind11']

class get_pybind_include(object):
    """Helper class to determine the pybind11 include path

    The purpose of this class is to postpone importing pybind11
    until it is actually installed, so that the ``get_include()``
    method can be invoked. """

    def __init__(self, user=False):
        self.user = user

    def search_pybind11_headers(self):
        import pybind11

        def recommended():
            return pybind11.get_include(self.user)

        def setuptools_temp_egg():
            # If users of setuptools drag in pybind11 only as a setup_require(ment), the pkg will be placed
            # temporarily into .eggs, but we can not use the headers directly. So we have to
            # link non-installed header files to correct subdirectory, so they can be used during compilation
            found = False
            for p in pybind11.__path__:
                if '.egg' in p:
                    found = True
            if not found:
                return ''

            header_src = os.path.abspath(os.path.join(pybind11.__path__[0], '..'))
            hdrs = []

            for _, _, filenames in os.walk(header_src):
                hdrs += [f for f in filenames if f.endswith('.h')]
            for h in sorted(hdrs):
                if 'detail' in h:
                    sub = 'detail'
                else:
                    sub = ''
                dest = os.path.join(pybind11.__path__[0], sub, os.path.basename(h))
                try:
                    os.link(h, dest)
                except OSError:
                    pass
            return header_src

        methods = (recommended(),
                   setuptools_temp_egg(),
                   )
        for m in methods:
            if os.path.exists(os.path.join(m, 'pybind11', 'pybind11.h')):
                return m
        return ''

    def __str__(self):
        result = self.search_pybind11_headers()
        if not result:
            raise RuntimeError()
        return result