Skip to content

Building osqp 1.x

Jump to bottom
Vineet Bansal edited this page Aug 13, 2024 · 7 revisions

The following build instructions illustrate how you can use the osqp git source tree to build osqp wheels that support different algebra backends.

The procedure has been tested out on Linux. The process should be identical on MacOS and Windows, though of course you will likely not be able to to build osqp-cuda wheels on MacOS, due to missing cuda-toolkit on Mac platforms.

Create a dedicated conda environment to build osqp. The osqp-python CI uses several python versions, but here we'll use python 3.9. Checkout and cd into the code.

conda create --name osqp-python-build python=3.9 pip
conda activate osqp-python-build

git clone https://github.com/osqp/osqp-python.git
cd osqp-python

Build wheels for builtin backend

python -m pip wheel . --no-deps --wheel-dir dist

The resulting wheel is available in the dist folder, and can be installed using pip install dist/osqp-1.*.whl

dist
├── osqp-1.0.0b2.dev68+gcaa6446-cp39-cp39-linux_x86_64.whl

The default wheel we built provides us with the osqp module, and we can check that we have the built-in algebra available:

python -c "from osqp import algebras_available; print(algebras_available())"
['builtin']

Build wheels for mkl backend

The process assumes that the mkl development library has been installed, and environment variables set so that the build process can discover these libraries (in particular, you will likely need to set MKL_ROOT, which on my machine is set to opt/intel/oneapi/mkl/latest). When working on a cluster, you may want to look for a module that populates the necessary environment variables (on our clusters, we do a module load intel-mkl/2024.0, for example).

If you're using a conda environment to build osqp, one way to get MKL is to try (osqp uses this in its CI):

  • conda install -c https://software.repos.intel.com/python/conda/ mkl-devel

On Windows:

  • conda install -c https://software.repos.intel.com/python/conda/ dpcpp_impl_win-64

However you decide to get MKL for your platform, the next step is to build the wrappers for the mkl backend:

python -m pip wheel backend/mkl --no-deps --wheel-dir dist

The resulting wheel is available in the dist folder, and can be installed using pip install dist/osqp_mkl-1.*.whl

dist
├── osqp-1.0.0b2.dev68+gcaa6446-cp39-cp39-linux_x86_64.whl
└── osqp_mkl-1.0.0b2.dev68+gcaa6446-cp39-cp39-linux_x86_64.whl

The mkl wheel we built provides us with the osqp_mkl module. When we try to do this:

10:45 $ python -c "import osqp_mkl"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: libmkl_rt.so.2: cannot open shared object file: No such file or directory

This is because we need the mkl libraries available to python at runtime to be able to use osqp-mkl. There are many ways for users to do this, so we don't enforce an mkl dependency in pip to install the osqp_mkl wheels. An easy way to do this in conda would be conda install anaconda::mkl, for example.

Note that while its possible to do an import osqp_mkl in python, we'll never import that module directly in our code, and just use import osqp. We can check that we have the mkl algebra in osqp available:

python -c "from osqp import algebras_available; print(algebras_available())"
['mkl', 'builtin']

Note that mkl appears before builtin, and will be the preferred backend for all osqp operations. We can verify this by running:

python -c "from osqp import default_algebra; print(default_algebra())"
mkl

This behavior can be overridden by setting the OSQP_ALGEBRA_BACKEND environment variable (which can take the values builtin, mkl, or cuda).

OSQP_ALGEBRA_BACKEND=builtin python -c "from osqp import default_algebra; print(default_algebra())"
builtin

Build wheels for cuda backend

This assumes you have the cuda-toolkit installed, and available at /usr/local/cuda. When working on a cluster, you may want to look for a module that populates the necessary environment variables (on our clusters, we do a module load cudatoolkit/12.4, for example).

python -m pip wheel backend/cuda --no-deps --wheel-dir dist

The resulting wheel is available in the dist folder, and can be installed using pip install dist/osqp_cuda-1.*.whl

dist
├── osqp-1.0.0b2.dev68+gcaa6446-cp39-cp39-linux_x86_64.whl
├── osqp_cuda-1.0.0b2.dev68+gcaa6446-cp39-cp39-linux_x86_64.whl
└── osqp_mkl-1.0.0b2.dev68+gcaa6446-cp39-cp39-linux_x86_64.whl

The cuda wheel we built provides us with the osqp_cuda module, and we can check that we have the cuda algebra available:

python -c "from osqp import algebras_available; print(algebras_available())"
['cuda', 'mkl', 'builtin']

Again, the default algebra can be overridden with the OSQP_ALGEBRA_BACKEND environment variable.

Install developer dependencies and run tests

Finally, to test that osqp is installed/working correct with all available algebras, run the tests.

pip install .[dev]
pytest

The tests run across all available algebras. The OSQP_ALGEBRA_BACKEND environment variable does not need to be set, and has no effect for the tests. If the mkl backend is available, then the tests are run for both the "direct" and "indirect" modes of mkl.

To pick exactly what algebras are tested, read on.

Note

Tests that use the mkl backend and the indirect mode are slow to run on head nodes of clusters, where cpu cores are a shared resource and thus cannot be monopolized. Tests involving the cuda algebra may not be possible to run on head nodes of clusters anyway (because of lack of GPUs there). Also, codegen tests in osqp require an internet connection to work properly, which compute nodes may or may not have (because the generated code gets compiled using python+cmake, which wants to fetch certain modules..)

For these and other unforeseen scenarios, fine-tuning of test parametrization is supported using the OSQP_TEST_ALGEBRA_INCLUDE and OSQP_TEST_ALGEBRA_SKIP environment variables (both optional). These variables can take space-delimited values that include builtin, mkl-direct, mkl-indirect, and cuda. Of course, this can be combined with the -k pattern selection that pytest itself supports.

For example:

  • Run all tests for all available algebras:

    pytest

  • Run all tests, but only for the builtin and mkl-direct algebras (if available):

    OSQP_TEST_ALGEBRA_INCLUDE="builtin mkl-direct" pytest

  • Run all tests, but skip the cuda algebra:

    OSQP_TEST_ALGEBRA_SKIP="cuda" pytest

  • Run all tests, but only for the builtin algebra, and skip the codegen tests:

    OSQP_TEST_ALGEBRA_INCLUDE="builtin" pytest -k "not codegen"

Clone this wiki locally