Friday, 29 June 2018

pybind11 and python sub-modules

In my last post, I introduced pybind11 and some basic examples. In this post I want to show how to use python sub-modules with your exported bindings. In particular, I want to show how we structured our bindings in sub-modules when the C++ code was in different libraries in our main project.

Python sub-module

A python sub-module is accessible from python like:

1
2
3
>>> from ork import peon
>>> peon.work_work()
I'm not that kind of Orc

In this example, ork is the main module and peon is the sub-module. In pure Python these might have the folder structure

1
2
3
4
ork
    __init__.py
    peon
        __init__.py

C++ Layout

For our code, we had a project that has multiple C++ libraries and we wanted to expose bindings from each library as a different sub-module of ork in Python:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
ork
    CMakeLists.txt
    peon
        CMakeLists.txt
        include/
        src/
    grunt
        CMakeLists.txt
        include/
        src/
    warlock
        CMakeLists.txt
        include/
        src/

So we wanted to export some functionality from ork.peon, ork.grunt, and ork.warlock to python.

Exporting the bindings

Single Shared Object

The easiest way to do this is to create a single shared object using pybind11. This will include all exported bindings for all the libraries you want to export.

A simplified example would be to add a new ork_bindings.cpp after your other libraries:

1
2
3
4
5
6
7
8
9
10
11
12
PYBIND11_MODULE(orc, mymodule) {
    mymodule.doc() = "Orks live here";
 
    py::module peon = mymodule.def_submodule("peon", "A peon is a submodule of 'ork'");
    peon.def("work_work", &Peon::work_work, "Do some work");
 
    py::module grunt = mymodule.def_submodule("grunt", "A grunt is a submodule of 'ork'");
    grunt.def("work_work", &Grunt::work_work, "Do some work");
 
    py::module warlock = mymodule.def_submodule("warlock", "A warlock is a submodule of 'ork'");
    warlock.def("work_work", &Warlock::work_work, "Do some work");
}

In your CMakeLists.txt you export the the module as:

1
2
3
4
5
6
7
pybind11_add_module(ork, ork_bindings.cpp)
target_add_library(ork
    PRIVATE
        peon
        grunt
        warlock
)

This works fine for a small amount of libraries and exported code. However, I didn't like this approach as it moved your export code away from your main code. This would make it easy to forget to add a new function to a binding and allow for consistency issues to creep into the project.

Multiple Shared Objects

Our chosen approach was to use a separate bindings library for each C++ library to be exported as a sub-module. Then use a Python module as the main module.

To do this we added the following to our code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
ork/
    CMakeLists.txt
    peon/
        CMakeLists.txt
        include/
        src/
            bindings.cpp
    grunt
        CMakeLists.txt
        include/
        src/
            bindings.cpp
    warlock
        CMakeLists.txt
        include/
        src/
            bindings.cpp

An example bindings.cpp for peon is:

1
2
3
4
5
6
7
PYBIND11_MODULE(pypeon, m)
{
    // rename to a submodule
    m.attr("__name__") = "ork.pypeon";
    m.doc() = "A peon is a submodule of 'ork'";
    m.def("work_work", &Peon::work_work, "Do some work");
}

These bindings were added in each CMakeLists.txt as:

1
2
3
4
5
pybind11_add_module(pypeon, src/bindings.cpp)
target_link_library(pypeon
    PRIVATE
        peon
)

When installing your library you then install as:

1
2
3
4
5
ork/
    __init__.py
    pypeon.[python_info].so
    pygrunt.[python_info].so
    pywarlock.[python_info].so

One of the main problems with this approach is the naming of the sub-modules. As the C++ libraries are called peon, grunt, and warlock, it is not possible to have another CMake target with the same name. Therefore, you have to have a slightly different name. In the above example, I have chosen to add py as a prefix for the sub-module names.

Conclusions

So far we have found our approach to work, even with the downside of having a prefix to the name. This allows us to make sure bindings code lives as close a possible to our C++ code and we can conditionally choose which modules to export using CMake options.

No comments:

Post a Comment