How can I build manually C++ extension with mingw-w64, Python and pybind11?
Question:
My final goal is to compile Python C++ extension from my C++ code. Currently to get started I am following a simple example from the first steps of pybind11
documentation. My working environment is a Windows 7 Professional 64 bit, mingw-w64 (x86_64-8.1.0-posix-seh-rt_v6-rev0) and Anaconda3 with Python 3.7.4 64 bit. I have 2 files. The first one is a C++ file — example.cpp
#include <pybind11/pybind11.h>
int add(int i, int j) {
return i + j;
}
PYBIND11_MODULE(example, m) {
m.doc() = "pybind11 example plugin"; // optional module docstring
m.def("add", &add, "A function which adds two numbers");
}
I compile the C++ file with the following command:
C:/mingw-w64/x86_64-8.1.0-posix-seh-rt_v6-rev0/mingw64/bin/g++.exe -shared -std=c++11 -DMS_WIN64 -fPIC -ID:UsersADASanaconda3Include -ID:UsersADASanaconda3Libraryinclude -ID:UsersADASanaconda3pkgspython-3.7.4-h5263a28_0include -Wall -LD:UsersADASanaconda3Lib -LD:UsersADASanaconda3pkgspython-3.7.4-h5263a28_0libs example.cpp -o example.dll -lPython37
The compilation result is successful and I am getting example.dll
file.
At the next step I run the following Python code — example.py:
import example
def main():
i, j = (1, 2)
res = example.add(i, j)
print("%d + %d = %d" % (i, j, res))
if __name__ == '__main__':
main()
And here I have got an issue. It seems the import example
line does not give me any warnings or errors but the line res = example.add(i, j)
gives me an error:
AttributeError: module 'example' has no attribute 'add'
Under Ubuntu 18.04 I successfully compiled and run in Python the example above but in my office I have only Windows 7.
Questions: what is wrong in either my setup or command line? Is it possible to fix this problem without changing current C++ compiler (mingw-w64 version 8.1) under Windows?
Answers:
It is unbelievable! The problem was just the file extension of the compiled file. As soon as I changed .dll
to .pyd
, the Python example (example.py
) is running without any issue!
So the new command line is:
C:/mingw-w64/x86_64-8.1.0-posix-seh-rt_v6-rev0/mingw64/bin/g++.exe -shared -std=c++11 -DMS_WIN64 -fPIC -ID:UsersADASanaconda3Include -ID:UsersADASanaconda3Libraryinclude -ID:UsersADASanaconda3pkgspython-3.7.4-h5263a28_0include -Wall -LD:UsersADASanaconda3Lib -LD:UsersADASanaconda3pkgspython-3.7.4-h5263a28_0libs example.cpp -o example.pyd -lPython37
Because I did some experiments with the command line arguments, I am going to check again all the compiler arguments to make sure it gives the successful result. I will let you know, if some changes are still required.
Update1:
According to Python3 default settings, the full extension of the compiled C++ file under Windows has to be .cp37-win_amd64.pyd
.
We can get the extension by terminal command:
python -c "from distutils import sysconfig; print(sysconfig.get_config_var('EXT_SUFFIX'))"
That is an equivalent of the python3-config --extension-suffix
from pybind11 documentation. The python3-config
script is not implemented in Windows environment (at least in Anaconda3 distribution).
Update2:
Under Linux OS (tested in Ubuntu 20.04), the command line is like
c++ -O3 -Wall -shared -std=c++11 -fPIC $(python3 -m pybind11 --includes) example.cpp -o example$(python3-config --extension-suffix)
My final goal is to compile Python C++ extension from my C++ code. Currently to get started I am following a simple example from the first steps of pybind11
documentation. My working environment is a Windows 7 Professional 64 bit, mingw-w64 (x86_64-8.1.0-posix-seh-rt_v6-rev0) and Anaconda3 with Python 3.7.4 64 bit. I have 2 files. The first one is a C++ file — example.cpp
#include <pybind11/pybind11.h>
int add(int i, int j) {
return i + j;
}
PYBIND11_MODULE(example, m) {
m.doc() = "pybind11 example plugin"; // optional module docstring
m.def("add", &add, "A function which adds two numbers");
}
I compile the C++ file with the following command:
C:/mingw-w64/x86_64-8.1.0-posix-seh-rt_v6-rev0/mingw64/bin/g++.exe -shared -std=c++11 -DMS_WIN64 -fPIC -ID:UsersADASanaconda3Include -ID:UsersADASanaconda3Libraryinclude -ID:UsersADASanaconda3pkgspython-3.7.4-h5263a28_0include -Wall -LD:UsersADASanaconda3Lib -LD:UsersADASanaconda3pkgspython-3.7.4-h5263a28_0libs example.cpp -o example.dll -lPython37
The compilation result is successful and I am getting example.dll
file.
At the next step I run the following Python code — example.py:
import example
def main():
i, j = (1, 2)
res = example.add(i, j)
print("%d + %d = %d" % (i, j, res))
if __name__ == '__main__':
main()
And here I have got an issue. It seems the import example
line does not give me any warnings or errors but the line res = example.add(i, j)
gives me an error:
AttributeError: module 'example' has no attribute 'add'
Under Ubuntu 18.04 I successfully compiled and run in Python the example above but in my office I have only Windows 7.
Questions: what is wrong in either my setup or command line? Is it possible to fix this problem without changing current C++ compiler (mingw-w64 version 8.1) under Windows?
It is unbelievable! The problem was just the file extension of the compiled file. As soon as I changed .dll
to .pyd
, the Python example (example.py
) is running without any issue!
So the new command line is:
C:/mingw-w64/x86_64-8.1.0-posix-seh-rt_v6-rev0/mingw64/bin/g++.exe -shared -std=c++11 -DMS_WIN64 -fPIC -ID:UsersADASanaconda3Include -ID:UsersADASanaconda3Libraryinclude -ID:UsersADASanaconda3pkgspython-3.7.4-h5263a28_0include -Wall -LD:UsersADASanaconda3Lib -LD:UsersADASanaconda3pkgspython-3.7.4-h5263a28_0libs example.cpp -o example.pyd -lPython37
Because I did some experiments with the command line arguments, I am going to check again all the compiler arguments to make sure it gives the successful result. I will let you know, if some changes are still required.
Update1:
According to Python3 default settings, the full extension of the compiled C++ file under Windows has to be .cp37-win_amd64.pyd
.
We can get the extension by terminal command:
python -c "from distutils import sysconfig; print(sysconfig.get_config_var('EXT_SUFFIX'))"
That is an equivalent of the python3-config --extension-suffix
from pybind11 documentation. The python3-config
script is not implemented in Windows environment (at least in Anaconda3 distribution).
Update2:
Under Linux OS (tested in Ubuntu 20.04), the command line is like
c++ -O3 -Wall -shared -std=c++11 -fPIC $(python3 -m pybind11 --includes) example.cpp -o example$(python3-config --extension-suffix)