Making an executable in Cython
Question:
Been playing with cython. Normally program in Python, but used C in a previous life.
I can’t figure out how to make a free-standing executable.
I’ve downloaded cython, and I can make a .pyx file (that’s just a normal Python file with a .pyx extension), that executes in the Python shell, using:
import pyximport; pyximport.install()
I can generate a .c file at the command line with: cython file.pyx
I can generate a .so file by building a standard setup.py and executing:
setup.py build_ext --inplace
I’ve tried making an executable out of the .so file using gcc with various options, but always have tons of missing files, headers, etc. Have tried pointing to headers from virtually everywhere, but with no success, and am not really familiar with what all the gcc options do, or even if I should be using gcc.
I’ve having a disconnect here with the fact that I can run my program in the Python shell, but not at the command line, (I don’t want users to have to get into the shell, import modules, etc).
What am I missing here?
Answers:
What you want is the --embed flag for the Cython compiler.
There isn’t a ton of documentation on it, but this is what I was able to find. It does link to a simple working example.
To compile the Cython source code to a C file that can then be compiled to an executable you use a command like cython myfile.pyx --embed and then compile with whichever C compiler you are using.
When you compile the C source code, you will still need to include the directory with the Python headers and link to the corresponding Python shared library on your system (a file named something like libpython27.so or libpython27.a if you are using Python 2.7).
Edit: Here are some more instructions on how to get the commands for including the proper headers and linking against the proper libraries.
As I said earlier, you need to run the Cython compiler like this:
cython <cython_file> --embed
To compile using gcc, you will need to find where the python headers are on your system (you can get this location by running distutils.sysconfig.get_python_inc() (you’ll have to import it first).
It is probably just the /include subdirectory in your Python installation directory.
You will also have to find the python shared library.
For Python 2.7 it would be libpython27.a on Windows or libpython2.7.so on Linux.
Your gcc command will then be
gcc <C_file_from_cython> -I<include_directory> -L<directory_containing_libpython> -l<name_of_libpython_without_lib_on_the_front> -o <output_file_name>
It may be wise to include the -fPIC flag.
On Windows 64 bit machines you will also have to include the flags -D MS_WIN64 that tells mingw to compile for 64 bit windows.
If you are compiling something that depends on NumPy, you will also need to include the directory containing the NumPy headers.
You can find this folder by running numpy.get_include() (again, after importing numpy).
Your gcc command then becomes
gcc <C_file_from_cython> -I<include_directory> -I<numpy_include_directory> -L<directory_containing_libpython> -l<name_of_libpython_without_lib_on_the_front> -o <output_file_name>
This gcc command option guide may be helpful.
Also, I would recommend you use Cython memory views if possible.
That will make it so that you won’t have to include the NumPy headers and include the NumPy pxd file in your Cython file.
It also makes slicing operations easier for the C compiler to optimize.
Tested this on Ubuntu:
Install Cython using the following (Python 2):
sudo apt-get install cython
For Python 3:
sudo apt-get install cython3
To compile the Python code into a C code run (change cython to cython3 for Python 3):
cython --embed -o example.c example.py
This will generate the example.c file.
Now compile the example.c file:
gcc -Os -I /usr/include/python2.7 example.c -lpython2.7 -o example
Run the file:
./example
Now for Python 3, something like this will work (not tested):
gcc -Os -I /usr/include/python3.6 example.c -lpython3.6 -o example
where python3.x is the version of Python installed on your machine.
This is a solution for Windows + MS Visual Studio 14 (as no-one mentioned the cl.exe parameters yet).
First use the embed parameter to produce the test.c file:
cython test.pyx --embed
Then compile it:
call "C:Program Files (x86)Microsoft Visual Studio 14.0VCvcvarsall.bat" x64
cl test.c /I C:Python37include /link C:Python37libspython37.lib
The output is small executable file test.exe (140 KB in my case for a print("Hello World")).
Notes:
-
for a GUI app (and no console app), see @longgangfan’s answer Can Cython compile to an EXE?, i.e. add /subsystem:windows /entry:wmainCRTStartup in the cl.exe parameters.
-
for more information about whether the source code is still there, see Are executables produced with Cython really free of the source code? and Does compiled standalone Cython executable still contain all original source code?.
-
also, the .exe won’t run out-of-the-box on any machine; some other files are required, see Minimal set of files required to distribute an embed-Cython-compiled code and make it work on any machine.
Building off of Rafael’s answer, here’s an explanation of how to make a ‘Hello, World’ executable with Cython and Python 3 (tested on Termux), followed by instructions for Python 2:
cython3 doens’t seem to be an option on my system, so here’s what I do instead:
The file is called test.py:
print("Hello, world!")
Then do this:
cython -3 --embed -o test.c test.py
Then you need to figure out what include path you need, and maybe your libs.
I’m using Python 3.10.6; so, I use a program called python3.10-config to get the information before inputting it in gcc:
python3.10-config --includes
For me, that outputs this:
-I/data/data/com.termux/files/usr/include/python3.10 -I/data/data/com.termux/files/usr/include/python3.10
It gave me the same path twice (I don’t know why), so just use that path (you only need to use it once, I think).
Then do
python3.10-config --libs
For me, that outputs this:
-lpython3.10 -lcrypt -ldl -lm -lm
I’m not sure that every last one of those is important for every program, since the ‘Hello, world’ one works if I only put -lpython3.10.
Anyway, then I do this and it gives me an executable:
gcc -Os -I/data/data/com.termux/files/usr/include/python3.10 test.c -lpython3.10 -o test
There’s a program called python2.7-config, too.
If I want to get an executable with Python 2.7.18, I can do this instead of the above:
cython -2 --embed -o test.c test.py
Then
python2.7-config --includes
python2.7-config --libs
That gives me these:
-I/data/data/com.termux/files/usr/include/python2.7 -I/data/data/com.termux/files/usr/include/python2.7-lpython2.7 -ldl -lm
Finally, I do this to get the executable:
gcc -Os -I/data/data/com.termux/files/usr/include/python2.7 test.c -lpython2.7 -o test
Been playing with cython. Normally program in Python, but used C in a previous life.
I can’t figure out how to make a free-standing executable.
I’ve downloaded cython, and I can make a .pyx file (that’s just a normal Python file with a .pyx extension), that executes in the Python shell, using:
import pyximport; pyximport.install()
I can generate a .c file at the command line with: cython file.pyx
I can generate a .so file by building a standard setup.py and executing:
setup.py build_ext --inplace
I’ve tried making an executable out of the .so file using gcc with various options, but always have tons of missing files, headers, etc. Have tried pointing to headers from virtually everywhere, but with no success, and am not really familiar with what all the gcc options do, or even if I should be using gcc.
I’ve having a disconnect here with the fact that I can run my program in the Python shell, but not at the command line, (I don’t want users to have to get into the shell, import modules, etc).
What am I missing here?
What you want is the --embed flag for the Cython compiler.
There isn’t a ton of documentation on it, but this is what I was able to find. It does link to a simple working example.
To compile the Cython source code to a C file that can then be compiled to an executable you use a command like cython myfile.pyx --embed and then compile with whichever C compiler you are using.
When you compile the C source code, you will still need to include the directory with the Python headers and link to the corresponding Python shared library on your system (a file named something like libpython27.so or libpython27.a if you are using Python 2.7).
Edit: Here are some more instructions on how to get the commands for including the proper headers and linking against the proper libraries.
As I said earlier, you need to run the Cython compiler like this:
cython <cython_file> --embed
To compile using gcc, you will need to find where the python headers are on your system (you can get this location by running distutils.sysconfig.get_python_inc() (you’ll have to import it first).
It is probably just the /include subdirectory in your Python installation directory.
You will also have to find the python shared library.
For Python 2.7 it would be libpython27.a on Windows or libpython2.7.so on Linux.
Your gcc command will then be
gcc <C_file_from_cython> -I<include_directory> -L<directory_containing_libpython> -l<name_of_libpython_without_lib_on_the_front> -o <output_file_name>
It may be wise to include the -fPIC flag.
On Windows 64 bit machines you will also have to include the flags -D MS_WIN64 that tells mingw to compile for 64 bit windows.
If you are compiling something that depends on NumPy, you will also need to include the directory containing the NumPy headers.
You can find this folder by running numpy.get_include() (again, after importing numpy).
Your gcc command then becomes
gcc <C_file_from_cython> -I<include_directory> -I<numpy_include_directory> -L<directory_containing_libpython> -l<name_of_libpython_without_lib_on_the_front> -o <output_file_name>
This gcc command option guide may be helpful.
Also, I would recommend you use Cython memory views if possible.
That will make it so that you won’t have to include the NumPy headers and include the NumPy pxd file in your Cython file.
It also makes slicing operations easier for the C compiler to optimize.
Tested this on Ubuntu:
Install Cython using the following (Python 2):
sudo apt-get install cython
For Python 3:
sudo apt-get install cython3
To compile the Python code into a C code run (change cython to cython3 for Python 3):
cython --embed -o example.c example.py
This will generate the example.c file.
Now compile the example.c file:
gcc -Os -I /usr/include/python2.7 example.c -lpython2.7 -o example
Run the file:
./example
Now for Python 3, something like this will work (not tested):
gcc -Os -I /usr/include/python3.6 example.c -lpython3.6 -o example
where python3.x is the version of Python installed on your machine.
This is a solution for Windows + MS Visual Studio 14 (as no-one mentioned the cl.exe parameters yet).
First use the embed parameter to produce the test.c file:
cython test.pyx --embed
Then compile it:
call "C:Program Files (x86)Microsoft Visual Studio 14.0VCvcvarsall.bat" x64
cl test.c /I C:Python37include /link C:Python37libspython37.lib
The output is small executable file test.exe (140 KB in my case for a print("Hello World")).
Notes:
-
for a GUI app (and no console app), see @longgangfan’s answer Can Cython compile to an EXE?, i.e. add
/subsystem:windows /entry:wmainCRTStartupin thecl.exeparameters. -
for more information about whether the source code is still there, see Are executables produced with Cython really free of the source code? and Does compiled standalone Cython executable still contain all original source code?.
-
also, the .exe won’t run out-of-the-box on any machine; some other files are required, see Minimal set of files required to distribute an embed-Cython-compiled code and make it work on any machine.
Building off of Rafael’s answer, here’s an explanation of how to make a ‘Hello, World’ executable with Cython and Python 3 (tested on Termux), followed by instructions for Python 2:
cython3 doens’t seem to be an option on my system, so here’s what I do instead:
The file is called test.py:
print("Hello, world!")
Then do this:
cython -3 --embed -o test.c test.py
Then you need to figure out what include path you need, and maybe your libs.
I’m using Python 3.10.6; so, I use a program called python3.10-config to get the information before inputting it in gcc:
python3.10-config --includes
For me, that outputs this:
-I/data/data/com.termux/files/usr/include/python3.10 -I/data/data/com.termux/files/usr/include/python3.10
It gave me the same path twice (I don’t know why), so just use that path (you only need to use it once, I think).
Then do
python3.10-config --libs
For me, that outputs this:
-lpython3.10 -lcrypt -ldl -lm -lm
I’m not sure that every last one of those is important for every program, since the ‘Hello, world’ one works if I only put -lpython3.10.
Anyway, then I do this and it gives me an executable:
gcc -Os -I/data/data/com.termux/files/usr/include/python3.10 test.c -lpython3.10 -o test
There’s a program called python2.7-config, too.
If I want to get an executable with Python 2.7.18, I can do this instead of the above:
cython -2 --embed -o test.c test.py
Then
python2.7-config --includes
python2.7-config --libs
That gives me these:
-I/data/data/com.termux/files/usr/include/python2.7 -I/data/data/com.termux/files/usr/include/python2.7-lpython2.7 -ldl -lm
Finally, I do this to get the executable:
gcc -Os -I/data/data/com.termux/files/usr/include/python2.7 test.c -lpython2.7 -o test