Writing a Python extension in Go (Golang)
Question:
I currently use Cython to link C and Python, and get speedup in slow bits of python code. However, I’d like to use goroutines to implement a really slow (and very parallelizable) bit of code, but it must be callable from python. (I’ve already seen this question)
I’m (sort of) happy to go via C (or Cython) to set up data structures etc if necessary, but avoiding this extra layer would be good from a bug fix/avoidance point of view.
What is the simplest way to do this without having to reinvent any wheels?
Answers:
Unfortunately, this is not currently possible. Go can run C code (and that C code can then call back into Go), but the main function has to be in Go, so the Go runtime can set things up.
There is a go-python package precisely to help you write Python extensions in Go:
this package provides an executable “go-python” which just loads
“python” and then call python.Py_Main(os.Args). the rational being
that under such an executable, go based extensions for C-Python would
be easier to implement (as this usually means calling into go from C
through some rather convoluted functions hops)
Update 2015: possible as of Go 1.5 https://blog.filippo.io/building-python-modules-with-go-1-5/
with Go 1.5 you can build .so objects and import them as Python modules, running Go code (instead of C) directly from Python.
See also https://github.com/go-python/gopy
gopy generates a CPython extension module from a go package.
I’ve written an extension to setuptools which allows you to write cpython extensions that interface with go: https://github.com/asottile/setuptools-golang
There’s a couple example extensions here:
The neat thing is these can be installed just like any other pip package and support both cpython and pypy.
PEP 513 manylinux1 wheels can also be built to provide pre-built wheels via the setuptools-golang-build-manylinux-wheels tool.
The approach is nearly identical to the one in @ColonelPanic’s answer but uses some additional tricks to enable python2 + python3 compatibility.
I currently use Cython to link C and Python, and get speedup in slow bits of python code. However, I’d like to use goroutines to implement a really slow (and very parallelizable) bit of code, but it must be callable from python. (I’ve already seen this question)
I’m (sort of) happy to go via C (or Cython) to set up data structures etc if necessary, but avoiding this extra layer would be good from a bug fix/avoidance point of view.
What is the simplest way to do this without having to reinvent any wheels?
Unfortunately, this is not currently possible. Go can run C code (and that C code can then call back into Go), but the main function has to be in Go, so the Go runtime can set things up.
There is a go-python package precisely to help you write Python extensions in Go:
this package provides an executable “go-python” which just loads
“python” and then call python.Py_Main(os.Args). the rational being
that under such an executable, go based extensions for C-Python would
be easier to implement (as this usually means calling into go from C
through some rather convoluted functions hops)
Update 2015: possible as of Go 1.5 https://blog.filippo.io/building-python-modules-with-go-1-5/
with Go 1.5 you can build .so objects and import them as Python modules, running Go code (instead of C) directly from Python.
See also https://github.com/go-python/gopy
gopy generates a CPython extension module from a go package.
I’ve written an extension to setuptools which allows you to write cpython extensions that interface with go: https://github.com/asottile/setuptools-golang
There’s a couple example extensions here:
The neat thing is these can be installed just like any other pip package and support both cpython and pypy.
PEP 513 manylinux1 wheels can also be built to provide pre-built wheels via the setuptools-golang-build-manylinux-wheels tool.
The approach is nearly identical to the one in @ColonelPanic’s answer but uses some additional tricks to enable python2 + python3 compatibility.