which build tool(s) do you recommend for python?
Question:
I’m starting a small/medium-sized python project, likely in Test Driven Development. My backgrounds are more in C and java than python (I used ant and makefile)
I know that in python you might not need a build tool, but I will, because I’ll use cython and PyInstaller (the thing should work on a couple of different UNIXes, without depending directly on python) and I’d like the convenience of selecting the tests form CLI, building the docs, maybe check dependencies, etc.
Somebody is complaining that python lacks a make-like tool.
I know that few tools exist, like scon and paver, but I’d like to hear from real users and not just their website. Does anybody use paver?
What’s about the usual setup.py that comes with many packages? I looked into a few to see if there is a common usage, but I didn’t find anything interesting (maybe I used the wrong examples)
Do you recommend sticking with things I already know (ant and Makefile) at least to start? If so, is there any ant extension you recommend for python (+cython+pyinstaller+pyUnit)?
EDIT: to avoid further answers like jwp’s one, note that, for this project, I absolutely need my program being a standalone executable because it is absolutely impossible to have a python VM on the target platform where the executable will run. I have exactly the same hw available for compiling, so luckly I don’t need to cross-compile (but I’d do the development on a more friendly Linux).
I’d also like to test if my code compile in Cython from the beginning, not to premature optimize, but just to be sure that I’m not going too far with the use of incompatible features, which would require a painful refactoring if Cython would be seriously needed.
So please focus on my actual question
Answers:
I’ve been using setuptools for this, sometimes in combination with Ant for more complex stuff, or for integrating with other builds. In general it has worked well for me, but I’ve heard some people at #python (freenode) complain about a generally low source quality for this package, and recommending the standard distutils.
One thing to really watch out with if you are used to Java (like me), is that python does not support split packages. That is, you cannot put site.db in src/site/db, site.view in src2/site/view, put src and src2 on PYTHONPATH and the expect everything to work. This caused major headaches for me with the build tools, it worked a bit sporadically 🙂
If it is at all possible, I’d suggest avoiding extension modules(C/cython) in the beginning. Get your all code written in Python, use a simple distutils based configuration, run your tests using -m (python -m mypkg.test.testall, or whatever; import unittest).
Once you get your project to a comfy state, then start tackling some optimizations with cython and the extra project management that comes with that. distutils can build extension modules, so I’m not sure you’ll need make/scons..
project-dir/
setup.py
mypkg/
__init__.py
mymod.py
test/
__init__.py
testall.py
testsomething_specific.py
One tool I love is virtualenv: http://pypi.python.org/pypi/virtualenv
from the site:
What It Does
virtualenv is a tool to create isolated Python environments.
The basic problem being addressed is one of dependencies and versions, and indirectly permissions. Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python2.4/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.
You can also try: http://pypi.python.org/pypi/zc.buildout
from the site:
The Buildout project provides support for creating applications, especially Python applications. It provides tools for assembling applications from multiple parts, Python or otherwise. An application may actually contain multiple programs, processes, and configuration settings.
Tried it a couple of times, excellent for deploying development envrionments, maybe it’s too much for your needs.
I’ve also been told and read in a couple of places that nose is a very cool testing tool
http://somethingaboutorange.com/mrl/projects/nose/0.11.1/, I’m looking for some free time to try it.
Hope It helps
Cheers,
Ale
Your requirements suggest rather Scons which, according to their website, has more control over variety of building tasks than Paver. In the latter you would end up using a lot of sh() which runs a regular command line programs.
Recently, I started using Paver which is really great to run tests, build documentations with Sphinx, but I use only pure Python.
If you want to see what’s possible with Paver I highly recommend those 2 articles:
converting-from-make-to-paver and writing-technical-documentation by Doug Hellmann and you definitely want to check his pavement.py configuration file.
to avoid further answers like jwp’s one, note that, for this project, I absolutely need my program being a standalone executable because it is absolutely impossible to have a python VM on the target platform where the executable will run. I have exactly the same hw available for compiling, so luckly I don’t need to cross-compile (but I’d do the development on a more friendly Linux).
I’d also like to test if my code compile in Cython from the beginning, not to premature optimize, but just to be sure that I’m not going too far with the use of incompatible features, which would require a painful refactoring if Cython would be seriously needed.
If all you’re looking for is to have a stand-alone executable, you don’t need to run Cython. There are a few good libraries for doing this:
However, to run Python code, you’re simply going to have to install an interpreter or virtual machine on the target machine in some form or fashion. The solutions I presented involve merely embedding the interpreter to make it easier to install. In fact, I’m pretty sure (but could be wrong) that Cython won’t allow you to build a stand-alone executables. It’s mainly meant for creating extensions to the Python interpreter.
However, there are a couple of other alternatives. If you have a Java interpreter on the target machine, you can run Jython. There’s also a IronPython for .net. However, you will still have to distribute the runtimes for these languages with your project.
In short, if you want a stand-alone executable with no dependencies, your only option is pretty much to go with straight C or C++.
I’m starting a small/medium-sized python project, likely in Test Driven Development. My backgrounds are more in C and java than python (I used ant and makefile)
I know that in python you might not need a build tool, but I will, because I’ll use cython and PyInstaller (the thing should work on a couple of different UNIXes, without depending directly on python) and I’d like the convenience of selecting the tests form CLI, building the docs, maybe check dependencies, etc.
Somebody is complaining that python lacks a make-like tool.
I know that few tools exist, like scon and paver, but I’d like to hear from real users and not just their website. Does anybody use paver?
What’s about the usual setup.py that comes with many packages? I looked into a few to see if there is a common usage, but I didn’t find anything interesting (maybe I used the wrong examples)
Do you recommend sticking with things I already know (ant and Makefile) at least to start? If so, is there any ant extension you recommend for python (+cython+pyinstaller+pyUnit)?
EDIT: to avoid further answers like jwp’s one, note that, for this project, I absolutely need my program being a standalone executable because it is absolutely impossible to have a python VM on the target platform where the executable will run. I have exactly the same hw available for compiling, so luckly I don’t need to cross-compile (but I’d do the development on a more friendly Linux).
I’d also like to test if my code compile in Cython from the beginning, not to premature optimize, but just to be sure that I’m not going too far with the use of incompatible features, which would require a painful refactoring if Cython would be seriously needed.
So please focus on my actual question
I’ve been using setuptools for this, sometimes in combination with Ant for more complex stuff, or for integrating with other builds. In general it has worked well for me, but I’ve heard some people at #python (freenode) complain about a generally low source quality for this package, and recommending the standard distutils.
One thing to really watch out with if you are used to Java (like me), is that python does not support split packages. That is, you cannot put site.db in src/site/db, site.view in src2/site/view, put src and src2 on PYTHONPATH and the expect everything to work. This caused major headaches for me with the build tools, it worked a bit sporadically 🙂
If it is at all possible, I’d suggest avoiding extension modules(C/cython) in the beginning. Get your all code written in Python, use a simple distutils based configuration, run your tests using -m (python -m mypkg.test.testall, or whatever; import unittest).
Once you get your project to a comfy state, then start tackling some optimizations with cython and the extra project management that comes with that. distutils can build extension modules, so I’m not sure you’ll need make/scons..
project-dir/
setup.py
mypkg/
__init__.py
mymod.py
test/
__init__.py
testall.py
testsomething_specific.py
One tool I love is virtualenv: http://pypi.python.org/pypi/virtualenv
from the site:
What It Does
virtualenv is a tool to create isolated Python environments.
The basic problem being addressed is one of dependencies and versions, and indirectly permissions. Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python2.4/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.
You can also try: http://pypi.python.org/pypi/zc.buildout
from the site:
The Buildout project provides support for creating applications, especially Python applications. It provides tools for assembling applications from multiple parts, Python or otherwise. An application may actually contain multiple programs, processes, and configuration settings.
Tried it a couple of times, excellent for deploying development envrionments, maybe it’s too much for your needs.
I’ve also been told and read in a couple of places that nose is a very cool testing tool
http://somethingaboutorange.com/mrl/projects/nose/0.11.1/, I’m looking for some free time to try it.
Hope It helps
Cheers,
Ale
Your requirements suggest rather Scons which, according to their website, has more control over variety of building tasks than Paver. In the latter you would end up using a lot of sh() which runs a regular command line programs.
Recently, I started using Paver which is really great to run tests, build documentations with Sphinx, but I use only pure Python.
If you want to see what’s possible with Paver I highly recommend those 2 articles:
converting-from-make-to-paver and writing-technical-documentation by Doug Hellmann and you definitely want to check his pavement.py configuration file.
to avoid further answers like jwp’s one, note that, for this project, I absolutely need my program being a standalone executable because it is absolutely impossible to have a python VM on the target platform where the executable will run. I have exactly the same hw available for compiling, so luckly I don’t need to cross-compile (but I’d do the development on a more friendly Linux).
I’d also like to test if my code compile in Cython from the beginning, not to premature optimize, but just to be sure that I’m not going too far with the use of incompatible features, which would require a painful refactoring if Cython would be seriously needed.
If all you’re looking for is to have a stand-alone executable, you don’t need to run Cython. There are a few good libraries for doing this:
However, to run Python code, you’re simply going to have to install an interpreter or virtual machine on the target machine in some form or fashion. The solutions I presented involve merely embedding the interpreter to make it easier to install. In fact, I’m pretty sure (but could be wrong) that Cython won’t allow you to build a stand-alone executables. It’s mainly meant for creating extensions to the Python interpreter.
However, there are a couple of other alternatives. If you have a Java interpreter on the target machine, you can run Jython. There’s also a IronPython for .net. However, you will still have to distribute the runtimes for these languages with your project.
In short, if you want a stand-alone executable with no dependencies, your only option is pretty much to go with straight C or C++.