Atlassian Bamboo with Django & Python – Possible?
Question:
At my company, we currently use Atlassian Bamboo for our continuous integration tool. We currently use Java for all of our projects, so it works great.
However, we are considering using a Django + Python for one of our new applications. I was wondering if it is possible to use Bamboo for this.
First off, let me say that I have a low level of familiarity with Bamboo, as I’ve only ever used it, not configured it (other than simple changes like changing the svn checkout directory for a build).
Obviously there isn’t a lot of point in just running a build (since Python projects don’t really build), but I’d like to be able to use Bamboo for running the test suite, as well as use bamboo to deploy the latest code to our various test environments the way we do with our Java projects.
Does Bamboo support this type of thing with a Python project?
Answers:
Bamboo essentially just runs a shell script, so this could just as easily be:
./manage.py test
as it typically is:
mvn clean install
or:
ant compile
You may have to massage to output of the Django test runner into traditional JUnit XML output, so that Bamboo can give you pretty graphs on how many tests passed. Look at this post about using xmlrunner.py to get Python working with Hudson. Also take a look at NoseXUnit.
You can even add a bootstrap for pip and virtualenv on a clean environment quite easily, which is cool:
wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py --root=${bamboo.build.working.directory}/tmp --ignore-installed
export PATH=${bamboo.build.working.directory}/tmp/usr/local/bin:$PATH
export PYTHONPATH=${bamboo.build.working.directory}/tmp/usr/local/lib/python2.7/dist-packages:$PYTHONPATH
pip install --root=${bamboo.build.working.directory}/tmp --ignore-installed virtualenv
virtualenv virtual_tmp
cd virtual_tmp
. bin/activate
echo Pip is located `which pip`
pip install django
pip install djangorestframework
Warning, source bin/activate does not work as the inline script tasks are stored into an sh file (so bash run it in sh compatibility mode).
Edit
Even better, we can run unit tests on the top of it, with xml outputs that can be parsed by the JUnit of bamboo:
pip install unittest-xml-reporting
python manage.py test --noinput --testrunner="xmlrunner.extra.djangotestrunner.XMLTestRunner"
If you use pytest you can simply use py.test --junitxml=/path/to/results/xml/file.xml
It turns out it is possible. There are two major integration tasks: test runner results and code coverage results. I assume normal Python 3 codebase and standard unittest test suite.
Test runner
Bamboo expects test runner results in JUnit XML format. There is separate test runner on the Cheese Shop able to produce such output, but it would require you to write a little code to run it, which is not nice. Better way which keeps the codebase intact is to use pytest‘s features.
Code coverage
Bamboo only supports the XML format of Atlassian Clover. Important note here is that you don’t need Atlassian Clover plugin enabled (and license for it which costs some bucks). Bamboo works on its own.
Python de facto standard code coverage tool, coverage, produces somewhat
Cobertura XML format, but there’s a converter. There’s a pytest plugin for integration with the coverage tool.
Solution
Here’s the Tox environment where I used pytest to make both Bamboo integrations work.
[tox]
envlist = py34
skipsdist = True
[testenv]
setenv = LANG=C.UTF-8
basepython = python3.4
deps = -r{toxinidir}/requirements.txt
[testenv:bamboo]
commands =
py.test --junitxml=results.xml
--cov=project_name --cov-config=tox.ini --cov-report=xml
--cov-report=html project_name/test
coverage2clover -i coverage.xml -o clover.xml
deps =
{[testenv]deps}
pytest
pytest-cov
coverage2clover
# read by pytest
[pytest]
python_files = *.py
# read by coverage
[run]
omit=project_name/test/*,project_name/__main__.py
Note that both pytest and pytest-cov use tox.ini for the configuration that is not supported on command line. It again saves your from having additional clutter in root of your repo. pytest tries to read tox.ini automatically. pytest-cov bypasses to .coveragerc, but because it’s also an INI file, tox.ini fits.
On Bamboo side add a script task that runs tox -e bamboo. Then add JUnit parse task to the job. In its dialogue, under Specify custom results directories put results.xml.
Coverage configuration is done other way.
- Open Miscellaneous tab of your job
- Check Use Clover to collect Code Coverage for this build
- Select Clover is already integrated into this build and a clover.xml file will be produced
- Type
clover.xml into Clover XML Location
At this point in your next build you will see total coverage and two charts: Coverage history and Lines of code history. It’s also nice to have interactive HTML produced by coverage tool, so you can drill down to certain line of code.
The settings made above (at least in Bamboo 5.7) has created Clover Report (System) in Artifact job’s tab. Open it and set htmlcov to Location field, and *.* to Copy pattern. Bamboo will now collect the HTML reports. You can see it at Clover tab of your plan.
At my company, we currently use Atlassian Bamboo for our continuous integration tool. We currently use Java for all of our projects, so it works great.
However, we are considering using a Django + Python for one of our new applications. I was wondering if it is possible to use Bamboo for this.
First off, let me say that I have a low level of familiarity with Bamboo, as I’ve only ever used it, not configured it (other than simple changes like changing the svn checkout directory for a build).
Obviously there isn’t a lot of point in just running a build (since Python projects don’t really build), but I’d like to be able to use Bamboo for running the test suite, as well as use bamboo to deploy the latest code to our various test environments the way we do with our Java projects.
Does Bamboo support this type of thing with a Python project?
Bamboo essentially just runs a shell script, so this could just as easily be:
./manage.py test
as it typically is:
mvn clean install
or:
ant compile
You may have to massage to output of the Django test runner into traditional JUnit XML output, so that Bamboo can give you pretty graphs on how many tests passed. Look at this post about using xmlrunner.py to get Python working with Hudson. Also take a look at NoseXUnit.
You can even add a bootstrap for pip and virtualenv on a clean environment quite easily, which is cool:
wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py --root=${bamboo.build.working.directory}/tmp --ignore-installed
export PATH=${bamboo.build.working.directory}/tmp/usr/local/bin:$PATH
export PYTHONPATH=${bamboo.build.working.directory}/tmp/usr/local/lib/python2.7/dist-packages:$PYTHONPATH
pip install --root=${bamboo.build.working.directory}/tmp --ignore-installed virtualenv
virtualenv virtual_tmp
cd virtual_tmp
. bin/activate
echo Pip is located `which pip`
pip install django
pip install djangorestframework
Warning, source bin/activate does not work as the inline script tasks are stored into an sh file (so bash run it in sh compatibility mode).
Edit
Even better, we can run unit tests on the top of it, with xml outputs that can be parsed by the JUnit of bamboo:
pip install unittest-xml-reporting
python manage.py test --noinput --testrunner="xmlrunner.extra.djangotestrunner.XMLTestRunner"
If you use pytest you can simply use py.test --junitxml=/path/to/results/xml/file.xml
It turns out it is possible. There are two major integration tasks: test runner results and code coverage results. I assume normal Python 3 codebase and standard unittest test suite.
Test runner
Bamboo expects test runner results in JUnit XML format. There is separate test runner on the Cheese Shop able to produce such output, but it would require you to write a little code to run it, which is not nice. Better way which keeps the codebase intact is to use pytest‘s features.
Code coverage
Bamboo only supports the XML format of Atlassian Clover. Important note here is that you don’t need Atlassian Clover plugin enabled (and license for it which costs some bucks). Bamboo works on its own.
Python de facto standard code coverage tool, coverage, produces somewhat
Cobertura XML format, but there’s a converter. There’s a pytest plugin for integration with the coverage tool.
Solution
Here’s the Tox environment where I used pytest to make both Bamboo integrations work.
[tox]
envlist = py34
skipsdist = True
[testenv]
setenv = LANG=C.UTF-8
basepython = python3.4
deps = -r{toxinidir}/requirements.txt
[testenv:bamboo]
commands =
py.test --junitxml=results.xml
--cov=project_name --cov-config=tox.ini --cov-report=xml
--cov-report=html project_name/test
coverage2clover -i coverage.xml -o clover.xml
deps =
{[testenv]deps}
pytest
pytest-cov
coverage2clover
# read by pytest
[pytest]
python_files = *.py
# read by coverage
[run]
omit=project_name/test/*,project_name/__main__.py
Note that both pytest and pytest-cov use tox.ini for the configuration that is not supported on command line. It again saves your from having additional clutter in root of your repo. pytest tries to read tox.ini automatically. pytest-cov bypasses to .coveragerc, but because it’s also an INI file, tox.ini fits.
On Bamboo side add a script task that runs tox -e bamboo. Then add JUnit parse task to the job. In its dialogue, under Specify custom results directories put results.xml.
Coverage configuration is done other way.
- Open Miscellaneous tab of your job
- Check Use Clover to collect Code Coverage for this build
- Select Clover is already integrated into this build and a clover.xml file will be produced
- Type
clover.xmlinto Clover XML Location
At this point in your next build you will see total coverage and two charts: Coverage history and Lines of code history. It’s also nice to have interactive HTML produced by coverage tool, so you can drill down to certain line of code.
The settings made above (at least in Bamboo 5.7) has created Clover Report (System) in Artifact job’s tab. Open it and set htmlcov to Location field, and *.* to Copy pattern. Bamboo will now collect the HTML reports. You can see it at Clover tab of your plan.