Jenkinsfile and Python virtualenv

Question

I am trying to setup a project that uses the shiny new Jenkins pipelines, more specifically a multibranch project.

I have a Jenkinsfile created in a test branch as below:

node {
    stage 'Preparing VirtualEnv'
    if (!fileExists('.env')){
        echo 'Creating virtualenv ...'
        sh 'virtualenv --no-site-packages .env'
    }
    sh '. .env/bin/activate'
    sh 'ls -all'
    if (fileExists('requirements/preinstall.txt')){
        sh 'pip install -r requirements/preinstall.txt'
    }
    sh 'pip install -r requirements/test.txt'
    stage 'Unittests'
    sh './manage.py test --noinput'
}

It’s worth noting that preinstall.txt will update pip.

I am getting error as below:

OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/pip'

Looks like it’s trying to update pip in global env instead of inside virtualenv, and looks like each sh step is on its own context, how do I make them to execute within the same context?

Asked By: James Lin

||

Source

Answer 1

What you are trying to do will not work. Every time you call the sh command, jenkins will create a new shell.

This means that if you use .env/bin/activate in a sh it will be only sourced in that shell session. The result is that in a new sh command you have to source the file again (if you take a closer look at the console output you will see that Jenkins will actually create temporary shell files each time you run the command.

So you should either source the .env/bin/activate file at the beginning of each shell command (you can use triple quotes for multiline strings), like so

if (fileExists('requirements/preinstall.txt')) {
    sh """
    . .env/bin/activate
    pip install -r requirements/preinstall.txt
    """
}
...
sh """
. .env/bin/activate
pip install -r requirements/test.txt
"""
}
stage("Unittests") {
    sh """
    . .env/bin/activate
    ./manage.py test --noinput
    """
}

or run it all in one shell

sh """
. .env/bin/activate
if [[ -f requirements/preinstall.txt ]]; then
    pip install -r requirements/preinstall.txt
fi
pip install -r requirements/test.txt
./manage.py test --noinput
"""

Answered By: Rik

Answer 2

Like Rik posted, virtualenvs don’t work well within the Jenkins Pipeline Environment, since a new shell is created for each command.

I created a plugin that makes this process a little less painful, which can be found here: https://wiki.jenkins.io/display/JENKINS/Pyenv+Pipeline+Plugin. It essentially just wraps each call in a way that activates the virtualenv prior to running the command. This in itself is tricky, as some methods of running multiple commands inline are split into two separate commands by Jenkins, causing the activated virtualenv no longer to apply.

Answered By: cstarner

Answer 3

I’m new to Jenkins files. Here’s how I’ve been working around the virtual environment issue. (I’m running Python3, Jenkins 2.73.1)

Caveat: Just to be clear, I’m not saying this is a good way to solve the problem, nor have I tested this enough to stand behind this approach, but here what is working for me today:

I’ve been playing around with bypassing the venv ‘activate’ by calling the virtual environment’s python interpreter directly. So instead of:

source ~/venv/bin/activate

one can use:

~/venv/bin/python3 my_script.py

I pass the path to my virtual environment python interpreter via the shell’s rc file (In my case, ~/.bashrc.) In theory, every shell Jenkins calls should read this resource file. In practice, I must restart Jenkins after making changes to the shell resource file.

HOME_DIR=~
export VENV_PATH="$HOME_DIR/venvs/my_venv"
export PYTHON_INTERPRETER="${VENV_PATH}/bin/python3"

My Jenkinsfile looks similar to this:

pipeline {
    agent {
        label 'my_slave'
    }

    stages {
        stage('Stage1') {
            steps {
                // sh 'echo $PYTHON_INTERPRETER'
                // sh 'env | sort'
                sh "$PYTHON_INTERPRETER my_script.py "
            }
        }
    }
}

So when the pipeline is run, the sh has the $PYTHON_INTERPRETER environment values set.

Note one shortcoming of this approach is that now the Jenkins file does not contain all the necessary information to run the script correctly. Hopefully this will get you off the ground.

Answered By: JS.

Jenkinsfile and Python virtualenv

Question:

Answers: