run a crontab job using an anaconda env
Question:
I want to have a cron job execute a python script using an already existing anaconda python environment called my_env. The only thing I can think to do is have the cron job run a script called my_script.bash which in turn activates the env and then runs the python script.
#!/bin/bash
source activate my_env
python ~/my_project/main.py
Trying to execute this script from the command lines doesn’t work:
$ sh scripts/my_script.bash
scripts/my_script.bash: 9: scripts/my_script.bash: source: not found
What do I need to do to make sure the proper environment is activated. Its ok to explain it to me like I’m 5.
Answers:
Don’t call sh but bash. source is a bash command.
- sh scripts/my_script.bash
+ bash scripts/my_script.bash
Or just
chmod +x scripts/my_script.bash
./scripts/my_script.bash
since you added the bash shebang.
I recently switched from canopy to Anaconda precisely to get away from having to activate an env in cron jobs.
Anaconda makes this very simple, based on the PATH enviornment variable. (I’m using miniconda not the full Anaconds install but I believe anaconda should work the same way)
There are two different approaches, I’ve tested;
-
Add a shebang in your python script, main.py
#!/home/users/user_name/miniconda2/envs/my_env/bin/python
-
Add PATH to the top of your crontab
PATH=/home/users/user_name/miniconda2/envs/my_env/bin
Update:
Jérôme’s answer and cbarrick’s comments are correct. I just got burned using the above approach in a Conda env which needed pynco, which needs the full conda environment to find proper the nco commands, such as ncks, ncrcat. Solved by running a bash script from cron which calls conda activate first.
In my case, I got this error when I ran this line of shell script: source activate my_env
activate: No such file or directory
So I changed source activate my_env to source /path/to/conda/bin/activate my_env. Then it starts working.
After MUCH fiddling I got crontab to activate my conda environment with conda activate my_env and run the Python interpreter within that environment.
Note I’m using Ubuntu 18.04.
Background
-
When the Anaconda installer initializes conda, it appends a snippet at the end of the ~/.bashrc file. This file is executed each time the user opens bash interactively. The snippet allows the user to run conda commands (ie conda activate my_env) from bash.
-
Anaconda installer v2020.02 appended the following conda snippet in ~/.bashrc:
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/opt/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/opt/anaconda3/etc/profile.d/conda.sh" ]; then
. "/opt/anaconda3/etc/profile.d/conda.sh"
else
export PATH="/opt/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
- The path
/opt/anaconda3/ to be replaced with the correct reference: usually /home/USERNAME/anaconda3/.
The problem
Sourcing ~/.bashrc in crontab -e won’t work (at least not on Ubuntu).
Explanation:
- On Ubuntu,
~/.bashrc has the following (or similar) line at the beginning of the file:
# If not running interactively, don't do anything
[ -z "$PS1" ] && return
- This means that if we try to source the
~/.bashrc file in crontab, the rest of the .bashrc file will not execute because crontab is not running interactively (see another post on this topic). Which means that the conda snippet mentioned above will never get executed by crontab even if we source ~/.bashrc.
_________ Working solution _________
The solution I have found is to copy the conda snippet to a separate file.
1. Copying the conda snippet from ~/.bashrc to ~/.bashrc_conda
Copy the snippet mentioned above to another file, for example ~/.bashrc_conda.
Ensure that:
- The user running the cronjob has permission to read this file.
- Other users cannot write to this file (security risk).
2. In crontab -e insert 2 lines to run bash instead of sh and to source ~/.bashrc_conda
Run crontab -e and add the following 2 lines before the cronjob:
SHELL=/bin/bash
BASH_ENV=~/.bashrc_conda
Explanation:
-
SHELL=/bin/bash means that crontab will run the cronjobs via bash instead of sh (default). See post.
-
BASH_ENV=~/.bashrc_conda sources the conda snippet to bash run by chrontab. See post and post.
3. In crontab -e insert in the cronjob line conda activate my_env; before the desired .py script execution
Example of entry for a script that would execute at noon 12:30 each day within the desired conda environment:
30 12 * * * conda activate my_env; python /path/to/script.py
Notice conda activate my_env; before the command to run the Python interpreter.
_______________
And voilà, it worked.
Any downsides?
If the conda snippet in .bashrc gets updated by a conda update, it will of course not be reflected in the separate .bashrc_conda file. One may need to check for updates from time to time.
One could also to append ; conda deactivate at the end of that cronjob, but this may be redundant.
As of May 2022, I just use a .bat in Windows 10 to activate myenv and then start my localhost or whatever script you need:
@echo off
set CONDAPATH=C:UsersMyNameanaconda3
set ENVNAME=myenv
if %ENVNAME%==base (set ENVPATH=%CONDAPATH%) else (set ENVPATH=%CONDAPATH%envs%ENVNAME%)
call %CONDAPATH%Scriptsactivate.bat %ENVPATH%
call cd /d d:/mysite
python manage.py runserver 0.0.0.0:8000
Use a full path to conda
A simple solution that worked for me was to specify a full path to conda in the crontab entry, and to use the conda run -n <env> option to execute the command in the required environment. In my case I wanted to launch a command on start up, so my crontab entry looked like this:
@reboot ~/miniconda3/bin/conda run -n <env> <command>
where <env> and <command> are substituted according to your own requirements.
This approach could be a bit simpler and more concise in some cases:
-
find the path of your conda environment
conda env list
-
add this as "PATH" on top of the crontab file:
PATH=/Users/your_user_name/opt/anaconda3/envs/your_env/bin
For more details check this
tutorial (Source)
I think conda has changed, and so none of these solutions work for me anymore, other than the one posted by @James to specify the full path to conda within crontab.
However, sometimes I want to run a full bash script from crontab, and activate a conda environment throughout that script. For that, the solution which worked was to source conda.sh, as posted by @effecor here: https://stackoverflow.com/a/65183109/1676378.
For me, this means adding the following line to the top of my bash script that is called by cron:
source /home/chris/miniconda3/etc/profile.d/conda.sh
Your conda.sh may be located somewhere else.
The underlying issue is explained here: https://github.com/conda/conda/issues/7980
As nehaljwani writes: "Functions are not exported by default to be made available in subshells."
I want to have a cron job execute a python script using an already existing anaconda python environment called my_env. The only thing I can think to do is have the cron job run a script called my_script.bash which in turn activates the env and then runs the python script.
#!/bin/bash
source activate my_env
python ~/my_project/main.py
Trying to execute this script from the command lines doesn’t work:
$ sh scripts/my_script.bash
scripts/my_script.bash: 9: scripts/my_script.bash: source: not found
What do I need to do to make sure the proper environment is activated. Its ok to explain it to me like I’m 5.
Don’t call sh but bash. source is a bash command.
- sh scripts/my_script.bash
+ bash scripts/my_script.bash
Or just
chmod +x scripts/my_script.bash
./scripts/my_script.bash
since you added the bash shebang.
I recently switched from canopy to Anaconda precisely to get away from having to activate an env in cron jobs.
Anaconda makes this very simple, based on the PATH enviornment variable. (I’m using miniconda not the full Anaconds install but I believe anaconda should work the same way)
There are two different approaches, I’ve tested;
-
Add a shebang in your python script, main.py
#!/home/users/user_name/miniconda2/envs/my_env/bin/python -
Add PATH to the top of your crontab
PATH=/home/users/user_name/miniconda2/envs/my_env/bin
Update:
Jérôme’s answer and cbarrick’s comments are correct. I just got burned using the above approach in a Conda env which needed pynco, which needs the full conda environment to find proper the nco commands, such as ncks, ncrcat. Solved by running a bash script from cron which calls conda activate first.
In my case, I got this error when I ran this line of shell script: source activate my_env
activate: No such file or directory
So I changed source activate my_env to source /path/to/conda/bin/activate my_env. Then it starts working.
After MUCH fiddling I got crontab to activate my conda environment with conda activate my_env and run the Python interpreter within that environment.
Note I’m using Ubuntu 18.04.
Background
-
When the Anaconda installer initializes conda, it appends a snippet at the end of the
~/.bashrcfile. This file is executed each time the user opensbashinteractively. The snippet allows the user to runcondacommands (ieconda activate my_env) frombash. -
Anaconda installer v2020.02 appended the following
condasnippet in~/.bashrc:
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/opt/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/opt/anaconda3/etc/profile.d/conda.sh" ]; then
. "/opt/anaconda3/etc/profile.d/conda.sh"
else
export PATH="/opt/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
- The path
/opt/anaconda3/to be replaced with the correct reference: usually/home/USERNAME/anaconda3/.
The problem
Sourcing ~/.bashrc in crontab -e won’t work (at least not on Ubuntu).
Explanation:
- On Ubuntu,
~/.bashrchas the following (or similar) line at the beginning of the file:
# If not running interactively, don't do anything
[ -z "$PS1" ] && return
- This means that if we try to source the
~/.bashrcfile incrontab, the rest of the.bashrcfile will not execute becausecrontabis not running interactively (see another post on this topic). Which means that thecondasnippet mentioned above will never get executed bycrontabeven if we source~/.bashrc.
_________ Working solution _________
The solution I have found is to copy the conda snippet to a separate file.
1. Copying the conda snippet from ~/.bashrc to ~/.bashrc_conda
Copy the snippet mentioned above to another file, for example ~/.bashrc_conda.
Ensure that:
- The user running the cronjob has permission to read this file.
- Other users cannot write to this file (security risk).
2. In crontab -e insert 2 lines to run bash instead of sh and to source ~/.bashrc_conda
Run crontab -e and add the following 2 lines before the cronjob:
SHELL=/bin/bash
BASH_ENV=~/.bashrc_conda
Explanation:
-
SHELL=/bin/bashmeans thatcrontabwill run the cronjobs viabashinstead ofsh(default). See post. -
BASH_ENV=~/.bashrc_condasources thecondasnippet tobashrun bychrontab. See post and post.
3. In crontab -e insert in the cronjob line conda activate my_env; before the desired .py script execution
Example of entry for a script that would execute at noon 12:30 each day within the desired conda environment:
30 12 * * * conda activate my_env; python /path/to/script.py
Notice conda activate my_env; before the command to run the Python interpreter.
_______________
And voilà, it worked.
Any downsides?
If the conda snippet in .bashrc gets updated by a conda update, it will of course not be reflected in the separate .bashrc_conda file. One may need to check for updates from time to time.
One could also to append ; conda deactivate at the end of that cronjob, but this may be redundant.
As of May 2022, I just use a .bat in Windows 10 to activate myenv and then start my localhost or whatever script you need:
@echo off
set CONDAPATH=C:UsersMyNameanaconda3
set ENVNAME=myenv
if %ENVNAME%==base (set ENVPATH=%CONDAPATH%) else (set ENVPATH=%CONDAPATH%envs%ENVNAME%)
call %CONDAPATH%Scriptsactivate.bat %ENVPATH%
call cd /d d:/mysite
python manage.py runserver 0.0.0.0:8000
Use a full path to conda
A simple solution that worked for me was to specify a full path to conda in the crontab entry, and to use the conda run -n <env> option to execute the command in the required environment. In my case I wanted to launch a command on start up, so my crontab entry looked like this:
@reboot ~/miniconda3/bin/conda run -n <env> <command>
where <env> and <command> are substituted according to your own requirements.
This approach could be a bit simpler and more concise in some cases:
-
find the path of your conda environment
conda env list -
add this as "PATH" on top of the crontab file:
PATH=/Users/your_user_name/opt/anaconda3/envs/your_env/bin
For more details check this
tutorial (Source)
I think conda has changed, and so none of these solutions work for me anymore, other than the one posted by @James to specify the full path to conda within crontab.
However, sometimes I want to run a full bash script from crontab, and activate a conda environment throughout that script. For that, the solution which worked was to source conda.sh, as posted by @effecor here: https://stackoverflow.com/a/65183109/1676378.
For me, this means adding the following line to the top of my bash script that is called by cron:
source /home/chris/miniconda3/etc/profile.d/conda.sh
Your conda.sh may be located somewhere else.
The underlying issue is explained here: https://github.com/conda/conda/issues/7980
As nehaljwani writes: "Functions are not exported by default to be made available in subshells."