Crontab to launch python script on raspberry pi causes import error
Question:
I am running Ubuntu Mate on my Raspberry PI. I wrote a web scraper in Python that I want to run once a day. I figured using sudo crontab -e would be a good way to do way. The problem that I’m having is when the cronjob launches my python script, the python script throws an import error and doesn’t run. However, when I execute the python script directly from the command line, it runs without issue.
I was reading that some people use a shell script to launch they python script, so I tried that as well. Again, it works as expected when I execute the shell script directly from the command line, but does not work when executed by the cronjob.
For testing purposes, right now I have the cronjob to execute every minute until I figure out what’s going on. Once it works as expected, I’ll change the time that it executes to something closer to what I want.
To give you an idea of what my files look like, check it out below. As you can see, all of the files that need to run have execute prilivages.
web_scraper.py
#!/usr/bin/env python
import click
import logging
import os
from datetime import datetime
from bs4 import BeautifulSoup as bs
import re
import urlparse
...
start.sh
#!/bin/bash
cd /home/elmer/
python web_scraper.py
sudo crontab -e
* * * * * sh /home/elmer/start.sh >> /home/elmer/cron.log 2>&1
cron.log
Traceback (most recent call last):
File "web_scraper.py", line 6, in <module>
import click
ls -lh
(py27)elmer@elmer-rpi:$ ls -lh
total 56K
-rw-rw-r-- 1 elmer elmer 2.9K Mar 17 20:38 cron.log
-rwxrwxrwx 1 elmer elmer 8.2K Mar 16 09:54 web_scraper.py
-rwxrwxrwx 1 elmer elmer 64 Mar 17 20:02 start.sh
Answers:
The problem is that you’re running your script as yourself, but then you put it in cron as root. Those two different users have very different environments.
You should just use crontab -e instead of sudo crontab -e to put it in your own account’s cron. You can also remove the sh from the start of your cron command line, as that is the default.
In case it still does not work, the solution is likely to echo $PYTHONPATH in your regular shell, then add that setting to cron:
PYTHONPATH=/your/path/here
* * * * * python /home/elmer/web_scraper.py >> /home/elmer/cron.log 2>&1
Another good reason not to use sudo crontab is that if you run your script as root, any bugs it has could ruin your system.
I don’t exactly know what causes this error, but what @John Zwinck pointed out seems to be the correct answer.
What was my problem
Similar to @CurtLH I had crontab starting my python script but raising an error message: finding no package X. In my case it was selenium. And further note, I’m running this on Raspbian OS, which is based on Debian.
Solution
First you want to find the path of your Python libraries / modules in the environment your working in. In my case, the script is located under the user directory.
For that, type python in your console, so you can execute the following:
>>> import sys
>>> sys.path
Which returns a list of things, get the fourth which should point to your site-packages. In my case, the string looked like that '/home/pi/.local/lib/python3.9/site-packages'
Then we append some things to our crontab file. By typing the following in your terminal you can open crontab:
$ sudo crontab -e
Be aware that there are two separate crontab files(as it seems to me). One which you can access by typing sudo infront of crontab the other without so(for every user I guess plus the root user). Further note that you can read the crontab file by $ (sudo) crontab -l. So when you got your sudo crontab open, by as I said sudo crontab -e, scroll down using arrow keys and create a new line at the bottom, where you add the following.
*/1 * * * * PYTHONPATH=/home/pi/.local/lib/python3.9/site-packages python /home/pi/script.py
*/1 * * * * says it should run the script every minute (just for testing)(helpful guide to this). Next we’re defining the place where it should search for the modules. The next is self-explanatory: use python to open the py-script which is located at …
If you want to document the print stuff from your script add the last bit of the following:
*/1 * * * * PYTHONPATH=/home/pi/.local/lib/python3.9/site-packages python /home/pi/script.py >> /var/log/script_output.log 2>&1
In fact, you can add any log-file that it should save to, by alternating the last directory. Be aware that when running your script for months, this file can grow substantially. Also, only insert spaces between the arguments you give (no ;) to that one line in you crontab file that does all the auto-execution of your script.
To save and close the crontab file press Ctrl + X, then Y and hit Enter and you will be back in the terminal.
Summary
I added the following to my root crontab file, accessible through $ sudo crontab -e to execute script every minute:
*/1 * * * * PYTHONPATH=/path/to/python/site-packages python /path/to/py-script.py >> /var/log/save_log_here.log 2>&1
Using Python to determine path that points to site-packages:
>>> import sys
>>> sys.path
I am running Ubuntu Mate on my Raspberry PI. I wrote a web scraper in Python that I want to run once a day. I figured using sudo crontab -e would be a good way to do way. The problem that I’m having is when the cronjob launches my python script, the python script throws an import error and doesn’t run. However, when I execute the python script directly from the command line, it runs without issue.
I was reading that some people use a shell script to launch they python script, so I tried that as well. Again, it works as expected when I execute the shell script directly from the command line, but does not work when executed by the cronjob.
For testing purposes, right now I have the cronjob to execute every minute until I figure out what’s going on. Once it works as expected, I’ll change the time that it executes to something closer to what I want.
To give you an idea of what my files look like, check it out below. As you can see, all of the files that need to run have execute prilivages.
web_scraper.py
#!/usr/bin/env python
import click
import logging
import os
from datetime import datetime
from bs4 import BeautifulSoup as bs
import re
import urlparse
...
start.sh
#!/bin/bash
cd /home/elmer/
python web_scraper.py
sudo crontab -e
* * * * * sh /home/elmer/start.sh >> /home/elmer/cron.log 2>&1
cron.log
Traceback (most recent call last):
File "web_scraper.py", line 6, in <module>
import click
ls -lh
(py27)elmer@elmer-rpi:$ ls -lh
total 56K
-rw-rw-r-- 1 elmer elmer 2.9K Mar 17 20:38 cron.log
-rwxrwxrwx 1 elmer elmer 8.2K Mar 16 09:54 web_scraper.py
-rwxrwxrwx 1 elmer elmer 64 Mar 17 20:02 start.sh
The problem is that you’re running your script as yourself, but then you put it in cron as root. Those two different users have very different environments.
You should just use crontab -e instead of sudo crontab -e to put it in your own account’s cron. You can also remove the sh from the start of your cron command line, as that is the default.
In case it still does not work, the solution is likely to echo $PYTHONPATH in your regular shell, then add that setting to cron:
PYTHONPATH=/your/path/here
* * * * * python /home/elmer/web_scraper.py >> /home/elmer/cron.log 2>&1
Another good reason not to use sudo crontab is that if you run your script as root, any bugs it has could ruin your system.
I don’t exactly know what causes this error, but what @John Zwinck pointed out seems to be the correct answer.
What was my problem
Similar to @CurtLH I had crontab starting my python script but raising an error message: finding no package X. In my case it was selenium. And further note, I’m running this on Raspbian OS, which is based on Debian.
Solution
First you want to find the path of your Python libraries / modules in the environment your working in. In my case, the script is located under the user directory.
For that, type python in your console, so you can execute the following:
>>> import sys
>>> sys.path
Which returns a list of things, get the fourth which should point to your site-packages. In my case, the string looked like that '/home/pi/.local/lib/python3.9/site-packages'
Then we append some things to our crontab file. By typing the following in your terminal you can open crontab:
$ sudo crontab -e
Be aware that there are two separate crontab files(as it seems to me). One which you can access by typing sudo infront of crontab the other without so(for every user I guess plus the root user). Further note that you can read the crontab file by $ (sudo) crontab -l. So when you got your sudo crontab open, by as I said sudo crontab -e, scroll down using arrow keys and create a new line at the bottom, where you add the following.
*/1 * * * * PYTHONPATH=/home/pi/.local/lib/python3.9/site-packages python /home/pi/script.py
*/1 * * * * says it should run the script every minute (just for testing)(helpful guide to this). Next we’re defining the place where it should search for the modules. The next is self-explanatory: use python to open the py-script which is located at …
If you want to document the print stuff from your script add the last bit of the following:
*/1 * * * * PYTHONPATH=/home/pi/.local/lib/python3.9/site-packages python /home/pi/script.py >> /var/log/script_output.log 2>&1
In fact, you can add any log-file that it should save to, by alternating the last directory. Be aware that when running your script for months, this file can grow substantially. Also, only insert spaces between the arguments you give (no ;) to that one line in you crontab file that does all the auto-execution of your script.
To save and close the crontab file press Ctrl + X, then Y and hit Enter and you will be back in the terminal.
Summary
I added the following to my root crontab file, accessible through $ sudo crontab -e to execute script every minute:
*/1 * * * * PYTHONPATH=/path/to/python/site-packages python /path/to/py-script.py >> /var/log/save_log_here.log 2>&1
Using Python to determine path that points to site-packages:
>>> import sys
>>> sys.path