Scheduled python code can't find module when run in Docker
Question:
I’m trying to run a python script inside a Docker container every 2 minutes. To achieve this, I am using a cron job. When I run the Python script on the host machine, it executes perfectly, but inside the docker container I keep getting
ModuleNotFoundError: No module named 'requests'
The container builds perfectly fine, and the requirements are installed correctly according to the build log. My Dockerfile and cron job are as follows:
Dockerfile
FROM python:latest
WORKDIR /root
COPY *.py .
COPY requirements.txt .
COPY cronjob.txt .
RUN pip install -r requirements.txt
RUN apt update
RUN apt install cron -y
RUN crontab -l | { cat; cat cronjob.txt; } | crontab -
ENV PYTHONPATH "$PYTHONPATH:/usr/local/lib/python3.11/site-packages"
CMD cron -f
cronjob.txt
*/2 * * * * python3 runSync.py > /proc/1/fd/1 2>/proc/1/fd/2
runSync.py
import requests
def run_sync(network, ips, port, endpoint):
for ip in ips:
out = ""
try:
url = "http://" + network.format(ip=ip, port=port) + endpoint
print("Resolving " + url + "...")
res = requests.get(url)
if res.status_code != 200:
raise ValueError("Unexpected status code: {code}".format(code=res.status_code))
print(res)
print("Success!")
# TODO: Better error handling
except requests.exceptions.Timeout:
out = "Connection to server timed out."
break
except requests.exceptions.ConnectionError:
out = "Unable to connect to server, please check internet connection."
break
except requests.exceptions.InvalidURL:
out = "Malformed URL ", url, " please check and try again."
break
except KeyboardInterrupt:
out = "Stopping..."
break
except ValueError as e:
out = "Encountered error: "{error}"".format(error=e.args[0])
break
if out != "":
return out
else:
return "Success"
if __name__ == '__main__':
defaultNetwork = "192.168.33.{ip:n}:{port:n}"
defaultPort = 8080
defaultEndpoint = "/api/system/sync"
defaultIPs = [29, 28, 12]
out = run_sync(defaultNetwork, defaultIPs, defaultPort, defaultEndpoint)
if out != "Success":
print(out)
exit(1)
else:
exit(0)
requirements.txt
requests==2.28.1
I’ve tried manually setting the PYTHONPATH (as seen in Dockerfile), as well as changing the application root to a couple of different options. I should be seeing the output of runSync.py in the docker logs, but instead I just get ModuleNotFoundError: No module named 'requests'
.
Answers:
The python:latest
image comes with both Python 3.9 (because Debian) and 3.11, and cron
environment seems to use the former. Specifying the full path to the python
executable in the cronjob.txt
should fix it:
*/2 * * * * /usr/local/bin/python3 runSync.py > /proc/1/fd/1 2>/proc/1/fd/2
or
*/2 * * * * /usr/local/bin/python3.11 runSync.py > /proc/1/fd/1 2>/proc/1/fd/2
You don’t need to mess with the PYTHONPATH
environment in the Dockerfile
after this change.
There’s no need for using cron
Try apscheduler, it’s a python module for shceduling tasks:
https://pypi.org/project/APScheduler/
short example:
from apscheduler.schedulers.blocking import BlockingScheduler
def some_job():
print("Doing something...")
scheduler = BlockingScheduler()
scheduler.add_job(some_job,"interval",minutes=2)
try:
scheduler.start()
except (KeyboardInterrupt, SystemExit):
pass
If you look at /etc/crontab
, it has :
PATH=/usr/local/bin:/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin
You can put above setting in your cronjob.txt, then you don’t have to use absolute path for python3.
Also, above setting make sure the PATH is the same on the command line and in crontab, this way it avoids problems for other commands as well.
I’m trying to run a python script inside a Docker container every 2 minutes. To achieve this, I am using a cron job. When I run the Python script on the host machine, it executes perfectly, but inside the docker container I keep getting
ModuleNotFoundError: No module named 'requests'
The container builds perfectly fine, and the requirements are installed correctly according to the build log. My Dockerfile and cron job are as follows:
Dockerfile
FROM python:latest
WORKDIR /root
COPY *.py .
COPY requirements.txt .
COPY cronjob.txt .
RUN pip install -r requirements.txt
RUN apt update
RUN apt install cron -y
RUN crontab -l | { cat; cat cronjob.txt; } | crontab -
ENV PYTHONPATH "$PYTHONPATH:/usr/local/lib/python3.11/site-packages"
CMD cron -f
cronjob.txt
*/2 * * * * python3 runSync.py > /proc/1/fd/1 2>/proc/1/fd/2
runSync.py
import requests
def run_sync(network, ips, port, endpoint):
for ip in ips:
out = ""
try:
url = "http://" + network.format(ip=ip, port=port) + endpoint
print("Resolving " + url + "...")
res = requests.get(url)
if res.status_code != 200:
raise ValueError("Unexpected status code: {code}".format(code=res.status_code))
print(res)
print("Success!")
# TODO: Better error handling
except requests.exceptions.Timeout:
out = "Connection to server timed out."
break
except requests.exceptions.ConnectionError:
out = "Unable to connect to server, please check internet connection."
break
except requests.exceptions.InvalidURL:
out = "Malformed URL ", url, " please check and try again."
break
except KeyboardInterrupt:
out = "Stopping..."
break
except ValueError as e:
out = "Encountered error: "{error}"".format(error=e.args[0])
break
if out != "":
return out
else:
return "Success"
if __name__ == '__main__':
defaultNetwork = "192.168.33.{ip:n}:{port:n}"
defaultPort = 8080
defaultEndpoint = "/api/system/sync"
defaultIPs = [29, 28, 12]
out = run_sync(defaultNetwork, defaultIPs, defaultPort, defaultEndpoint)
if out != "Success":
print(out)
exit(1)
else:
exit(0)
requirements.txt
requests==2.28.1
I’ve tried manually setting the PYTHONPATH (as seen in Dockerfile), as well as changing the application root to a couple of different options. I should be seeing the output of runSync.py in the docker logs, but instead I just get ModuleNotFoundError: No module named 'requests'
.
The python:latest
image comes with both Python 3.9 (because Debian) and 3.11, and cron
environment seems to use the former. Specifying the full path to the python
executable in the cronjob.txt
should fix it:
*/2 * * * * /usr/local/bin/python3 runSync.py > /proc/1/fd/1 2>/proc/1/fd/2
or
*/2 * * * * /usr/local/bin/python3.11 runSync.py > /proc/1/fd/1 2>/proc/1/fd/2
You don’t need to mess with the PYTHONPATH
environment in the Dockerfile
after this change.
There’s no need for using cron
Try apscheduler, it’s a python module for shceduling tasks:
https://pypi.org/project/APScheduler/
short example:
from apscheduler.schedulers.blocking import BlockingScheduler
def some_job():
print("Doing something...")
scheduler = BlockingScheduler()
scheduler.add_job(some_job,"interval",minutes=2)
try:
scheduler.start()
except (KeyboardInterrupt, SystemExit):
pass
If you look at /etc/crontab
, it has :
PATH=/usr/local/bin:/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin
You can put above setting in your cronjob.txt, then you don’t have to use absolute path for python3.
Also, above setting make sure the PATH is the same on the command line and in crontab, this way it avoids problems for other commands as well.