Run a python script with supervisor
Question:
I copied from here to run my Python code as a daemon.
For extra uptime. I thought it would be a better Idea to use supervisor to keep this daemon running.
I did this.
python_deamon.conf
[program:python_deamon]
directory=/usr/local/python_deamon/
command=/usr/local/python_venv/bin/python daemon_runnner.py start
stderr_logfile=/var/log/gunicorn.log
stdout_logfile=/var/log/gunicorn.log
autostart=true
autorestart=true
The problem is that though supervisor successfully starts the python_daemon it keeps retrying.
2015-09-23 16:10:45,592 CRIT Supervisor running as root (no user in config file)
2015-09-23 16:10:45,592 WARN Included extra file "/etc/supervisor/conf.d/python_daemon.conf" during parsing
2015-09-23 16:10:45,592 INFO RPC interface 'supervisor' initialized
2015-09-23 16:10:45,592 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2015-09-23 16:10:45,592 INFO supervisord started with pid 13880
2015-09-23 16:10:46,595 INFO spawned: 'python_deamon' with pid 17884
2015-09-23 16:10:46,611 INFO exited: python_deamon (exit status 1; not expected)
2015-09-23 16:10:47,614 INFO spawned: 'python_deamon' with pid 17885
2015-09-23 16:10:47,630 INFO exited: python_deamon (exit status 1; not expected)
2015-09-23 16:10:49,635 INFO spawned: 'python_deamon' with pid 17888
2015-09-23 16:10:49,656 INFO exited: python_deamon (exit status 1; not expected)
2015-09-23 16:10:52,662 INFO spawned: 'python_deamon' with pid 17891
2015-09-23 16:10:52,680 INFO exited: python_deamon (exit status 1; not expected)
2015-09-23 16:10:53,681 INFO gave up: python_deamon entered FATAL state, too many start retries too quickly
Just for the record the after overriding run() method I never return anything.
Is it possible to do what I am trying to do or am I being dumb ?
P.S: I know that the root cause of the whole problem is that since run() never return anything supervisor keeps trying to start it and hence thinks that the process failed and gives the status as FATAL Exited too quickly (process log may have details).
My actual question is am I doing it right ? or can this be done this way ?
P.P.S: Stand alone(without supervisor) daemon_runnner.py runs fine with and without sudo permissions.
Answers:
Your script is failing with an exit status. Supervisor is simply trying to restart the script.
Supervisor is started with root permissions, perhaps it is giving those permissions to your script and this is causing it to fail (a change in the source directory or something). Check what happens when you run your daemon as root without Supervisor.
We really need more information to know why it is failing.
Not sure if the issue is the same with daemon runner, but if you use the daemon context directly and use supervisord, you need to set context.detach_process to False
try to set startsecs = 0:
[program:foo]
command = ls
startsecs = 0
autorestart = false
http://supervisord.org/configuration.html
startsecs
The total number of seconds which the program needs to stay running after a startup to consider the start successful. If the program does not stay up for this many seconds after it has started, even if it exits with an “expected” exit code (see exitcodes), the startup will be considered a failure. Set to 0 to indicate that the program needn’t stay running for any particular amount of time.
This is what I normally do:
- Create a
service.conf file which describes the new Python script. This script references the shell script which is the one in reality launching the Python script. This .conf file lives inside /etc/supervisor/conf.d/
- Create a shell script which launches the Python script. Change permissions to executable.
chmod 755 service.sh. In this script we actually launch the Python script.
- Configure log_stderr and stderr_logfile to verify issue.
- Update supervisor using reload and then check status:
supervisor> status
alexad RUNNING pid 32657, uptime 0:21:05
service.conf
[program:alexad]
; Set full path to celery program if using virtualenv
command=sh /usr/local/src/gonzo/supervisorctl/alexad.sh
directory=/usr/local/src/gonzo/services/alexa
log_stdout=true ; if true, log program stdout (default true)
log_stderr=true ; if true, log program stderr (default false)
stderr_logfile=/usr/local/src/gonzo/log/alexad.err
logfile=/usr/local/src/gonzo/log/alexad.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; Set Celery priority higher than default (999)
priority=500
service.sh
#!/bin/bash
cd /usr/local/src/gonzo/services/alexa
exec python reader.py
You should check your supervisor program logs which live here normally /var/log/supervisor/
In my case, I had a ModuleNotFoundError: No module named 'somemodule'
I found this odd cause when I ran the script directly it worked fine.
After attempting to run the script with sudo I realized what was happening.
Python imports are specific to the user, not to the folder. So if you python or pip install with your current logged-in user and try running the same script as sudo or some other user it will probably return ModuleNotFoundError: No module named 'somemodule'
Supervisor by default runs as root
I solved this by setting the user in the supervisor config file to the current user which in my case ubuntu:
[program:some_program]
directory=/home/ubuntu/scripts/
command=/usr/bin/python3 myscript.py
autostart=true
autorestart=true
user=ubuntu
stderr_logfile=/var/log/supervisor/myscriptlogs.err.log
stdout_logfile=/var/log/supervisor/myscriptlogs.out.log
As a side note, it’s also important to make sure your Supervisor command= is calling the script from the version of Python you intend.
cd to the folder where your script lives and run which python and use that
I copied from here to run my Python code as a daemon.
For extra uptime. I thought it would be a better Idea to use supervisor to keep this daemon running.
I did this.
python_deamon.conf
[program:python_deamon]
directory=/usr/local/python_deamon/
command=/usr/local/python_venv/bin/python daemon_runnner.py start
stderr_logfile=/var/log/gunicorn.log
stdout_logfile=/var/log/gunicorn.log
autostart=true
autorestart=true
The problem is that though supervisor successfully starts the python_daemon it keeps retrying.
2015-09-23 16:10:45,592 CRIT Supervisor running as root (no user in config file)
2015-09-23 16:10:45,592 WARN Included extra file "/etc/supervisor/conf.d/python_daemon.conf" during parsing
2015-09-23 16:10:45,592 INFO RPC interface 'supervisor' initialized
2015-09-23 16:10:45,592 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2015-09-23 16:10:45,592 INFO supervisord started with pid 13880
2015-09-23 16:10:46,595 INFO spawned: 'python_deamon' with pid 17884
2015-09-23 16:10:46,611 INFO exited: python_deamon (exit status 1; not expected)
2015-09-23 16:10:47,614 INFO spawned: 'python_deamon' with pid 17885
2015-09-23 16:10:47,630 INFO exited: python_deamon (exit status 1; not expected)
2015-09-23 16:10:49,635 INFO spawned: 'python_deamon' with pid 17888
2015-09-23 16:10:49,656 INFO exited: python_deamon (exit status 1; not expected)
2015-09-23 16:10:52,662 INFO spawned: 'python_deamon' with pid 17891
2015-09-23 16:10:52,680 INFO exited: python_deamon (exit status 1; not expected)
2015-09-23 16:10:53,681 INFO gave up: python_deamon entered FATAL state, too many start retries too quickly
Just for the record the after overriding run() method I never return anything.
Is it possible to do what I am trying to do or am I being dumb ?
P.S: I know that the root cause of the whole problem is that since run() never return anything supervisor keeps trying to start it and hence thinks that the process failed and gives the status as FATAL Exited too quickly (process log may have details).
My actual question is am I doing it right ? or can this be done this way ?
P.P.S: Stand alone(without supervisor) daemon_runnner.py runs fine with and without sudo permissions.
Your script is failing with an exit status. Supervisor is simply trying to restart the script.
Supervisor is started with root permissions, perhaps it is giving those permissions to your script and this is causing it to fail (a change in the source directory or something). Check what happens when you run your daemon as root without Supervisor.
We really need more information to know why it is failing.
Not sure if the issue is the same with daemon runner, but if you use the daemon context directly and use supervisord, you need to set context.detach_process to False
try to set startsecs = 0:
[program:foo]
command = ls
startsecs = 0
autorestart = false
http://supervisord.org/configuration.html
startsecs
The total number of seconds which the program needs to stay running after a startup to consider the start successful. If the program does not stay up for this many seconds after it has started, even if it exits with an “expected” exit code (see exitcodes), the startup will be considered a failure. Set to 0 to indicate that the program needn’t stay running for any particular amount of time.
This is what I normally do:
- Create a
service.conffile which describes the new Python script. This script references the shell script which is the one in reality launching the Python script. This.conffile lives inside/etc/supervisor/conf.d/ - Create a shell script which launches the Python script. Change permissions to executable.
chmod 755 service.sh. In this script we actually launch the Python script. - Configure log_stderr and stderr_logfile to verify issue.
- Update supervisor using reload and then check status:
supervisor> status
alexad RUNNING pid 32657, uptime 0:21:05
service.conf
[program:alexad]
; Set full path to celery program if using virtualenv
command=sh /usr/local/src/gonzo/supervisorctl/alexad.sh
directory=/usr/local/src/gonzo/services/alexa
log_stdout=true ; if true, log program stdout (default true)
log_stderr=true ; if true, log program stderr (default false)
stderr_logfile=/usr/local/src/gonzo/log/alexad.err
logfile=/usr/local/src/gonzo/log/alexad.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; Set Celery priority higher than default (999)
priority=500
service.sh
#!/bin/bash
cd /usr/local/src/gonzo/services/alexa
exec python reader.py
You should check your supervisor program logs which live here normally /var/log/supervisor/
In my case, I had a ModuleNotFoundError: No module named 'somemodule'
I found this odd cause when I ran the script directly it worked fine.
After attempting to run the script with sudo I realized what was happening.
Python imports are specific to the user, not to the folder. So if you python or pip install with your current logged-in user and try running the same script as sudo or some other user it will probably return ModuleNotFoundError: No module named 'somemodule'
Supervisor by default runs as root
I solved this by setting the user in the supervisor config file to the current user which in my case ubuntu:
[program:some_program]
directory=/home/ubuntu/scripts/
command=/usr/bin/python3 myscript.py
autostart=true
autorestart=true
user=ubuntu
stderr_logfile=/var/log/supervisor/myscriptlogs.err.log
stdout_logfile=/var/log/supervisor/myscriptlogs.out.log
As a side note, it’s also important to make sure your Supervisor command= is calling the script from the version of Python you intend.
cd to the folder where your script lives and run which python and use that