Have Supervisord Periodically restart child processes
Question:
I am using Supervisor (3.0a12) on ubuntu 12.04 to manage php gearman workers. Sometimes the workers get caught in a weird state where they use tons of cpu and ram. While I am figuring this issue out I thought it would be nice to have Supervisor automatically kill and refresh workers occasionally. I looked at http://supervisord.org/configuration.html the configuration documentation and didn’t seem to see any options that would allow for this.
Does anyone know if it is possible to have supervisord periodically restart all processes it governs??
Answers:
The superlance package offers a memmon plugin for supervisor. memmon monitors memory usage for programs under supervisor control.
You configure memmon as a supervisor eventlistener:
[eventlistener:memmon]
command=memmon -a 200MB
events=TICK_60
The above configuration sets memmon to restart any program under supervisor control if it exceeds 200MB memory usage. It checks every 60 seconds.
You can configure memmon to monitor specific programs or program groups, setting limits for each.
You could use crontab to pass commands directly to supervisorctl. For example, the following will restart a process every 20 minutes.
0,20,40 * * * * /path/to/supervisorctl restart [supervisor_process]
There’s an easy supervisor-only solution. Make another supervisor process that sleeps for the appropriate amount of time and then restarts the processes you want.
[program:my_program]
process_name = python something.py
[program:periodicrestarter]
command = sh -c "echo 'restart my_program' | supervisorctl;sleep 600"
; restarts all (sleeps first because this restarts itself):
;command = sh -c "sleep 600; echo 'restart all' | supervisorctl"
autorestart = true
I am using Supervisor (3.0a12) on ubuntu 12.04 to manage php gearman workers. Sometimes the workers get caught in a weird state where they use tons of cpu and ram. While I am figuring this issue out I thought it would be nice to have Supervisor automatically kill and refresh workers occasionally. I looked at http://supervisord.org/configuration.html the configuration documentation and didn’t seem to see any options that would allow for this.
Does anyone know if it is possible to have supervisord periodically restart all processes it governs??
The superlance package offers a memmon plugin for supervisor. memmon monitors memory usage for programs under supervisor control.
You configure memmon as a supervisor eventlistener:
[eventlistener:memmon]
command=memmon -a 200MB
events=TICK_60
The above configuration sets memmon to restart any program under supervisor control if it exceeds 200MB memory usage. It checks every 60 seconds.
You can configure memmon to monitor specific programs or program groups, setting limits for each.
You could use crontab to pass commands directly to supervisorctl. For example, the following will restart a process every 20 minutes.
0,20,40 * * * * /path/to/supervisorctl restart [supervisor_process]
There’s an easy supervisor-only solution. Make another supervisor process that sleeps for the appropriate amount of time and then restarts the processes you want.
[program:my_program]
process_name = python something.py
[program:periodicrestarter]
command = sh -c "echo 'restart my_program' | supervisorctl;sleep 600"
; restarts all (sleeps first because this restarts itself):
;command = sh -c "sleep 600; echo 'restart all' | supervisorctl"
autorestart = true