Jesse's Software Engineering Blog
Process Monitoring with SupervisorD
The ability to monitor services on a web server is crucial for helping to ensure high availability. Another common process is to have daemon scripts which are designed to continually run for things such as a monitoring job queue. Being able to help prevent failure for such processes is extremely important.
SupervisorD is a daemon program which handles the monitoring of any crucial services or ever running scripts at the process level. By controlling the instantiation of processes supervisor can ensure those processes are continually running and restart them when something goes awry. Another great feature of supervisor is the ability to control which processes are started on server boot, to prevent server restarts from wreaking havoc on an application stack.
It’s important to note that when setting up scripts to be managed by supervisor that they are written to continually run in the foreground i.e. with a infinite loop, etc. Otherwise, if scripts only need to be executed on a time interval using cron would likely be a better fit.
To install supervisor
sudo service supervisor install # treat like a service sudo service supervisor status sudo service supervisor start
While the Linux OS typically manages the starting and stopping of services on reboot, it can be beneficial to disable that functionality so that the Linux OS and supervisor are not competing to start the same process. To see all services and change their auto start properties (on Ubuntu)
service --status-all # enable / disable update-rc.d <service-name> defaults update-rc.d -f <service-name> remove # sometimes needed with upstart services echo "manual" | sudo tee -a /etc/init/php5-fpm.override
To ensure supervisor boots on startup
update-rc.d supervisor defaults
The main supervisor conf file simply sets up some of the socket values and then includes all the conf file from the conf.d directory, therefore all custom configs are placed in conf.d as a .conf file
If you have some script, called daemon.js which we needs to always be running, simply create a config file for it
# /etc/supervisor/conf.d/daemon.conf [program:daemon] command=nodejs daemon.js directory=/opt autostart=true autorestart=true startretries=3 stderr_logfile=/var/log/supervisor.daemon.err.log stdout_logfile=/var/log/supervisor.daemon.out.log user=www-data environment=PARAMETER1='this is a parameter,PARAMETER2='another parameter
- program – notifies supervisor the the program “daemon” needs to be controlled
- directory – directory supervisor should cd to before running command
- user – user to run the process as
- environment – parameters to pass to the process
Now that config file has been written it can be loaded in supervisor and monitored.
Interaction with supervisor typically is done via a command line tool, supervisorctl.
# reload config files supervisorctl reread # activate new config files supervisorctl update # see status of monitored processes supervisorctl status
Once the processes are managed by supervisor they can be managed like an Ubuntu process
supervisorctl start <process-name> supervisorctl stop <process-name> supervisorctl restart <process-name>
One last convenience of supervisor is there is an optional web UI which prevents having to SSH into a server to see process statuses. Simply add the following to the main supervisor config file. It’s important to prevent access to this page however as it shows potentially sensitive information, as well as allows for the starting/stopping of internal processes of a server.
# /etc/supervisor/supervisord.conf [inet_http_server] port = 9001 username = user password = pass