Skip to content

Jesses Software Engineering Blog

Aug 11

Jesse

Process Monitoring with SupervisorD

The ability to monitor services on a web server is crucial for helping to ensure high availability. Another common process is to have daemon scripts which are designed to continually run for things such as a monitoring job queue. Being able to help prevent failure for such processes is extremely important.

SupervisorD

SupervisorD is a daemon program which handles the monitoring of any crucial services or ever running scripts at the process level. By controlling the instantiation of processes supervisor can ensure those processes are continually running and restart them when something goes awry. Another great feature of supervisor is the ability to control which processes are started on server boot, to prevent server restarts from wreaking havoc on an application stack.

It’s important to note that when setting up scripts to be managed by supervisor that they are written to continually run in the foreground i.e. with a infinite loop, etc. Otherwise, if scripts only need to be executed on a time interval using cron would likely be a better fit.

Set Up

To install supervisor

sudo service supervisor install

# treat like a service
sudo service supervisor status
sudo service supervisor start

While the Linux OS typically manages the starting and stopping of services on reboot, it can be beneficial to disable that functionality so that the Linux OS and supervisor are not competing to start the same process. To see all services and change their auto start properties (on Ubuntu)

service --status-all

# enable / disable
update-rc.d <service-name> defaults
update-rc.d -f <service-name> remove

# sometimes needed with upstart services
echo "manual" | sudo tee -a /etc/init/php5-fpm.override

To ensure supervisor boots on startup

update-rc.d supervisor defaults

Configuration

The main supervisor conf file simply sets up some of the socket values and then includes all the conf file from the conf.d directory, therefore all custom configs are placed in conf.d as a .conf file

/etc/supervisor/supervisord.conf
/etc/supervisor/conf.d

If you have some script, called daemon.js which we needs to always be running, simply create a config file for it

# /etc/supervisor/conf.d/daemon.conf
[program:daemon]
command=nodejs daemon.js
directory=/opt
autostart=true
autorestart=true
startretries=3
stderr_logfile=/var/log/supervisor.daemon.err.log
stdout_logfile=/var/log/supervisor.daemon.out.log
user=www-data
environment=PARAMETER1='this is a parameter,PARAMETER2='another parameter
  • program – notifies supervisor the the program “daemon” needs to be controlled
  • directory – directory supervisor should cd to before running command
  • user – user to run the process as
  • environment – parameters to pass to the process

Now that config file has been written it can be loaded in supervisor and monitored.

Usage

Interaction with supervisor typically is done via a command line tool, supervisorctl.

# reload config files
supervisorctl reread
# activate new config files
supervisorctl update
# see status of monitored processes
supervisorctl status

Once the processes are managed by supervisor they can be managed like an Ubuntu process

supervisorctl start <process-name>
supervisorctl stop <process-name>
supervisorctl restart <process-name>

Web UI

One last convenience of supervisor is there is an optional web UI which prevents having to SSH into a server to see process statuses. Simply add the following to the main supervisor config file. It’s important to prevent access to this page however as it shows potentially sensitive information, as well as allows for the starting/stopping of internal processes of a server.

# /etc/supervisor/supervisord.conf
[inet_http_server]
port = 9001
username = user 
password = pass 

Blog Powered By Wordpress