Provided by: supervisor_4.2.5-3_all 
NAME
supervisor - Supervisor Documentation
Supervisor is a client/server system that allows its users to monitor and control a number of processes
on UNIX-like operating systems.
It shares some of the same goals of programs like launchd, daemontools, and runit. Unlike some of these
programs, it is not meant to be run as a substitute for init as "process id 1". Instead it is meant to be
used to control processes related to a project or a customer, and is meant to start like any other
program at boot time.
NARRATIVE DOCUMENTATION
Introduction
Overview
Supervisor is a client/server system that allows its users to control a number of processes on UNIX-like
operating systems. It was inspired by the following:
Convenience
It is often inconvenient to need to write rc.d scripts for every single process instance. rc.d
scripts are a great lowest-common-denominator form of process initialization/autostart/management, but
they can be painful to write and maintain. Additionally, rc.d scripts cannot automatically restart a
crashed process and many programs do not restart themselves properly on a crash. Supervisord starts
processes as its subprocesses, and can be configured to automatically restart them on a crash. It can
also automatically be configured to start processes on its own invocation.
Accuracy
It's often difficult to get accurate up/down status on processes on UNIX. Pidfiles often lie.
Supervisord starts processes as subprocesses, so it always knows the true up/down status of its
children and can be queried conveniently for this data.
Delegation
Users who need to control process state often need only to do that. They don't want or need
full-blown shell access to the machine on which the processes are running. Processes which listen on
"low" TCP ports often need to be started and restarted as the root user (a UNIX misfeature). It's
usually the case that it's perfectly fine to allow "normal" people to stop or restart such a process,
but providing them with shell access is often impractical, and providing them with root access or sudo
access is often impossible. It's also (rightly) difficult to explain to them why this problem exists.
If supervisord is started as root, it is possible to allow "normal" users to control such processes
without needing to explain the intricacies of the problem to them. Supervisorctl allows a very
limited form of access to the machine, essentially allowing users to see process status and control
supervisord-controlled subprocesses by emitting "stop", "start", and "restart" commands from a simple
shell or web UI.
Process Groups
Processes often need to be started and stopped in groups, sometimes even in a "priority order". It's
often difficult to explain to people how to do this. Supervisor allows you to assign priorities to
processes, and allows user to emit commands via the supervisorctl client like "start all", and
"restart all", which starts them in the preassigned priority order. Additionally, processes can be
grouped into "process groups" and a set of logically related processes can be stopped and started as a
unit.
Features
Simple
Supervisor is configured through a simple INI-style config file that’s easy to learn. It provides many
per-process options that make your life easier like restarting failed processes and automatic log
rotation.
Centralized
Supervisor provides you with one place to start, stop, and monitor your processes. Processes can be
controlled individually or in groups. You can configure Supervisor to provide a local or remote
command line and web interface.
Efficient
Supervisor starts its subprocesses via fork/exec and subprocesses don’t daemonize. The operating
system signals Supervisor immediately when a process terminates, unlike some solutions that rely on
troublesome PID files and periodic polling to restart failed processes.
Extensible
Supervisor has a simple event notification protocol that programs written in any language can use to
monitor it, and an XML-RPC interface for control. It is also built with extension points that can be
leveraged by Python developers.
Compatible
Supervisor works on just about everything except for Windows. It is tested and supported on Linux, Mac
OS X, Solaris, and FreeBSD. It is written entirely in Python, so installation does not require a C
compiler.
Proven
While Supervisor is very actively developed today, it is not new software. Supervisor has been around
for years and is already in use on many servers.
Supervisor Components
supervisord
The server piece of supervisor is named supervisord. It is responsible for starting child programs at
its own invocation, responding to commands from clients, restarting crashed or exited subprocesseses,
logging its subprocess stdout and stderr output, and generating and handling "events" corresponding to
points in subprocess lifetimes.
The server process uses a configuration file. This is typically located in /etc/supervisord.conf.
This configuration file is a "Windows-INI" style config file. It is important to keep this file
secure via proper filesystem permissions because it may contain unencrypted usernames and passwords.
supervisorctl
The command-line client piece of the supervisor is named supervisorctl. It provides a shell-like
interface to the features provided by supervisord. From supervisorctl, a user can connect to
different supervisord processes, get status on the subprocesses controlled by, stop and start
subprocesses of, and get lists of running processes of a supervisord.
The command-line client talks to the server across a UNIX domain socket or an internet (TCP) socket.
The server can assert that the user of a client should present authentication credentials before it
allows him to perform commands. The client process typically uses the same configuration file as the
server but any configuration file with a [supervisorctl] section in it will work.
Web Server
A (sparse) web user interface with functionality comparable to supervisorctl may be accessed via a
browser if you start supervisord against an internet socket. Visit the server URL (e.g.
http://localhost:9001/) to view and control process status through the web interface after activating
the configuration file's [inet_http_server] section.
XML-RPC Interface
The same HTTP server which serves the web UI serves up an XML-RPC interface that can be used to
interrogate and control supervisor and the programs it runs. See xml_rpc.
Platform Requirements
Supervisor has been tested and is known to run on Linux (Ubuntu 9.10), Mac OS X (10.4/10.5/10.6), and
Solaris (10 for Intel) and FreeBSD 6.1. It will likely work fine on most UNIX systems.
Supervisor will not run at all under any version of Windows.
Supervisor is known to work with Python 2.4 or later but will not work under any version of Python 3.
Running Supervisor
This section makes reference to a BINDIR when explaining how to run the supervisord and supervisorctl
commands. This is the "bindir" directory that your Python installation has been configured with. For
example, for an installation of Python installed via ./configure --prefix=/usr/local/py; make; make
install, BINDIR would be /usr/local/py/bin. Python interpreters on different platforms use a different
BINDIR. Look at the output of setup.py install if you can't figure out where yours is.
Adding a Program
Before supervisord will do anything useful for you, you'll need to add at least one program section to
its configuration. The program section will define a program that is run and managed when you invoke the
supervisord command. To add a program, you'll need to edit the supervisord.conf file.
One of the simplest possible programs to run is the UNIX cat program. A program section that will run
cat when the supervisord process starts up is shown below.
[program:foo]
command=/bin/cat
This stanza may be cut and pasted into the supervisord.conf file. This is the simplest possible program
configuration, because it only names a command. Program configuration sections have many other
configuration options which aren't shown here. See programx_section for more information.
Running supervisord
To start supervisord, run $BINDIR/supervisord. The resulting process will daemonize itself and detach
from the terminal. It keeps an operations log at $CWD/supervisor.log by default.
You may start the supervisord executable in the foreground by passing the -n flag on its command line.
This is useful to debug startup problems.
WARNING:
When supervisord starts up, it will search for its configuration file in default locations including
the current working directory. If you are security-conscious you will probably want to specify a "-c"
argument after the supervisord command specifying an absolute path to a configuration file to ensure
that someone doesn't trick you into running supervisor from within a directory that contains a rogue
supervisord.conf file. A warning is emitted when supervisor is started as root without this -c
argument.
To change the set of programs controlled by supervisord, edit the supervisord.conf file and kill -HUP or
otherwise restart the supervisord process. This file has several example program definitions.
The supervisord command accepts a number of command-line options. Each of these command line options
overrides any equivalent value in the configuration file.
supervisord Command-Line Options
-c FILE, --configuration=FILE
The path to a supervisord configuration file.
-n, --nodaemon
Run supervisord in the foreground.
-h, --help
Show supervisord command help.
-u USER, --user=USER
UNIX username or numeric user id. If supervisord is started as the root user, setuid to this user
as soon as possible during startup.
-m OCTAL, --umask=OCTAL
Octal number (e.g. 022) representing the umask that should be used by supervisord after it starts.
-d PATH, --directory=PATH
When supervisord is run as a daemon, cd to this directory before daemonizing.
-l FILE, --logfile=FILE
Filename path to use as the supervisord activity log.
-y BYTES, --logfile_maxbytes=BYTES
Max size of the supervisord activity log file before a rotation occurs. The value is
suffix-multiplied, e.g "1" is one byte, "1MB" is 1 megabyte, "1GB" is 1 gigabyte.
-y NUM, --logfile_backups=NUM
Number of backup copies of the supervisord activity log to keep around. Each logfile will be of
size logfile_maxbytes.
-e LEVEL, --loglevel=LEVEL
The logging level at which supervisor should write to the activity log. Valid levels are trace,
debug, info, warn, error, and critical.
-j FILE, --pidfile=FILE
The filename to which supervisord should write its pid file.
-i STRING, --identifier=STRING
Arbitrary string identifier exposed by various client UIs for this instance of supervisor.
-q PATH, --childlogdir=PATH
A path to a directory (it must already exist) where supervisor will write its AUTO -mode child
process logs.
-k, --nocleanup
Prevent supervisord from performing cleanup (removal of old AUTO process log files) at startup.
-a NUM, --minfds=NUM
The minimum number of file descriptors that must be available to the supervisord process before it
will start successfully.
-t, --strip_ansi
Strip ANSI escape sequences from all child log process.
-v, --version
Print the supervisord version number out to stdout and exit.
--profile_options=LIST
Comma-separated options list for profiling. Causes supervisord to run under a profiler, and
output results based on the options, which is a comma-separated list of the following: cumulative,
calls, callers. E.g. cumulative,callers.
--minprocs=NUM
The minimum number of OS process slots that must be available to the supervisord process before it
will start successfully.
supervisorctl Command-Line Options
-c, --configuration
Configuration file path (default /etc/supervisord.conf)
-h, --help
Print usage message and exit
-i, --interactive
Start an interactive shell after executing commands
-s,--serverurl URL
URL on which supervisord server is listening (default "http://localhost:9001").
-u, --username
Username to use for authentication with server
-p, --password
Password to use for authentication with server
-r, --history-file
Keep a readline history (if readline is available)
action [arguments]
Actions are commands like "tail" or "stop". If -i is specified or no action is specified on the command
line, a "shell" interpreting actions typed interactively is started. Use the action "help" to find out
about available actions.
Running supervisorctl
To start supervisorctl, run $BINDIR/supervisorctl. A shell will be presented that will allow you to
control the processes that are currently managed by supervisord. Type "help" at the prompt to get
information about the supported commands.
The supervisorctl executable may be invoked with "one time" commands when invoked with arguments from a
command line. An example: supervisorctl stop all. If arguments are present on the command-line, it will
prevent the interactive shell from being invoked. Instead, the command will be executed and
supervisorctl will exit.
If supervisorctl is invoked in interactive mode against a supervisord that requires authentication, you
will be asked for authentication credentials.
Signals
The supervisord program may be sent signals which cause it to perform certain actions while it's running.
You can send any of these signals to the single supervisord process id. This process id can be found in
the file represented by the pidfile parameter in the [supervisord] section of the configuration file (by
default it's $CWD/supervisord.pid).
Signal Handlers
SIGTERM
supervisord and all its subprocesses will shut down. This may take several seconds.
SIGINT
supervisord and all its subprocesses will shut down. This may take several seconds.
SIGQUIT
supervisord and all its subprocesses will shut down. This may take several seconds.
SIGHUP
supervisord will stop all processes, reload the configuration from the first config file it finds, and
restart all processes.
SIGUSR2
supervisord will close and reopen the main activity log and all child log files.
Runtime Security
The developers have done their best to assure that use of a supervisord process running as root cannot
lead to unintended privilege escalation. But caveat emptor. Supervisor is not as paranoid as something
like DJ Bernstein's daemontools, inasmuch as supervisord allows for arbitrary path specifications in its
configuration file to which data may be written. Allowing arbitrary path selections can create
vulnerabilities from symlink attacks. Be careful when specifying paths in your configuration. Ensure
that the supervisord configuration file cannot be read from or written to by unprivileged users and that
all files installed by the supervisor package have "sane" file permission protection settings.
Additionally, ensure that your PYTHONPATH is sane and that all Python standard library files have
adequate file permission protections.
Running supervisord automatically on startup
If you are using a distribution-packaged version of Supervisor, it should already be integrated into the
service management infrastructure of your distribution.
There are user-contributed scripts for various operating systems at:
https://github.com/Supervisor/initscripts
There are some answers at Serverfault in case you get stuck: How to automatically start supervisord on
Linux (Ubuntu)
Configuration File
The Supervisor configuration file is conventionally named supervisord.conf. It is used by both
supervisord and supervisorctl. If either application is started without the -c option (the option which
is used to tell the application the configuration filename explicitly), the application will look for a
file named supervisord.conf within the following locations, in the specified order. It will use the
first file it finds.
1. $CWD/supervisord.conf
2. $CWD/etc/supervisord.conf
3. /etc/supervisord.conf
4. ../etc/supervisord.conf (Relative to the executable)
5. ../supervisord.conf (Relative to the executable)
NOTE:
Some distributions have packaged Supervisor with their own customizations. These modified versions of
Supervisor may load the configuration file from locations other than those described here. Notably,
Ubuntu packages have been found that use /etc/supervisor/supervisord.conf.
File Format
supervisord.conf is a Windows-INI-style (Python ConfigParser) file. It has sections (each denoted by a
[header]) and key / value pairs within the sections. The sections and their allowable values are
described below.
Environment Variables
Environment variables that are present in the environment at the time that supervisord is started can be
used in the configuration file using the Python string expression syntax %(ENV_X)s:
[program:example]
command=/usr/bin/example --loglevel=%(ENV_LOGLEVEL)s
In the example above, the expression %(ENV_LOGLEVEL)s would be expanded to the value of the environment
variable LOGLEVEL.
NOTE:
In Supervisor 3.2 and later, %(ENV_X)s expressions are supported in all options. In prior versions,
some options support them, but most do not. See the documentation for each option below.
[unix_http_server] Section Settings
The supervisord.conf file contains a section named [unix_http_server] under which configuration
parameters for an HTTP server that listens on a UNIX domain socket should be inserted. If the
configuration file has no [unix_http_server] section, a UNIX domain socket HTTP server will not be
started. The allowable configuration values are as follows.
[unix_http_server] Section Values
file
A path to a UNIX domain socket (e.g. /tmp/supervisord.sock) on which supervisor will listen for
HTTP/XML-RPC requests. supervisorctl uses XML-RPC to communicate with supervisord over this port.
This option can include the value %(here)s, which expands to the directory in which the supervisord
configuration file was found.
Default: None.
Required: No.
Introduced: 3.0
chmod
Change the UNIX permission mode bits of the UNIX domain socket to this value at startup.
Default: 0700
Required: No.
Introduced: 3.0
chown
Change the user and group of the socket file to this value. May be a UNIX username (e.g. chrism) or a
UNIX username and group separated by a colon (e.g. chrism:wheel).
Default: Use the username and group of the user who starts supervisord.
Required: No.
Introduced: 3.0
username
The username required for authentication to this HTTP server.
Default: No username required.
Required: No.
Introduced: 3.0
password
The password required for authentication to this HTTP server. This can be a cleartext password, or
can be specified as a SHA-1 hash if prefixed by the string {SHA}. For example,
{SHA}82ab876d1387bfafe46cc1c8a2ef074eae50cb1d is the SHA-stored version of the password "thepassword".
Note that hashed password must be in hex format.
Default: No password required.
Required: No.
Introduced: 3.0
[unix_http_server] Section Example
[unix_http_server]
file = /tmp/supervisor.sock
chmod = 0777
chown= nobody:nogroup
username = user
password = 123
[inet_http_server] Section Settings
The supervisord.conf file contains a section named [inet_http_server] under which configuration
parameters for an HTTP server that listens on a TCP (internet) socket should be inserted. If the
configuration file has no [inet_http_server] section, an inet HTTP server will not be started. The
allowable configuration values are as follows.
[inet_http_server] Section Values
port
A TCP host:port value or (e.g. 127.0.0.1:9001) on which supervisor will listen for HTTP/XML-RPC
requests. supervisorctl will use XML-RPC to communicate with supervisord over this port. To listen
on all interfaces in the machine, use :9001 or *:9001.
Default: No default.
Required: Yes.
Introduced: 3.0
username
The username required for authentication to this HTTP server.
Default: No username required.
Required: No.
Introduced: 3.0
password
The password required for authentication to this HTTP server. This can be a cleartext password, or
can be specified as a SHA-1 hash if prefixed by the string {SHA}. For example,
{SHA}82ab876d1387bfafe46cc1c8a2ef074eae50cb1d is the SHA-stored version of the password "thepassword".
Note that hashed password must be in hex format.
Default: No password required.
Required: No.
Introduced: 3.0
[inet_http_server] Section Example
[inet_http_server]
port = 127.0.0.1:9001
username = user
password = 123
[supervisord] Section Settings
The supervisord.conf file contains a section named [supervisord] in which global settings related to the
supervisord process should be inserted. These are as follows.
[supervisord] Section Values
logfile
The path to the activity log of the supervisord process. This option can include the value %(here)s,
which expands to the directory in which the supervisord configuration file was found.
Default: $CWD/supervisord.log
Required: No.
Introduced: 3.0
logfile_maxbytes
The maximum number of bytes that may be consumed by the activity log file before it is rotated (suffix
multipliers like "KB", "MB", and "GB" can be used in the value). Set this value to 0 to indicate an
unlimited log size.
Default: 50MB
Required: No.
Introduced: 3.0
logfile_backups
The number of backups to keep around resulting from activity log file rotation. If set to 0, no
backups will be kept.
Default: 10
Required: No.
Introduced: 3.0
loglevel
The logging level, dictating what is written to the supervisord activity log. One of critical, error,
warn, info, debug, trace, or blather. Note that at log level debug, the supervisord log file will
record the stderr/stdout output of its child processes and extended info info about process state
changes, which is useful for debugging a process which isn't starting properly. See also:
activity_log_levels.
Default: info
Required: No.
Introduced: 3.0
pidfile
The location in which supervisord keeps its pid file. This option can include the value %(here)s,
which expands to the directory in which the supervisord configuration file was found.
Default: $CWD/supervisord.pid
Required: No.
Introduced: 3.0
umask
The umask of the supervisord process.
Default: 022
Required: No.
Introduced: 3.0
nodaemon
If true, supervisord will start in the foreground instead of daemonizing.
Default: false
Required: No.
Introduced: 3.0
minfds
The minimum number of file descriptors that must be available before supervisord will start
successfully. A call to setrlimit will be made to attempt to raise the soft and hard limits of the
supervisord process to satisfy minfds. The hard limit may only be raised if supervisord is run as
root. supervisord uses file descriptors liberally, and will enter a failure mode when one cannot be
obtained from the OS, so it's useful to be able to specify a minimum value to ensure it doesn't run
out of them during execution. This option is particularly useful on Solaris, which has a low
per-process fd limit by default.
Default: 1024
Required: No.
Introduced: 3.0
minprocs
The minimum number of process descriptors that must be available before supervisord will start
successfully. A call to setrlimit will be made to attempt to raise the soft and hard limits of the
supervisord process to satisfy minprocs. The hard limit may only be raised if supervisord is run as
root. supervisord will enter a failure mode when the OS runs out of process descriptors, so it's
useful to ensure that enough process descriptors are available upon supervisord startup.
Default: 200
Required: No.
Introduced: 3.0
nocleanup
Prevent supervisord from clearing any existing AUTO child log files at startup time. Useful for
debugging.
Default: false
Required: No.
Introduced: 3.0
childlogdir
The directory used for AUTO child log files. This option can include the value %(here)s, which
expands to the directory in which the supervisord configuration file was found.
Default: value of Python's tempfile.get_tempdir()
Required: No.
Introduced: 3.0
user
Instruct supervisord to switch users to this UNIX user account before doing any meaningful processing.
The user can only be switched if supervisord is started as the root user. If supervisord can't switch
users, it will still continue but will write a log message at the critical level saying that it can't
drop privileges.
Default: do not switch users
Required: No.
Introduced: 3.0
directory
When supervisord daemonizes, switch to this directory. This option can include the value %(here)s,
which expands to the directory in which the supervisord configuration file was found.
Default: do not cd
Required: No.
Introduced: 3.0
strip_ansi
Strip all ANSI escape sequences from child log files.
Default: false
Required: No.
Introduced: 3.0
environment
A list of key/value pairs in the form KEY="val",KEY2="val2" that will be placed in the supervisord
process' environment (and as a result in all of its child process' environments). This option can
include the value %(here)s, which expands to the directory in which the supervisord configuration file
was found. Values containing non-alphanumeric characters should be quoted (e.g.
KEY="val:123",KEY2="val,456"). Otherwise, quoting the values is optional but recommended. To escape
percent characters, simply use two. (e.g. URI="/first%%20name") Note that subprocesses will inherit
the environment variables of the shell used to start supervisord except for the ones overridden here
and within the program's environment option. See subprocess_environment.
Default: no values
Required: No.
Introduced: 3.0
identifier
The identifier string for this supervisor process, used by the RPC interface.
Default: supervisor
Required: No.
Introduced: 3.0
[supervisord] Section Example
[supervisord]
logfile = /tmp/supervisord.log
logfile_maxbytes = 50MB
logfile_backups=10
loglevel = info
pidfile = /tmp/supervisord.pid
nodaemon = false
minfds = 1024
minprocs = 200
umask = 022
user = chrism
identifier = supervisor
directory = /tmp
nocleanup = true
childlogdir = /tmp
strip_ansi = false
environment = KEY1="value1",KEY2="value2"
[supervisorctl] Section Settings
The configuration file may contain settings for the supervisorctl interactive shell program. These
options are listed below.
[supervisorctl] Section Values
serverurl
The URL that should be used to access the supervisord server, e.g. http://localhost:9001. For UNIX
domain sockets, use unix:///absolute/path/to/file.sock.
Default: http://localhost:9001
Required: No.
Introduced: 3.0
username
The username to pass to the supervisord server for use in authentication. This should be same as
username from the supervisord server configuration for the port or UNIX domain socket you're
attempting to access.
Default: No username
Required: No.
Introduced: 3.0
password
The password to pass to the supervisord server for use in authentication. This should be the cleartext
version of password from the supervisord server configuration for the port or UNIX domain socket
you're attempting to access. This value cannot be passed as a SHA hash. Unlike other passwords
specified in this file, it must be provided in cleartext.
Default: No password
Required: No.
Introduced: 3.0
prompt
String used as supervisorctl prompt.
Default: supervisor
Required: No.
Introduced: 3.0
history_file
A path to use as the readline persistent history file. If you enable this feature by choosing a path,
your supervisorctl commands will be kept in the file, and you can use readline (e.g. arrow-up) to
invoke commands you performed in your last supervisorctl session.
Default: No file
Required: No.
Introduced: 3.0a5
[supervisorctl] Section Example
[supervisorctl]
serverurl = unix:///tmp/supervisor.sock
username = chris
password = 123
prompt = mysupervisor
[program:x] Section Settings
The configuration file must contain one or more program sections in order for supervisord to know which
programs it should start and control. The header value is composite value. It is the word "program",
followed directly by a colon, then the program name. A header value of [program:foo] describes a program
with the name of "foo". The name is used within client applications that control the processes that are
created as a result of this configuration. It is an error to create a program section that does not have
a name. The name must not include a colon character or a bracket character. The value of the name is
used as the value for the %(program_name)s string expression expansion within other values where
specified.
NOTE:
A [program:x] section actually represents a "homogeneous process group" to supervisor (as of 3.0).
The members of the group are defined by the combination of the numprocs and process_name parameters in
the configuration. By default, if numprocs and process_name are left unchanged from their defaults,
the group represented by [program:x] will be named x and will have a single process named x in it.
This provides a modicum of backwards compatibility with older supervisor releases, which did not treat
program sections as homogeneous process group definitions.
But for instance, if you have a [program:foo] section with a numprocs of 3 and a process_name
expression of %(program_name)s_%(process_num)02d, the "foo" group will contain three processes, named
foo_00, foo_01, and foo_02. This makes it possible to start a number of very similar processes using
a single [program:x] section. All logfile names, all environment strings, and the command of programs
can also contain similar Python string expressions, to pass slightly different parameters to each
process.
[program:x] Section Values
command
The command that will be run when this program is started. The command can be either absolute (e.g.
/path/to/programname) or relative (e.g. programname). If it is relative, the supervisord's
environment $PATH will be searched for the executable. Programs can accept arguments, e.g.
/path/to/program foo bar. The command line can use double quotes to group arguments with spaces in
them to pass to the program, e.g. /path/to/program/name -p "foo bar". Note that the value of command
may include Python string expressions, e.g. /path/to/programname --port=80%(process_num)02d might
expand to /path/to/programname --port=8000 at runtime. String expressions are evaluated against a
dictionary containing the keys group_name, host_node_name, process_num, program_name, here (the
directory of the supervisord config file), and all supervisord's environment variables prefixed with
ENV_. Controlled programs should themselves not be daemons, as supervisord assumes it is responsible
for daemonizing its subprocesses (see nondaemonizing_of_subprocesses).
Default: No default.
Required: Yes.
Introduced: 3.0
process_name
A Python string expression that is used to compose the supervisor process name for this process. You
usually don't need to worry about setting this unless you change numprocs. The string expression is
evaluated against a dictionary that includes group_name, host_node_name, process_num, program_name,
and here (the directory of the supervisord config file).
Default: %(program_name)s
Required: No.
Introduced: 3.0
numprocs
Supervisor will start as many instances of this program as named by numprocs. Note that if numprocs >
1, the process_name expression must include %(process_num)s (or any other valid Python string
expression that includes process_num) within it.
Default: 1
Required: No.
Introduced: 3.0
numprocs_start
An integer offset that is used to compute the number at which numprocs starts.
Default: 0
Required: No.
Introduced: 3.0
priority
The relative priority of the program in the start and shutdown ordering. Lower priorities indicate
programs that start first and shut down last at startup and when aggregate commands are used in
various clients (e.g. "start all"/"stop all"). Higher priorities indicate programs that start last
and shut down first.
Default: 999
Required: No.
Introduced: 3.0
autostart
If true, this program will start automatically when supervisord is started.
Default: true
Required: No.
Introduced: 3.0
startsecs
The total number of seconds which the program needs to stay running after a startup to consider the
start successful (moving the process from the STARTING state to the RUNNING state). Set to 0 to
indicate that the program needn't stay running for any particular amount of time.
NOTE:
Even if a process exits with an "expected" exit code (see exitcodes), the start will still be
considered a failure if the process exits quicker than startsecs.
Default: 1
Required: No.
Introduced: 3.0
startretries
The number of serial failure attempts that supervisord will allow when attempting to start the program
before giving up and putting the process into an FATAL state. See process_states for explanation of
the FATAL state.
Default: 3
Required: No.
Introduced: 3.0
autorestart
Specifies if supervisord should automatically restart a process if it exits when it is in the RUNNING
state. May be one of false, unexpected, or true. If false, the process will not be autorestarted.
If unexpected, the process will be restarted when the program exits with an exit code that is not one
of the exit codes associated with this process' configuration (see exitcodes). If true, the process
will be unconditionally restarted when it exits, without regard to its exit code.
NOTE:
autorestart controls whether supervisord will autorestart a program if it exits after it has
successfully started up (the process is in the RUNNING state).
supervisord has a different restart mechanism for when the process is starting up (the process is
in the STARTING state). Retries during process startup are controlled by startsecs and
startretries.
Default: unexpected
Required: No.
Introduced: 3.0
exitcodes
The list of "expected" exit codes for this program used with autorestart. If the autorestart
parameter is set to unexpected, and the process exits in any other way than as a result of a
supervisor stop request, supervisord will restart the process if it exits with an exit code that is
not defined in this list.
Default: 0,2
Required: No.
Introduced: 3.0
stopsignal
The signal used to kill the program when a stop is requested. This can be any of TERM, HUP, INT,
QUIT, KILL, USR1, or USR2.
Default: TERM
Required: No.
Introduced: 3.0
stopwaitsecs
The number of seconds to wait for the OS to return a SIGCHILD to supervisord after the program has
been sent a stopsignal. If this number of seconds elapses before supervisord receives a SIGCHILD from
the process, supervisord will attempt to kill it with a final SIGKILL.
Default: 10
Required: No.
Introduced: 3.0
stopasgroup
If true, the flag causes supervisor to send the stop signal to the whole process group and implies
killasgroup is true. This is useful for programs, such as Flask in debug mode, that do not propagate
stop signals to their children, leaving them orphaned.
Default: false
Required: No.
Introduced: 3.0b1
killasgroup
If true, when resorting to send SIGKILL to the program to terminate it send it to its whole process
group instead, taking care of its children as well, useful e.g with Python programs using
multiprocessing.
Default: false
Required: No.
Introduced: 3.0a11
user
Instruct supervisord to use this UNIX user account as the account which runs the program. The user
can only be switched if supervisord is run as the root user. If supervisord can't switch to the
specified user, the program will not be started.
NOTE:
The user will be changed using setuid only. This does not start a login shell and does not change
environment variables like USER or HOME. See subprocess_environment for details.
Default: Do not switch users
Required: No.
Introduced: 3.0
redirect_stderr
If true, cause the process' stderr output to be sent back to supervisord on its stdout file descriptor
(in UNIX shell terms, this is the equivalent of executing /the/program 2>&1).
NOTE:
Do not set redirect_stderr=true in an [eventlistener:x] section. Eventlisteners use stdout and
stdin to communicate with supervisord. If stderr is redirected, output from stderr will interfere
with the eventlistener protocol.
Default: false
Required: No.
Introduced: 3.0, replaces 2.0's log_stdout and log_stderr
stdout_logfile
Put process stdout output in this file (and if redirect_stderr is true, also place stderr output in
this file). If stdout_logfile is unset or set to AUTO, supervisor will automatically choose a file
location. If this is set to NONE, supervisord will create no log file. AUTO log files and their
backups will be deleted when supervisord restarts. The stdout_logfile value can contain Python string
expressions that will evaluated against a dictionary that contains the keys group_name,
host_node_name, process_num, program_name, and here (the directory of the supervisord config file).
NOTE:
It is not possible for two processes to share a single log file (stdout_logfile) when rotation
(stdout_logfile_maxbytes) is enabled. This will result in the file being corrupted.
Default: AUTO
Required: No.
Introduced: 3.0, replaces 2.0's logfile
stdout_logfile_maxbytes
The maximum number of bytes that may be consumed by stdout_logfile before it is rotated (suffix
multipliers like "KB", "MB", and "GB" can be used in the value). Set this value to 0 to indicate an
unlimited log size.
Default: 50MB
Required: No.
Introduced: 3.0, replaces 2.0's logfile_maxbytes
stdout_logfile_backups
The number of stdout_logfile backups to keep around resulting from process stdout log file rotation.
If set to 0, no backups will be kept.
Default: 10
Required: No.
Introduced: 3.0, replaces 2.0's logfile_backups
stdout_capture_maxbytes
Max number of bytes written to capture FIFO when process is in "stdout capture mode" (see
capture_mode). Should be an integer (suffix multipliers like "KB", "MB" and "GB" can used in the
value). If this value is 0, process capture mode will be off.
Default: 0
Required: No.
Introduced: 3.0, replaces 2.0's logfile_backups
stdout_events_enabled
If true, PROCESS_LOG_STDOUT events will be emitted when the process writes to its stdout file
descriptor. The events will only be emitted if the file descriptor is not in capture mode at the time
the data is received (see capture_mode).
Default: 0
Required: No.
Introduced: 3.0a7
stderr_logfile
Put process stderr output in this file unless redirect_stderr is true. Accepts the same value types
as stdout_logfile and may contain the same Python string expressions.
NOTE:
It is not possible for two processes to share a single log file (stderr_logfile) when rotation
(stderr_logfile_maxbytes) is enabled. This will result in the file being corrupted.
Default: AUTO
Required: No.
Introduced: 3.0
stderr_logfile_maxbytes
The maximum number of bytes before logfile rotation for stderr_logfile. Accepts the same value types
as stdout_logfile_maxbytes.
Default: 50MB
Required: No.
Introduced: 3.0
stderr_logfile_backups
The number of backups to keep around resulting from process stderr log file rotation. If set to 0, no
backups will be kept.
Default: 10
Required: No.
Introduced: 3.0
stderr_capture_maxbytes
Max number of bytes written to capture FIFO when process is in "stderr capture mode" (see
capture_mode). Should be an integer (suffix multipliers like "KB", "MB" and "GB" can used in the
value). If this value is 0, process capture mode will be off.
Default: 0
Required: No.
Introduced: 3.0
stderr_events_enabled
If true, PROCESS_LOG_STDERR events will be emitted when the process writes to its stderr file
descriptor. The events will only be emitted if the file descriptor is not in capture mode at the time
the data is received (see capture_mode).
Default: false
Required: No.
Introduced: 3.0a7
environment
A list of key/value pairs in the form KEY="val",KEY2="val2" that will be placed in the child process'
environment. The environment string may contain Python string expressions that will be evaluated
against a dictionary containing group_name, host_node_name, process_num, program_name, and here (the
directory of the supervisord config file). Values containing non-alphanumeric characters should be
quoted (e.g. KEY="val:123",KEY2="val,456"). Otherwise, quoting the values is optional but
recommended. Note that the subprocess will inherit the environment variables of the shell used to
start "supervisord" except for the ones overridden here. See subprocess_environment.
Default: No extra environment
Required: No.
Introduced: 3.0
directory
A file path representing a directory to which supervisord should temporarily chdir before exec'ing the
child.
Default: No chdir (inherit supervisor's)
Required: No.
Introduced: 3.0
umask
An octal number (e.g. 002, 022) representing the umask of the process.
Default: No special umask (inherit supervisor's)
Required: No.
Introduced: 3.0
serverurl
The URL passed in the environment to the subprocess process as SUPERVISOR_SERVER_URL (see
supervisor.childutils) to allow the subprocess to easily communicate with the internal HTTP server.
If provided, it should have the same syntax and structure as the [supervisorctl] section option of the
same name. If this is set to AUTO, or is unset, supervisor will automatically construct a server URL,
giving preference to a server that listens on UNIX domain sockets over one that listens on an internet
socket.
Default: AUTO
Required: No.
Introduced: 3.0
[program:x] Section Example
[program:cat]
command=/bin/cat
process_name=%(program_name)s
numprocs=1
directory=/tmp
umask=022
priority=999
autostart=true
autorestart=unexpected
startsecs=10
startretries=3
exitcodes=0,2
stopsignal=TERM
stopwaitsecs=10
stopasgroup=false
killasgroup=false
user=chrism
redirect_stderr=false
stdout_logfile=/a/path
stdout_logfile_maxbytes=1MB
stdout_logfile_backups=10
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
stderr_logfile=/a/path
stderr_logfile_maxbytes=1MB
stderr_logfile_backups=10
stderr_capture_maxbytes=1MB
stderr_events_enabled=false
environment=A="1",B="2"
serverurl=AUTO
[include] Section Settings
The supervisord.conf file may contain a section named [include]. If the configuration file contains an
[include] section, it must contain a single key named "files". The values in this key specify other
configuration files to be included within the configuration.
[include] Section Values
files
A space-separated sequence of file globs. Each file glob may be absolute or relative. If the file
glob is relative, it is considered relative to the location of the configuration file which includes
it. A "glob" is a file pattern which matches a specified pattern according to the rules used by the
Unix shell. No tilde expansion is done, but *, ?, and character ranges expressed with [] will be
correctly matched. Recursive includes from included files are not supported.
Default: No default (required)
Required: Yes.
Introduced: 3.0
[include] Section Example
[include]
files = /an/absolute/filename.conf /an/absolute/*.conf foo.conf config??.conf
[group:x] Section Settings
It is often useful to group "homogeneous" process groups (aka "programs") together into a "heterogeneous"
process group so they can be controlled as a unit from Supervisor's various controller interfaces.
To place programs into a group so you can treat them as a unit, define a [group:x] section in your
configuration file. The group header value is a composite. It is the word "group", followed directly by
a colon, then the group name. A header value of [group:foo] describes a group with the name of "foo".
The name is used within client applications that control the processes that are created as a result of
this configuration. It is an error to create a group section that does not have a name. The name must
not include a colon character or a bracket character.
For a [group:x], there must be one or more [program:x] sections elsewhere in your configuration file, and
the group must refer to them by name in the programs value.
If "homogeneous" process groups (represented by program sections) are placed into a "heterogeneous" group
via [group:x] section's programs line, the homogeneous groups that are implied by the program section
will not exist at runtime in supervisor. Instead, all processes belonging to each of the homogeneous
groups will be placed into the heterogeneous group. For example, given the following group
configuration:
[group:foo]
programs=bar,baz
priority=999
Given the above, at supervisord startup, the bar and baz homogeneous groups will not exist, and the
processes that would have been under them will now be moved into the foo group.
[group:x] Section Values
programs
A comma-separated list of program names. The programs which are listed become members of the group.
Default: No default (required)
Required: Yes.
Introduced: 3.0
priority
A priority number analogous to a [program:x] priority value assigned to the group.
Default: 999
Required: No.
Introduced: 3.0
[group:x] Section Example
[group:foo]
programs=bar,baz
priority=999
[fcgi-program:x] Section Settings
Supervisor can manage groups of FastCGI processes that all listen on the same socket. Until now,
deployment flexibility for FastCGI was limited. To get full process management, you could use
mod_fastcgi under Apache but then you were stuck with Apache's inefficient concurrency model of one
process or thread per connection. In addition to requiring more CPU and memory resources, the
process/thread per connection model can be quickly saturated by a slow resource, preventing other
resources from being served. In order to take advantage of newer event-driven web servers such as
lighttpd or nginx which don't include a built-in process manager, you had to use scripts like cgi-fcgi or
spawn-fcgi. These can be used in conjunction with a process manager such as supervisord or daemontools
but require each FastCGI child process to bind to its own socket. The disadvantages of this are:
unnecessarily complicated web server configuration, ungraceful restarts, and reduced fault tolerance.
With fewer sockets to configure, web server configurations are much smaller if groups of FastCGI
processes can share sockets. Shared sockets allow for graceful restarts because the socket remains bound
by the parent process while any of the child processes are being restarted. Finally, shared sockets are
more fault tolerant because if a given process fails, other processes can continue to serve inbound
connections.
With integrated FastCGI spawning support, Supervisor gives you the best of both worlds. You get
full-featured process management with groups of FastCGI processes sharing sockets without being tied to a
particular web server. It's a clean separation of concerns, allowing the web server and the process
manager to each do what they do best.
NOTE:
The socket manager in Supervisor was originally developed to support FastCGI processes but it is not
limited to FastCGI. Other protocols may be used as well with no special configuration. Any program
that can access an open socket from a file descriptor (e.g. with socket.fromfd in Python) can use the
socket manager. Supervisor will automatically create the socket, bind, and listen before forking the
first child in a group. The socket will be passed to each child on file descriptor number 0 (zero).
When the last child in the group exits, Supervisor will close the socket.
All the options available to [program:x] sections are also respected by fcgi-program sections.
[fcgi-program:x] Section Values
[fcgi-program:x] sections have a single key which [program:x] sections do not have.
socket
The FastCGI socket for this program, either TCP or UNIX domain socket. For TCP sockets, use this
format: tcp://localhost:9002. For UNIX domain sockets, use unix:///absolute/path/to/file.sock.
String expressions are evaluated against a dictionary containing the keys "program_name" and "here"
(the directory of the supervisord config file).
Default: No default.
Required: Yes.
Introduced: 3.0
socket_owner
For UNIX domain sockets, this parameter can be used to specify the user and group for the FastCGI
socket. May be a UNIX username (e.g. chrism) or a UNIX username and group separated by a colon (e.g.
chrism:wheel).
Default: Uses the user and group set for the fcgi-program
Required: No.
Introduced: 3.0
socket_mode
For UNIX domain sockets, this parameter can be used to specify the permission mode.
Default: 0700
Required: No.
Introduced: 3.0
Consult [program:x] Section Settings for other allowable keys, delta the above constraints and additions.
[fcgi-program:x] Section Example
[fcgi-program:fcgiprogramname]
command=/usr/bin/example.fcgi
socket=unix:///var/run/supervisor/%(program_name)s.sock
socket_owner=chrism
socket_mode=0700
process_name=%(program_name)s_%(process_num)02d
numprocs=5
directory=/tmp
umask=022
priority=999
autostart=true
autorestart=unexpected
startsecs=1
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopasgroup=false
killasgroup=false
stopwaitsecs=10
user=chrism
redirect_stderr=true
stdout_logfile=/a/path
stdout_logfile_maxbytes=1MB
stdout_logfile_backups=10
stdout_events_enabled=false
stderr_logfile=/a/path
stderr_logfile_maxbytes=1MB
stderr_logfile_backups=10
stderr_events_enabled=false
environment=A="1",B="2"
serverurl=AUTO
[eventlistener:x] Section Settings
Supervisor allows specialized homogeneous process groups ("event listener pools") to be defined within
the configuration file. These pools contain processes that are meant to receive and respond to event
notifications from supervisor's event system. See events for an explanation of how events work and how
to implement programs that can be declared as event listeners.
Note that all the options available to [program:x] sections are respected by eventlistener sections
except for stdout_capture_maxbytes and stderr_capture_maxbytes (event listeners cannot emit process
communication events, see capture_mode).
[eventlistener:x] Section Values
[eventlistener:x] sections have a few keys which [program:x] sections do not have.
buffer_size
The event listener pool's event queue buffer size. When a listener pool's event buffer is overflowed
(as can happen when an event listener pool cannot keep up with all of the events sent to it), the
oldest event in the buffer is discarded.
events
A comma-separated list of event type names that this listener is "interested" in receiving
notifications for (see event_types for a list of valid event type names).
result_handler
A pkg_resources entry point string that resolves to a Python callable. The default value is
supervisor.dispatchers:default_handler. Specifying an alternate result handler is a very uncommon
thing to need to do, and as a result, how to create one is not documented.
Consult [program:x] Section Settings for other allowable keys, delta the above constraints and additions.
[eventlistener:x] Section Example
[eventlistener:theeventlistenername]
command=/bin/eventlistener
process_name=%(program_name)s_%(process_num)02d
numprocs=5
events=PROCESS_STATE
buffer_size=10
directory=/tmp
umask=022
priority=-1
autostart=true
autorestart=unexpected
startsecs=1
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
stopasgroup=false
killasgroup=false
user=chrism
redirect_stderr=false
stdout_logfile=/a/path
stdout_logfile_maxbytes=1MB
stdout_logfile_backups=10
stdout_events_enabled=false
stderr_logfile=/a/path
stderr_logfile_maxbytes=1MB
stderr_logfile_backups=10
stderr_events_enabled=false
environment=A="1",B="2"
serverurl=AUTO
[rpcinterface:x] Section Settings
Adding rpcinterface:x settings in the configuration file is only useful for people who wish to extend
supervisor with additional custom behavior.
In the sample config file, there is a section which is named [rpcinterface:supervisor]. By default it
looks like the following.
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
The [rpcinterface:supervisor] section must remain in the configuration for the standard setup of
supervisor to work properly. If you don't want supervisor to do anything it doesn't already do out of
the box, this is all you need to know about this type of section.
However, if you wish to add rpc interface namespaces in order to customize supervisor, you may add
additional [rpcinterface:foo] sections, where "foo" represents the namespace of the interface (from the
web root), and the value named by supervisor.rpcinterface_factory is a factory callable which should have
a function signature that accepts a single positional argument supervisord and as many keyword arguments
as required to perform configuration. Any extra key/value pairs defined within the [rpcinterface:x]
section will be passed as keyword arguments to the factory.
Here's an example of a factory function, created in the __init__.py file of the Python package
my.package.
from my.package.rpcinterface import AnotherRPCInterface
def make_another_rpcinterface(supervisord, **config):
retries = int(config.get('retries', 0))
another_rpc_interface = AnotherRPCInterface(supervisord, retries)
return another_rpc_interface
And a section in the config file meant to configure it.
[rpcinterface:another]
supervisor.rpcinterface_factory = my.package:make_another_rpcinterface
retries = 1
[rpcinterface:x] Section Values
supervisor.rpcinterface_factory
pkg_resources "entry point" dotted name to your RPC interface's factory function.
Default: N/A
Required: No.
Introduced: 3.0
[rpcinterface:x] Section Example
[rpcinterface:another]
supervisor.rpcinterface_factory = my.package:make_another_rpcinterface
retries = 1
Subprocesses
supervisord's primary purpose is to create and manage processes based on data in its configuration file.
It does this by creating subprocesses. Each subprocess spawned by supervisor is managed for the entirety
of its lifetime by supervisord (supervisord is the parent process of each process it creates). When a
child dies, supervisor is notified of its death via the SIGCHLD signal, and it performs the appropriate
operation.
Nondaemonizing of Subprocesses
Programs meant to be run under supervisor should not daemonize themselves. Instead, they should run in
the foreground. They should not detach from the terminal from which they are started.
The easiest way to tell if a program will run in the foreground is to run the command that invokes the
program from a shell prompt. If it gives you control of the terminal back, but continues running, it's
daemonizing itself and that will almost certainly be the wrong way to run it under supervisor. You want
to run a command that essentially requires you to press Ctrl-C to get control of the terminal back. If
it gives you a shell prompt back after running it without needing to press Ctrl-C, it's not useful under
supervisor. All programs have options to be run in the foreground but there's no "standard way" to do
it; you'll need to read the documentation for each program.
Below are configuration file examples that are known to start common programs in "foreground" mode under
Supervisor.
Examples of Program Configurations
Here are some "real world" program configuration examples:
Apache 2.2.6
[program:apache2]
command=/path/to/httpd -c "ErrorLog /dev/stdout" -DFOREGROUND
redirect_stderr=true
Two Zope 2.X instances and one ZEO server
[program:zeo]
command=/path/to/runzeo
priority=1
[program:zope1]
command=/path/to/instance/home/bin/runzope
priority=2
redirect_stderr=true
[program:zope2]
command=/path/to/another/instance/home/bin/runzope
priority=2
redirect_stderr=true
Postgres 8.X
[program:postgres]
command=/path/to/postmaster
; we use the "fast" shutdown signal SIGINT
stopsignal=INT
redirect_stderr=true
OpenLDAP slapd
[program:slapd]
command=/path/to/slapd -f /path/to/slapd.conf -h ldap://0.0.0.0:8888
redirect_stderr=true
Other Examples
Other examples of shell scripts that could be used to start services under supervisord can be found at
http://thedjbway.b0llix.net/services.html. These examples are actually for daemontools but the premise
is the same for supervisor.
Another collection of recipes for starting various programs in the foreground is available from
http://smarden.org/runit/runscripts.html.
pidproxy Program
Some processes (like mysqld) ignore signals sent to the actual process which is spawned by supervisord.
Instead, a "special" thread/process is created by these kinds of programs which is responsible for
handling signals. This is problematic because supervisord can only kill a process which it creates
itself. If a process created by supervisord creates its own child processes, supervisord cannot kill
them.
Fortunately, these types of programs typically write a "pidfile" which contains the "special" process'
PID, and is meant to be read and used in order to kill the process. As a workaround for this case, a
special pidproxy program can handle startup of these kinds of processes. The pidproxy program is a small
shim that starts a process, and upon the receipt of a signal, sends the signal to the pid provided in a
pidfile. A sample configuration program entry for a pidproxy-enabled program is provided below.
[program:mysql]
command=/path/to/pidproxy /path/to/pidfile /path/to/mysqld_safe
The pidproxy program is put into your configuration's $BINDIR when supervisor is installed (it is a
"console script").
Subprocess Environment
Subprocesses will inherit the environment of the shell used to start the supervisord program. Several
environment variables will be set by supervisord itself in the child's environment also, including
SUPERVISOR_ENABLED (a flag indicating the process is under supervisor control), SUPERVISOR_PROCESS_NAME
(the config-file-specified process name for this process) and SUPERVISOR_GROUP_NAME (the
config-file-specified process group name for the child process).
These environment variables may be overridden within the [supervisord] section config option named
environment (applies to all subprocesses) or within the per- [program:x] section environment config
option (applies only to the subprocess specified within the [program:x] section). These "environment"
settings are additive. In other words, each subprocess' environment will consist of:
The environment variables set within the shell used to start supervisord...
... added-to/overridden-by ...
... the environment variables set within the environment global
config option ...
... added-to/overridden-by ...
... supervisor-specific environment variables
(SUPERVISOR_ENABLED, SUPERVISOR_PROCESS_NAME, SUPERVISOR_GROUP_NAME) ..
... added-to/overridden-by ...
... the environment variables set within the per-process
"environment" config option.
No shell is executed by supervisord when it runs a subprocess, so environment variables such as USER,
PATH, HOME, SHELL, LOGNAME, etc. are not changed from their defaults or otherwise reassigned. This is
particularly important to note when you are running a program from a supervisord run as root with a user=
stanza in the configuration. Unlike cron, supervisord does not attempt to divine and override
"fundamental" environment variables like USER, PATH, HOME, and LOGNAME when it performs a setuid to the
user defined within the user= program config option. If you need to set environment variables for a
particular program that might otherwise be set by a shell invocation for a particular user, you must do
it explicitly within the environment= program config option. An example of setting these environment
variables is as below.
[program:apache2]
command=/home/chrism/bin/httpd -c "ErrorLog /dev/stdout" -DFOREGROUND
user=chrism
environment=HOME="/home/chrism",USER="chrism"
Process States
A process controlled by supervisord will be in one of the below states at any given time. You may see
these state names in various user interface elements in clients.
STOPPED (0)
The process has been stopped due to a stop request or has never been started.
STARTING (10)
The process is starting due to a start request.
RUNNING (20)
The process is running.
BACKOFF (30)
The process entered the STARTING state but subsequently exited too quickly to move to the RUNNING
state.
STOPPING (40)
The process is stopping due to a stop request.
EXITED (100)
The process exited from the RUNNING state (expectedly or unexpectedly).
FATAL (200)
The process could not be started successfully.
UNKNOWN (1000)
The process is in an unknown state (supervisord programming error).
Each process run under supervisor progresses through these states as per the following directed graph.
[image: Subprocess State Transition Graph] [image] Subprocess State Transition Graph.UNINDENT
A process is in the STOPPED state if it has been stopped adminstratively or if it has never been
started.
When an autorestarting process is in the BACKOFF state, it will be automatically restarted by
supervisord. It will switch between STARTING and BACKOFF states until it becomes evident that it
cannot be started because the number of startretries has exceeded the maximum, at which point it will
transition to the FATAL state. Each start retry will take progressively more time.
When a process is in the EXITED state, it will automatically restart:
• never if its autorestart parameter is set to false.
• unconditionally if its autorestart parameter is set to true.
• conditionally if its autorestart parameter is set to unexpected. If it exited with an exit code that
doesn't match one of the exit codes defined in the exitcodes configuration parameter for the process,
it will be restarted.
A process automatically transitions from EXITED to RUNNING as a result of being configured to autorestart
conditionally or unconditionally. The number of transitions between RUNNING and EXITED is not limited in
any way: it is possible to create a configuration that endlessly restarts an exited process. This is a
feature, not a bug.
An autorestarted process will never be automatically restarted if it ends up in the FATAL state (it must
be manually restarted from this state).
A process transitions into the STOPPING state via an administrative stop request, and will then end up in
the STOPPED state.
A process that cannot be stopped successfully will stay in the STOPPING state forever. This situation
should never be reached during normal operations as it implies that the process did not respond to a
final SIGKILL signal sent to it by supervisor, which is "impossible" under UNIX.
State transitions which always require user action to invoke are these:
FATAL -> STARTING
RUNNING -> STOPPING
State transitions which typically, but not always, require user action to invoke are these, with
exceptions noted:
STOPPED -> STARTING (except at supervisord startup if process is configured to autostart)
EXITED -> STARTING (except if process is configured to autorestart)
All other state transitions are managed by supervisord automatically.
Logging
One of the main tasks that supervisord performs is logging. supervisord logs an activity log detailing
what it's doing as it runs. It also logs child process stdout and stderr output to other files if
configured to do so.
Activity Log
The activity log is the place where supervisord logs messages about its own health, its subprocess' state
changes, any messages that result from events, and debug and informational messages. The path to the
activity log is configured via the logfile parameter in the [supervisord] section of the configuration
file, defaulting to $CWD/supervisord.log. Sample activity log traffic is shown in the example below.
Some lines have been broken to better fit the screen.
Sample Activity Log Output
2007-09-08 14:43:22,886 DEBG 127.0.0.1:Medusa (V1.11) started at Sat Sep 8 14:43:22 2007
Hostname: kingfish
Port:9001
2007-09-08 14:43:22,961 INFO RPC interface 'supervisor' initialized
2007-09-08 14:43:22,961 CRIT Running without any HTTP authentication checking
2007-09-08 14:43:22,962 INFO supervisord started with pid 27347
2007-09-08 14:43:23,965 INFO spawned: 'listener_00' with pid 27349
2007-09-08 14:43:23,970 INFO spawned: 'eventgen' with pid 27350
2007-09-08 14:43:23,990 INFO spawned: 'grower' with pid 27351
2007-09-08 14:43:24,059 DEBG 'listener_00' stderr output:
/Users/chrism/projects/supervisor/supervisor2/dev-sandbox/bin/python:
can't open file '/Users/chrism/projects/supervisor/supervisor2/src/supervisor/scripts/osx_eventgen_listener.py':
[Errno 2] No such file or directory
2007-09-08 14:43:24,060 DEBG fd 7 closed, stopped monitoring <PEventListenerDispatcher at 19910168 for
<Subprocess at 18892960 with name listener_00 in state STARTING> (stdout)>
2007-09-08 14:43:24,060 INFO exited: listener_00 (exit status 2; not expected)
2007-09-08 14:43:24,061 DEBG received SIGCHLD indicating a child quit
The activity log "level" is configured in the config file via the loglevel parameter in the [supervisord]
ini file section. When loglevel is set, messages of the specified priority, plus those with any higher
priority are logged to the activity log. For example, if loglevel is error, messages of error and
critical priority will be logged. However, if loglevel is warn, messages of warn, error, and critical
will be logged.
Activity Log Levels
The below table describes the logging levels in more detail, ordered in highest priority to lowest. The
"Config File Value" is the string provided to the loglevel parameter in the [supervisord] section of
configuration file and the "Output Code" is the code that shows up in activity log output lines.
┌───────────────────┬─────────────┬──────────────────────────────┐
│ Config File Value │ Output Code │ Description │
├───────────────────┼─────────────┼──────────────────────────────┤
│ critical │ CRIT │ Messages that indicate a │
│ │ │ condition that requires │
│ │ │ immediate user attention, a │
│ │ │ supervisor state change, or │
│ │ │ an error in supervisor │
│ │ │ itself. │
├───────────────────┼─────────────┼──────────────────────────────┤
│ error │ ERRO │ Messages that indicate a │
│ │ │ potentially ignorable error │
│ │ │ condition (e.g. unable to │
│ │ │ clear a log directory). │
├───────────────────┼─────────────┼──────────────────────────────┤
│ warn │ WARN │ Messages that indicate an │
│ │ │ anomalous condition which │
│ │ │ isn't an error. │
├───────────────────┼─────────────┼──────────────────────────────┤
│ info │ INFO │ Normal informational output. │
│ │ │ This is the default log │
│ │ │ level if none is explicitly │
│ │ │ configured. │
├───────────────────┼─────────────┼──────────────────────────────┤
│ debug │ DEBG │ Messages useful for users │
│ │ │ trying to debug process │
│ │ │ configuration and │
│ │ │ communications behavior │
│ │ │ (process output, listener │
│ │ │ state changes, event │
│ │ │ notifications). │
├───────────────────┼─────────────┼──────────────────────────────┤
│ trace │ TRAC │ Messages useful for │
│ │ │ developers trying to debug │
│ │ │ supervisor plugins, and │
│ │ │ information about HTTP and │
│ │ │ RPC requests and responses. │
├───────────────────┼─────────────┼──────────────────────────────┤
│ blather │ BLAT │ Messages useful for │
│ │ │ developers trying to debug │
│ │ │ supervisor itself. │
└───────────────────┴─────────────┴──────────────────────────────┘
Activity Log Rotation
The activity log is "rotated" by supervisord based on the combination of the logfile_maxbytes and the
logfile_backups parameters in the [supervisord] section of the configuration file. When the activity log
reaches logfile_maxbytes bytes, the current log file is moved to a backup file and a new activity log
file is created. When this happens, if the number of existing backup files is greater than or equal to
logfile_backups, the oldest backup file is removed and the backup files are renamed accordingly. If the
file being written to is named supervisord.log, when it exceeds logfile_maxbytes, it is closed and
renamed to supervisord.log.1, and if files supervisord.log.1, supervisord.log.2 etc. exist, then they are
renamed to supervisord.log.2, supervisord.log.3 etc. respectively. If logfile_maxbytes is 0, the
logfile is never rotated (and thus backups are never made). If logfile_backups is 0, no backups will be
kept.
Child Process Logs
The stdout of child processes spawned by supervisor, by default, is captured for redisplay to users of
supervisorctl and other clients. If no specific logfile-related configuration is performed in a
[program:x], [fcgi-program:x], or [eventlistener:x] section in the configuration file, the following is
true:
• supervisord will capture the child process' stdout and stderr output into temporary files. Each stream
is captured to a separate file. This is known as AUTO log mode.
• AUTO log files are named automatically and placed in the directory configured as childlogdir of the
[supervisord] section of the config file.
• The size of each AUTO log file is bounded by the {streamname}_logfile_maxbytes value of the program
section (where {streamname} is "stdout" or "stderr"). When it reaches that number, it is rotated (like
the activity log), based on the {streamname}_logfile_backups.
The configuration keys that influence child process logging in [program:x] and [fcgi-program:x] sections
are these:
redirect_stderr, stdout_logfile, stdout_logfile_maxbytes, stdout_logfile_backups,
stdout_capture_maxbytes, stderr_logfile, stderr_logfile_maxbytes, stderr_logfile_backups and
stderr_capture_maxbytes.
One may set stdout_logfile or stderr_logfile to the special string "syslog". In this case, logs will be
routed to the syslog service instead of being saved to files.
[eventlistener:x] sections may not specify redirect_stderr, stdout_capture_maxbytes, or
stderr_capture_maxbytes, but otherwise they accept the same values.
The configuration keys that influence child process logging in the [supervisord] config file section are
these: childlogdir, and nocleanup.
Capture Mode
Capture mode is an advanced feature of Supervisor. You needn't understand capture mode unless you want
to take actions based on data parsed from subprocess output.
If a [program:x] section in the configuration file defines a non-zero stdout_capture_maxbytes or
stderr_capture_maxbytes parameter, each process represented by the program section may emit special
tokens on its stdout or stderr stream (respectively) which will effectively cause supervisor to emit a
PROCESS_COMMUNICATION event (see events for a description of events).
The process communications protocol relies on two tags, one which commands supervisor to enter "capture
mode" for the stream and one which commands it to exit. When a process stream enters "capture mode",
data sent to the stream will be sent to a separate buffer in memory, the "capture buffer", which is
allowed to contain a maximum of capture_maxbytes bytes. During capture mode, when the buffer's length
exceeds capture_maxbytes bytes, the earliest data in the buffer is discarded to make room for new data.
When a process stream exits capture mode, a PROCESS_COMMUNICATION event subtype is emitted by supervisor,
which may be intercepted by event listeners.
The tag to begin "capture mode" in a process stream is <!--XSUPERVISOR:BEGIN-->. The tag to exit capture
mode is <!--XSUPERVISOR:END-->. The data between these tags may be arbitrary, and forms the payload of
the PROCESS_COMMUNICATION event. For example, if a program is set up with a stdout_capture_maxbytes of
"1MB", and it emits the following on its stdout stream:
<!--XSUPERVISOR:BEGIN-->Hello!<!--XSUPERVISOR:END-->
In this circumstance, supervisord will emit a PROCESS_COMMUNICATIONS_STDOUT event with data in the
payload of "Hello!".
An example of a script (written in Python) which emits a process communication event is in the scripts
directory of the supervisor package, named sample_commevent.py.
The output of processes specified as "event listeners" ([eventlistener:x] sections) is not processed this
way. Output from these processes cannot enter capture mode.
Extending Supervisor's XML-RPC API
Supervisor can be extended with new XML-RPC APIs. Several third-party plugins already exist that can be
wired into your Supervisor configuration. You may additionally write your own. Extensible XML-RPC
interfaces is an advanced feature, introduced in version 3.0. You needn't understand it unless you wish
to use an existing third-party RPC interface plugin or if you wish to write your own RPC interface
plugin.
Configuring XML-RPC Interface Factories
An additional RPC interface is configured into a supervisor installation by adding a [rpcinterface:x]
section in the Supervisor configuration file.
In the sample config file, there is a section which is named [rpcinterface:supervisor]. By default it
looks like this:
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
This section must remain in the configuration for the standard setup of supervisor to work properly. If
you don't want supervisor to do anything it doesn't already do out of the box, this is all you need to
know about this type of section.
However, if you wish to add additional XML-RPC interface namespaces to a configuration of supervisor, you
may add additional [rpcinterface:foo] sections, where "foo" represents the namespace of the interface
(from the web root), and the value named by supervisor.rpcinterface_factory is a factory callable written
in Python which should have a function signature that accepts a single positional argument supervisord
and as many keyword arguments as required to perform configuration. Any key/value pairs defined within
the rpcinterface:foo section will be passed as keyword arguments to the factory. Here's an example of a
factory function, created in the package my.package.
def make_another_rpcinterface(supervisord, **config):
retries = int(config.get('retries', 0))
another_rpc_interface = AnotherRPCInterface(supervisord, retries)
return another_rpc_interface
And a section in the config file meant to configure it.
[rpcinterface:another]
supervisor.rpcinterface_factory = my.package:make_another_rpcinterface
retries = 1
Glossary
daemontools
A process control system by D.J. Bernstein.
launchd
A process control system used by Apple as process 1 under Mac OS X.
runit A process control system.
Superlance
A package which provides various event listener implementations that plug into Supervisor which
can help monitor process memory usage and crash status: http://pypi.python.org/pypi/superlance.
umask Abbreviation of user mask: sets the file mode creation mask of the current process. See
http://en.wikipedia.org/wiki/Umask.
API DOCUMENTATION
XML-RPC API Documentation
To use the XML-RPC interface, connect to supervisor's HTTP port with any XML-RPC client library and run
commands against it. An example of doing this using Python's xmlrpclib client library is as follows.
import xmlrpclib
server = xmlrpclib.Server('http://localhost:9001/RPC2')
You may call methods against supervisord and its subprocesses by using the supervisor namespace. An
example is provided below.
server.supervisor.getState()
You can get a list of methods supported by the supervisord XML-RPC interface by using the XML-RPC
system.listMethods API:
server.system.listMethods()
You can see help on a method by using the system.methodHelp API against the method:
server.system.methodHelp('supervisor.shutdown')
The supervisord XML-RPC interface also supports the XML-RPC multicall API.
You can extend supervisord functionality with new XML-RPC API methods by adding new top-level RPC
interfaces as necessary. See rpcinterface_factories.
NOTE:
Any XML-RPC method call may result in a fault response. This includes errors caused by the client
such as bad arguments, and any errors that make supervisord unable to fulfill the request. Many
XML-RPC client programs will raise an exception when a fault response is encountered.
Status and Control
class supervisor.rpcinterface.SupervisorNamespaceRPCInterface(supervisord)
getAPIVersion()
Return the version of the RPC API used by supervisord
@return string version version id
This API is versioned separately from Supervisor itself. The API version returned by
getAPIVersion only changes when the API changes. Its purpose is to help the client
identify with which version of the Supervisor API it is communicating.
When writing software that communicates with this API, it is highly recommended that you
first test the API version for compatibility before making method calls.
NOTE:
The getAPIVersion method replaces getVersion found in Supervisor versions prior to
3.0a1. It is aliased for compatibility but getVersion() is deprecated and support
will be dropped from Supervisor in a future version.
getSupervisorVersion()
Return the version of the supervisor package in use by supervisord
@return string version version id
getIdentification()
Return identifiying string of supervisord
@return string identifier identifying string
This method allows the client to identify with which Supervisor instance it is
communicating in the case of environments where multiple Supervisors may be running.
The identification is a string that must be set in Supervisor’s configuration file. This
method simply returns that value back to the client.
getState()
Return current state of supervisord as a struct
@return struct A struct with keys int statecode, string statename
This is an internal value maintained by Supervisor that determines what Supervisor
believes to be its current operational state.
Some method calls can alter the current state of the Supervisor. For example, calling
the method supervisor.shutdown() while the station is in the RUNNING state places the
Supervisor in the SHUTDOWN state while it is shutting down.
The supervisor.getState() method provides a means for the client to check Supervisor's
state, both for informational purposes and to ensure that the methods it intends to call
will be permitted.
The return value is a struct:
{'statecode': 1,
'statename': 'RUNNING'}
The possible return values are:
─────────────────────────────────────────────────────────
statecode statename Description
─────────────────────────────────────────────────────────
2 FATAL Supervisor has experienced a
serious error.
─────────────────────────────────────────────────────────
1 RUNNING Supervisor is working
normally.
─────────────────────────────────────────────────────────
0 RESTARTING Supervisor is in the process
of restarting.
─────────────────────────────────────────────────────────
-1 SHUTDOWN Supervisor is in the process
of shutting down.
┌───────────┬────────────┬──────────────────────────────┐
│ │ │ │
--
PLUGINS
INDICES AND TABLES
• genindex
• modindex
• search
AUTHOR
This man page was created by Orestis Ioannou <orestis@oioannou.com> using the official documentation.
COPYRIGHT
2004-2015, Agendaless Consulting and Contributors
3.2.0 December 10, 2015 SUPERVISOR(1)