Provided by: masakari-monitors-common_17.0.1-0ubuntu1_all 
NAME
masakari-monitors - masakari-monitors 17.0.1
Contents:
MASAKARI-MONITORS
Monitors for Masakari
Monitors for Masakari provides Virtual Machine High Availability (VMHA) service for OpenStack clouds by
automatically detecting the failure events such as VM process down, provisioning process down, and
nova-compute host failure. If it detect the events, it sends notifications to the masakari-api.
Original version of Masakari: https://github.com/ntt-sic/masakari
Tokyo Summit Session: https://www.youtube.com/watch?v=BmjNKceW_9A
Monitors for Masakari is distributed under the terms of the Apache License, Version 2.0. The full terms
and conditions of this license are detailed in the LICENSE file.
• Free software: Apache license
• Documentation: https://docs.openstack.org/masakari-monitors
• Source: https://git.openstack.org/cgit/openstack/masakari-monitors
• Bugs: https://bugs.launchpad.net/masakari-monitors
Configure masakari-monitors
1. Clone masakari using:
$ git clone https://github.com/openstack/masakari-monitors.git
2. Create masakarimonitors directory in /etc/.
3. Run setup.py from masakari-monitors:
$ sudo python setup.py install
4. Copy masakarimonitors.conf and process_list.yaml files from masakari-monitors/etc/ to
/etc/masakarimonitors folder and make necessary changes to the masakarimonitors.conf and
process_list.yaml files. To generate the sample masakarimonitors.conf file, run the following command
from the top level of the masakari-monitors directory:
$ tox -egenconfig
5. To run masakari-processmonitor, masakari-hostmonitor and masakari-instancemonitor simply use following
binary:
$ masakari-processmonitor
$ masakari-hostmonitor
$ masakari-instancemonitor
Features
• TODO
INSTALLATION
At the command line:
$ pip install masakari-monitors
Or, if you have virtualenvwrapper installed:
$ mkvirtualenv masakari-monitors
$ pip install masakari-monitors
USAGE
Monitors for Masakari:
masakari-hostmonitor
Monitor Overview
The masakari-hostmonitor provides compute node High Availability for OpenStack clouds by automatically
detecting compute nodes failure via monitor driver.
How does it work based on pacemaker & corosync?
• Pacemaker or pacemaker-remote is required to install into compute nodes to form a pacemaker cluster.
• The compute node’s status is depending on the heartbeat between the compute node and the cluster. Once
the node lost the heartbeat, masakari-hostmonitor in other nodes will detect the failure and send
notifications to masakari-api.
How does it work based on consul?
• If the nodes in the cloud have multiple interfaces to connect to management network, tenant network or
storage network, monitor driver based on consul is another choice. Consul agents are required to
install into all noedes, which make up multiple consul clusters.
Here is an example to show how to make up one consul cluster.
Consul Usage
Consul overview
Consul is a service mesh solution providing a full featured control plane with service discovery,
configuration, and segmentation functionality. Each of these features can be used individually as
needed, or they can be used together to build a full service mesh.
The Consul agent is the core process of Consul. The Consul agent maintains membership information,
registers services, runs checks, responds to queries, and more.
Consul clients can provide any number of health checks, either associated with a given service or with
the local node. This information can be used by an operator to monitor cluster health.
Please refer to Consul Agent Overview.
Test Environment
There are three controller nodes and two compute nodes in the test environment. Every node has three
network interfaces. The first interface is used for management, with an ip such as ‘192.168.101.*’. The
second interface is used to connect to storage, with an ip such as ‘192.168.102.*’. The third interface
is used for tenant, with an ip such as ‘192.168.103.*’.
Download Consul
Download Consul package for CentOS. Other OS please refer to Download Consul.
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo yum -y install Consul
Configure Consul agent
Consul agent must runs on every node. Consul server agent runs on controller nodes, while Consul client
agent runs on compute nodes, which makes up one Consul cluster.
The following is an example of a config file for Consul server agent which binds to management interface
of the host.
management.json
{
"bind_addr": "192.168.101.1",
"datacenter": "management",
"data_dir": "/tmp/consul_m",
"log_level": "INFO",
"server": true,
"bootstrap_expect": 3,
"node_name": "node01",
"addresses": {
"http": "192.168.101.1"
},
"ports": {
"http": 8500,
"serf_lan": 8501
},
"retry_join": ["192.168.101.1:8501", "192.168.101.2:8501", "192.168.101.3:8501"]
}
The following is an example of a config file for Consul client agent which binds to management
interface of the host.
management.json
{
"bind_addr": "192.168.101.4",
"datacenter": "management",
"data_dir": "/tmp/consul_m",
"log_level": "INFO",
"node_name": "node04",
"addresses": {
"http": "192.168.101.4"
},
"ports": {
"http": 8500,
"serf_lan": 8501
},
"retry_join": ["192.168.101.1:8501", "192.168.101.2:8501", "192.168.101.3:8501"]
}
Use the tenant or storage interface ip and ports when config agent in tenant or storage datacenter.
Please refer to Consul Agent Configuration.
Start Consul agent
The Consul agent is started by the following command.
# Consul agent –config-file management.json
Test Consul installation
After all Consul agents installed and started, you can see all nodes in the cluster by the following
command.
# Consul members -http-addr=192.168.101.1:8500
Node Address Status Type Build Protocol DC
node01 192.168.101.1:8501 alive server 1.10.2 2 management
node02 192.168.101.2:8501 alive server 1.10.2 2 management
node03 192.168.101.3:8501 alive server 1.10.2 2 management
node04 192.168.101.4:8501 alive client 1.10.2 2 management
node05 192.168.101.5:8501 alive client 1.10.2 2 management
• The compute node’s status is depending on assembly of multiple interfaces connectivity status, which
are retrieved from multiple consul clusters. Then it sends notifition to trigger host failure recovery
according to defined HA strategy - host states and the corresponding actions.
Related configurations
This section in masakarimonitors.conf shows an example of how to configure the hostmonitor if you choice
monitor driver based on pacemaker.
[host]
# Driver that hostmonitor uses for monitoring hosts.
monitoring_driver = default
# Monitoring interval(in seconds) of node status.
monitoring_interval = 60
# Do not check whether the host is completely down.
# Possible values:
# * True: Do not check whether the host is completely down.
# * False: Do check whether the host is completely down.
# If ipmi RA is not set in pacemaker, this value should be set True.
disable_ipmi_check = False
# Timeout value(in seconds) of the ipmitool command.
ipmi_timeout = 5
# Number of ipmitool command retries.
ipmi_retry_max = 3
# Retry interval(in seconds) of the ipmitool command.
ipmi_retry_interval = 10
# Only monitor pacemaker-remotes, ignore the status of full cluster
# members.
restrict_to_remotes = False
# Standby time(in seconds) until activate STONITH.
stonith_wait = 30
# Timeout value(in seconds) of the tcpdump command when monitors
# the corosync communication.
tcpdump_timeout = 5
# The name of interface that corosync is using for mutual communication
# between hosts.
# If there are multiple interfaces, specify them in comma-separated
# like 'enp0s3,enp0s8'.
# The number of interfaces you specify must be equal to the number of
# corosync_multicast_ports values and must be in correct order with
# relevant ports in corosync_multicast_ports.
corosync_multicast_interfaces = enp0s3,enp0s8
# The port numbers that corosync is using for mutual communication
# between hosts.
# If there are multiple port numbers, specify them in comma-separated
# like '5405,5406'.
# The number of port numbers you specify must be equal to the number of
# corosync_multicast_interfaces values and must be in correct order with
# relevant interfaces in corosync_multicast_interfaces.
corosync_multicast_ports = 5405,5406
If you want to use or test monitor driver based on consul, please modify following configuration.
[host]
# Driver that hostmonitor uses for monitoring hosts.
monitoring_driver = consul
[consul]
# Addr for local consul agent in management datacenter.
# The addr is make up of the agent's bind_addr and http port,
# such as '192.168.101.1:8500'.
agent_manage = $(CONSUL_MANAGEMENT_ADDR)
# Addr for local consul agent in tenant datacenter.
agent_tenant = $(CONSUL_TENANT_ADDR)
# Addr for local consul agent in storage datacenter.
agent_storage = $(CONSUL_STORAGE_ADDR)
# Config file for consul health action matrix.
matrix_config_file = /etc/masakarimonitors/matrix.yaml
The matrix_config_file shows the HA strategy. Matrix is combined by host health and actions. The ‘health:
[x, x, x]’, repreasents assembly status of SEQUENCE. Action, means which actions it will trigger if host
health turns into, while ‘recovery’ means it will trigger one host failure recovery workflow. User can
define the HA strategy according to the physical environment. For example, if there is just 1 cluster to
monitor management network connectivity, the user just need to configurate $(CONSUL_MANAGEMENT_ADDR) in
consul section of the hostmontior’ configuration file, and change the HA strategy in
/etc/masakarimonitors/matrix.yaml as following:
sequence: ['manage']
matrix:
- health: ['up']
action: []
- health: ['down']
action: ['recovery']
Then the hostmonitor by consul works as same as the hostmonitor by pacemaker.
masakari-instancemonitor
Monitor Overview
The masakari-instancemonitor provides Virtual Machine High Availability for OpenStack clouds by
automatically detecting VMs domain events via libvirt. If it detects specific libvirt events, it sends
notifications to the masakari-api.
How does it work?
• It runs libvirt event loop in a background thread.
• Invoking libvirt.virEventRegisterDefaultImpl() will register libvirt’s default event loop
implementation.
• Invoking libvirt.virEventRunDefaultImpl() will perform one iteration of the libvirt default event
loop.
• Invoking conn.domainEventRegisterAny() will register event callbacks against libvirt connection
instances. The callbacks registered will be triggered from the execution context of
libvirt.virEventRunDefaultImpl(), which will send notifications to the masakari-api.
• It will reconnect to libvirt and reprocess if disconnected.
Related configurations
This section in masakarimonitors.conf shows an example of how to configure the monitor.
[libvirt]
# Override the default libvirt URI.
connection_uri = qemu:///system
masakari-introspectiveinstancemonitor
Monitor Overview
The masakari-introspectiveinstancemonitor provides Virtual Machine HA for OpenStack clouds by
automatically detecting the system-level failure events via QEMU Guest Agent. If it detects VM heartbeat
failure events, it sends notifications to the masakari-api.
How does it work?
• libvirt and QEMU Guest Agent are used as the underlying protocol for messaging to and from VM.
• The host-side qemu-agent sockets are used to detemine whether VMs are configured with QEMU Guest
Agent.
• qemu-guest-ping is used as the monitoring heartbeat.
• For the future release, we can pass through arbitrary guest agent commands to check the health of the
applications inside a VM.
Related configurations
This section in masakarimonitors.conf shows an example of how to configure the monitor.
[libvirt]
# Override the default libvirt URI.
connection_uri = qemu:///system
[introspectiveinstancemonitor]
# Guest monitoring interval of VM status (in seconds).
# * The value should not be too low as there should not be false negative
# * for reporting QEMU_GUEST_AGENT failures
# * VM needs time to do powering-off.
# * guest_monitoring_interval should be greater than
# * the time to SHUTDOWN VM gracefully.
guest_monitoring_interval = 10
# Guest monitoring timeout (in seconds).
guest_monitoring_timeout = 2
# Failure threshold before sending notification.
guest_monitoring_failure_threshold = 3
# The file path of qemu guest agent sock.
qemu_guest_agent_sock_path = \
/var/lib/libvirt/qemu/org\.qemu\.guest_agent\..*\.instance-.*\.sock
masakari-processmonitor
Monitor Overview
The masakari-processmonitor, provides key process High Availability for OpenStack clouds by automatically
detecting the process failure. If it detects process failure, it sends notifications to masakari-api.
If your OpenStack service runs in container(pod), this processmonitor will not work as expected. It is
recommended not to deploy processmonitor.
How does it work?
• Processes to be monitored should be pre-configured in process_list.yaml file.
Define one process to be monitored as follows:
process_name: [Name of the process as it in 'ps -ef'.]
start_command: [Start command of the process.]
pre_start_command: [Command which is executed before start_command.]
post_start_command: [Command which is executed after start_command.]
restart_command: [Restart command of the process.]
pre_restart_command: [Command which is executed before restart_command.]
post_restart_command: [Command which is executed after restart_command.]
run_as_root: [Bool value whether to execute commands as root authority.]
Sample of definitions is shown as follows:
# nova-compute
process_name: /usr/local/bin/nova-compute
start_command: systemctl start nova-compute
pre_start_command:
post_start_command:
restart_command: systemctl restart nova-compute
pre_restart_command:
post_restart_command:
run_as_root: True
• If masakari-processmonitor detects one process failure, it will try to restart it firstly. After
several retries failed, it sends notification to masakari-api.
Related configurations
This section in masakarimonitors.conf shows an example of how to configure the monitor.
[process]
# Interval in seconds for checking a process.
check_interval = 5
# Number of retries when the failure of restarting a process.
restart_retries = 3
# Interval in seconds for restarting a process.
restart_interval = 5
# The file path of process list.
process_list_path = /etc/masakarimonitors/process_list.yaml
MASAKARI MONITORS CONFIGURATION OPTIONS
The following is an overview of all available configuration options in masakari-monitors. To see sample
configuration file, see Masakari Monitors Sample Configuration File.
DEFAULT
tempdir
Type string
Default
<None>
Explicitly specify the temporary working directory.
monkey_patch
Type boolean
Default
False
Determine if monkey patching should be applied.
Related options:
• monkey_patch_modules: This must have values set for this option to have
any effect
monkey_patch_modules
Type list
Default
[]
List of modules/decorators to monkey patch.
This option allows you to patch a decorator for all functions in specified modules.
Related options:
• monkey_patch: This must be set to True for this option to have any effect
hostname
Type string
Default
lcy02-amd64-081
Hostname, FQDN or IP address of this host. Must be valid within AMQP key.
Possible values:
• String with hostname, FQDN or IP address. Default is hostname of this host.
DEPRECATED VARIATIONS
┌─────────┬──────┐
│ Group │ Name │
├─────────┼──────┤
│ DEFAULT │ host │
└─────────┴──────┘
instancemonitor_manager
Type string
Default
masakarimonitors.instancemonitor.instance.InstancemonitorManager
Full class name for the Manager for instancemonitor.
introspectiveinstancemonitor_manager
Type string
Default
masakarimonitors.introspectiveinstancemonitor.instance.IntrospectiveInstanceMonitorManager
Full class name for introspectiveinstancemonitor.
processmonitor_manager
Type string
Default
masakarimonitors.processmonitor.process.ProcessmonitorManager
Full class name for the Manager for processmonitor.
hostmonitor_manager
Type string
Default
masakarimonitors.hostmonitor.host.HostmonitorManager
Full class name for the Manager for hostmonitor.
debug
Type boolean
Default
False
Mutable
This option can be changed without restarting.
If set to true, the logging level will be set to DEBUG instead of the default INFO level.
log_config_append
Type string
Default
<None>
Mutable
This option can be changed without restarting.
The name of a logging configuration file. This file is appended to any existing logging
configuration files. For details about logging configuration files, see the Python logging module
documentation. Note that when logging configuration files are used then all logging configuration
is set in the configuration file and other logging configuration options are ignored (for example,
log-date-format).
DEPRECATED VARIATIONS
┌─────────┬────────────┐
│ Group │ Name │
├─────────┼────────────┤
│ DEFAULT │ log-config │
├─────────┼────────────┤
│ DEFAULT │ log_config │
└─────────┴────────────┘
log_date_format
Type string
Default
%Y-%m-%d %H:%M:%S
Defines the format string for %(asctime)s in log records. Default: the value above . This option
is ignored if log_config_append is set.
log_file
Type string
Default
<None>
(Optional) Name of log file to send logging output to. If no default is set, logging will go to
stderr as defined by use_stderr. This option is ignored if log_config_append is set.
DEPRECATED VARIATIONS
┌─────────┬─────────┐
│ Group │ Name │
├─────────┼─────────┤
│ DEFAULT │ logfile │
└─────────┴─────────┘
log_dir
Type string
Default
<None>
(Optional) The base directory used for relative log_file paths. This option is ignored if
log_config_append is set.
DEPRECATED VARIATIONS
┌─────────┬────────┐
│ Group │ Name │
├─────────┼────────┤
│ DEFAULT │ logdir │
└─────────┴────────┘
watch_log_file
Type boolean
Default
False
Uses logging handler designed to watch file system. When log file is moved or removed this handler
will open a new log file with specified path instantaneously. It makes sense only if log_file
option is specified and Linux platform is used. This option is ignored if log_config_append is
set.
use_syslog
Type boolean
Default
False
Use syslog for logging. Existing syslog format is DEPRECATED and will be changed later to honor
RFC5424. This option is ignored if log_config_append is set.
use_journal
Type boolean
Default
False
Enable journald for logging. If running in a systemd environment you may wish to enable journal
support. Doing so will use the journal native protocol which includes structured metadata in
addition to log messages.This option is ignored if log_config_append is set.
syslog_log_facility
Type string
Default
LOG_USER
Syslog facility to receive log lines. This option is ignored if log_config_append is set.
use_json
Type boolean
Default
False
Use JSON formatting for logging. This option is ignored if log_config_append is set.
use_stderr
Type boolean
Default
False
Log output to standard error. This option is ignored if log_config_append is set.
use_eventlog
Type boolean
Default
False
Log output to Windows Event Log.
WARNING:
This option is deprecated for removal. Its value may be silently ignored in the future.
Reason Windows support is no longer maintained.
log_rotate_interval
Type integer
Default
1
The amount of time before the log files are rotated. This option is ignored unless
log_rotation_type is set to “interval”.
log_rotate_interval_type
Type string
Default
days
Valid Values
Seconds, Minutes, Hours, Days, Weekday, Midnight
Rotation interval type. The time of the last file change (or the time when the service was
started) is used when scheduling the next rotation.
max_logfile_count
Type integer
Default
30
Maximum number of rotated log files.
max_logfile_size_mb
Type integer
Default
200
Log file maximum size in MB. This option is ignored if “log_rotation_type” is not set to “size”.
log_rotation_type
Type string
Default
none
Valid Values
interval, size, none
Log rotation type.
Possible values
interval
Rotate logs at predefined time intervals.
size Rotate logs once they reach a predefined size.
none Do not rotate log files.
logging_context_format_string
Type string
Default
%(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [%(global_request_id)s
%(request_id)s %(user_identity)s] %(instance)s%(message)s
Format string to use for log messages with context. Used by oslo_log.formatters.ContextFormatter
logging_default_format_string
Type string
Default
%(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [-] %(instance)s%(message)s
Format string to use for log messages when context is undefined. Used by
oslo_log.formatters.ContextFormatter
logging_debug_format_suffix
Type string
Default
%(funcName)s %(pathname)s:%(lineno)d
Additional data to append to log message when logging level for the message is DEBUG. Used by
oslo_log.formatters.ContextFormatter
logging_exception_prefix
Type string
Default
%(asctime)s.%(msecs)03d %(process)d ERROR %(name)s %(instance)s
Prefix each line of exception output with this format. Used by
oslo_log.formatters.ContextFormatter
logging_user_identity_format
Type string
Default
%(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s
Defines the format string for %(user_identity)s that is used in logging_context_format_string.
Used by oslo_log.formatters.ContextFormatter
default_log_levels
Type list
Default
['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO',
'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN',
'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN',
'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN',
'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN',
'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO']
List of package logging levels in logger=LEVEL pairs. This option is ignored if log_config_append
is set.
publish_errors
Type boolean
Default
False
Enables or disables publication of error events.
instance_format
Type string
Default
"[instance: %(uuid)s] "
The format for an instance that is passed with the log message.
instance_uuid_format
Type string
Default
"[instance: %(uuid)s] "
The format for an instance UUID that is passed with the log message.
rate_limit_interval
Type integer
Default
0
Interval, number of seconds, of log rate limiting.
rate_limit_burst
Type integer
Default
0
Maximum number of logged messages per rate_limit_interval.
rate_limit_except_level
Type string
Default
CRITICAL
Log level name used by rate limiting: CRITICAL, ERROR, INFO, WARNING, DEBUG or empty string. Logs
with level greater or equal to rate_limit_except_level are not filtered. An empty string means
that all levels are filtered.
fatal_deprecations
Type boolean
Default
False
Enables or disables fatal status of deprecations.
api
Configuration options for sending notifications.
region
Type string
Default
RegionOne
Region name.
api_version
Type string
Default
v1
Masakari API Version.
api_interface
Type string
Default
public
Interface of endpoint.
auth_url
Type unknown type
Default
<None>
Authentication URL
system_scope
Type unknown type
Default
<None>
Scope for system operations
domain_id
Type unknown type
Default
<None>
Domain ID to scope to
domain_name
Type unknown type
Default
<None>
Domain name to scope to
project_id
Type unknown type
Default
<None>
Project ID to scope to
DEPRECATED VARIATIONS
┌───────┬───────────┐
│ Group │ Name │
├───────┼───────────┤
│ api │ tenant-id │
├───────┼───────────┤
│ api │ tenant_id │
└───────┴───────────┘
project_name
Type unknown type
Default
<None>
Project name to scope to
DEPRECATED VARIATIONS
┌───────┬─────────────┐
│ Group │ Name │
├───────┼─────────────┤
│ api │ tenant-name │
├───────┼─────────────┤
│ api │ tenant_name │
└───────┴─────────────┘
project_domain_id
Type unknown type
Default
<None>
Domain ID containing project
project_domain_name
Type unknown type
Default
<None>
Domain name containing project
trust_id
Type unknown type
Default
<None>
ID of the trust to use as a trustee use
default_domain_id
Type unknown type
Default
<None>
Optional domain ID to use with v3 and v2 parameters. It will be used for both the user and project
domain in v3 and ignored in v2 authentication.
default_domain_name
Type unknown type
Default
<None>
Optional domain name to use with v3 API and v2 parameters. It will be used for both the user and
project domain in v3 and ignored in v2 authentication.
user_id
Type unknown type
Default
<None>
User id
username
Type unknown type
Default
<None>
Username
DEPRECATED VARIATIONS
─────────────────────
Group Name
─────────────────────
api user-name
─────────────────────
api user_name
┌───────┬───────────┐
│ │ │
user_domain_id │ │ │
│ │ │
Type unknown type │ │ │
│ │ │
Default │ │ │
<None> │ │ │
│ │ │
User’s domain id │ │ │
│ │ │
user_domain_name │ │ │
│ │ │
Type unknown type │ │ │
│ │ │
Default │ │ │
<None> │ │ │
│ │ │
User’s domain name │ │ │
│ │ │
password │ │ │
│ │ │
Type unknown type │ │ │
│ │ │
Default │ │ │
<None> │ │ │
│ │ │
User’s password │ │ │
│ │ │
callback │ │ │
retry_max │ │ │
│ │ │
Type integer │ │ │
│ │ │
Default │ │ │
12 │ │ │
│ │ │
--
DEPRECATED VARIATIONS
─────────────────────────────────────────
Group Name
─────────────────────────────────────────
DEFAULT osapi_max_request_body_size
─────────────────────────────────────────
DEFAULT max_request_body_size
┌─────────┬─────────────────────────────┐
│ │ │
enable_proxy_headers_parsing │ │ │
│ │ │
Type boolean │ │ │
│ │ │
Default │ │ │
False │ │ │
│ │ │
--
MASAKARI MONITORS SAMPLE CONFIGURATION│FILE │ │
--
AUTHOR
unknown
COPYRIGHT
2024, OpenStack Foundation
17.0.1 Apr 05, 2024 MASAKARI-MONITORS(1)