Tribblix: manual page: corosync.conf.5

COROSYNC_CONF(5) Corosync Cluster Engine Programmer's Manual

NAME

corosync.conf - corosync executive configuration file

SYNOPSIS

/etc/corosync/corosync.conf

DESCRIPTION

The corosync.conf instructs the corosync executive about various
parameters needed to control the corosync executive. Empty lines and
lines starting with # character are ignored. The configuration file
consists of bracketed top level directives. The possible directive
choices are:

totem { }
This top level directive contains configuration options for
the totem protocol.

logging { }
This top level directive contains configuration options for
logging.

quorum { }
This top level directive contains configuration options for
quorum.

nodelist { }
This top level directive contains configuration options for
nodes in cluster.

qb { } This top level directive contains configuration options
related to libqb.

resources { }
This top level directive contains configuration options for
resources.

Within the totem directive, for UDP transport, an interface section
is required.

For UDPU transport, an interface section is not needed and it is
recommended that the nodelist is used to define cluster nodes.

Within the interface sub-directive of totem there are four parameters
which are required. There is one parameter which is optional.

ringnumber
This specifies the ring number for the interface. When using
the redundant ring protocol, each interface should specify
separate ring numbers to uniquely identify to the membership
protocol which interface to use for which redundant ring. The
ringnumber must start at 0.

bindnetaddr
This specifies the network address the corosync executive
should bind to.

bindnetaddr should be an IP address configured on the system,
or a network address.

For example, if the local interface is 192.168.5.92 with
netmask 255.255.255.0, you should set bindnetaddr to
192.168.5.92 or 192.168.5.0. If the local interface is
192.168.5.92 with netmask 255.255.255.192, set bindnetaddr to
192.168.5.92 or 192.168.5.64, and so forth.

This may also be an IPV6 address, in which case IPV6
networking will be used. In this case, the exact address must
be specified and there is no automatic selection of the
network interface within a specific subnet as with IPv4.

If IPv6 networking is used, the nodeid field in nodelist must
be specified.

broadcast
This is optional and can be set to yes. If it is set to yes,
the broadcast address will be used for communication. If this
option is set, mcastaddr should not be set.

mcastaddr
This is the multicast address used by corosync executive. The
default should work for most networks, but the network
administrator should be queried about a multicast address to
use. Avoid 224.x.x.x because this is a "config" multicast
address.

This may also be an IPV6 multicast address, in which case IPV6
networking will be used. If IPv6 networking is used, the
nodeid field in nodelist must be specified.

It's not needed to use this option if cluster_name option is
used. If both options are used, mcastaddr has higher priority.

mcastport
This specifies the UDP port number. It is possible to use the
same multicast address on a network with the corosync services
configured for different UDP ports. Please note corosync uses
two UDP ports mcastport (for mcast receives) and mcastport - 1
(for mcast sends). If you have multiple clusters on the same
network using the same mcastaddr please configure the
mcastports with a gap.

ttl This specifies the Time To Live (TTL). If you run your cluster
on a routed network then the default of "1" will be too small.
This option provides a way to increase this up to 255. The
valid range is 0..255. Note that this is only valid on
multicast transport types.

Within the totem directive, there are seven configuration options of
which one is required, five are optional, and one is required when
IPV6 is configured in the interface subdirective. The required
directive controls the version of the totem configuration. The
optional option unless using IPV6 directive controls identification
of the processor. The optional options control secrecy and
authentication, the redundant ring mode of operation and maximum
network MTU field.

version
This specifies the version of the configuration file.
Currently the only valid version for this directive is 2.

clear_node_high_bit
This configuration option is optional and is only relevant
when no nodeid is specified. Some corosync clients require a
signed 32 bit nodeid that is greater than zero however by
default corosync uses all 32 bits of the IPv4 address space
when generating a nodeid. Set this option to yes to force the
high bit to be zero and therefor ensure the nodeid is a
positive signed 32 bit integer.

WARNING: The clusters behavior is undefined if this option is
enabled on only a subset of the cluster (for example during a
rolling upgrade).

crypto_hash
This specifies which HMAC authentication should be used to
authenticate all messages. Valid values are none (no
authentication), md5, sha1, sha256, sha384 and sha512.

The default is sha1.

crypto_cipher
This specifies which cipher should be used to encrypt all
messages. Valid values are none (no encryption), aes256,
aes192, aes128 and 3des. Enabling crypto_cipher, requires
also enabling of crypto_hash.

The default is aes256.

secauth
This specifies that HMAC/SHA1 authentication should be used to
authenticate all messages. It further specifies that all data
should be encrypted with the nss library and aes256 encryption
algorithm to protect data from eavesdropping.

Enabling this option adds a encryption header to every message
sent by totem which reduces total throughput. Also encryption
and authentication consume extra CPU cycles in corosync.

The default is on.

WARNING: This parameter is deprecated. It's recommended to use
combination of crypto_cipher and crypto_hash.

rrp_mode
This specifies the mode of redundant ring, which may be none,
active, or passive. Currently only 'passive' is supported or
tested (using 'active' is not recommended). Active
replication offers slightly lower latency from transmit to
delivery in faulty network environments but with less
performance. Passive replication may nearly double the speed
of the totem protocol if the protocol doesn't become cpu
bound. The final option is none, in which case only one
network interface will be used to operate the totem protocol.

If only one interface directive is specified, none is
automatically chosen. If multiple interface directives are
specified, only active or passive may be chosen.

The maximum number of interface directives that is allowed for
either modes (active or passive) is 2.

When using multiple interfaces, make sure to use different
multicast address/port (port for same address must differ by
at least two) pair for each interface (this is checked by
parser) to make rrp works.

netmtu This specifies the network maximum transmit unit. To set this
value beyond 1500, the regular frame MTU, requires ethernet
devices that support large, or also called jumbo, frames. If
any device in the network doesn't support large frames, the
protocol will not operate properly. The hosts must also have
their mtu size set from 1500 to whatever frame size is
specified here.

Please note while some NICs or switches claim large frame
support, they support 9000 MTU as the maximum frame size
including the IP header. Setting the netmtu and host MTUs to
9000 will cause totem to use the full 9000 bytes of the frame.
Then Linux will add a 18 byte header moving the full frame
size to 9018. As a result some hardware will not operate
properly with this size of data. A netmtu of 8982 seems to
work for the few large frame devices that have been tested.
Some manufacturers claim large frame support when in fact they
support frame sizes of 4500 bytes.

When sending multicast traffic, if the network frequently
reconfigures, chances are that some device in the network
doesn't support large frames.

Choose hardware carefully if intending to use large frame
support.

The default is 1500.

transport
This directive controls the transport mechanism used. If the
interface to which corosync is binding is an RDMA interface
such as RoCEE or Infiniband, the "iba" parameter may be
specified. To avoid the use of multicast entirely, a unicast
transport parameter "udpu" can be specified. This requires
specifying the list of members in nodelist directive, that
could potentially make up the membership before deployment.

The default is udp. The transport type can also be set to
udpu or iba.

cluster_name
This specifies the name of cluster and it's used for automatic
generating of multicast address.

config_version
This specifies version of config file. This is converted to
unsigned 64-bit int. By default it's 0. Option is used to
prevent joining old nodes with not up-to-date configuration.
If value is not 0, and node is going for first time (only for
first time, join after split doesn't follow this rules) from
single-node membership to multiple nodes membership, other
nodes config_versions are collected. If current node
config_version is not equal to highest of collected versions,
corosync is terminated.

ip_version
Specifies version of IP to use for communication. Value can be
one of ipv4 or ipv6. Default (if unspecified) is ipv4.

Within the totem directive, there are several configuration
options which are used to control the operation of the
protocol. It is generally not recommended to change any of
these values without proper guidance and sufficient testing.
Some networks may require larger values if suffering from
frequent reconfigurations. Some applications may require
faster failure detection times which can be achieved by
reducing the token timeout.

token This timeout is used directly or as a base for real token
timeout calculation (explained in token_coefficient section).
Token timeout specifies in milliseconds until a token loss is
declared after not receiving a token. This is the time spent
detecting a failure of a processor in the current
configuration. Reforming a new configuration takes about 50
milliseconds in addition to this timeout.

For real token timeout used by totem it's possible to read
cmap value of runtime.config.totem.token key.

The default is 1000 milliseconds.

token_coefficient
This value is used only when nodelist section is specified and
contains at least 3 nodes. If so, real token timeout is then
computed as token + (number_of_nodes - 2) * token_coefficient.
This allows cluster to scale without manually changing token
timeout every time new node is added. This value can be set to
0 resulting in effective removal of this feature.

The default is 650 milliseconds.

token_retransmit
This timeout specifies in milliseconds after how long before
receiving a token the token is retransmitted. This will be
automatically calculated if token is modified. It is not
recommended to alter this value without guidance from the
corosync community.

The default is 238 milliseconds.

hold This timeout specifies in milliseconds how long the token
should be held by the representative when the protocol is
under low utilization. It is not recommended to alter this
value without guidance from the corosync community.

The default is 180 milliseconds.

token_retransmits_before_loss_const
This value identifies how many token retransmits should be
attempted before forming a new configuration. It is also used
for token_retransmit and hold calculations.

The default is 4 retransmissions.

join This timeout specifies in milliseconds how long to wait for
join messages in the membership protocol.

The default is 50 milliseconds.

send_join
This timeout specifies in milliseconds an upper range between
0 and send_join to wait before sending a join message. For
configurations with less than 32 nodes, this parameter is not
necessary. For larger rings, this parameter is necessary to
ensure the NIC is not overflowed with join messages on
formation of a new ring. A reasonable value for large rings
(128 nodes) would be 80msec. Other timer values must also
change if this value is changed. Seek advice from the
corosync mailing list if trying to run larger configurations.

The default is 0 milliseconds.

consensus
This timeout specifies in milliseconds how long to wait for
consensus to be achieved before starting a new round of
membership configuration. The minimum value for consensus
must be 1.2 * token. This value will be automatically
calculated at 1.2 * token if the user doesn't specify a
consensus value.

For two node clusters, a consensus larger than the join
timeout but less than token is safe. For three node or larger
clusters, consensus should be larger than token. There is an
increasing risk of odd membership changes, which still
guarantee virtual synchrony, as node count grows if consensus
is less than token.

The default is 1200 milliseconds.

merge This timeout specifies in milliseconds how long to wait before
checking for a partition when no multicast traffic is being
sent. If multicast traffic is being sent, the merge detection
happens automatically as a function of the protocol.

The default is 200 milliseconds.

downcheck
This timeout specifies in milliseconds how long to wait before
checking that a network interface is back up after it has been
downed.

The default is 1000 milliseconds.

fail_recv_const
This constant specifies how many rotations of the token
without receiving any of the messages when messages should be
received may occur before a new configuration is formed.

The default is 2500 failures to receive a message.

seqno_unchanged_const
This constant specifies how many rotations of the token
without any multicast traffic should occur before the hold
timer is started.

The default is 30 rotations.

heartbeat_failures_allowed
[HeartBeating mechanism] Configures the optional HeartBeating
mechanism for faster failure detection. Keep in mind that
engaging this mechanism in lossy networks could cause faulty
loss declaration as the mechanism relies on the network for
heartbeating.

So as a rule of thumb use this mechanism if you require
improved failure in low to medium utilized networks.

This constant specifies the number of heartbeat failures the
system should tolerate before declaring heartbeat failure e.g
3. Also if this value is not set or is 0 then the heartbeat
mechanism is not engaged in the system and token rotation is
the method of failure detection

The default is 0 (disabled).

max_network_delay
[HeartBeating mechanism] This constant specifies in
milliseconds the approximate delay that your network takes to
transport one packet from one machine to another. This value
is to be set by system engineers and please don't change if
not sure as this effects the failure detection mechanism using
heartbeat.

The default is 50 milliseconds.

window_size
This constant specifies the maximum number of messages that
may be sent on one token rotation. If all processors perform
equally well, this value could be large (300), which would
introduce higher latency from origination to delivery for very
large rings. To reduce latency in large rings(16+), the
defaults are a safe compromise. If 1 or more slow
processor(s) are present among fast processors, window_size
should be no larger than 256000 / netmtu to avoid overflow of
the kernel receive buffers. The user is notified of this by
the display of a retransmit list in the notification logs.
There is no loss of data, but performance is reduced when
these errors occur.

The default is 50 messages.

max_messages
This constant specifies the maximum number of messages that
may be sent by one processor on receipt of the token. The
max_messages parameter is limited to 256000 / netmtu to
prevent overflow of the kernel transmit buffers.

The default is 17 messages.

miss_count_const
This constant defines the maximum number of times on receipt
of a token a message is checked for retransmission before a
retransmission occurs. This parameter is useful to modify for
switches that delay multicast packets compared to unicast
packets. The default setting works well for nearly all modern
switches.

The default is 5 messages.

rrp_problem_count_timeout
This specifies the time in milliseconds to wait before
decrementing the problem count by 1 for a particular ring to
ensure a link is not marked faulty for transient network
failures.

The default is 2000 milliseconds.

rrp_problem_count_threshold
This specifies the number of times a problem is detected with
a link before setting the link faulty. Once a link is set
faulty, no more data is transmitted upon it. Also, the
problem counter is no longer decremented when the problem
count timeout expires.

A problem is detected whenever all tokens from the proceeding
processor have not been received within the
rrp_token_expired_timeout. The rrp_problem_count_threshold *
rrp_token_expired_timeout should be atleast 50 milliseconds
less then the token timeout, or a complete reconfiguration may
occur.

The default is 10 problem counts.

rrp_problem_count_mcast_threshold
This specifies the number of times a problem is detected with
multicast before setting the link faulty for passive rrp mode.
This variable is unused in active rrp mode.

The default is 10 times rrp_problem_count_threshold.

rrp_token_expired_timeout
This specifies the time in milliseconds to increment the
problem counter for the redundant ring protocol after not
having received a token from all rings for a particular
processor.

This value will automatically be calculated from the token
timeout and problem_count_threshold but may be overridden. It
is not recommended to override this value without guidance
from the corosync community.

The default is 47 milliseconds.

rrp_autorecovery_check_timeout
This specifies the time in milliseconds to check if the failed
ring can be auto-recovered.

The default is 1000 milliseconds.

block_unlisted_ips
Allow UDPU to drop packets from IP addresses that are not
known (nodes which don't exist in the nodelist) to corosync.
Value is yes or no.

This feature is mainly to protect against the joining of nodes
with outdated configurations after a cluster split. Another
use case is to allow the atomic merge of two independent
clusters.

Changing the default value is not recommended, the overhead is
tiny and an existing cluster may fail if corosync is started
on an unlisted node with an old configuration.

The default value is yes.

cancel_token_hold_on_retransmit
Allows Corosync to hold token by representative when there is
too much retransmit messages. This allows network to process
increased load without overloading it. Used mechanism is same
as described for hold directive.

Some deployments may prefer to never hold token when there is
retransmit messages. If so, option should be set to yes.

The default value is no.

Within the logging directive, there are several configuration options
which are all optional.

The following 3 options are valid only for the top level logging
directive:

timestamp
This specifies that a timestamp is placed on all log messages.

The default is off.

fileline
This specifies that file and line should be printed.

The default is off.

function_name
This specifies that the code function name should be printed.

The default is off.

blackbox
This specifies that blackbox functionality should be enabled.

The default is on.

The following options are valid both for top level logging directive
and they can be overridden in logger_subsys entries.

to_stderr

to_logfile

to_syslog
These specify the destination of logging output. Any
combination of these options may be specified. Valid options
are yes and no.

The default is syslog and stderr.

Please note, if you are using to_logfile and want to rotate
the file, use logrotate(8) with the option copytruncate. eg.
/var/log/corosync.log {
missingok
compress
notifempty
daily
rotate 7
copytruncate
}

logfile
If the to_logfile directive is set to yes , this option
specifies the pathname of the log file.

No default.

logfile_priority
This specifies the logfile priority for this particular
subsystem. Ignored if debug is on. Possible values are:
alert, crit, debug (same as debug = on), emerg, err, info,
notice, warning.

The default is: info.

syslog_facility
This specifies the syslog facility type that will be used for
any messages sent to syslog. options are daemon, local0,
local1, local2, local3, local4, local5, local6 & local7.

The default is daemon.

syslog_priority
This specifies the syslog level for this particular subsystem.
Ignored if debug is on. Possible values are: alert, crit,
debug (same as debug = on), emerg, err, info, notice, warning.

The default is: info.

debug This specifies whether debug output is logged for this
particular logger. Also can contain value trace, what is
highest level of debug information.

The default is off.

Within the logging directive, logger_subsys directives are optional.

Within the logger_subsys sub-directive, all of the above logging
configuration options are valid and can be used to override the
default settings. The subsys entry, described below, is mandatory to
identify the subsystem.

subsys This specifies the subsystem identity (name) for which logging
is specified. This is the name used by a service in the
log_init() call. E.g. 'CPG'. This directive is required.

Within the quorum directive it is possible to specify the quorum
algorithm to use with the

provider
directive. At the time of writing only corosync_votequorum is
supported. See votequorum(5) for configuration options.

Within the nodelist directive it is possible to specify specific
information about nodes in cluster. Directive can contain only node
sub-directive, which specifies every node that should be a member of
the membership, and where non-default options are needed. Every node
must have at least ring0_addr field filled.

For UDPU, every node that should be a member of the membership must
be specified.

Possible options are:

ringX_addr
This specifies IP address of one of the nodes. X is ring
number.

nodeid This configuration option is optional when using IPv4 and
required when using IPv6. This is a 32 bit value specifying
the node identifier delivered to the cluster membership
service. If this is not specified with IPv4, the node id will
be determined from the 32 bit IP address the system to which
the system is bound with ring identifier of 0. The node
identifier value of zero is reserved and should not be used.

name This optional configuration option provides a unified way for
the client software (e.g. pacemaker) atop, respectively the
end users, to guide establishing a nominal
(self-)identification for each node in case neither respective
ringX_addr specifies a network hostname nor other means are
available/effective in this process. Option is not used by
Corosync itself.

Within the qb directive it is possible to specify options for libqb.

Possible option is:

ipc_type
This specifies type of IPC to use. Can be one of native
(default), shm and socket. Native means one of shm or socket,
depending on what is supported by OS. On systems with support
for both, SHM is selected. SHM is generally faster, but need
to allocate ring buffer file in /dev/shm.

Within the resources directive it is possible to specify options for
resources.

Possible option is:

watchdog_device
(Valid only if Corosync was compiled with watchdog support.)
Watchdog device to use. The default value is /dev/watchdog.
The special value "off" disables watchdog usage.

In a cluster with properly configured power fencing a watchdog
provides no additional value. On the other hand, slow
watchdog communication may incur multi-second delays in the
Corosync main loop, potentially breaking down membership.
IPMI watchdogs are particularly notorious in this regard: read
about kipmid_max_busy_us in IPMI.txt in the Linux kernel
documentation.

FILES

/etc/corosync/corosync.conf
The corosync executive configuration file.

NAME

SYNOPSIS

DESCRIPTION

FILES

SEE ALSO