Heartbeat (computing)

Short description: Synchronization primitive for fault tolerance

In computer science, a heartbeat is a periodic signal generated by hardware or software to indicate normal operation or to synchronize other parts of a computer system. Heartbeat mechanism is one of the common techniques in mission critical systems for providing high availability and fault tolerance of network services by detecting the network or systems failures of nodes or daemons which belongs to a network cluster—administered by a master server—for the purpose of automatic adaptation and rebalancing of the system by using the remaining redundant nodes on the cluster to take over the load of failed nodes for providing constant services. Usually a heartbeat is sent between machines at a regular interval in the order of seconds; a heartbeat message. If the endpoint does not receive a heartbeat for a time—usually a few heartbeat intervals—the machine that should have sent the heartbeat is assumed to have failed. Heartbeat messages are typically sent non-stop on a periodic or recurring basis from the originator's start-up until the originator's shutdown. When the destination identifies a lack of heartbeat messages during an anticipated arrival period, the destination may determine that the originator has failed, shutdown, or is generally no longer available.

Heartbeat protocol

A heartbeat protocol is generally used to negotiate and monitor the availability of a resource, such as a floating IP address, and the procedure involves sending network packets to all the nodes in the cluster to verify its reachability. Typically when a heartbeat starts on a machine, it will perform an election process with other machines on the heartbeat network to determine which machine, if any, owns the resource. On heartbeat networks of more than two machines, it is important to take into account partitioning, where two halves of the network could be functioning but not able to communicate with each other. In a situation such as this, it is important that the resource is only owned by one machine, not one machine in each partition.

As a heartbeat is intended to be used to indicate the health of a machine, it is important that the heartbeat protocol and the transport that it runs on are as reliable as possible. Causing a failover because of a false alarm may, depending on the resource, be highly undesirable. It is also important to react quickly to an actual failure, further signifying the reliability of the heartbeat messages. For this reason, it is often desirable to have a heartbeat running over more than one transport; for instance, an Ethernet segment using UDP/IP, and a serial link.

A "cluster membership" of a node is a property of network reachability: if the master can communicate with the node [math]\displaystyle{ x }[/math], it's considered a member of the cluster and "dead" otherwise. A heartbeat program as a whole consist of various subsystems:

• Heartbeat Subsystem (HS): The subsystem that monitors the node's presence within the cluster through a series of keepalive or "hear-beat messages".
• Cluster Manager (CM): The subsystem within the cluster—usually the master server—which keeps track of the "cluster members" and records which resources are on which nodes.
• Cluster Transition (CT): When a node joins or leaves the cluster, this subsystem is responsible for keeping track of such occurrences for the purpose of triggering events to rebalancing and reconfiguring the master to distribute the load.

Heartbeat messages are sent in a periodic manner through techniques such as broadcast or multicasts in larger clusters. Since CMs have transactions across the cluster, the most common pattern is to send heartbeat messages to all the nodes and "await" responses in non-blocking fashion. Since the heartbeat or keepalive messages are the overwhelming majority of non-application related cluster control messages—which also goes to all the members of the cluster—major critical systems also include non-IP protocols like serial ports to deliver heartbeats.

Design and implementation

Every CM on the master server maintains a finite-state machine with three states for each node it administers: Down, Init, and Alive. Whenever a new node joins, the CM changes the state of the node from Down to Init and broadcasts a "boot-up message", which the node receives the executes set of start-up procedures. It then responses with an acknowledgment message, CM then includes the node as the member of the cluster and transitions the state of the node from Init to Alive. Every node in the Alive state would receive a periodic broadcast heartbeat message from the HS subsystem and expects an acknowledgment message back within a timeout range. If CM didn't receive an acknowledgment heartbeat message back, the node is considered unavailable, and a state transition from Alive to Down takes place for that node by CM. The procedures or scripts to run, and actions to take between each state transition is an implementation detail of the system.

Heartbeat network

Heartbeat network is a private network which is shared only by the nodes in the cluster, and is not accessible from outside the cluster. It is used by cluster nodes in order to monitor each node's status and communicate with each other messages necessary for maintaining the operation of the cluster. The heartbeat method uses the FIFO nature of the signals sent across the network. By making sure that all messages have been received, the system ensures that events can be properly ordered.

In this communications protocol every node sends back a message in a given interval, say delta, in effect confirming that it is alive and has a heartbeat. These messages are viewed as control messages that help determine that the network includes no delayed messages. A receiver node called a "sync", maintains an ordered list of the received messages. Once a message with a timestamp later than the given marked time is received from every node, the system determines that all messages have been received since the FIFO property ensures that the messages are ordered.

In general, it is difficult to select a delta that is optimal for all applications. If delta is too small, it requires too much overhead and if it is large it results in performance degradation as everything waits for the next heartbeat signal.

See also

• Watchdog timer, electronic timer that is used to detect and recover from computer malfunctions
• Heartbleed vulnerability

References

• Nikoletseas, Sotiris, ed (2011). "Theoretical Aspects of Distributed Computing in Sensor Networks". Monographs in Theoretical Computer Science. An EATCS Series. Berlin, Heidelberg: Springer Berlin Heidelberg. doi:10.1007/978-3-642-14849-1. ISBN 978-3-642-14848-4. Bibcode: 2011tadc.book.....N. 
• Hou, Zonghao; Huang, Yongxiang (29 March 2003). "Design and implementation of heartbeat in multi-machine environment". 17th International Conference on Advanced Information Networking and Applications, 2003. AINA 2003.. China: IEEE Xplore. doi:10.1109/AINA.2003.1192949. ISBN 0-7695-1906-7. 
• Robertson, Alan (2000). "Linux-HA Heartbeat System Design". USENIX Annual Technical Conference. SUSE Labs. https://www.usenix.org/legacy/publications/library/proceedings/als00/2000papers/papers/full_papers/robertson/robertson.pdf. 
• Li, Fei-Fei; Yu, Xiang-Zhan; Wu, Gang (11 July 2009). "Design and Implementation of High Availability Distributed System Based on Multi-level Heartbeat Protocol". 2009 IITA International Conference on Control, Automation and Systems Engineering (case 2009). China: IEEE. doi:10.1109/CASE.2009.115. ISBN 978-0-7695-3728-3. https://ieeexplore.ieee.org/document/5194396.