HEAL: Online Incremental Recovery for Leaderless Distributed Systems Across Persistency Models

TL;DR

This paper presents HEAL, the first online incremental recovery scheme for leaderless distributed systems, addressing rapid fault recovery with minimal throughput impact. It defines a DDP-based framework with five persistency models and two visibility/durability concepts, introducing Partially_persisted lists and Missed_updates buffers to enable proactive, non-redundant recovery using a Buddy node. Built on MINOS, HEAL re-uses failed-durable state and allows the recovering node to participate in ongoing writes as a ShadowReplica, dramatically reducing recovery latency. Empirical results on a 6-node cluster show HEAL achieving about $120\ \mathrm{ms}$ recovery with ~${8.7}\%$ throughput degradation under $<\text{Lin},\text{Synch}>$ and outperforming both nonincremental leaderless recovery and a leader-based ZooKeeper scheme by large margins, highlighting strong practical impact for resilient, high-throughput distributed storage.

Abstract

Ensuring resilience in distributed systems has become an acute concern. In today's environment, it is crucial to develop light-weight mechanisms that recover a distributed system from faults quickly and with only a small impact on the live-system throughput. To address this need, this paper proposes a new low-overhead, general recovery scheme for modern non-transactional leaderless distributed systems. We call our scheme HEAL. On a node failure, HEAL performs an optimized online incremental recovery. This paper presents HEAL's algorithms for settings with Linearizable consistency and different memory persistency models. We implement HEAL on a 6-node Intel cluster. Our experiments running TAOBench workloads show that HEAL is very effective. HEAL recovers the cluster in 120 milliseconds on average, while reducing the throughput of the running workload by an average of 8.7%. In contrast, a conventional recovery scheme for leaderless systems needs 360 seconds to recover, reducing the throughput of the system by 16.2%. Finally, compared to an incremental recovery scheme for a state-of-the-art leader-based system, HEAL reduces the average recovery latency by 20.7x and the throughput degradation by 62.4%.

Figures (9)

• Figure 1: Recovery latency in a state-of-the-art leaderless system.
• Figure 2: Fault detection, reconfiguration, and recovery.
• Figure 3: Durability points in an update under $<$Lin, Synch$>$.
• Figure 4: HEAL recovery algorithm.
• Figure 5: Handling multiple concurrent failures. In the figure, TLi means timeline $i$.