bittide: Control Time, Not Flows

TL;DR

The paper tackles the challenge of achieving deterministic timing in distributed systems without relying on a global wall-clock reference. It presents a hardware implementation of bittide, a decentralized clock synchronization mechanism that enforces logical synchrony by aligning node frequencies through elastic buffers and a local control loop, with zero in-band signaling overhead. The 8-node FPGA prototype demonstrates network-wide frequency alignment, bounded buffer excursions, and stable logical latencies across fully connected, hourglass, and cube topologies, including a long 2 km fiber link. The work couples a simple, provably stable control algorithm with domain-difference counting hardware and a lightweight mathematical model, showing robustness to physical latencies and offering a scalable foundation for predictable distributed computation and ahead-of-time scheduling in large-scale systems.

Abstract

This paper presents the first hardware implementation of bittide, a decentralized clock synchronization mechanism for achieving logical synchrony in distributed systems. We detail the design and implementation of an 8-node bittide network using off-the-shelf FPGA boards and adjustable clock sources. Through experiments with various network topologies, including fully connected, hourglass, and cube, we demonstrate the effectiveness of bittide in aligning node frequencies and bounding buffer excursions. We collect and analyze frequency, buffer occupancy, and logical latency data, validating the hardware's performance against theoretical predictions and simulations. Our results show that bittide achieves tight frequency alignment, robustly handles varying physical latencies, and establishes a consistent notion of logical time across the network, enabling predictable distributed computation at scale with zero in-band overhead.

Figures (18)

• Figure 1: Two nodes showing the receive buffers as part of the links.
• Figure 2: Physical bittide hardware setup. In front are oscilloscopes, showing (synchronized) clock signals. Behind them are the clock synthesizer boards. In the back are the FPGA development boards.
• Figure 3: Part of the clock control algorithm. Note the use of delayI, which automatically inserts the appropriate number of registers to align the signals.
• Figure 4: Block diagram of clock control circuit of a single node. Thin lines indicate clock signals, while thick lines carry data. Different colors indicate different clock domains. Here CDR denotes clock and data recovery, EB denotes elastic buffer, and DDC blocks are domain difference counting. The ILA is the integrated logic analyzer, and $\textup{C}_\textit{ext}$ the adjustable clock board.
• Figure 5: Domain Difference Counting Setup