Dynamically Sharded Ledgers on a Distributed Hash Table

TL;DR

ScaleGraph addresses the scalability bottleneck of distributed ledgers by introducing dynamic sharding guided by a distributed hash table (DHT) proximity model. It combines Kademlia-based proximity shards with a modified Sync HotStuff consensus to validate transactions via the union of sender- and receiver-associated validator groups, enabling fine-grained parallelism through a partial order of events. The work provides a theoretical security analysis, extensive simulations of shard sizing and failure probabilities, and demonstrates that synchronous consensus enables smaller, more secure shards while maintaining high concurrency. The approach promises improved throughput and data locality for large-scale micro-transactions in the Machine Economy, while highlighting the need for further formal safety proofs and more accurate probabilistic models for overlapping shards.

Abstract

Distributed ledger technology such as blockchain is considered essential for supporting large numbers of micro-transactions in the Machine Economy, which is envisioned to involve billions of connected heterogeneous and decentralized cyber-physical systems. This stresses the need for performance and scalability of distributed ledger technologies. Sharding divides the blockchain network into multiple committees and is a common approach to improve scalability. However, with current sharding approaches, costly cross-shard verification is needed to prevent double-spending. This paper proposes a novel and more scalable distributed ledger method named ScaleGraph that implements dynamic sharding by using routing and logical proximity concepts from distributed hash tables. ScaleGraph addresses cyber security in terms of integrity, availability, and trust, to support frequent micro-transactions between autonomous devices. Benefits of ScaleGraph include a total storage space complexity of O(t), where t is the global number of transactions (assuming a constant replication degree). This space is sharded over n nodes so that each node needs O(t/n) storage, which provides a high level of concurrency and data localization as compared to other delegated consensus proposals. ScaleGraph allows for a dynamic grouping of validators which are selected based on a distance metric. We analyze the consensus requirements in such a dynamic setting and show that a synchronous consensus protocol allows shards to be smaller than an asynchronous one, and likely yields better performance. Moreover, we provide an experimental analysis of security aspects regarding the required size of the consensus groups with ScaleGraph. Our analysis shows that dynamic sharding based on proximity concepts brings attractive scalability properties in general, especially when the fraction of corrupt nodes is small.

Figures (6)

• Figure 1: (a) shows a collection of block chains, each associated with one of the accounts A, B, C, and D. (b) shows the DAG representing the partial order of all the transactions in those chains. Since transactions/blocks appear twice in (a), once in each chain for the two affected accounts, they have been color coded to make the pairs easier to see.
• Figure 2: The squares represent accounts A, B, C, and D. The circles represent validator nodes. Lines between accounts and nodes show which nodes form shards associated with the respective accounts. The large rectangles show the block chains stored at some of the nodes. Since the same transaction/block appear multiple times, they have been color coded to make the duplicates easier to identify. See also Figure \ref{['fig:blockdag']}, which more cleanly shows the block chain for each account. Each account is connected to 4 nodes, implying a shard size of 4.
• Figure 3: Required shard sizes for different network sizes and different fractions $F$ of Byzantine nodes present in the network, assuming a consensus protocol with $f < \frac{1}{2}$. Shard sizes for $F=\frac{1}{5}$, $F=\frac{1}{4}$, and $F=\frac{1}{3}$ are shown in blue (plus), orange (triangles), and green (discs), respectively. The number of shards is $m=2N$ and the results are based on $20 \cdot 5000 = 100000$ iterations.
• Figure 4: Comparing required shard sizes for different network sizes and different combinations of consensus protocol fault tolerances ($f < \frac{1}{2}$ in blue, $f < \frac{1}{3}$ in orange) and fractions of Byzantine node present in the network. The number of shards is $m=2N$ and the results are based on $20 \cdot 5000 = 100000$ iterations.
• Figure 5: Failure probability (on a logarithmic scale) for different shard sizes. The parameters are $N=2000$, $F=\frac{1}{4}$, $f<\frac{1}{2}$, and $m=2N$.