Non-local redundancy: Erasure coding and dispersed replicas for robust retrieval in the Swarm peer-to-peer network

TL;DR

This paper describes in detail how erasure codes are implemented in the Swarm system, and introduces erasure codes, and proposes a number of retrieval strategies applicable to erasure-coded files.

Abstract

This paper describes in detail how erasure codes are implemented in the Swarm system. First, in Section 1, we introduce erasure codes, and show how to apply them to files in Swarm (Section 2). In Section 3, we introduce security levels of data availability and derive their respective parameterisations. In Section 4, we describe a construct that enables cross-neighbourhood redundancy for singleton chunks and which completes erasure coding. Finally, in 5, we propose a number of retrieval strategies applicable to erasure-coded files.

Figures (7)

• Figure 1: The Swarm tree is the data structure encoding how a document is split into chunks.
• Figure 2: The Swarm tree with extra parity chunks using $m=112$ out of $n=128$ RS encoding. Chunks $P_{0}$ through $P_{15}$ are parity data for chunks $H_0$ through $H_{111}$ on every level of intermediate chunks.
• Figure 3: The point at $k=17$ along the binomial distribution, where the probability of exceeding this many errors becomes less than $\alpha = 10\%$. Here, the total number of chunks is $n = 128$, and the per-chunk error rate is $\epsilon = 0.1$.
• Figure 4: The number of parities needed (ordinate) as a function of the per-chunk error rate $\epsilon$ (abscissa), for keeping the probability of overall data corruption below given limits (colours) and for $n = 64$ chunks (left panel) and $n = 128$ chunks (right panel).
• Figure 5: The number of parities required (ordinate) to keep the probability of overall data corruption at a given level (abscissa), for various values of the per-chunk error rate $\epsilon$ (colours) and for $n = 64$ chunks (left panel) and $n = 128$ chunks (right panel).