Transformer-Based Decoding in Concatenated Coding Schemes Under Synchronization Errors

TL;DR

The paper tackles decoding in DNA data storage where multiple reads of a codeword suffer insertions, deletions, and substitutions (IDS). It introduces BCJRFormer, a transformer-based inner decoder that mirrors BCJR performance with quadratic scaling in the number of copies and extends to ConvBCJRFormer for convolutional inner codes; it then replaces the outer BP with ECCT to form an efficient end-to-end transformer pipeline. The authors demonstrate that this pipeline achieves lower error rates than traditional bcjr/bp in many regimes and scales to larger cluster sizes inaccessible to BCJR, while ConvBCJRFormer provides a principled approach toward joint inner-outer decoding for linear codes. The work also analyzes complexity, presents training strategies, and provides comprehensive experiments across marker and convolutional inner codes, highlighting practical implications for scalable DNA data storage decoding.

Abstract

We consider the reconstruction of a codeword from multiple noisy copies that are independently corrupted by insertions, deletions, and substitutions. This problem arises, for example, in DNA data storage. A common code construction uses a concatenated coding scheme that combines an outer linear block code with an inner code, which can be either a nonlinear marker code or a convolutional code. Outer decoding is done with Belief Propagation, and inner decoding is done with the Bahl-Cocke-Jelinek-Raviv (BCJR) algorithm. However, the BCJR algorithm scales exponentially with the number of noisy copies, which makes it infeasible to reconstruct a codeword from more than about four copies. In this work, we introduce BCJRFormer, a transformer-based neural inner decoder. BCJRFormer achieves error rates comparable to the BCJR algorithm for binary and quaternary single-message transmissions of marker codes. Importantly, BCJRFormer scales quadratically with the number of noisy copies. This property makes BCJRFormer well-suited for DNA data storage, where multiple reads of the same DNA strand occur. To lower error rates, we replace the Belief Propagation outer decoder with a transformer-based decoder. Together, these modifications yield an efficient and performant end-to-end transformer-based pipeline for decoding multiple noisy copies affected by insertion, deletion, and substitution errors. Additionally, we propose a novel cross-attending transformer architecture called ConvBCJRFormer. This architecture extends BCJRFormer to decode transmissions of convolutional codewords, serving as an initial step toward joint inner and outer decoding for more general linear code classes.

Figures (19)

• Figure 1: Overview of BCJRFormer for jointly decoding marker sequences over the IDS channel.
• Figure 2: State transitions for a codeword symbol $x_i$ transmitted through the ids channel.
• Figure 3: Generator matrix $G \in \{ 0, 1\}^{9 \times 18}$ for the $(2, 1, 2)$ convolutional code with generator polynomials $[5, 7]_8$, used for an outer codeword with $n_{\text{out}} = 7$. White entries indicate unmasked positions.
• Figure 4: Joint decoding of concatenated codes using BCJRFormer as the inner decoder and ECCT as the outer decoder.
• Figure 5: Embedding, architecture, and loss calculation for ConvBCJRFormer. Visualization inspired by the article park2025crossmpt.