NMP-PaK: Near-Memory Processing Acceleration of Scalable De Novo Genome Assembly

TL;DR

NMP-PaK presents a hardware-software co-design that leverages channel-level near-memory processing to accelerate De Bruijn graph-based de novo genome assembly. By combining a pipelined systolic PE design, integrated cross-dimms communication, and a hybrid CPU-NMP workflow with customized batch processing, it tackles three challenges: prohibitive memory footprints, memory-latency bottlenecks, and irregular data patterns inherent to PaKman-style assembly. The approach yields a $14\times$ memory footprint reduction, $16\times$ CPU-baseline speedup, and $8.3\times$ higher throughput under the same resources, while offering favorable memory bandwidth utilization and manageable area/power overhead. These results demonstrate that near-memory processing can make large-scale genome assembly more practical for personalized medicine and point to scalable hardware-software co-design paths for memory-bound genomics workloads.

Abstract

De novo assembly enables investigations of unknown genomes, paving the way for personalized medicine and disease management. However, it faces immense computational challenges arising from the excessive data volumes and algorithmic complexity. While state-of-the-art de novo assemblers utilize distributed systems for extreme-scale genome assembly, they demand substantial computational and memory resources. They also fail to address the inherent challenges of de novo assembly, including a large memory footprint, memory-bound behavior, and irregular data patterns stemming from complex, interdependent data structures. Given these challenges, de novo assembly merits a custom hardware solution, though existing approaches have not fully addressed the limitations. We propose NMP-PaK, a hardware-software co-design that accelerates scalable de novo genome assembly through near-memory processing (NMP). Our channel-level NMP architecture addresses memory bottlenecks while providing sufficient scratchpad space for processing elements. Customized processing elements maximize parallelism while efficiently handling large data structures that are both dynamic and interdependent. Software optimizations include customized batch processing to reduce the memory footprint and hybrid CPU-NMP processing to address hardware underutilization caused by irregular data patterns. NMP-PaK conducts the same genome assembly while incurring a 14X smaller memory footprint compared to the state-of-the-art de novo assembly. Moreover, NMP-PaK delivers a 16X performance improvement over the CPU baseline, with a 2.4X reduction in memory operations. Consequently, NMP-PaK achieves 8.3X greater throughput than state-of-the-art de novo assembly under the same resource constraints, showcasing its superior computational efficiency.

Figures (15)

• Figure 1: De Bruijn graph-based genome assembly georganas2018extreme: It generates contigs (e.g., an unknown virus) from DNA samples. (e.g., an infected mouse).
• Figure 2: PaKman assembly algorithm procedure ghosh2020pakman. A,B: k-mers are generated using a sliding window of size 32. C: MacroNodes are constructed and form the PaK-graphs. D: Iterative Compaction makes the PaK-graphs more compact. E: Contigs are generated from the compacted PaK-graphs.
• Figure 3: (a) MacroNode structure and the construction process using k-mers. (b) Creation of PaK-graph edges: one k-mer is used to create two MacroNodes, with that k-mer itself serving as an edge of the PaK-graph.
• Figure 4: Example of the Iterative Compaction process.
• Figure 5: Runtime breakdown of the PaKman algorithm (10% of the full human genome, 64 threads). Iterative Compaction dominates the overall assembly time.