LinearPartition: Linear-Time Approximation of RNA Folding Partition Function and Base Pairing Probabilities

TL;DR

The paper tackles the cubic-time bottleneck of classical RNA partition-function calculations by introducing LinearPartition, a left-to-right beam-pruning dynamic-programming algorithm that approximates the partition function and base-pair probabilities in near-linear time. It builds on the LinearFold framework and employs inside/outside phases with a beam size $b$ (default 100) to reduce state空间 to $O(nb)$ and time to $O(nb^2)$, achieving substantial speedups and linear memory. Empirical results show LinearPartition dramatically outperforms Vienna RNAfold and CONTRAfold in runtime on long sequences while producing base-pair probabilities that correlate more closely with ground-truth structures, and it provides at least comparable, often improved, downstream MEA and ThreshKnot predictions, especially for long-range interactions. The method also demonstrates strong approximation quality with small ensemble-energy differences and low RMSD, suggesting beam pruning acts as a beneficial regularizer; extensions to pseudoknots and stochastic sampling could broaden its impact on large-scale RNA analytics.

Abstract

RNA secondary structure prediction is widely used to understand RNA function. Recently, there has been a shift away from the classical minimum free energy (MFE) methods to partition function-based methods that account for folding ensembles and can therefore estimate structure and base pair probabilities. However, the classical partition function algorithm scales cubically with sequence length, and is therefore a slow calculation for long sequences. This slowness is even more severe than cubic-time MFE-based methods due to a larger constant factor in runtime. Inspired by the success of our recently proposed LinearFold algorithm that predicts the approximate MFE structure in linear time, we design a similar linear-time heuristic algorithm, LinearPartition, to approximate the partition function and base pairing probabilities, which is shown to be orders of magnitude faster than Vienna RNAfold and CONTRAfold (e.g., 2.5 days vs. 1.3 minutes on a sequence with length 32,753 nt). More interestingly, the resulting base pairing probabilities are even better correlated with the ground truth structures. LinearPartition also leads to a small accuracy improvement when used for downstream structure prediction on families with the longest length sequences (16S and 23S rRNA), as well as a substantial improvement on long-distance base pairs (500+ nt apart).

Figures (17)

• Figure 1: An RNA can fold into multiple structures at equilibrium. A--B: Two secondary structures of Tebowned RNA: TBWN-A and TBWN-B Cordero+Das:2015. C: upper triangle shows the estimated base pairing probability matrix for this RNA using Vienna RNAfold, where darker red squares represent higher probility base pairs; the lower triangle shows the two different structures; D: Comparison between classical, local, and left-to-right algorithms for MFE and partition function calculation. LinearFold and LinearPartition enjoy linear runtime because of a left-to-right order that enables heuristic beam pruning, and both become exact $O(n^3)$ algorithms without pruning. "Span" denotes the maximum pair distance allowed ($\infty$ means no limit); it is a small constant in local methods (e.g., default $L$=70 $\text{\it nt}$ in RNAplfold).
• Figure 2: Although the total number of possible base pairings scales $O(n^2)$ with the sequence length $n$ (using the probability matrix from Vienna RNAfold as an example), with any reasonable threshold $\theta$, the number of surviving pairings (in colors for different $\theta$) grows linearly, suggesting our approximation, only computing $O(n)$ pairings, is reasonable.
• Figure 3: Partition function calculation pseudocode of a simplified version of the LinearPartition algorithm (the inside phase). See Fig. \ref{['fig:beam_prune_alg']} for the pseudocode of beam pruning (line \ref{['line:beamprune']}). The base-pairing probabilities are computed with the combination of the outside phase (Fig. \ref{['fig:outside']}). The actual algorithm using the Turner model is available on https://github.com/LinearFold/LinearPartition.
• Figure 4: Total runtime and memory usage of computing both the partition function and base pairing probabilities. A: Runtime comparisons on the ArchiveII dataset; the curve-fittings were log-log in gnuplot with $n > 10^3$. B: Runtime comparisons on the RNAcentral dataset (log scale). The partition function computation takes about half of the total time shown here. C: Memory usage comparisons on the RNAcentral dataset (log scale).
• Figure 5: A: Ensemble defect (expected number of incorrectly predicted nucleotides; lower is better) comparison between Vienna RNAfold and LinearPartition on the ArchiveII dataset. B: Ensemble defect difference for each family. LinearPartition has lower ensemble defects for longer families: on average 56.3 less incorrectly predicted nucleotides on 23S rRNA and 8.3 less over all families. C--F: An example of E. coli 23S rRNA (shaded point in A). C: Circular plot of the ground truth. D--F: Base pair probabilities from Vienna RNAplfold (with default window size $70$), RNAfold and LinearPartition, respectively; Blue denotes pairs in the known structure and Red denotes predicted pairs not in the known structure. The darkness of the line indicates pairing probability. G--J: Circular plots of C. ellipsoidea Group I Intron. See Fig. \ref{['fig:example']} for another view of this example.