Knowledge from Large-Scale Protein Contact Prediction Models Can Be Transferred to the Data-Scarce RNA Contact Prediction Task

TL;DR

This paper demonstrates that knowledge from a large-scale protein contact prediction transformer can be transferred to the data-scarce RNA contact prediction task. By translating RNA MSAs into a protein-language representation, leveraging the Co-evolution Transformer’s multi-layer attention, and using a mid-fusion ConvNet architecture, the approach achieves substantial gains over RNA baselines and prior RNA-specific transfer methods. Key contributions include a three-stage transfer framework, analysis of transfer strategies and fusion designs, and evidence of robust performance across multiple nucleotide-to-amino-acid translations. The work highlights a new direction in cross-biomolecule transfer learning with potential to enhance RNA 3D structure prediction and downstream biological applications, while outlining limitations related to data scale and sequence length.

Abstract

RNA, whose functionality is largely determined by its structure, plays an important role in many biological activities. The prediction of pairwise structural proximity between each nucleotide of an RNA sequence can characterize the structural information of the RNA. Historically, this problem has been tackled by machine learning models using expert-engineered features and trained on scarce labeled datasets. Here, we find that the knowledge learned by a protein-coevolution Transformer-based deep neural network can be transferred to the RNA contact prediction task. As protein datasets are orders of magnitude larger than those for RNA contact prediction, our findings and the subsequent framework greatly reduce the data scarcity bottleneck. Experiments confirm that RNA contact prediction through transfer learning using a publicly available protein model is greatly improved. Our findings indicate that the learned structural patterns of proteins can be transferred to RNAs, opening up potential new avenues for research.

Figures (6)

• Figure 1: Our study is focused on RNA contact prediction, i.e., predicting the contact map matrix for an RNA sequence. The contact map indicates the proximity between each nucleotide, with those closer than a threshold (10 Å) being deemed in contact. Correct predictions of the contact map can benefit downstream tasks, e.g., by acting as constraints for filtering 3D RNA structure predictions.
• Figure 2: Overview of our three-stage method (from top to bottom). Adapted Feature Extraction: First, a projection layer is used to translate the RNA MSA sequences into protein language (e.g., from nucleotide "AUCG" to amino acids "HETL"). Then, we leverage a fixed large-scale pre-trained protein contact prediction transformer model (called Co-evolution Transformer model (CoT)) to extract attentive (i.e., contribution) features at different layers. Feature Fusion: Features from different layers are processed by separate convolution blocks before being concatenated. Classification: The aggregated features are sent into a standard Convolutional Network (ConvNet) classifier with three layers of convolution.
• Figure 3: Common baselines for transferring protein CoT to RNA contact prediction.
• Figure 4: Different feature fusion strategies. Our final model (shown in Figure \ref{['fig:overview']}) uses the mid-fusion design.
• Figure B.1: Visual comparison of contact map predictions by CoCoNet and ours in 6 sample test RNAs. The grey dots denote ground truth contacts, the yellow dots in the upper triangle denote correct predictions by CoCoNet, and the purple dots in the lower triangle denote the correct predictions by ours. The red box highlights correct predictions by our model that are missed by CoCoNet.