SeqProFT: Sequence-only Protein Property Prediction with LoRA Finetuning

TL;DR

SeqProFT demonstrates that parameter-efficient finetuning via LoRA on protein language models enables smaller models to match or outperform larger ones on diverse protein-property tasks using only sequence data. By integrating a contact-map–augmented attention head, the approach leverages partial structural information to boost classification accuracy, while maintaining low training costs and fast inference. Comprehensive ablations show robustness across ranks and model sizes, with attention analyses indicating LoRA guides the model toward biologically meaningful sequence features. The work provides a practical blueprint for deploying efficient sequence-only predictions in resource-constrained settings and offers interpretability insights valuable for applications like drug discovery.

Abstract

Protein language models (PLMs) have demonstrated remarkable capabilities in learning relationships between protein sequences and functions. However, finetuning these large models requires substantial computational resources, often with suboptimal task-specific results. This study investigates how parameter-efficient finetuning via LoRA can enhance protein property prediction while significantly reducing computational demands. By applying LoRA to ESM-2 and ESM-C models of varying sizes and evaluating 10 diverse protein property prediction tasks, we demonstrate that smaller models with LoRA adaptation can match or exceed the performance of larger models without adaptation. Additionally, we integrate contact map information through a multi-head attention mechanism, improving model comprehension of structural features. Our systematic analysis reveals that LoRA finetuning enables faster convergence, better performance, and more efficient resource utilization, providing practical guidance for protein research applications in resource-constrained environments. The code is available at https://github.com/jiankliu/SeqProFT.

Figures (2)

• Figure 1: Architecture of SeqProFT. A: Overview showing pretrained ESM-2 model with LoRA adaptation and downstream head (blue: frozen parameters; yellow: trainable parameters). B: ESM-2 structure with LoRA applied to attention layers. C: LoRA training process with frozen pretrained weights and tunable low-rank matrices. D: Downstream networks: D1 shows Simple MLP Head (SMH); D2 shows Multi-head Attention Head with optional contact map processing (MAH, CM-MAH). E: Process for generating task-specific contact maps from attention scores.
• Figure 2: Attention analysis of Nuclease P1 (PDB ID: 1AK0). (a) Layer-stacked attention heatmaps without LoRA finetuning. (b) Layer-stacked attention heatmaps with LoRA finetuning. The x-axis in (a) and (b) represents token positions in the protein sequence, and the y-axis represents transformer layers (a total of 33 layers). Higher attention weights are shown in brighter colors. (c) Comparison of per-layer attention entropy between models with and without LoRA fine-tuning. Lower entropy indicates a more focused attention distribution. (d) Attention weight distribution of the last transformer layer (with LoRA). The residue positions with high attention are labeled with a red number. (e) 3D structure of Nuclease P1. Known substrate binding site residues (H116, D120, H126, and H149) are highlighted in orange. Other high-attention residues (H60 and C217) are highlighted in purple.