Design Proteins Using Large Language Models: Enhancements and Comparative Analyses

TL;DR

The paper investigates whether medium-sized LLMs (7–8B parameters) can be repurposed to design functional protein sequences using a small, human-focused dataset. The authors retrain tokenizers with Byte-Pair Encoding on UniRef50 Homo sapiens, then fine-tune four LLMs with a cross-entropy objective to predict amino acid tokens, effectively mapping language modeling to protein generation $L = -\sum_{t=1}^{N} \log p_{model}(x_{t+1} | x_1, x_2, ..., x_t)$. Evaluation uses AlphaFold2-based pLDDT, FoldSeek TM-score and intra-RMSD, Rosetta-Relax REU, and PyMOL inter-RMSD to quantify structural quality and stability; results show P-Mistral often surpasses protein-focused baselines trained on millions of sequences. The work demonstrates that compact, cost-efficient LLMs can achieve competitive design performance and will release the trained models to promote reproducibility and future enhancements.

Abstract

Pre-trained LLMs have demonstrated substantial capabilities across a range of conventional natural language processing (NLP) tasks, such as summarization and entity recognition. In this paper, we explore the application of LLMs in the generation of high-quality protein sequences. Specifically, we adopt a suite of pre-trained LLMs, including Mistral-7B1, Llama-2-7B2, Llama-3-8B3, and gemma-7B4, to produce valid protein sequences. All of these models are publicly available.5 Unlike previous work in this field, our approach utilizes a relatively small dataset comprising 42,000 distinct human protein sequences. We retrain these models to process protein-related data, ensuring the generation of biologically feasible protein structures. Our findings demonstrate that even with limited data, the adapted models exhibit efficiency comparable to established protein-focused models such as ProGen varieties, ProtGPT2, and ProLLaMA, which were trained on millions of protein sequences. To validate and quantify the performance of our models, we conduct comparative analyses employing standard metrics such as pLDDT, RMSD, TM-score, and REU. Furthermore, we commit to making the trained versions of all four models publicly available, fostering greater transparency and collaboration in the field of computational biology.

Figures (8)

• Figure 1: A comprehensive overview of our methodology employed for training, evaluating, and validating the protein sequence generation model. We initially retrained tokenizers for four distinct large language models --- Mistral-7B, Llama-2-7B, Llama-3-8B, and gemma-7B --- using the UniRef50-Homo sapiens dataset employing the Byte-Pair Encoding (BPE) technique. Subsequently, we fine-tuned these models on a filtered subset of the UniRef50-Homo sapiens dataset, aiming to minimize the loss associated with predicting subsequent protein sequences. For evaluation, model output was validated using AlphaFold 2 to construct 3D protein structures, followed by assessments of the generated protein structural accuracy using metrics such as per-residue confidence score (pLDDT) from AlphaFold 2, RMSD (Root Mean Square Deviation), and TM-Score to compare topological similarities with known protein structures applied using FoldSeek. Additional evaluation included the use of Rosetta- Relax for analyzing the energetic profiles of the generated proteins. Finally, protein structural comparisons within each dataset were conducted using PyMOL to calculate the intra-dataset RMSD.
• Figure 2: Illustration of High TM-Score and low Intra RMSD Compared to Low TM-Score and high Intra RMSD
• Figure 3: Examples of the 3D structure of proteins generated by each introduced model
• Figure 4: Violin plot of pLDDT
• Figure 5: Violin plot of TM-Score