Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self Distillation Fine-Tuning of Protein Language Models Improves Versatility in Protein Design

Amin Tavakoli, Raswanth Murugan, Ozan Gokdemir, Arvind Ramanathan, Frances Arnold, Anima Anandkumar

TL;DR

The paper tackles the scarcity of high-quality annotations for protein sequence design by introducing a self-distillation supervised fine-tuning framework that uses a curated filtering pipeline to generate training data from the model’s own outputs. It demonstrates that SFT on a GenSLM backbone, guided by sequential filters, improves both targeted design metrics (length, active-site conservation, pLDDT) and emergent properties (stability proxies, PLP docking) for TrpB sequences. Two sampling strategies balance novelty and functionality, showing the approach can expand sequence space while retaining plausible, functional candidates. Overall, the work provides a data-efficient, generalizable workflow for refining PLMs in protein design without external wet-lab data.

Abstract

Supervised fine-tuning (SFT) is a standard approach for adapting large language models to specialized domains, yet its application to protein sequence modeling and protein language models (PLMs) remains ad hoc. This is in part because high-quality annotated data are far more difficult to obtain for proteins than for natural language. We present a simple and general recipe for fast SFT of PLMs, designed to improve the fidelity, reliability, and novelty of generated protein sequences. Unlike existing approaches that require costly precompiled experimental datasets for SFT, our method leverages the PLM itself, integrating a lightweight curation pipeline with domain-specific filters to construct high-quality training data. These filters can independently refine a PLM's output and identify candidates for in vitro evaluation; when combined with SFT, they enable PLMs to generate more stable and functional enzymes, while expanding exploration into protein sequence space beyond natural variants. Although our approach is agnostic to both the choice of protein language model (PLM) and the protein system, we demonstrate its effectiveness with a genome-scale PLM (GenSLM) applied to the tryptophan synthase enzyme family. The supervised fine-tuned model generates sequences that are not only more novel but also display improved characteristics across both targeted design constraints and emergent protein property measures.

Self Distillation Fine-Tuning of Protein Language Models Improves Versatility in Protein Design

TL;DR

The paper tackles the scarcity of high-quality annotations for protein sequence design by introducing a self-distillation supervised fine-tuning framework that uses a curated filtering pipeline to generate training data from the model’s own outputs. It demonstrates that SFT on a GenSLM backbone, guided by sequential filters, improves both targeted design metrics (length, active-site conservation, pLDDT) and emergent properties (stability proxies, PLP docking) for TrpB sequences. Two sampling strategies balance novelty and functionality, showing the approach can expand sequence space while retaining plausible, functional candidates. Overall, the work provides a data-efficient, generalizable workflow for refining PLMs in protein design without external wet-lab data.

Abstract

Supervised fine-tuning (SFT) is a standard approach for adapting large language models to specialized domains, yet its application to protein sequence modeling and protein language models (PLMs) remains ad hoc. This is in part because high-quality annotated data are far more difficult to obtain for proteins than for natural language. We present a simple and general recipe for fast SFT of PLMs, designed to improve the fidelity, reliability, and novelty of generated protein sequences. Unlike existing approaches that require costly precompiled experimental datasets for SFT, our method leverages the PLM itself, integrating a lightweight curation pipeline with domain-specific filters to construct high-quality training data. These filters can independently refine a PLM's output and identify candidates for in vitro evaluation; when combined with SFT, they enable PLMs to generate more stable and functional enzymes, while expanding exploration into protein sequence space beyond natural variants. Although our approach is agnostic to both the choice of protein language model (PLM) and the protein system, we demonstrate its effectiveness with a genome-scale PLM (GenSLM) applied to the tryptophan synthase enzyme family. The supervised fine-tuned model generates sequences that are not only more novel but also display improved characteristics across both targeted design constraints and emergent protein property measures.

Paper Structure

This paper contains 10 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Materials and Methods
  3. Protein Language Model
  4. Fine-tuning on Tryptophan Synthase Family
  5. Filtering Pipeline
  6. Results
  7. Conclusion
  8. Appendix
  9. Enzyme Family
  10. Reference Set

Figures (3)

  • Figure 1: A: The general framework of the supervised fine-tuning pipeline using the artificially generated sequences. B: The sequence of the filters used to curate data for SFT.
  • Figure 2: (a) Stability scores. (b) PLP docking scores. In both cases, the plots show reversed magnitude, so higher values are better. The distribution shift is observed in both structural stability and the PLP Docking scores of generated samples. post SFT training
  • Figure 3: SFT-Novel generates more sequences in lower maxID regions compared to the vanilla GenSLM model, resulting in greater sequence diversity.