FoldToken: Learning Protein Language via Vector Quantization and Beyond

TL;DR

This work addresses the modality gap between protein sequences and 3D structures by learning a discrete protein language with FoldTokenizer, producing FoldToken symbols that encode sequence and structure. A GPT-style model, FoldGPT, then uses this language for autoregressive sequence-structure co-generation, enabling tasks like general backbone inpainting and antibody design. A key contribution is SoftCVQ, a Soft Conditional Vector Quantizer that balances high-quality reconstruction with robust generation, outperforming prior VQ methods. The approach demonstrates strong performance on reconstruction and generative benchmarks and opens a path toward broader applications of discrete protein languages in structure-aware generation and design.

Abstract

Is there a foreign language describing protein sequences and structures simultaneously? Protein structures, represented by continuous 3D points, have long posed a challenge due to the contrasting modeling paradigms of discrete sequences. We introduce \textbf{FoldTokenizer} to represent protein sequence-structure as discrete symbols. This innovative approach involves projecting residue types and structures into a discrete space, guided by a reconstruction loss for information preservation. We refer to the learned discrete symbols as \textbf{FoldToken}, and the sequence of FoldTokens serves as a new protein language, transforming the protein sequence-structure into a unified modality. We apply the created protein language on general backbone inpainting and antibody design tasks, building the first GPT-style model (\textbf{FoldGPT}) for sequence-structure co-generation with promising results. Key to our success is the substantial enhancement of the vector quantization module, Soft Conditional Vector Quantization (\textbf{SoftCVQ}).

Figures (6)

• Figure 1: The overall framework of FoldToken.
• Figure 2: Baseline vector quantization methods.
• Figure 3: Proposed vector quantization methods.
• Figure 4: Binary VQ-ID is crucial for convergence.
• Figure 5: Torsion angle computation.