HELM-BERT: A Transformer for Medium-sized Peptide Property Prediction

TL;DR

HELM-BERT introduces an encoder-based language model trained on HELM notation to capture monomer-level chemistry and macrocyclic topology in therapeutic peptides. Pre-trained on 39,079 diverse peptides, it outperforms SMILES-based baselines on cyclic peptide membrane permeability and peptide–protein interaction tasks, approaching protein-language-model performance with far less data. Ablation and embedding analyses show disentangled attention is essential and that HELM representations better encode topology, suggesting topology-aware modeling as a key advantage for peptide drug discovery. The work emphasizes the potential of HELM-enabled hierarchical representations and points to standardization and graph-based extensions as promising directions for broader scalability.

Abstract

Therapeutic peptides have emerged as a pivotal modality in modern drug discovery, occupying a chemically and topologically rich space. While accurate prediction of their physicochemical properties is essential for accelerating peptide development, existing molecular language models rely on representations that fail to capture this complexity. Atom-level SMILES notation generates long token sequences and obscures cyclic topology, whereas amino-acid-level representations cannot encode the diverse chemical modifications central to modern peptide design. To bridge this representational gap, the Hierarchical Editing Language for Macromolecules (HELM) offers a unified framework enabling precise description of both monomer composition and connectivity, making it a promising foundation for peptide language modeling. Here, we propose HELM-BERT, the first encoder-based peptide language model trained on HELM notation. Based on DeBERTa, HELM-BERT is specifically designed to capture hierarchical dependencies within HELM sequences. The model is pre-trained on a curated corpus of 39,079 chemically diverse peptides spanning linear and cyclic structures. HELM-BERT significantly outperforms state-of-the-art SMILES-based language models in downstream tasks, including cyclic peptide membrane permeability prediction and peptide-protein interaction prediction. These results demonstrate that HELM's explicit monomer- and topology-aware representations offer substantial data-efficiency advantages for modeling therapeutic peptides, bridging a long-standing gap between small-molecule and protein language models.

Figures (5)

• Figure 1: Overview of HELM-BERT architecture. Input peptides are converted to HELM notation, tokenized into monomer-level tokens, and subjected to span masking, where contiguous spans of tokens are masked (gray) during pre-training. The HELM-BERT encoder comprises a hybrid first layer combining disentangled self-attention with nGiE, followed by five transformer blocks with disentangled attention. The EMD receives the output of Layer 5, injects absolute position embeddings ($\mathbf{P}^{\text{abs}}$), and applies two weight-tied iterative refinement steps using the same parameters as Layer 6. The MLM projection head predicts the masked tokens (green).
• Figure 2: Pre-training loss curves. Training and validation MLM loss over the course of pre-training. The model was trained for 127 epochs with early stopping (patience = 20).
• Figure 3: Pre-training MLM loss curves under architectural ablations. Validation loss for HELM-BERT and architectural variants over training epochs.
• Figure 4: t-SNE projections of pre-trained embeddings colored by structure type.
• Figure 5: t-SNE projections of PPI dataset splits in aCSM complex space. Colors indicate fold assignment. The Cluster-based Split shows distinct spatial separation between folds, reflecting heterogeneous protein cluster distributions.