tcrLM: a lightweight protein language model for predicting T cell receptor and epitope binding specificity

TL;DR

This work tackles the challenge of predicting TCR-antigen bindings amid vast TCR diversity by introducing tcrLM, a lightweight BERT-based masked language model trained on $113{,}888{,}692$ TCR CDR3 sequences with random masking and virtual adversarial training. The pretraining yields rich TCR representations that, when combined with antigen embeddings, enable accurate pTCR binding prediction and robust generalization across independent, external, and COVID-19 test sets, outperforming state-of-the-art methods and larger protein language models. Notably, predicted pTCR scores correlate with immunotherapy outcomes in a melanoma cohort, suggesting clinical utility for predicting treatment responses and informing personalized immunotherapy. The study highlights the potential of targeted, data-efficient language-model pretraining on domain-specific repertoires to improve immunotherapy design and vaccine strategies.

Abstract

The anti-cancer immune response relies on the bindings between T-cell receptors (TCRs) and antigens, which elicits adaptive immunity to eliminate tumor cells. This ability of the immune system to respond to novel various neoantigens arises from the immense diversity of TCR repository. However, TCR diversity poses a significant challenge on accurately predicting antigen-TCR bindings. In this study, we introduce a lightweight masked language model, termed tcrLM, to address this challenge. Our approach involves randomly masking segments of TCR sequences and training tcrLM to infer the masked segments, thereby enabling the extraction of expressive features from TCR sequences. To further enhance robustness, we incorporate virtual adversarial training into tcrLM. We construct the largest TCR CDR3 sequence set with more than 100 million distinct sequences, and pretrain tcrLM on these sequences. The pre-trained encoder is subsequently applied to predict TCR-antigen binding specificity. We evaluate model performance on three test datasets: independent, external, and COVID-19 test set. The results demonstrate that tcrLM not only surpasses existing TCR-antigen binding prediction methods, but also outperforms other mainstream protein language models. More interestingly, tcrLM effectively captures the biochemical properties and positional preference of amino acids within TCR sequences. Additionally, the predicted TCR-neoantigen binding scores indicates the immunotherapy responses and clinical outcomes in a melanoma cohort. These findings demonstrate the potential of tcrLM in predicting TCR-antigen binding specificity, with significant implications for advancing immunotherapy and personalized medicine.

Figures (6)

• Figure 1: Illustrative diagram of tcrLM model. (a) Masked language model with the encoder composed of stacked attention-based modules. (b) Prediction network includes the pre-trained encoder to generate embeddings and projection head for pTCR binding prediction. (c) Detailed architecture of the projection head.
• Figure 2: Performance evaluation of tcrLM and five comparative methods on independent test set. (a-b) The ROC and Precision-Recall curves achieved by tcrLM and competing methods on the independent test set, respectively. (c) Comprehensive performance comparison between tcrLM and five competing methods. (d) Positive predictive value (PPV) for the top 100, top 1000, and top 5000 predictions. (e) Model ablation experiments for verifying the effectiveness of pretrained encoder and virtual adversarial training.
• Figure 3: Performance evaluation of tcrLM and four comparative methods on external and COVID-19 test set. (a-b) Performance metrics of tcrLM and competing methods on external and COVID-19 test set, respectively. (c-d) Positive predictive value (PPV) for the top 100, top 1000, and top 5000 predicted samples on external and COVID-19 test sets, respectively.
• Figure 4: Comprehensive evaluation of tcrLM and other four protein language models. (a) Perplexity and training time of three tcrLM variants on our established large-scale dataset comprising over 100 million TCR CDR3 sequences. (b) Comparison of parameter sizes of three tcrLM variants and four other protein language models. (c) Loss curves of tcrLM-M and competing protein language models. (d) Performance comparison of tcrLM-M and competing protein language models on independent, external and COVID-19 test sets.
• Figure 5: Correlation between predicted pTCR scores and immunotherapy outcomes in a melanoma cohort. (a-b) Violin and boxplots of predicted pTCR binding scores regarding the different immunotherapy response groups (p-value<0.01, $F$-tests). (c-d) Overall survival and progression free survival curves between stratified patient groups with high- and low-confidence pTCR bindings.