Generative power of a protein language model trained on multiple sequence alignments

TL;DR

The paper addresses generating novel protein-family sequences by leveraging protein language models trained on MSAs. It introduces an iterative masking scheme that uses MSA Transformer’s masked language modeling objective to produce synthetic MSAs, and benchmarks them against bmDCA Potts models across homology (HMMER), coevolution (DCA energy), and structure (pLDDT and RMSD) scores. Across large, deep MSAs, the MSA Transformer–generated sequences score as well as or better than natural sequences and often rival or exceed bmDCA-generated sequences, including experimentally validated cases; for small families, the approach outperforms bmDCA and better reproduces higher-order statistics and the overall distribution in sequence space. The results support MSA Transformer as a strong candidate for protein sequence generation and design, highlighting coevolution-aware deep learning models as a complementary path to structure-based and evolution-guided protein design.

Abstract

Computational models starting from large ensembles of evolutionarily related protein sequences capture a representation of protein families and learn constraints associated to protein structure and function. They thus open the possibility for generating novel sequences belonging to protein families. Protein language models trained on multiple sequence alignments, such as MSA Transformer, are highly attractive candidates to this end. We propose and test an iterative method that directly employs the masked language modeling objective to generate sequences using MSA Transformer. We demonstrate that the resulting sequences score as well as natural sequences, for homology, coevolution and structure-based measures. For large protein families, our synthetic sequences have similar or better properties compared to sequences generated by Potts models, including experimentally-validated ones. Moreover, for small protein families, our generation method based on MSA Transformer outperforms Potts models. Our method also more accurately reproduces the higher-order statistics and the distribution of sequences in sequence space of natural data than Potts models. MSA Transformer is thus a strong candidate for protein sequence generation and protein design.

Figures (22)

• Figure 1: Comparison of homology, coevolution, and structure-based scores between natural sequences and sequences generated by MSA Transformer or bmDCA. For each Pfam family in \ref{['tab:dataset']}, we compare a natural MSA from Pfam and three synthetic MSAs of the same depth. The first synthetic MSA was obtained using MSA Transformer via our iterative masking procedure, and the second and third ones were generated by a Potts model inferred from the natural MSA using bmDCA with two different pairs $(\lambda,\,T)$ of regularization strength $\lambda$ and sampling temperature $T$. For each of the four scores described in " \ref{['sec:Scores']} ", we show the distributions of score values among sequences in each MSA as a violin plot. Higher score values are better for all scores except RMSD (bottom panel), where smaller values indicate a closer match to an experimental structure. Top panel: For each Pfam family, HMMER scores are divided by the highest score found in the natural MSA. Note that sequences below HMMER's default homology detection score (E-value larger than 10), and whose HMMER score is thus 0, are not shown (the median over families of the fraction of such sequences is $2\%$ for bmDCA($10^{-2}, 1.00$)-generated MSAs, while there are no such sequences among the MSA-Transformer--generated ones). Second panel: Statistical energy scores are defined as minus the bmDCA statistical energies. To accommodate the highly family-dependent ranges of these scores, for each Pfam family we show their values after shifting by the mean score in the natural MSA, and normalizing by the standard deviation of natural MSA scores. Third panel: AlphaFold's pLDDT confidence scores. Bottom panel: RMSD of predicted structures with respect to the experimental structures in \ref{['tab:dataset']}. Structural scores (pLDDT and RMSD) were computed on $200$ randomly chosen sequences from each MSA. All kernel-smoothed histograms are normalized such that all violins have the same maximal width. Outliers (less than 1% in all cases) were discarded for legibility.
• Figure 2: Homology and coevolution scores vs. distance to the natural MSA, for protein families PF00072 and PF00153. We show contour plots of the HMMER score and the statistical energy score (defined as minus the DCA statistical energy, shifted by its mean value in the natural MSA) versus the Hamming distance of each sequence to the closest natural sequence (which is not itself, in the case of natural sequences). Results are shown for natural sequences and for sequences generated using MSA Transformer and bmDCA (the same two $(\lambda,T)$ pairs as in \ref{['fig:violin_h']} are used for bmDCA). The lightest contours shown include $99\%$ of the cumulative probability mass.
• Figure 3: Application of our sequence generation method based on MSA Transformer to small protein families. We consider 7 small protein families, with natural MSAs that comprise from 9 to a few hundreds of sequences, see \ref{['tab:dataset_bis']}. As in \ref{['fig:violin_h']}, for each family, we compare the natural MSA and three synthetic MSAs of the same depth. In all cases, we show violin plots of the same four scores as for large families in \ref{['fig:violin_h']}, as well as of the Hamming distance to the closest natural sequence, which is not itself in the case of natural sequences ("Distance"). For the three smallest families (left panel; fewer than 40 sequences), we also show the score of each individual sequence as a swarm plot. Note that while we employ the same sampling temperatures $T$ as in \ref{['fig:violin_h']} for bmDCA, here, we use regularization strength $\lambda=10^{-2}$ throughout, due to MSA shallowness (see " \ref{['subsec:bmDCA']} ").
• Figure 4: Similarity of statistics between synthetic and natural MSAs. To compare the statistics of synthetic and natural MSAs at various orders, we compute r20 scores Haldane2018McGee2021, and plot them versus the number of different MSA columns that are considered (see " \ref{['sec:Statistics']} " for details). All families in \ref{['tab:dataset']} are considered. For each of them, the reference MSA comprises either half of the natural MSA (with sequences selected uniformly at random), or 30,000 sequences from it if the natural MSA depth is larger than 60,000. The null model compares the other half of the natural MSA to this reference MSA. It yields an estimate of the expected r20 scores due only to finite-size effects in a model-free, purely data-driven way.
• Figure 5: Neighbors of natural and synthetic sequences, for families PF00072 and PF00153. We show the distribution of the number of neighbors of sequences in the natural MSA, and the distribution of the number of neighbors of the closest natural sequence to each of our generated sequences. Given a sequence in a natural MSA, its number of neighbors is the number of natural sequences that are within a (normalized) Hamming distance $\delta = 0.2$ from it. The moving average of the results is shown, using a window representing $5\%$ of the total number of points.