Lost in Tokenization: Context as the Key to Unlocking Biomolecular Understanding in Scientific LLMs

TL;DR

The paper tackles the tokenization dilemma in biomolecular reasoning by Sci-LLMs and proposes a context-driven paradigm that leverages high-level textual context derived from established bioinformatics tools. It systematically compares three input modes—sequence-only, context-only, and sequence+context—across multiple models, finding that context-only inputs consistently outperform alternatives, while raw sequences degrade performance. The authors implement a multi-step context-generation pipeline (InterProScan, BLAST, and a ProTrek fallback) to produce dense, human-readable annotations that feed language models, reframing Sci-LLMs as reasoning engines over expert knowledge rather than sequence decoders. While the approach shows robust generalization and temporal stability, it acknowledges limitations for truly novel proteins and emphasizes the need for further exploration of context-driven reasoning in diverse biomolecular domains. The work provides a foundation for hybrid scientific AI agents that prioritize knowledge synthesis over direct sequence interpretation, with code available at the provided repository.

Abstract

Scientific Large Language Models (Sci-LLMs) have emerged as a promising frontier for accelerating biological discovery. However, these models face a fundamental challenge when processing raw biomolecular sequences: the tokenization dilemma. Whether treating sequences as a specialized language, risking the loss of functional motif information, or as a separate modality, introducing formidable alignment challenges, current strategies fundamentally limit their reasoning capacity. We challenge this sequence-centric paradigm by positing that a more effective strategy is to provide Sci-LLMs with high-level structured context derived from established bioinformatics tools, thereby bypassing the need to interpret low-level noisy sequence data directly. Through a systematic comparison of leading Sci-LLMs on biological reasoning tasks, we tested three input modes: sequence-only, context-only, and a combination of both. Our findings are striking: the context-only approach consistently and substantially outperforms all other modes. Even more revealing, the inclusion of the raw sequence alongside its high-level context consistently degrades performance, indicating that raw sequences act as informational noise, even for models with specialized tokenization schemes. These results suggest that the primary strength of existing Sci-LLMs lies not in their nascent ability to interpret biomolecular syntax from scratch, but in their profound capacity for reasoning over structured, human-readable knowledge. Therefore, we argue for reframing Sci-LLMs not as sequence decoders, but as powerful reasoning engines over expert knowledge. This work lays the foundation for a new class of hybrid scientific AI agents, repositioning the developmental focus from direct sequence interpretation towards high-level knowledge synthesis. The code is available at https://github.com/opendatalab-raiser/CoKE.

Figures (10)

• Figure 1: Paradigms for integrating biomolecular sequences into Sci-LLMs. (a) The sequence-as-language approach, tokenization fragments sequences into atomic symbols, potentially destroying functional motifs. (b) The sequence-as-modality approach preserves structure via specialized encoders but suffers from semantic misalignment with natural language. (c) The context-driven approach leverages bioinformatics tools to provide high-density, natively aligned textual context.
• Figure 2: The visualization of representation spaces.
• Figure 3: Visualization of representation spaces at different stages within the Evolla-10B model.
• Figure 4: Comparison of Evolla-10B and our approach across the easy, medium, and hard subsets.
• Figure 5: Analysis of model performance versus protein's first publication year.