Identifying and Analyzing Performance-Critical Tokens in Large Language Models

TL;DR

The paper investigates how large language models learn from demonstrations by identifying performance-critical tokens in in-context prompts. It categorizes prompt tokens into template, stopword, and content tokens, and uses representation- and token-level ablations to show that template and stopword tokens directly drive performance while content tokens contribute mainly indirectly by aggregating information into the critical tokens. The authors find that lexical meaning, repetition, and structural cues characterize performance-critical tokens and that these effects persist across model sizes and tasks, challenging the view that only label words are central. These insights inform prompt design and robustness, suggesting targeted token-level considerations to stabilize and improve ICL under varying prompts and demonstrations. Code and data availability further support reproducibility and practical adoption of these findings.

Abstract

In-context learning (ICL) has emerged as an effective solution for few-shot learning with large language models (LLMs). However, how LLMs leverage demonstrations to specify a task and learn a corresponding computational function through ICL is underexplored. Drawing from the way humans learn from content-label mappings in demonstrations, we categorize the tokens in an ICL prompt into content, stopword, and template tokens. Our goal is to identify the types of tokens whose representations directly influence LLM's performance, a property we refer to as being performance-critical. By ablating representations from the attention of the test example, we find that the representations of informative content tokens have less influence on performance compared to template and stopword tokens, which contrasts with the human attention to informative words. We give evidence that the representations of performance-critical tokens aggregate information from the content tokens. Moreover, we demonstrate experimentally that lexical meaning, repetition, and structural cues are the main distinguishing characteristics of these tokens. Our work sheds light on how large language models learn to perform tasks from demonstrations and deepens our understanding of the roles different types of tokens play in large language models.

Figures (3)

• Figure 1: An illustration of 4-way text classification on AGNews with different parts of its 4-shot ICL demonstrations masked with respect to the attention of the test example. Specifically, (b) masking the representations of what we call the template and stopword tokens from the attention of the test example leads to a significant drop in performance while (c) masking representations of the content tokens leaves the performance relatively unchanged. The dashed lines represent the attention between every pair of tokens while those from the test example to the ICL prompt are unshaded.
• Figure 2: An illustrative example of the token-level ablation we use to analyze the working mechanism of performance-critical tokens.
• Figure 3: Results of the token-level ablation where, for example, $-$stop refers to the ablation where stopword tokens are dropped from the ICL prompt. Models without template tokens consistently yielded an accuracy of 0% and are thus omitted from this figure.