FrugalPrompt: Reducing Contextual Overhead in Large Language Models via Token Attribution

TL;DR

FrugalPrompt tackles the rising cost and latency of large language models by compressing prompts at the token level using saliency scores from GlobEnc and DecompX. The method ranks tokens by semantic importance and retains only the top-$k$% in their original order, producing sparse prompts without training. Empirical results show that a $20\%$ token reduction preserves performance for sentiment analysis, commonsense QA, and summarization across frontier LLMs, while mathematical reasoning suffers markedly due to its dependence on complete token continuity. The work highlights task-dependent trade-offs, potential task contamination in benchmarks, and provides open-source code to enable broader adoption of efficient prompt prompting in real-world applications.

Abstract

Large language models (LLMs) owe much of their stellar performance to expansive input contexts, yet such verbosity inflates monetary costs, carbon footprint, and inference-time latency. Much of this overhead manifests from the redundant low-utility tokens present in typical prompts, as only a fraction of tokens typically carries the majority of the semantic weight. We address this inefficiency by introducing FrugalPrompt, a novel prompt compression framework for LLMs, which retains only the most semantically significant tokens. Leveraging two state-of-the-art token attribution methods, GlobEnc and DecompX, we assign salience scores to every token in an input sequence, rank them to preserve the top-k% tokens in their original order, and obtain a sparse frugalized prompt. We evaluate the approach across four NLP tasks: Sentiment Analysis, Commonsense QA, Summarization, and Mathematical Reasoning, using a suite of frontier LLMs. For the first three tasks, a 20% prompt reduction incurs only a marginal loss in task performance, demonstrating that contemporary LLMs can reconstruct elided context from high-salience cues. In contrast, performance on mathematical reasoning deteriorates sharply, reflecting a stronger dependence on complete token continuity. Further analysis with bottom-k% and random-k% tokens reveals asymmetric performance patterns that may suggest potential task contamination effects, wherein models may resort to shallow memorized patterns from pretraining exposure for conventional NLP tasks. We posit that our work contributes to a more nuanced understanding of LLM behavior in performance-efficiency trade-offs, and delineate the boundary between tasks tolerant to contextual sparsity and those requiring exhaustive context. Our source code and models are available at: https://github.com/Starscream-11813/Frugal-ICL.

Figures (3)

• Figure 1: Overview of the FrugalPrompt pipeline. The input prompt with tokens $\langle t_1, t_2,\dots,t_n\rangle$ is passed to our task-specific token attribution module to generate the saliency scores for each token $\langle s_1, s_2,\dots,s_n\rangle$. The tokens are ordered based on their saliency in a permutation $\pi$ and the top $p=k\%*n$ tokens are filtered while maintaining the order of the text to form the reduced text $\langle t_{\pi(1)}, t_{\pi(2)},\dots,t_{\pi(p)}\rangle$. The reduced text is inferred by a frozen LLM to generate the answer tokens $\langle a_1, a_2,\dots,a_n\rangle$.
• Figure 2: The performance difference between the reduced tokens and baseline performance using 100% tokens of metrics across three tasks: text classification (CLS), question-answering (QA), and reasoning (RSN).
• Figure 3: The performance difference between the reduced tokens and baseline performance using 100% tokens across the summarization metrics.