An Information Theoretic Perspective on Agentic System Design

TL;DR

The paper reframes compressor-predictor agentic LM systems through an information-theoretic lens, introducing mutual information between raw context and compressed representations conditioned on a query and a rate-distortion perspective to quantify compression quality independently of task specifics. Empirical results across five datasets show that larger, more capable compressors not only improve accuracy but also increase token efficiency, with information rate tightly predicting downstream performance. A key practical takeaway is to front-load compute into local compressors, enabling smaller cloud predictors and substantial cost savings without sacrificing performance. The work culminates in a scalable Deep Research demonstration where on-device compressors achieve near frontier-LM accuracy at a fraction of API costs, while outlining guidelines and limitations for future information-theoretic design of agentic systems.

Abstract

Agentic language model (LM) systems power modern applications like "Deep Research" and "Claude Code," and leverage multi-LM architectures to overcome context limitations. Beneath their apparent diversity lies a recurring pattern: smaller "compressor" LMs (that can even run locally) distill raw context into compact text that is then consumed by larger "predictor" LMs. Despite their popularity, the design of compressor-predictor systems remains largely ad hoc, with little guidance on how compressor and predictor choices shape downstream performance. In practice, attributing gains to compression versus prediction requires costly, task-specific pairwise sweeps. We argue that these agentic system design questions are, at root, information-theoretic. Viewing the compressor LM as a noisy channel, we introduce a simple estimator of mutual information between the context and its compression to quantify compression quality in a task-independent way. We show that mutual information strongly predicts downstream performance, independent of any specific task. Through an information-theoretic framework, we perform a comprehensive empirical analysis across five datasets and three model families. Results reveal that larger compressors not only are more accurate, but also more token-efficient, conveying more bits of information per token. A 7B Qwen-2.5 compressor, for instance, is $1.6\times$ more accurate, $4.6\times$ more concise, and conveys $5.5\times$ more bits of mutual information per token than its 1.5B sibling. Across datasets, scaling compressors is substantially more effective than scaling predictors, enabling larger on-device compressors to pair with smaller cloud predictors. Applied to a Deep Research system, these principles enable local compressors as small as 3B parameters to recover $99\%$ of frontier-LM accuracy at $26\%$ of API costs.

Figures (24)

• Figure 1: Why compressors matter. Many agentic LM systems rely on compressors, and personal devices are growing powerful enough to host them. (Left) A compressor condenses a long input $X$ into a shorter summary $Z$, which a predictor ingests to extract the final answer $Y$. (Right) Consumer hardware can now run increasingly large open-weight LMs, shown for Google Pixel phones and Apple MacBook laptops under FP16 precision with memory estimates from Modal modal2024vram. LM-Arena ranks indicate relative performance.
• Figure 2: Downstream accuracy, compression length, and compute cost scale with compressor size (Top: LongHealth; Bottom: FinanceBench). We scale compressor model size and reports a different metric on the $y$-axis of each column: (Left) accuracy, with the black dotted line showing the GPT-4o model baseline. (Middle) compression length, (Right) GFLOPs-per-compression. Vertical bars denote standard errors. Larger compressors produce shorter outputs with higher downstream accuracy. Similar trends hold on QASPER (Appendix \ref{['sec:qasper-results-appendix']}), Wildchat (Appendix \ref{['sec:wildchat-results-appendix']}) and FineWeb (Appendix \ref{['sec:fineweb-results-appendix']}).
• Figure 3: Scaling compressors is more effective than scaling predictors on LongHealth. The $y$-axis reports accuracy and $x$-axis shows total compute cost in FLOPs-per-generation (log-scale). We compare compressor LMs from two families: (Left)Qwen-2.5, (Right)Llama-3. We scale predictor (marker color) and compressor (marker label) sizes and measure the total FLOPs-per-generation and downstream accuracy on QA tasks. Appendix \ref{['sec:financebench-mi-appendix']} shows consistent trends on FinanceBench.
• Figure 4: Larger compressors generate outputs that carry more information about their inputs (conditioned on the query) on LongHealth. We scale compressor model size and estimate the (Left) mutual information, and (Right) bit efficiency (bits of mutual information per token; higher is better) carried by their outputs. Larger compressor model sizes compress documents with higher mutual information and bit efficiency. The black dotted line represents the theoretical maximum of the mutual information estimator at the natural logarithm $\log(N)$, where $N$ is the number of documents mutual information is computed across. We find consistent trends on FinanceBench (Appendix \ref{['sec:financebench-mi-appendix']}) and QASPER (Appendix \ref{['sec:qasper-results-appendix']}).
• Figure 5: Scaling behavior of compressor model size hold across instructed conciseness (Compressor = Qwen-2.5). We ablate over different levels of compression conciseness by varying the compression prompt instructions. We measure (Left) accuracy, (Middle) GFLOPs-per-generation, and estimate (Right) mutual information. We find that accuracy and mutual information are largely unaffected by conciseness instructions. Compressors instructed to be more concise are more token-efficient, and thus compute-efficient. Trends in accuracy, compute cost, and mutual information as we scale compressor hold across conciseness constraints. Appendix \ref{['sec:financebench-prompting-appendix']} shows analogous results on FinanceBench.

Theorems & Definitions (2)

• Theorem B.1
• proof