Context Discipline and Performance Correlation: Analyzing LLM Performance and Quality Degradation Under Varying Context Lengths
Ahilan Ayyachamy Nadar Ponnusamy, Karthic Chandran, M Maruf Hossain
TL;DR
This work addresses the problem of context-length scalability in LLM deployment by quantifying the trade-off between inference performance and reasoning quality when long, noisy contexts saturate the KV cache. Using dense Llama-3.1-70B-Instruct and Qwen1.5-14B-Chat, plus Mixtral MoE, the authors benchmark performance and accuracy across context regimes of 4k, 10k, and 15k words, revealing a non-linear latency increase driven by memory bandwidth limits while preserving high accuracy (≈97.5–98.5%). The study highlights an MoE-specific anomaly at moderate context lengths, where sparse routing adds latency before KV-cache overhead dominates, and demonstrates that context engineering is essential for sustaining throughput in real-time deployments. Overall, the paper provides actionable insights into IO-aware attention and KV-cache optimization, guiding hardware and software design for long-context LLM applications with practical implications for production systems.
Abstract
The scaling trend in Large Language Models (LLMs) has prioritized increasing the maximum context window to facilitate complex, long-form reasoning and document analysis. However, managing this expanded context introduces severe computational overhead. This paper investigates the critical trade-off between system performance and model quality when dense transformer architectures--specifically Llama-3.1-70B and Qwen1.5-14B--are exposed to large volumes of irrelevant and distracting context. The research identifies a non-linear performance degradation tied to the growth of the Key-Value (KV) cache. Furthermore, an extended analysis of the Mixture-of-Experts (MoE) architecture reveals unique behavioral anomalies at varying context scales, suggesting that architectural benefits may be masked by infrastructure bottlenecks at high token volumes.