Sparse Brains are Also Adaptive Brains: Cognitive-Load-Aware Dynamic Activation for LLMs

TL;DR

CLADA tackles inefficiency in dense LLMs by introducing cognitive-load-aware dynamic activation that combines global statistical sparsity with local semantic adaptability. The method uses offline base-threshold optimization plus online token-level cognitive signals derived from surprisal and entropy to adjust activations, requiring no retraining. Empirical results across six models and nine benchmarks show about 20% average speedup with less than 2% accuracy loss, outperforming prior dynamic activation approaches like Griffin and TT. The work also establishes a formal link between neurolinguistic ERP components and LLM efficiency via regression analysis, bridging cognitive science and scalable AI.

Abstract

Dense large language models(LLMs) face critical efficiency bottlenecks as they rigidly activate all parameters regardless of input complexity. While existing sparsity methods(static pruning or dynamic activation) address this partially, they either lack adaptivity to contextual or model structural demands or incur prohibitive computational overhead. Inspired by human brain's dual-process mechanisms - predictive coding (N400) for backbone sparsity and structural reanalysis (P600) for complex context - we propose CLADA, a \textit{\textbf{C}ognitive-\textbf{L}oad-\textbf{A}ware \textbf{D}ynamic \textbf{A}ctivation} framework that synergizes statistical sparsity with semantic adaptability. Our key insight is that LLM activations exhibit two complementary patterns: 1) \textit{Global statistical sparsity} driven by sequence-level prefix information, and 2) \textit{Local semantic adaptability} modulated by cognitive load metrics(e.g., surprisal and entropy). CLADA employs a hierarchical thresholding strategy: a baseline from offline error-controlled optimization ensures 40\%+ sparsity, dynamically adjusted by real-time cognitive signals. Evaluations across six mainstream LLMs and nine benchmarks demonstrate that CLADA achieves \textbf{~20\% average speedup with <2\% accuracy drop}, outperforming Griffin (5\%+ degradation) and TT (negligible speedup). Crucially, we establish the first formal connection between neurolinguistic event-related potential (ERP) components and LLM efficiency mechanisms through multi-level regression analysis ($R^2=0.17$ for sparsity-adaptation synergy). Requiring no retraining or architectural changes, CLADA offers a deployable solution for resource-aware LLM inference while advancing biologically-inspired AI design. Our code is available at \href{https://github.com/Oldify/CLADA}{CLADA}.

Figures (7)

• Figure 1: A comparative analysis of dynamic activation methods, highlighting CLADA's advantages in minimal computational overhead, substantial inference speedup, the ability to maximize model capability, and robust generalization ability across various LLM architectures.
• Figure 2: Training-Dependent DA
• Figure 3: Training-Free DA
• Figure 4: Activation of NLS-A
• Figure 5: Activation of NLS-B