Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decoding

TL;DR

This work tackles knowledge-intensive NLP challenges in Retrieval-Augmented Generation by introducing PORAG, a policy-optimized framework, and ATLAS, a dynamic retrieval timing and content selector. PORAG leverages GRPO-based reinforcement learning with QLoRA adapters and dual reward heads to directly optimize retrieval fidelity and response quality, while ATLAS uses MLAG and Layerwise Representation Pooling to trigger retrieval and craft precise queries. The approach is augmented with CRITIC for selective KV-cache compression and test-time scaling to balance reasoning depth with compute, yielding faster, more memory-efficient long-context inference. Across HotpotQA, Gorilla, and PubMedQA, PORAG+ATLAS achieves substantial gains in accuracy and factual grounding with notable reductions in hallucinations and API errors, supported by extensive ablations and cross-LLM experiments. The results demonstrate practical impact for scalable, robust RAG systems applicable to diverse domains and deployment constraints, with optimized decoding and latency optimizations further enhancing real-world viability.

Abstract

We present a comprehensive framework for enhancing Retrieval-Augmented Generation (RAG) systems through dynamic retrieval strategies and reinforcement fine-tuning. This approach significantly improves large language models on knowledge-intensive tasks, including opendomain question answering and complex reasoning. Our framework integrates two complementary techniques: Policy-Optimized RetrievalAugmented Generation (PORAG), which optimizes the use of retrieved information, and Adaptive Token-Layer Attention Scoring (ATLAS), which dynamically determines retrieval timing and content based on contextual needs. Together, these techniques enhance both the utilization and relevance of retrieved content, improving factual accuracy and response quality. Designed as a lightweight solution compatible with any Transformer-based LLM without requiring additional training, our framework excels in knowledge-intensive tasks, boosting output accuracy in RAG settings. We further propose CRITIC, a novel method to selectively compress key-value caches by token importance, mitigating memory bottlenecks in long-context applications. The framework also incorporates test-time scaling techniques to dynamically balance reasoning depth and computational resources, alongside optimized decoding strategies for faster inference. Experiments on benchmark datasets show that our framework reduces hallucinations, strengthens domain-specific reasoning, and achieves significant efficiency and scalability gains over traditional RAG systems. This integrated approach advances the development of robust, efficient, and scalable RAG systems across diverse applications.