ArcAligner: Adaptive Recursive Aligner for Compressed Context Embeddings in RAG

TL;DR

ArcAligner addresses the bottleneck of long-document RAG by integrating an adaptive recursive alignment module into LLM layers that refines compressed context through a gated, multi-step process. The method uses a context-slot interface with selective LoRA updates and a gate-controlled recursion depth, trained in three stages to achieve robust alignment and task grounding under compression. Empirical results on knowledge-intensive QA benchmarks show consistent improvements over embedding-based baselines, particularly for multi-hop and long-tail questions, with analyses detailing gate behavior and reconstruction fidelity. This approach enables efficient, scalable RAG with highly compressed evidence and has public code available for practical adoption.

Abstract

Retrieval-Augmented Generation (RAG) helps LLMs stay accurate, but feeding long documents into a prompt makes the model slow and expensive. This has motivated context compression, ranging from token pruning and summarization to embedding-based compression. While researchers have tried ''compressing'' these documents into smaller summaries or mathematical embeddings, there is a catch: the more you compress the data, the more the LLM struggles to understand it. To address this challenge, we propose ArcAligner (Adaptive recursive context *Aligner*), a lightweight module integrated into the language model layers to help the model better utilize highly compressed context representations for downstream generation. It uses an adaptive ''gating'' system that only adds extra processing power when the information is complex, keeping the system fast. Across knowledge-intensive QA benchmarks, ArcAligner consistently beats compression baselines at comparable compression rates, especially on multi-hop and long-tail settings. The source code is publicly available.

Figures (10)

• Figure 1: Comparison of different methods.
• Figure 2: Illustration of the ArcAligner framework. The left part is an illustration of our training and inference process, where the context is compressed and progressively refined within the LLM while the input query remains unchanged. The right part is a flow of the token through each layer, passing through the LoRA and the gate, where the context tokens are adaptively and recursively refined.
• Figure 3: Accuracy across different backbone language models on four QA datasets. Grouped bars compare Naive, StandardRAG, and ArcAligner(Ours) under two backbones: Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.2.
• Figure 4: Reconstruction perplexity (PPL) after the pretraining stage, evaluated on a held-out pretraining test set with 10k samples. Results are reported for two backbone language models, Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.2. Lower PPL indicates better reconstruction fidelity.
• Figure 5: Gate behavior analysis across layers. The figure shows the token-level average number of refinement loops triggered by the gate at each layer evaluated on four datasets.