Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parallel Context-of-Experts Decoding for Retrieval Augmented Generation

Giulio Corallo, Paolo Papotti

TL;DR

Retrieval Augmented Generation often sacrifices cross-document reasoning when using independent KV caches or incurs steep latency with long concatenated prompts. The authors introduce Parallel Context-of-Experts Decoding (Pced), a training-free decoding framework that treats each retrieved document as a separate expert and aggregates evidence at decode time through retrieval-aware contrastive decoding, guided by per-document priors. Pced demonstrates strong performance on LOFT and LongBench, often matching or exceeding full-context baselines while delivering substantial speedups, and shows robust cross-document reasoning with large candidate pools. The work highlights practical deployment considerations, including logits access requirements and storage-cost trade-offs for offline KV caches, and identifies limitations tied to retrieval quality and API-access restrictions.

Abstract

Retrieval Augmented Generation faces a trade-off: concatenating documents in a long prompt enables multi-document reasoning but creates prefill bottlenecks, while encoding document KV caches separately offers speed but breaks cross-document interaction. We propose Parallel Context-of-Experts Decoding (Pced), a training-free framework that shifts evidence aggregation from the attention mechanism to the decoding. Pced treats retrieved documents as isolated "experts", synchronizing their predictions via a novel retrieval-aware contrastive decoding rule that weighs expert logits against the model prior. This approach recovers cross-document reasoning capabilities without constructing a shared attention across documents.

Parallel Context-of-Experts Decoding for Retrieval Augmented Generation

TL;DR

Retrieval Augmented Generation often sacrifices cross-document reasoning when using independent KV caches or incurs steep latency with long concatenated prompts. The authors introduce Parallel Context-of-Experts Decoding (Pced), a training-free decoding framework that treats each retrieved document as a separate expert and aggregates evidence at decode time through retrieval-aware contrastive decoding, guided by per-document priors. Pced demonstrates strong performance on LOFT and LongBench, often matching or exceeding full-context baselines while delivering substantial speedups, and shows robust cross-document reasoning with large candidate pools. The work highlights practical deployment considerations, including logits access requirements and storage-cost trade-offs for offline KV caches, and identifies limitations tied to retrieval quality and API-access restrictions.

Abstract

Retrieval Augmented Generation faces a trade-off: concatenating documents in a long prompt enables multi-document reasoning but creates prefill bottlenecks, while encoding document KV caches separately offers speed but breaks cross-document interaction. We propose Parallel Context-of-Experts Decoding (Pced), a training-free framework that shifts evidence aggregation from the attention mechanism to the decoding. Pced treats retrieved documents as isolated "experts", synchronizing their predictions via a novel retrieval-aware contrastive decoding rule that weighs expert logits against the model prior. This approach recovers cross-document reasoning capabilities without constructing a shared attention across documents.
Paper Structure (31 sections, 7 equations, 6 figures, 7 tables)

This paper contains 31 sections, 7 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Retrieval-aware contrastive decoding.
  5. Experimental Setup
  6. Results and Discussion
  7. Conclusion
  8. Dependence on access to model logits.
  9. Sensitivity to retrieval quality.
  10. Storage-Computation Trade-offs.
  11. Evaluation Setup
  12. Prompt Definitions.
  13. LOFT Benchmark
  14. LOFT-RAG Templates
  15. LOFT-ICL Templates
  16. ...and 16 more sections

Figures (6)

  • Figure 1: Parallel Context-of-Experts Decoding (Pced) runs one expert per retrieved document (and a no-context, amateur prior) in parallel and chooses each next token based on retrieval support, enabling evidence to be stitched across documents without joint attention.
  • Figure 2: HotpotQA expert trace. Green dots illustrate the model hopping between multiple gold documents.
  • Figure 3: Latency Benchmarks. Comparison of TTFT scalability across Top-$k$ values (left) and total end-to-end latency with 65k context (right).
  • Figure 4: Prompt template configuration for LOFT-RAG tasks.
  • Figure 5: Prompt template configuration for LOFT-ICL tasks.
  • ...and 1 more figures