CORD: Balancing COnsistency and Rank Distillation for Robust Retrieval-Augmented Generation

TL;DR

CORD addresses position bias in retrieval-augmented generation by jointly enforcing consistency across perturbed context orders and preserving useful retriever rank information through adaptive distillation. It introduces an interpolation-based perturbation space $\mathbf{c}^ extprime_\alpha$ and learns with a combination of a consistency loss and a distillation loss, while adaptively selecting perturbation strength using scenario signals and retriever scores. The approach yields improved performance over baselines on MS MARCO, HotpotQA, NaturalQuestions, and MN, with ablations confirming the value of adaptive teacher selection and score-aware sampling. Overall, CORD offers a practical framework for robust RAG by balancing consistency with rank priors, enhancing grounding and reducing reliance on brittle context ordering.

Abstract

With the adoption of retrieval-augmented generation (RAG), large language models (LLMs) are expected to ground their generation to the retrieved contexts. Yet, this is hindered by position bias of LLMs, failing to evenly attend to all contexts. Previous work has addressed this by synthesizing contexts with perturbed positions of gold segment, creating a position-diversified train set. We extend this intuition to propose consistency regularization with augmentation and distillation. First, we augment each training instance with its position perturbation to encourage consistent predictions, regardless of ordering. We also distill behaviors of this pair, although it can be counterproductive in certain RAG scenarios where the given order from the retriever is crucial for generation quality. We thus propose CORD, balancing COnsistency and Rank Distillation. CORD adaptively samples noise-controlled perturbations from an interpolation space, ensuring both consistency and respect for the rank prior. Empirical results show this balance enables CORD to outperform consistently in diverse RAG benchmarks.

Figures (4)

• Figure 1: Enforcing consistency with (1) augmentation (green) and (2) distillation (blue).
• Figure 2: (Left) In2 only uses $\mathbf{c}^\prime$. (Right) We augment the given order $\mathbf{c}$ (top) with perturbed ranking $\mathbf{c}^\prime$ (bottom) and use both.
• Figure 3: Interpolated sample space for scenario A and B from Table \ref{['tab:risk']}, where (A, left) perturbation leads to a large drop in probability of ground-truth $\hat{y}$, and (B, right) with no such drop.
• Figure 4: (Top) On MS MARCO, the interpolated noise-controlled perturbation $\mathbf{c}^\prime_\alpha$ (dark blue) is much more likely to be paired with the given order $\mathbf{c}$, than $\mathbf{c}^\prime$ (light blue). (Bottom) The gap is much smaller on MN.