Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hit-RAG: Learning to Reason with Long Contexts via Preference Alignment

Junming Liu, Yuqi Li, Shiping Wen, Zhigang Zeng, Tingwen Huang

TL;DR

A multi-stage preference alignment framework designed to resolve cognitive bottlenecks through a progressive optimization pipeline, enabling models to bridge the gap between context acquisition and accurate reasoning while surpassing much larger counterparts in long-context scenarios.

Abstract

Despite the promise of Retrieval-Augmented Generation in grounding Multimodal Large Language Models with external knowledge, the transition to extensive contexts often leads to significant attention dilution and reasoning hallucinations. The surge in information density causes critical evidence to be submerged by voluminous noise, which complicates the discernment of relevant fragments within a dense input. In this paper, we propose \textbf{Hit-RAG}, a multi-stage preference alignment framework designed to resolve these cognitive bottlenecks through a progressive optimization pipeline. Our approach systematically refines the utilization of external evidence via three distinct stages. First, Supervised Fine-tuning establishes baseline context awareness to minimize information neglect. Next, Discriminative Preference Alignment enhances robustness against misleading distractors. Finally, Group-Relative Policy Optimization stabilizes logical synthesis to prevent reasoning collapse. Extensive evaluations on eight benchmarks demonstrate that Hit-RAG consistently yields substantial performance gains, enabling models to bridge the gap between context acquisition and accurate reasoning while surpassing much larger counterparts in long-context scenarios.

Hit-RAG: Learning to Reason with Long Contexts via Preference Alignment

TL;DR

A multi-stage preference alignment framework designed to resolve cognitive bottlenecks through a progressive optimization pipeline, enabling models to bridge the gap between context acquisition and accurate reasoning while surpassing much larger counterparts in long-context scenarios.

Abstract

Despite the promise of Retrieval-Augmented Generation in grounding Multimodal Large Language Models with external knowledge, the transition to extensive contexts often leads to significant attention dilution and reasoning hallucinations. The surge in information density causes critical evidence to be submerged by voluminous noise, which complicates the discernment of relevant fragments within a dense input. In this paper, we propose \textbf{Hit-RAG}, a multi-stage preference alignment framework designed to resolve these cognitive bottlenecks through a progressive optimization pipeline. Our approach systematically refines the utilization of external evidence via three distinct stages. First, Supervised Fine-tuning establishes baseline context awareness to minimize information neglect. Next, Discriminative Preference Alignment enhances robustness against misleading distractors. Finally, Group-Relative Policy Optimization stabilizes logical synthesis to prevent reasoning collapse. Extensive evaluations on eight benchmarks demonstrate that Hit-RAG consistently yields substantial performance gains, enabling models to bridge the gap between context acquisition and accurate reasoning while surpassing much larger counterparts in long-context scenarios.
Paper Structure (23 sections, 3 equations, 2 figures, 6 tables)

This paper contains 23 sections, 3 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Retrieval-Augmented Generation
  4. Multimodal Large Language Models
  5. Alignment for Long-Context Multimodal Reasoning
  6. Methodology
  7. Data Construction Protocol
  8. Supervised Grounding for Long-Context Pressure
  9. Preference Alignment for Noise and Bias Robustness
  10. Policy Refinement via Group Relative Optimization
  11. Experiments
  12. Experimental Setup
  13. Datasets.
  14. Data Construction.
  15. Baselines.
  16. ...and 8 more sections

Figures (2)

  • Figure 1: A comparison of reasoning paradigms under long-context RAG. Traditional models often suffer from reasoning collapse due to passive reliance on noisy distractors. In contrast, our Hit-RAG achieves reflective contextual reasoning, enabling the model to critically discern information through multi-stage optimization.
  • Figure 2: The Hit-RAG optimization pipeline. (0) Dataset Building: Retrieving Top-$K$ documents and scaling $K$ to augment long context for enhanced robustness. (1) Supervised Fine-Tuning: Training the model to ignore irrelevant documents and locate gold evidence. (2) Reinforcement Learning Optimization: Implementing DPO to mitigate blind reliance on noisy context and GRPO to resolve reasoning collapse through outcome veracity rewards.