PA-RAG: RAG Alignment via Multi-Perspective Preference Optimization

TL;DR

PA-RAG addresses the misalignment of end-to-end RAG generators with retrieval-grounded tasks by introducing a two-phase training pipeline: instruction fine-tuning to establish basic RAG capabilities, and staged Direct Preference Optimization to learn from multiple perspectives—informativeness, robustness, and citation quality. By constructing high-quality instruction data with citation rewrites and building diverse preference data under varying document qualities, PA-RAG trains generators to fully utilize relevant documents, resist noisy context, and cite sources accurately. Across four QA datasets and three LLM backbones, PA-RAG yields substantial gains in correctness and citation quality, outperforming SFT and pipeline baselines and demonstrating robust generalization and improved citation grounding. The work provides practical training data and demonstrates that sequential, perspective-aligned optimization is crucial for aligning RAG systems with end-to-end architectures and real-world retrieval variability.

Abstract

The emergence of Retrieval-augmented generation (RAG) has alleviated the issues of outdated and hallucinatory content in the generation of large language models (LLMs), yet it still reveals numerous limitations. When a general-purpose LLM serves as the RAG generator, it often suffers from inadequate response informativeness, response robustness, and citation quality. Past approaches to tackle these limitations, either by incorporating additional steps beyond generating responses or optimizing the generator through supervised fine-tuning (SFT), still failed to align with the RAG requirement thoroughly. Consequently, optimizing the RAG generator from multiple preference perspectives while maintaining its end-to-end LLM form remains a challenge. To bridge this gap, we propose Multiple Perspective Preference Alignment for Retrieval-Augmented Generation (PA-RAG), a method for optimizing the generator of RAG systems to align with RAG requirements comprehensively. Specifically, we construct high-quality instruction fine-tuning data and multi-perspective preference data by sampling varied quality responses from the generator across different prompt documents quality scenarios. Subsequently, we optimize the generator using SFT and Direct Preference Optimization (DPO). Extensive experiments conducted on four question-answer datasets across three LLMs demonstrate that PA-RAG can significantly enhance the performance of RAG generators. Our code and datasets are available at https://github.com/wujwyi/PA-RAG.

Figures (3)

• Figure 1: PA-RAG retains the end-to-end form of the generator while enabling the generator to learn multi-perspective preference information.
• Figure 2: Overview of instruction fine-tuning data construction.
• Figure 3: Overview of preference data construction.