Divide-Then-Align: Honest Alignment based on the Knowledge Boundary of RAG

TL;DR

This work tackles the problem that Retrieval-Augmented Generation (RAG) and RAFT models often provide answers even when neither the model’s parametric knowledge nor retrieved passages suffice. It introduces Divide-Then-Align (DTA), a post-training framework that partitions queries into four knowledge quadrants based on two boundaries: the parametric knowledge boundary $ m KB_{param}$ and the retrieval boundary $ m KB_r$. By constructing quadrant-specific preference data and training with Direct Preference Optimization (DPO) plus supervised fine-tuning and a quadrant-classification loss, DTA enables honest abstention with the response “I don’t know” when a query lies outside both boundaries, while preserving high answer quality for answerable cases. Experiments on Natural Questions, TriviaQA, and WebQuestions show that DTA outperforms RAFT-based baselines and calibrations across nine metrics, balancing accuracy with principled abstention, and demonstrating robustness across biomedical and counterfactual contexts. The approach advances trustworthy RAG systems by aligning behavior with explicit knowledge boundaries and provides a comprehensive evaluation framework for abstention capability in open-domain QA.

Abstract

Large language models (LLMs) augmented with retrieval systems have significantly advanced natural language processing tasks by integrating external knowledge sources, enabling more accurate and contextually rich responses. To improve the robustness of such systems against noisy retrievals, Retrieval-Augmented Fine-Tuning (RAFT) has emerged as a widely adopted method. However, RAFT conditions models to generate answers even in the absence of reliable knowledge. This behavior undermines their reliability in high-stakes domains, where acknowledging uncertainty is critical. To address this issue, we propose Divide-Then-Align (DTA), a post-training approach designed to endow RAG systems with the ability to respond with "I don't know" when the query is out of the knowledge boundary of both the retrieved passages and the model's internal knowledge. DTA divides data samples into four knowledge quadrants and constructs tailored preference data for each quadrant, resulting in a curated dataset for Direct Preference Optimization (DPO). Experimental results on three benchmark datasets demonstrate that DTA effectively balances accuracy with appropriate abstention, enhancing the reliability and trustworthiness of retrieval-augmented systems.

Figures (5)

• Figure 1: Knowledge Boundary of RAG. A query can be divided into four quadrants based on the model's parametric knowledge boundary ($\mathrm{KB}_{\mathrm{param}}$) and the knowledge boundary of the retrieval passages ($\mathrm{KB}_r$). The queries that fall into ✘✘ should be answered with "I don't know" instead of generating potentially hallucinatory answers.
• Figure 2: The pipeline of knowledge quadrants division and preference dataset construction. GT denotes the ground truth answer; IDK represents "I don't know" response; WA1 and WA2 are wrong answers generated by the LLM (WA = Wrong Answer); "If Wrong" indicates the condition where the model generates an incorrect response. The symbol ">" indicates a preference relationship where the left option is preferred over the right option. The preference construction (right) shows how different response types (GT, IDK, WA1, WA2) are ranked based on the knowledge quadrant the query falls into. $\mathrm{KB}_{\mathrm{param}}$ means the LLM's parametric knowledge boundary and $\mathrm{KB}_r$ means the retrieval knowledge boundary.
• Figure 3: Experiments across DPO data size. (IDK ratio=0.7, loss weights $\beta$=1.0, $\gamma$=0.5)
• Figure 4: Experiments across multi-objective loss weights. (DPO data size=5k, IDK ratio=0.7)
• Figure 5: Experiments across IDK ratio. (DPO data size=5k, loss weights $\beta$=1.0, $\gamma$=0.5)