Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KnowPO: Knowledge-aware Preference Optimization for Controllable Knowledge Selection in Retrieval-Augmented Language Models

Ruizhe Zhang, Yongxin Xu, Yuzhen Xiao, Runchuan Zhu, Xinke Jiang, Xu Chu, Junfeng Zhao, Yasha Wang

TL;DR

KnowPO addresses knowledge conflicts in retrieval-augmented language models by creating a knowledge-aware preference framework that distinguishes adherence to external evidence from resilience to noise. It systematically constructs contradictory and irrelevant contexts, defines two error modes (Contextual Overinclusion and Contextual Ignorance), and trains models through instruction tuning complemented by Direct Preference Optimization to align behavior with user preferences. Empirical results across multiple base models and datasets show substantial gains in both adherence and robustness, with strong generalization to out-of-domain medical data. The approach offers a practical path toward robust, knowledge-grounded LLMs in real retrieval scenarios.

Abstract

By integrating external knowledge, Retrieval-Augmented Generation (RAG) has become an effective strategy for mitigating the hallucination problems that large language models (LLMs) encounter when dealing with knowledge-intensive tasks. However, in the process of integrating external non-parametric supporting evidence with internal parametric knowledge, inevitable knowledge conflicts may arise, leading to confusion in the model's responses. To enhance the knowledge selection of LLMs in various contexts, some research has focused on refining their behavior patterns through instruction-tuning. Nonetheless, due to the absence of explicit negative signals and comparative objectives, models fine-tuned in this manner may still exhibit undesirable behaviors such as contextual ignorance and contextual overinclusion. To this end, we propose a Knowledge-aware Preference Optimization strategy, dubbed KnowPO, aimed at achieving adaptive knowledge selection based on contextual relevance in real retrieval scenarios. Concretely, we proposed a general paradigm for constructing knowledge conflict datasets, which comprehensively cover various error types and learn how to avoid these negative signals through preference optimization methods. Simultaneously, we proposed a rewriting strategy and data ratio optimization strategy to address preference imbalances. Experimental results show that KnowPO outperforms previous methods for handling knowledge conflicts by over 37\%, while also exhibiting robust generalization across various out-of-distribution datasets.

KnowPO: Knowledge-aware Preference Optimization for Controllable Knowledge Selection in Retrieval-Augmented Language Models

TL;DR

KnowPO addresses knowledge conflicts in retrieval-augmented language models by creating a knowledge-aware preference framework that distinguishes adherence to external evidence from resilience to noise. It systematically constructs contradictory and irrelevant contexts, defines two error modes (Contextual Overinclusion and Contextual Ignorance), and trains models through instruction tuning complemented by Direct Preference Optimization to align behavior with user preferences. Empirical results across multiple base models and datasets show substantial gains in both adherence and robustness, with strong generalization to out-of-domain medical data. The approach offers a practical path toward robust, knowledge-grounded LLMs in real retrieval scenarios.

Abstract

By integrating external knowledge, Retrieval-Augmented Generation (RAG) has become an effective strategy for mitigating the hallucination problems that large language models (LLMs) encounter when dealing with knowledge-intensive tasks. However, in the process of integrating external non-parametric supporting evidence with internal parametric knowledge, inevitable knowledge conflicts may arise, leading to confusion in the model's responses. To enhance the knowledge selection of LLMs in various contexts, some research has focused on refining their behavior patterns through instruction-tuning. Nonetheless, due to the absence of explicit negative signals and comparative objectives, models fine-tuned in this manner may still exhibit undesirable behaviors such as contextual ignorance and contextual overinclusion. To this end, we propose a Knowledge-aware Preference Optimization strategy, dubbed KnowPO, aimed at achieving adaptive knowledge selection based on contextual relevance in real retrieval scenarios. Concretely, we proposed a general paradigm for constructing knowledge conflict datasets, which comprehensively cover various error types and learn how to avoid these negative signals through preference optimization methods. Simultaneously, we proposed a rewriting strategy and data ratio optimization strategy to address preference imbalances. Experimental results show that KnowPO outperforms previous methods for handling knowledge conflicts by over 37\%, while also exhibiting robust generalization across various out-of-distribution datasets.
Paper Structure (31 sections, 1 equation, 2 figures, 6 tables)

This paper contains 31 sections, 1 equation, 2 figures, 6 tables.

Table of Contents

  1. I. Introduction
  2. II. Related Work
  3. Knowledge Conflicts.
  4. Retrieval-Augmented Generation.
  5. Knowledge Editing.
  6. III. Methodology
  7. A. Task Definition
  8. B. Contradictory Knowledge
  9. C. Context Formulation
  10. D. Error Type Analyse
  11. Contextual Overinclusion Error.
  12. Contextual Ignorance Error.
  13. E. Training Method
  14. Instruction Tuning.
  15. Direct Preference Optimization.
  16. ...and 16 more sections

Figures (2)

  • Figure 1: An illustrative example of how does LLM behave when encountering knowledge conflicts in RAG scenarios.
  • Figure 2: (Left.) The adherence ratio of LLM when encountering conflicting context with different prior probability on dataset RGB with Baichuan2-7B-Chat. (Right.) Hyper-parameter study with data ratio $\mathcal{R}_{error}$ on Squad2.0-Eval with Baichuan2-7B-Chat.