Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Trustworthy Alignment of Retrieval-Augmented Large Language Models via Reinforcement Learning

Zongmeng Zhang, Yufeng Shi, Jinhua Zhu, Wengang Zhou, Xiang Qi, Peng Zhang, Houqiang Li

TL;DR

This work proposes a reinforcement learning based algorithm Trustworthy-Alignment, theoretically and experimentally demonstrating large language models' capability of reaching a trustworthy status without explicit supervision on how to respond.

Abstract

Trustworthiness is an essential prerequisite for the real-world application of large language models. In this paper, we focus on the trustworthiness of language models with respect to retrieval augmentation. Despite being supported with external evidence, retrieval-augmented generation still suffers from hallucinations, one primary cause of which is the conflict between contextual and parametric knowledge. We deem that retrieval-augmented language models have the inherent capabilities of supplying response according to both contextual and parametric knowledge. Inspired by aligning language models with human preference, we take the first step towards aligning retrieval-augmented language models to a status where it responds relying merely on the external evidence and disregards the interference of parametric knowledge. Specifically, we propose a reinforcement learning based algorithm Trustworthy-Alignment, theoretically and experimentally demonstrating large language models' capability of reaching a trustworthy status without explicit supervision on how to respond. Our work highlights the potential of large language models on exploring its intrinsic abilities by its own and expands the application scenarios of alignment from fulfilling human preference to creating trustworthy agents.

Trustworthy Alignment of Retrieval-Augmented Large Language Models via Reinforcement Learning

TL;DR

This work proposes a reinforcement learning based algorithm Trustworthy-Alignment, theoretically and experimentally demonstrating large language models' capability of reaching a trustworthy status without explicit supervision on how to respond.

Abstract

Trustworthiness is an essential prerequisite for the real-world application of large language models. In this paper, we focus on the trustworthiness of language models with respect to retrieval augmentation. Despite being supported with external evidence, retrieval-augmented generation still suffers from hallucinations, one primary cause of which is the conflict between contextual and parametric knowledge. We deem that retrieval-augmented language models have the inherent capabilities of supplying response according to both contextual and parametric knowledge. Inspired by aligning language models with human preference, we take the first step towards aligning retrieval-augmented language models to a status where it responds relying merely on the external evidence and disregards the interference of parametric knowledge. Specifically, we propose a reinforcement learning based algorithm Trustworthy-Alignment, theoretically and experimentally demonstrating large language models' capability of reaching a trustworthy status without explicit supervision on how to respond. Our work highlights the potential of large language models on exploring its intrinsic abilities by its own and expands the application scenarios of alignment from fulfilling human preference to creating trustworthy agents.

Paper Structure

This paper contains 38 sections, 4 theorems, 23 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Preliminary Study
  3. Retrieval-Augmented Generation
  4. Knowledge Conflicts
  5. Tendency Analysis
  6. Trustworthy Alignment
  7. LLM as an MDP
  8. Objective of Trustworthy-Alignment
  9. Theoretical Analysis
  10. Experiments
  11. Setup
  12. Datasets
  13. Metrics
  14. Baselines
  15. Analyses
  16. ...and 23 more sections

Key Result

Lemma 3.2

(Order Preserving) There exists a potential function $\Psi(\cdot)$ such that $\Psi(s_{t + 1}) - \Psi(s_t) = r_t' - r_t$. Given state $s \in \mathcal{S}$ and policy $\tilde{\pi}$, for any actions $a_1, a_2 \in \mathcal{A}$, when $\gamma = 1$, if $Q^{\tilde{\pi}}_{\mathcal{M}'}(s, a_1) > Q^{\tilde{\pi

Figures (4)

  • Figure 1: Average answer probabilities of the substituted answer $p^s$, original answer $p^o$ and none response $p^n$ for Llama-2-7b, Llama-2-7b-chat Touvron-2023-Llama and Vicuna-7b-v1.5 Zheng-2023-Judging with instruction-based and opinion-based prompting strategies. Generating tokens other than option codes is also feasible so that the sum of three probabilities is less than 1. Details of prompts are listed in \ref{['app:prompts']}.
  • Figure 2: Curves of training episodic reward, $p^s$, $p^o$ and $M_R$ on held-out evaluation dataset in the training procedure of Trustworthy-Alignment on Natural Questions. The solid curves represent the mean value and the shaded areas correspond to the standard deviations of 3 runs with different seeds. The curve of episodic reward is smoothed by averaging within a sliding window. For a clear presentation of task performance, the KL penalty is not included in the curve of episodic reward but visualized alone in \ref{['app:supp_results']}.
  • Figure 3: Win rate of Trustworthy-Alignment (Ours) and supervised fine-tuning (SFT) in human evaluation for alignment tax and impact on fluency. We present the mean win rate along with error bars that represent the standard deviations.
  • Figure 4: Curves of token mean KL, token mean return, episode length and $\tau$ in the training procedure of Trustworthy-Alignment on Natural Questions. The solid curves represent the mean value and the shaded areas correspond to the standard deviations of 3 runs with different seeds. The curve of token mean KL, token mean return and episode length are smoothed by averaging within a sliding window.

Theorems & Definitions (8)

  • Definition 3.1
  • Lemma 3.2
  • Theorem 3.3
  • Definition 1.1
  • Lemma 1.2
  • proof
  • Theorem 1.3
  • proof