Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignment

Tiejin Chen, Xiaoou Liu, Vishnu Nandam, Kuan-Ru Liou, Hua Wei

TL;DR

Confronting noisy and inconsistent preference data in RLHF-based LLM alignment, this work introduces Conformal Feedback Alignment (CFA), which grounds answer-level reliability in Conformal Prediction to generate set-based confidence weights. CFA constructs conformal prediction sets with controllable coverage $1-\alpha$ and derives set-wise reliability weights that are applied to both PPO-style and DPO-style training, enabling uncertainty-aware alignment. Empirical results across multiple models and datasets show that CFA improves robustness and data efficiency, outperforming baselines that focus solely on preference-level uncertainty. The framework is compatible with white-box and black-box CP and highlights the importance of answer quality in alignment, with potential extensions to multimodal feedback and human-in-the-loop calibration.

Abstract

Preference-based alignment like Reinforcement Learning from Human Feedback (RLHF) learns from pairwise preferences, yet the labels are often noisy and inconsistent. Existing uncertainty-aware approaches weight preferences, but ignore a more fundamental factor: the reliability of the \emph{answers} being compared. To address the problem, we propose Conformal Feedback Alignment (CFA), a framework that grounds preference weighting in the statistical guarantees of Conformal Prediction (CP). CFA quantifies answer-level reliability by constructing conformal prediction sets with controllable coverage and aggregates these reliabilities into principled weights for both DPO- and PPO-style training. Experiments across different datasets show that CFA improves alignment robustness and data efficiency, highlighting that modeling \emph{answer-side} uncertainty complements preference-level weighting and yields more robust, data-efficient alignment. Codes are provided here.

Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignment

TL;DR

Confronting noisy and inconsistent preference data in RLHF-based LLM alignment, this work introduces Conformal Feedback Alignment (CFA), which grounds answer-level reliability in Conformal Prediction to generate set-based confidence weights. CFA constructs conformal prediction sets with controllable coverage and derives set-wise reliability weights that are applied to both PPO-style and DPO-style training, enabling uncertainty-aware alignment. Empirical results across multiple models and datasets show that CFA improves robustness and data efficiency, outperforming baselines that focus solely on preference-level uncertainty. The framework is compatible with white-box and black-box CP and highlights the importance of answer quality in alignment, with potential extensions to multimodal feedback and human-in-the-loop calibration.

Abstract

Preference-based alignment like Reinforcement Learning from Human Feedback (RLHF) learns from pairwise preferences, yet the labels are often noisy and inconsistent. Existing uncertainty-aware approaches weight preferences, but ignore a more fundamental factor: the reliability of the \emph{answers} being compared. To address the problem, we propose Conformal Feedback Alignment (CFA), a framework that grounds preference weighting in the statistical guarantees of Conformal Prediction (CP). CFA quantifies answer-level reliability by constructing conformal prediction sets with controllable coverage and aggregates these reliabilities into principled weights for both DPO- and PPO-style training. Experiments across different datasets show that CFA improves alignment robustness and data efficiency, highlighting that modeling \emph{answer-side} uncertainty complements preference-level weighting and yields more robust, data-efficient alignment. Codes are provided here.
Paper Structure (31 sections, 9 equations, 8 figures, 3 tables, 1 algorithm)

This paper contains 31 sections, 9 equations, 8 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. LLM Alignment
  4. Uncertainty Quantification
  5. Preliminaries
  6. Preference-Based Alignment
  7. Reward-Model-Based Methods (PPO-style).
  8. Reward-Free Methods (DPO-style).
  9. Conformal Prediction
  10. Method
  11. Overview
  12. Set-wise Uncertainty Estimation via CP
  13. Uncertainty-Aware Alignment
  14. Reward-Model-Based Optimization (PPO-Style)
  15. Reward-Free Optimization (DPO-Style)
  16. ...and 16 more sections

Figures (8)

  • Figure 1: The pipeline of our method. In our paper, we use conformal prediction, which can be applied for both black-box and white-box settings, to estimate the uncertainty and then use a weighted loss to conduct alignment.
  • Figure 2: Performance comparison on Llama2-7B and the Summarize dataset with different sizes of model parameters.
  • Figure 3: Performance comparison on Llama2-7B and the Summarize dataset with different sizes of training samples.
  • Figure 4: Performance comparison on Llama2-7B and the Summarize dataset using CFA and other methods.
  • Figure 5: Prompt for Evaluation in our Test Stage.
  • ...and 3 more figures