Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

References Indeed Matter? Reference-Free Preference Optimization for Conversational Query Reformulation

Doyoung Kim, Youngjun Lee, Joeun Kim, Jihwan Bang, Hwanjun Song, Susik Yoon, Jae-Gil Lee

TL;DR

Confronting the practicality gap in conversational query reformulation, this paper presents DualReform, a reference-free framework that generates pseudo reference passages through response-based inference and refines responses via CQR’s dual role. By iteratively refining pseudo references and optimizing the CQR model, DualReform achieves retrieval accuracy close to reference-based upper bounds while outperforming prior reference-free methods across general and specialized domains. The approach eliminates the need for ground-truth references, reducing annotation costs while maintaining strong retrieval and generation performance. This work advances practical CQR by enabling effective preference optimization directly on raw conversational data, with implications for retrieval-augmented generation in diverse real-world settings.

Abstract

Conversational query reformulation (CQR) has become indispensable for improving retrieval in dialogue-based applications. However, existing approaches typically rely on reference passages for optimization, which are impractical to acquire in real-world scenarios. To address this limitation, we introduce a novel reference-free preference optimization framework DualReform that generates pseudo reference passages from commonly-encountered conversational datasets containing only queries and responses. DualReform attains this goal through two key innovations: (1) response-based inference, where responses serve as proxies to infer pseudo reference passages, and (2) response refinement via the dual-role of CQR, where a CQR model refines responses based on the shared objectives between response refinement and CQR. Despite not relying on reference passages, DualReform achieves 96.9--99.1% of the retrieval accuracy attainable only with reference passages and surpasses the state-of-the-art method by up to 31.6%.

References Indeed Matter? Reference-Free Preference Optimization for Conversational Query Reformulation

TL;DR

Confronting the practicality gap in conversational query reformulation, this paper presents DualReform, a reference-free framework that generates pseudo reference passages through response-based inference and refines responses via CQR’s dual role. By iteratively refining pseudo references and optimizing the CQR model, DualReform achieves retrieval accuracy close to reference-based upper bounds while outperforming prior reference-free methods across general and specialized domains. The approach eliminates the need for ground-truth references, reducing annotation costs while maintaining strong retrieval and generation performance. This work advances practical CQR by enabling effective preference optimization directly on raw conversational data, with implications for retrieval-augmented generation in diverse real-world settings.

Abstract

Conversational query reformulation (CQR) has become indispensable for improving retrieval in dialogue-based applications. However, existing approaches typically rely on reference passages for optimization, which are impractical to acquire in real-world scenarios. To address this limitation, we introduce a novel reference-free preference optimization framework DualReform that generates pseudo reference passages from commonly-encountered conversational datasets containing only queries and responses. DualReform attains this goal through two key innovations: (1) response-based inference, where responses serve as proxies to infer pseudo reference passages, and (2) response refinement via the dual-role of CQR, where a CQR model refines responses based on the shared objectives between response refinement and CQR. Despite not relying on reference passages, DualReform achieves 96.9--99.1% of the retrieval accuracy attainable only with reference passages and surpasses the state-of-the-art method by up to 31.6%.
Paper Structure (42 sections, 4 theorems, 20 equations, 19 figures, 14 tables, 1 algorithm)

This paper contains 42 sections, 4 theorems, 20 equations, 19 figures, 14 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preference Optimization
  4. Conversational Query Reformulation
  5. Preliminaries
  6. Conversational Query Reformulation
  7. Reference-Based Preference Optimization for CQR
  8. DualReform: "Reference-Free" Preference Optimization
  9. Problem Statement
  10. Response Refinement by CQR's Dual-Role
  11. Overview of DualReform
  12. Step I: Pseudo Reference Generation
  13. Step II: Preference Optimization
  14. Evaluation
  15. Experiment Setting
  16. ...and 27 more sections

Key Result

Lemma 4.2

Suppose that Assumption assume:expansion holds. Then, the training error of a CQR model $\theta_{single}$ trained under the single-role configuration is bounded by the quality of pseudo reference passages generated by the LLM such that

Figures (19)

  • Figure 1: Overview of DualReform. (a) Preference optimization framework for CQR with reference passages as a key component for generating preference feedback over candidate queries. (b) Key idea of DualReform: Inferring pseudo reference passages through response refinement, addressing ambiguities and omissions in raw responses by incorporating conversational context.
  • Figure 2: Dual-role of the CQR model.
  • Figure 3: Empirical comparison of single- and dual-role variants. All variants employ Llama3.1-8b-inst as their backbones and use the prompt detailed in Prompt \ref{['prompt_temp']}. Llama+ICL variant additionally employs in-context learning.
  • Figure 4: Overall flow of DualReform.
  • Figure 5: Examples of refined responses generated by different methods on TopiOCQA. Fragments strongly aligned with the reference passage are highlighted in blue, while fragments with weaker connections (e.g., off-topic elements referring to previous conversation topics) are marked in red.
  • ...and 14 more figures

Theorems & Definitions (10)

  • Lemma 4.2: Single-Role Bound
  • Lemma 4.3: Dual-Role Bound
  • Theorem 4.4: Error Bound Difference
  • proof
  • Definition 4.5: Pseudo Reference Passages
  • Lemma A.3: Pseudo Label Denoising Bound wei2021theoretical
  • Definition B.1
  • Definition B.2
  • Definition B.3
  • Definition B.4