Preference-based Learning with Retrieval Augmented Generation for Conversational Question Answering
Magdalena Kaiser, Gerhard Weikum
TL;DR
The paper addresses ConvQA's need to interpret incomplete questions and retrieve faithful evidence without heavy intermediate supervision. It introduces PRAISE, a three-stage pipeline with Question Understanding, Evidence Retrieval and Filtering, and Answer Generation, each trained via Direct Preference Optimization using weak labels derived from final answers. The approach uses parameter-efficient adapters on a Llama-3-8B-Instruct backbone and demonstrates state-of-the-art results on the ConvMix benchmark, with a 15.5-point improvement in P@1 over strong baselines. It also provides insightful analyses of component contributions, domain robustness, and runtime/memory considerations, suggesting broad applicability of automatic preference-based training for multi-subtask NLP pipelines.
Abstract
Conversational Question Answering (ConvQA) involves multiple subtasks, i) to understand incomplete questions in their context, ii) to retrieve relevant information, and iii) to generate answers. This work presents PRAISE, a pipeline-based approach for ConvQA that trains LLM adapters for each of the three subtasks. As labeled training data for individual subtasks is unavailable in practice, PRAISE learns from its own generations using the final answering performance as feedback signal without human intervention and treats intermediate information, like relevant evidence, as weakly labeled data. We apply Direct Preference Optimization by contrasting successful and unsuccessful samples for each subtask. In our experiments, we show the effectiveness of this training paradigm: PRAISE shows improvements per subtask and achieves new state-of-the-art performance on a popular ConvQA benchmark, by gaining 15.5 percentage points increase in precision over baselines.