WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback

TL;DR

WildFeedback introduces a three-part framework for aligning LLMs with real user preferences by mining in-situ feedback from multi-turn conversations. It identifies SAT/DSAT feedback signals, constructs explicit preference data, and employs a checklist-guided evaluation to ensure alignment with genuine user expectations. Applied to WildChat, the framework yields 20,281 preference samples and demonstrates consistent improvements on standard benchmarks and a dedicated checklist-based evaluation. The work also discusses spurious preferences and selection bias as limitations, offering safety measures and data-balancing strategies to mitigate these issues.

Abstract

As large language models (LLMs) continue to advance, aligning these models with human preferences has emerged as a critical challenge. Traditional alignment methods, relying on human or LLM annotated datasets, are limited by their resource-intensive nature, inherent subjectivity, misalignment with real-world user preferences, and the risk of feedback loops that amplify model biases. To overcome these limitations, we introduce WildFeedback, a novel framework that leverages in-situ user feedback during conversations with LLMs to create preference datasets automatically. Given a corpus of multi-turn user-LLM conversation, WildFeedback identifies and classifies user feedback to LLM responses between conversation turns. The user feedback is then used to create examples of preferred and dispreferred responses according to users' preference. Our experiments demonstrate that LLMs fine-tuned on WildFeedback dataset exhibit significantly improved alignment with user preferences, as evidenced by both traditional benchmarks and our proposed checklist-guided evaluation. By incorporating in-situ feedback from actual users, WildFeedback addresses the scalability, subjectivity, and bias challenges that plague existing approaches, marking a significant step toward developing LLMs that are more responsive to the diverse and evolving needs of their users.

Figures (5)

• Figure 1: Overview of WildFeedback. (1) We begin by applying user satisfaction estimation to identify conversations and utterances that contain feedback signals. (2) We extract the entire conversation history leading up to a DSAT (dissatisfaction) signal as the prompt, and the response that triggers the DSAT as the dispreferred response. (3) Finally, we summarize the user’s preferences based on the identified feedback signals and guide the generation of the preferred response
• Figure 2: Comparison of in-situ user alignment across datasets generated by different models. "Win/Tie/Lose" represents the percentage of instances where the preferred responses win/tie/lose compared to the dispreferred responses in the WildFeedback dataset, prior to filtering. The comparison is made both with and without providing GPT-4 with summarized user preferences as checklists to guide its evaluation. With checklists, the preferred responses can be better distinguished.
• Figure 3: Preference evaluation on the WildFeedback test set, with or without the checklist. All numbers are the percentages of win/tie/lose. WF/UF On-policy/GPT-4 refers to the model trained on the on-policy/GPT-4 version of WildFeedback/UltraFeedback. Base models here refers to the off-the-shelf instruct models. Models trained on WildFeedback consistently outperformed all the baselines.
• Figure 4: The interface used for annotating SAT and DSAT signals.
• Figure 5: The interface used for annotating checklist-guided evaluation.