MUSIC: MUlti-Step Instruction Contrast for Multi-Turn Reward Models

TL;DR

This work tackles the challenge of evaluating multi-turn conversations by introducing MUSIC, an unsupervised data augmentation that synthesizes contrastive dialogue pairs whose quality differences span multiple turns via an instruction-contrast mechanism. By augmenting the Skywork preference dataset and training a Gemma-2-9B-Instruct-based multi-turn reward model, MUSIC-improved RM performance is demonstrated in multi-turn Best-of-N evaluation judged by a sophisticated LLM while preserving, and occasionally improving, single-turn benchmarks like RewardBench. Across datasets and settings, the approach shows that distributing the quality signal across turns yields more robust multi-turn evaluation signals without incurring human annotation costs. The findings highlight MUSIC as a scalable method to enhance multi-turn RM training and evaluation, with potential for longer histories and larger base models.

Abstract

Evaluating the quality of multi-turn conversations is crucial for developing capable Large Language Models (LLMs), yet remains a significant challenge, often requiring costly human evaluation. Multi-turn reward models (RMs) offer a scalable alternative and can provide valuable signals for guiding LLM training. While recent work has advanced multi-turn \textit{training} techniques, effective automated \textit{evaluation} specifically for multi-turn interactions lags behind. We observe that standard preference datasets, typically contrasting responses based only on the final conversational turn, provide insufficient signal to capture the nuances of multi-turn interactions. Instead, we find that incorporating contrasts spanning \textit{multiple} turns is critical for building robust multi-turn RMs. Motivated by this finding, we propose \textbf{MU}lti-\textbf{S}tep \textbf{I}nstruction \textbf{C}ontrast (MUSIC), an unsupervised data augmentation strategy that synthesizes contrastive conversation pairs exhibiting differences across multiple turns. Leveraging MUSIC on the Skywork preference dataset, we train a multi-turn RM based on the Gemma-2-9B-Instruct model. Empirical results demonstrate that our MUSIC-augmented RM outperforms baseline methods, achieving higher alignment with judgments from advanced proprietary LLM judges on multi-turn conversations, crucially, without compromising performance on standard single-turn RM benchmarks.

Figures (2)

• Figure 1: Overview of the MUSIC data augmentation procedure. Given seed contexts from existing datasets, we generate multi-turn rollouts where LLM simulators generate contrastive pairs, and use a contrastive instruction prompt to induce quality degradation in the rejected branch. The augmented preference pairs are used to train a multi-turn reward model along with the original dataset. Black arrows represent ephemeral changes that are provided to the assistant once, but not persisted. For each augmented pair, the chosen example consists of turns with blue borders, while the rejected example consists of turns with red borders.
• Figure 2: Winrates comparing conversations generated via Best-of-N ($N \in \{2, 4, 8\}$) guided by the MUSIC-Augmented RM versus the Baseline (non-augmented) RM, evaluated by Gemini 1.5 Pro on subsets of Anthropic HH and UltraInteract. Comparisons against greedy decoding are also shown.