PersoDPO: Scalable Preference Optimization for Instruction-Adherent, Persona-Grounded Dialogue via Multi-LLM Evaluation
Saleh Afzoon, MohammadHossein Ahmadi, Usman Naseem, Amin Beheshti
TL;DR
PersoDPO introduces a scalable, annotation-free framework that jointly optimizes coherence, persona alignment, and instruction adherence in persona-grounded dialogue by deriving preference signals from diverse open- and closed-source LLM outputs and training with a score-weighted DPO objective. The method combines metric-based signals (coherence and personalization) with a Length-Format Compliance signal to improve instructability, enabling automatic construction of high-quality preference pairs without manual labeling. Evaluations on the FoCus dataset show PersoDPO outperforms strong open-source baselines and a vanilla DPO variant across multiple metrics, while also reducing failure rates and improving response times. This approach provides a practical pathway to deploy more personalized, consistent, and instruction-adherent dialogue systems at scale.
Abstract
Personalization and contextual coherence are two essential components in building effective persona-grounded dialogue systems. These aspects play a crucial role in enhancing user engagement and ensuring responses are more relevant and consistent with user identity. However, recent studies indicate that open-source large language models (LLMs) continue to struggle to generate responses that are both contextually grounded and aligned with persona cues, despite exhibiting strong general conversational abilities like fluency and naturalness. We present PersoDPO, a scalable preference optimisation framework that uses supervision signals from automatic evaluations of responses generated by both closed-source and open-source LLMs to fine-tune dialogue models. The framework integrates evaluation metrics targeting coherence and personalization, along with a length-format compliance feature to promote instruction adherence. These signals are combined to automatically construct high-quality preference pairs without manual annotation, enabling a scalable and reproducible training pipeline. Experiments on the FoCus dataset show that an open-source language model fine-tuned with the PersoDPO framework consistently outperforms strong open-source baselines and a standard Direct Preference Optimization (DPO) variant across multiple evaluation dimensions.