Multi-Level Aware Preference Learning: Enhancing RLHF for Complex Multi-Instruction Tasks
Ruopei Sun, Jianfeng Cai, Jinhua Zhu, Kangwen Zhao, Dongyun Xue, Wengang Zhou, Li Li, Houqiang Li
TL;DR
MAPL addresses the gap in RLHF for complex multi-instruction tasks by exploiting latent prompt signals through intra-sample and inter-sample preference learning. It constructs two complementary data streams and losses, $\,\mathcal{L}_{intra}$ and $\mathcal{L}_{inter}$, and integrates them with traditional $\,\mathcal{L}_{BT}$ to form $\mathcal{L}_{MAPL}$, applicable to Reward Modeling and Direct Preference Optimization. Empirical results across Llama-3.1-Instruct, Qwen-2 variants, and MT-Bench show improved multi-instruction adherence while preserving semantic quality, outperforming M-LIFT and MuSC baselines. The approach avoids reliance on LLM-generated instructions and demonstrates robustness through ablations and case studies, with future work focusing on efficiency and wider instruction sets. Overall, MAPL enhances practical multi-instruction alignment by leveraging both within- and between-preference signals in a programmatic, scalable manner.
Abstract
RLHF has emerged as a predominant approach for aligning artificial intelligence systems with human preferences, demonstrating exceptional and measurable efficacy in instruction following tasks; however, it exhibits insufficient compliance capabilities when confronted with complex multi-instruction tasks. Conventional approaches rely heavily on human annotation or more sophisticated large language models, thereby introducing substantial resource expenditure or potential bias concerns. Meanwhile, alternative synthetic methods that augment standard preference datasets often compromise the model's semantic quality. Our research identifies a critical oversight in existing techniques, which predominantly focus on comparing responses while neglecting valuable latent signals embedded within prompt inputs, and which only focus on preference disparities at the intra-sample level, while neglecting to account for the inter-sample level preference differentials that exist among preference data. To leverage these previously neglected indicators, we propose a novel Multi-level Aware Preference Learning (MAPL) framework, capable of enhancing multi-instruction capabilities. Specifically, for any given response in original preference data pairs, we construct varied prompts with a preference relation under different conditions, in order to learn intra-sample level preference disparities. Furthermore, for any given original preference pair, we synthesize multi-instruction preference pairs to capture preference discrepancies at the inter-sample level. Building on the two datasets constructed above, we consequently devise two sophisticated training objective functions. Subsequently, our framework integrates seamlessly into both Reward Modeling and Direct Preference Optimization paradigms. Through rigorous evaluation across multiple benchmarks, we empirically validate the efficacy of our framework.