Aligning to Thousands of Preferences via System Message Generalization

TL;DR

The Multifaceted Collection is created, a preference dataset with 192k combinations of values beyond generic helpfulness and harmlessness, spanning 65k user instructions, and a 7B LLM called Janus is trained and tested, demonstrating that training with a vast array of system messages could also enhance alignment to the general public's preference.

Abstract

Although humans inherently have diverse values, current large language model (LLM) alignment methods often assume that aligning LLMs with the general public's preferences is optimal. A major challenge in adopting a more individualized approach to LLM alignment is its lack of scalability, as it involves repeatedly acquiring preference data and training new reward models and LLMs for each individual's preferences. To address these challenges, we propose a new paradigm where users specify what they value most within the system message, steering the LLM's generation behavior to better align with the user's intentions. However, a naive application of such an approach is non-trivial since LLMs are typically trained on a uniform system message (e.g., "You are a helpful assistant") which limits their ability to generalize to diverse, unseen system messages. To improve this generalization, we create the Multifaceted Collection, a preference dataset with 192k combinations of values beyond generic helpfulness and harmlessness, spanning 65k user instructions. Using this dataset, we train a 7B LLM called Janus and test it on 921 prompts from 5 benchmarks (AlpacaEval 2.0, FLASK, Koala, MT-Bench, and Self-Instruct) by adding various unseen system messages that reflect user preferences. Janus achieves tie+win rate of 75.2%, 72.4%, and 66.4% against Mistral 7B Instruct v0.2, GPT-3.5 Turbo, and GPT-4, respectively. Unexpectedly, on three benchmarks focused on response helpfulness (AlpacaEval 2.0, MT-Bench, Arena Hard Auto v0.1), Janus also outperforms LLaMA 3 8B Instruct by a +4.0%, +0.1%, +3.0% margin, underscoring that training with a vast array of system messages could also enhance alignment to the general public's preference as well. Our code, dataset, benchmark, and models are available at https://github.com/kaistAI/Janus.

Figures (14)

• Figure 1: Previous LLMs are trained with homogeneous system messages reflecting general helpfulness and harmlessness. We propose training LLMs with diverse system messages, each representing an individual’s multifaceted preferences, to generalize to unseen system messages. The resulting model, Janus 7B, is adept at generating personalized responses for personalized system messages.
• Figure 2: Multifaceted Collection construction process. For each instruction, value descriptions are augmented from general to specific, allowing for multiple facets to branch out. We combine value from various dimensions into a system message to materialize preferences into model input. Following the system message and instruction, a proprietary LLM generates a gold response for training.
• Figure 3: Human comparison of Janus against Mistral Instruct 7B v0.2, GPT-3.5-Turbo-0125, and GPT-4-0613 on Multifaceted Bench.
• Figure 4: Length distribution of responses generated by LLMs and reference answer on Multifaceted Bench.
• Figure 5: Test-time system message ablation results on Multifaceted Bench.