MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time

TL;DR

Experimental results show that LLMs optimized on the meticulously constructed MetaAlign Dataset can effectively align with any preferences specified at the inference stage, validating the feasibility of MetaAlign.

Abstract

Large Language Models (LLMs) acquire extensive knowledge and remarkable abilities from extensive text corpora, making them powerful tools for various applications. To make LLMs more usable, aligning them with human preferences is essential. Existing alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), typically embed predefined preferences directly within the model's parameters. These methods, however, often result in a static alignment that can not account for the diversity of human preferences in practical applications. In response to this challenge, we propose an effective method, \textbf{MetaAlign}, which aims to help LLMs dynamically align with various explicit or implicit preferences specified at inference time. Experimental results show that LLMs optimized on our meticulously constructed MetaAlign Dataset can effectively align with any preferences specified at the inference stage, validating the feasibility of MetaAlign. We hope that our work can provide some insights into the alignment of language models.

Figures (6)

• Figure 1: Examples of the commonly used dialog template (top) and our three-tier dialog template (bottom). We introduce the "Meta-Prompt", which consists of System Info and User Info, to guide the model in aligning with human preferences during inference time.
• Figure 2: Compared to the previous alignment method (linked by pink arrows), our proposed MetaAlign Framework (linked by cyan arrows) build a MetaAligned LLM which could aligns with different preferences by simply modifying the meta-prompt, without the need to train separate models for each preference.
• Figure 3: When User Info and System Info conflict, we define the model's behavior using a Priority Matrix.
• Figure 4: The proportion of different sub-datasets in the MetaAlign Dataset.
• Figure 5: Visualization of the representations of harmful queries concatenated with nine different meta-prompts on V-Llama2-7B and MetaAlign-Llama2-7B*.