CodecLM: Aligning Language Models with Tailored Synthetic Data

TL;DR

CodecLM presents a general framework for tailoring synthetic instruction data to specific downstream distributions by encoding seed instructions into instruction metadata and decoding them into tailored prompts. Through Self-Rubrics and Contrastive Filtering, it adaptively refines and selects high-quality instruction–response pairs, enabling task-specific LLM alignment without human annotation. Empirical results across multiple open-domain benchmarks and backbones demonstrate state-of-the-art performance and data efficiency, with robust ablations illustrating the value of each component. The approach offers a flexible pathway for customized instruction-following capabilities, while acknowledging ethical considerations and limitations tied to strong LLM dependence and distribution mismatch.

Abstract

Instruction tuning has emerged as the key in aligning large language models (LLMs) with specific task instructions, thereby mitigating the discrepancy between the next-token prediction objective and users' actual goals. To reduce the labor and time cost to collect or annotate data by humans, researchers start to explore the use of LLMs to generate instruction-aligned synthetic data. Recent works focus on generating diverse instructions and applying LLM to increase instruction complexity, often neglecting downstream use cases. It remains unclear how to tailor high-quality data to elicit better instruction-following abilities in different target instruction distributions and LLMs. To this end, we introduce CodecLM, a general framework for adaptively generating high-quality synthetic data for LLM alignment with different downstream instruction distributions and LLMs. Drawing on the Encode-Decode principles, we use LLMs as codecs to guide the data generation process. We first encode seed instructions into metadata, which are concise keywords generated on-the-fly to capture the target instruction distribution, and then decode metadata to create tailored instructions. We also introduce Self-Rubrics and Contrastive Filtering during decoding to tailor data-efficient samples. Extensive experiments on four open-domain instruction following benchmarks validate the effectiveness of CodecLM over the current state-of-the-arts.

Figures (6)

• Figure 1: Overview of CodecLM. We first encode seed instructions into metadata to capture the underlying distribution of instructions. This metadata is then decoded through Self-Rubrics and Contrastive Filtering to tailor high-quality synthetic instructions that are aligned with the target instruction distribution. Intermediate instructions and responses are omitted in the figure for clarity.
• Figure 2: Overview of the proposed CodecLM. First, the strong LLM $f_s$ encodes the seed instruction into instruction metadata, specifying its use case and skills required for responses. Next, $f_s$ decodes metadata into basic instructions. Meanwhile, Self-Rubrics leverages $f_s$ to generate rubrics and actions to improve the basic instruction, tailoring them for the downstream task. Finally, Contrastive Filtering uses a scoring function $S$ to compares $f_s$ and $f_t$'s responses. The most effective pairs are selected for aligning the LLM, while less effective instructions are sent for further improvement. In this figure, the strong LLM's response is winning against the target one's, so we select the corresponding pair for instruction tuning the target LLM.
• Figure 3: Data proportion from each iteration and the corresponding CRR performance at each iteration.
• Figure 4: Metadata matching proportion vs. CRR.
• Figure 5: Scaling with model size and data quantity.