PIKA: Expert-Level Synthetic Datasets for Post-Training Alignment from Scratch

TL;DR

PiKa tackles data efficiency in open-source LLM alignment by synthesizing expert-level instruction data through persona-driven prompts, multi-path responses, and reward-model filtering. The PiKa-SFT dataset uses only $30{,}000$ examples yet achieves superior performance on AlpacaEval 2.0 and Arena-Hard—often surpassing models trained on orders of magnitude more data and even beating proprietary baselines—while extending gains to Qwen2.5 backbones. The approach demonstrates strong data efficiency and data quality, positioning PiKa as a scalable path for open-source post-training alignment and offering a competitive alternative to large proprietary datasets. Overall, PiKa shows that carefully constructed synthetic data can achieve high-quality instruction-following with substantially less data, encouraging broader participation in LLM alignment research.

Abstract

Reinforcement Learning from Human Feedback (RLHF) has become a cornerstone for aligning large language models (LLMs). However, its effectiveness depends on high-quality instruction data. Most existing alignment datasets are either private or require costly human annotation, which limits reproducibility and scalability. Even with Reinforcement Learning from AI Feedback (RLAIF), concerns about data quality remain. Moreover, it is unclear how much data is actually required to fine-tune a base model into a strong instruction-following model. Current approaches often rely on over 300k examples even at the supervised fine-tuning (SFT) stage, yet they still underperform compared to proprietary models, creating barriers for academic and resource-limited communities. To address this gap, we introduce PiKa, a data-efficient family of expert-level alignment datasets. In particular, the PiKa-SFT dataset uses only 30k SFT examples, far fewer than state-of-the-art datasets like Magpie. Through evaluations by fine-tuning Llama-3-8B-Base on PiKa and other public datasets, we show that PiKa-SFT outperforms models trained on much larger data. On AlpacaEval 2.0 and Arena-Hard benchmarks, PiKa-SFT fine-tuning even surpasses the official Llama-3-8B-Instruct model trained on over 10 million proprietary examples. We further extend our study by training the Qwen2.5 series (0.5B to 7B) on PiKa-SFT, achieving consistent gains. These findings demonstrate that high-quality alignment can be achieved with significantly less data, offering a scalable path for open-source LLM alignment. Code and data: https://github.com/SJY8460/PiKa.

Figures (6)

• Figure 1: Overview of the PiKa pipeline for synthesizing expert-level alignment data. Step 1: An expert-level persona covering multiple domains is given to an aligned LLM, which auto-regressively generates high-quality and knowledge-intensive instructions. Step 2: For each instruction, the LLM produces multiple candidate responses. Step 3: A reward model scores these responses. For SFT, we select the instruction and its highest-scoring response as a training pair. For DPO, we construct preference data by pairing the highest- and lowest-scoring responses with the same instruction.
• Figure 2: Comparison between PiKa and Magpie-Pro. (a) Minimum input embedding distances distribution. (b) Instruction lengths distribution. (c) Response lengths Distribution.
• Figure 3: GPT-4o–based evaluation of PiKa's instruction difficulty, feasibility and instruction-response quality compared with Magpie-Pro.
• Figure 4: Example instructions from Magpie-Pro and PiKa datasets. PiKa instructions are generally more complex and domain-specific, while Magpie-Pro tends to focus on simpler, open-ended prompts. We also include detailed instruction-response pairs in Appendix \ref{['example']}.
• Figure 5: Comparison across three dataset families: UltraFeedback, Magpie-Pro, and PiKa. (a) Dataset sizes. (b) AlpacaEval 2.0 win rates. (c) Arena-Hard win rates. .