CrowdSelect: Synthetic Instruction Data Selection with Multi-LLM Wisdom

TL;DR

This work addresses the data-culling bottleneck in instruction tuning by introducing CrowdSelect, a data-selection framework that leverages offline Multi-LLM Wisdom. It defines three core metrics—Difficulty, Separability, and Stability—computed from multiple LLM responses and reward scores, and integrates them with clustering to preserve diversity. Empirical results show that these foundation metrics, alone and in combination, yield strong gains across FFT and LoRA fine-tuning on MT-Bench and Arena-Hard, achieving new state-of-the-art performance across four base models. The approach demonstrates robustness to tuning method and reward-model variations, offering a scalable and effective pathway to improve instruction-following capabilities with significantly fewer high-quality examples. The work provides practical insights for future data-selection design and offers code access for reproducibility.

Abstract

Distilling advanced Large Language Models' instruction-following capabilities into smaller models using a selected subset has become a mainstream approach in model training. While existing synthetic instruction data selection strategies rely mainly on single-dimensional signals (i.e., reward scores, model perplexity), they fail to capture the complexity of instruction-following across diverse fields. Therefore, we investigate more diverse signals to capture comprehensive instruction-response pair characteristics and propose three foundational metrics that leverage Multi-LLM wisdom, informed by (1) diverse LLM responses and (2) reward model assessment. Building upon base metrics, we propose CrowdSelect, an integrated metric incorporating a clustering-based approach to maintain response diversity. Our comprehensive experiments demonstrate that our foundation metrics consistently improve performance across 4 base models on MT-bench and Arena-Hard. CrowdSelect, efficiently incorporating all metrics, achieves state-of-the-art performance in both Full and LoRA fine-tuning, showing improvements of 4.81% on Arena-Hard and 11.1% on MT-bench with Llama-3.2-3b-instruct. We hope our findings will bring valuable insights for future research in this direction. Code are available at https://github.com/listentm/crowdselect.

Figures (6)

• Figure 1: A demonstration of instruction tuning with selected synthetic instruction-response pairs.
• Figure 2: The overall pipeline of our CrowdSelect, which innovatively leverages metrics calculated from multiple facets of instructions using pre-collected synthesized responses from various LLMs and their corresponding reward model scores. We enhance data selection through clustering for diversity and metric combination to explore the method's potential. Finally, we evaluate the effectiveness of our selected instruction subset through FFT or LoRA fine-tuning hu2021lora for efficient instruction tuning.
• Figure 3: Overall results demonstrate that our foundation metrics and CrowdSelect consistently outperform baseline methods by a significant margin across FFT settings of four models, with particularly strong performance improvements on Llama-3b-instruct.
• Figure 4: Results show that small elite datasets behaves on par with a large dataset, corresponding to the experiment results in cao2023instruction. Our implementation (line in Red) achieves reasonably good results.
• Figure 5: Example of our selected instruction and correspond answer.