Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinement

TL;DR

This paper tackles the bottleneck of high-quality SFT data for large language models by introducing Condor, a two-stage pipeline that first generates knowledge-rich, diverse SFT data via a World Knowledge Tree-guided synthesis stage (Condor Void) and then refines it through Self-Reflection Refinement (Condor Refine). Using a single model, Condor produces $|oldsymbol{D_V}|\napprox 200{,}000$ initial QA pairs and $|oldsymbol{D_R}| approx 200{,}000$ refined pairs, with the base model achieving strong gains from as few as 20K Condor-generated samples and further improvements up to scales of $72B$ parameters through self-iteration. The work demonstrates substantial improvements on subjective human-preference benchmarks and competitive knowledge-based QA performance, while also exploring data scaling and self-iteration as pathways to even larger benefits. The findings suggest a promising direction for scalable, knowledge-driven data synthesis in SFT, though challenges such as multi-round iterations and potential hallucinations warrant further investigation.

Abstract

The quality of Supervised Fine-Tuning (SFT) data plays a critical role in enhancing the conversational capabilities of Large Language Models (LLMs). However, as LLMs become more advanced, the availability of high-quality human-annotated SFT data has become a significant bottleneck, necessitating a greater reliance on synthetic training data. In this work, we introduce Condor, a novel two-stage synthetic data generation framework that incorporates World Knowledge Tree and Self-Reflection Refinement to produce high-quality SFT data at scale. Our experimental results demonstrate that a base model fine-tuned on only 20K Condor-generated samples achieves superior performance compared to counterparts. The additional refinement stage in Condor further enables iterative self-improvement for LLMs at various scales (up to 72B), validating the effectiveness of our approach. Furthermore, our investigation into the scaling for synthetic data in post-training reveals substantial unexplored potential for performance improvements, opening promising avenues for future research.

Figures (7)

• Figure 1: Comparison between Official Models and Condor-Refined Models. The performances are obtained from CompassJudger-1-32B. Different families of popular LLMs benefit greatly from the Condor Refine engine.
• Figure 2: The Pipeline of Condor Engine. The pipeline is mainly divided into (1) Data synthesis: LLM is inspired from the World Knowledge Tree with different task diversity and difficulty to construct the original question-answer dataset $\boldsymbol{{D}_{V}}$; (2) Data refinement: LLM is utilized to reflect on the synthesized data and generate critiques for refinement and improve the quality of responses, producing the dataset $\boldsymbol{{D}_{R}}$.
• Figure 3: Average Score on Human-preference Benchmarks. CompassJudger-1-32B and GPT-4o provide a consistent trend in scoring.
• Figure 4: Performance Comparison under Different Proportions of Training Data. "Average" represents the average score across 8 benchmarks, and the results show a clear and consistent improvement as the size of the training data increases (Detailed scores in \ref{['tab:totalnumber']}).
• Figure 5: Performance Results under Different Proportion of WKT Tags and defined Tasks. Tasks have a more significant influence than tags, as evidenced by the significant drop with a small proportion of tasks.