SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models

TL;DR

This work tackles the challenge of instruction-following in LLMs by identifying how independently sampled responses introduce distracting variations in preference learning. It introduces SPaR, a self-play framework where an actor and a refiner iteratively generate, judge, and refine responses using tree-search refinements to produce high-quality, comparison-focused data. Through three iterations on LLaMA3-8B-Instruct and scaling to larger models, SPaR achieves GPT-4-level performance on IFEval while preserving general abilities, with inference-time tree search further boosting results. The approach demonstrates strong scalability, robust refinement capability, and meaningful ablations showing the necessity of each component, offering a practical pathway to continuous, autonomous improvement in instruction-following without relying on external labels. The publicly released code and data further enable adoption and extension in the community.

Abstract

Instruction-following is a fundamental capability of language models, requiring the model to recognize even the most subtle requirements in the instructions and accurately reflect them in its output. Such an ability is well-suited for and often optimized by preference learning. However, existing methods often directly sample multiple independent responses from the model when creating preference pairs. Such practice can introduce content variations irrelevant to whether the instruction is precisely followed (e.g., different expressions about the same semantic), interfering with the goal of teaching models to recognize the key differences that lead to improved instruction following. In light of this, we introduce SPaR, a self-play framework integrating tree-search self-refinement to yield valid and comparable preference pairs free from distractions. By playing against itself, an LLM employs a tree-search strategy to refine its previous responses with respect to the instruction while minimizing unnecessary variations. Our experiments show that a LLaMA3-8B model, trained over three iterations guided by SPaR, surpasses GPT-4-Turbo on the IFEval benchmark without losing general capabilities. Furthermore, SPaR demonstrates promising scalability, greatly enhancing models like GLM-4-9B and LLaMA3-70B. We also identify how inference scaling in tree search would impact model performance. Our code and data are publicly available at https://github.com/thu-coai/SPaR.

Figures (11)

• Figure 1: An example of the interfering factors (story content) in independently sampled multiple responses (Left). Refined response pairs exclude these factors, highlight the key difference (ending sentence), and lead to improved performance on iteratively trained LLaMA3-8B-Instruct (Right).
• Figure 2: SPaR iterative training framework. At iteration $t$, the refiner $R_t$ first judges the generated responses from the actor $M_t$ to collect negative data. Next, a tree-search algorithm is employed to refine these imperfect responses. Finally, using the data from the above steps, we can optimize the actor and refiner for the next iteration, aiming for continuous self-improvement.
• Figure 3: Comparison with baseline methods across iterations (Cf. Figure \ref{['fig: mistral baseline']} for SPaR-7B). SPaR-8B consistently surpasses all baselines.
• Figure 4: Synthetic data experiment results: Character Sequence Generation (left) and Start/End Story Generation (right). For Character Sequence Generation, interfering pairs show rapid learning of the uppercase ratio (interfering factor) but perform worse than refinement pairs. In the Start/End Story Generation task, refinement pairs outperform interfering pairs, which even underperform the original model at step 0.
• Figure 5: Comparison of decoding strategies.