Divide-Verify-Refine: Can LLMs Self-Align with Complex Instructions?

TL;DR

Divide-Verify-Refine (DVR) addresses the challenge of following complex, multi-constraint instructions without retraining. It introduces a tool-assisted, three-stage approach that divides constraints, uses external tools for rigorous verification with textual guidance, and refines outputs via a memory-enabled dynamic few-shot repository. Empirical results on open-source LLMs show substantial improvements in constraint adherence and generalization across constraint types, supported by a new complex-instruction dataset. The framework offers a scalable path toward reliable instruction compliance in high-stakes deployments while preserving fluency and readability.

Abstract

Recent studies show LLMs struggle with complex instructions involving multiple constraints (e.g., length, format, sentiment). Existing works address this issue by fine-tuning, which heavily relies on fine-tuning data quality and is computational expensive. An alternative is leveraging LLMs' self-correction to refine responses for better constraint adherence. However, this is limited by the feedback quality, as LLMs cannot generate reliable feedback or detect errors. Moreover, its effectiveness relies on few-shot examples illustrating response modifications. As constraints in complex instructions are diverse, manually crafting such examples for each constraint type can be labor-intensive and sub-optimal. To address these two challenges, we propose the Divide-Verify-Refine (DVR) framework with three steps: (1) Divide complex instructions into single constraints and prepare appropriate tools; (2) Verify responses using tools that provide rigorous check and textual guidance (e.g., Python toolkit for format checks or pre-trained classifiers for content analysis); (3) Refine: To maximize refinement effectiveness, we propose dynamic few-shot prompting, where a refinement repository collects successful refinements, and these examples are selectively retrieved for future refinements. Recognizing the lack of complexity in existing datasets, we create a new dataset of complex instructions. DVR doubles Llama3.1-8B's constraint adherence and triples Mistral-7B's performance.

Figures (8)

• Figure 1: The LLMs hallucinate and struggle to give reliable feedback.
• Figure 2: The DVR framework: (a) Divide: The LLMs decompose constraints and instantiate tools for each constraint, (b) Verify: Tools will give feedback on the response, (c) Refine: The refinement repository provides past refinement process as few-shot examples. The current refinement process will be stored in the repository.
• Figure 3: Distribution of satisfied constraints number per instruction (level 6).
• Figure 4: Ablation study on Mistral-7B, Llama3.1-8B and Llama3-8B.
• Figure 5: Parameter study on ComplexInstruct.