Chain-of-Specificity: An Iteratively Refining Method for Eliciting Knowledge from Large Language Models

TL;DR

Chain-of-Specificity (CoS) tackles constrained instruction following in LLMs by iteratively emphasizing specific constraints and eliciting latent knowledge to refine outputs. The method is validated on CoScript, EXPLORE-INSTRUCT, and the newly introduced ConstrainSPEC dataset, with automatic and human evaluations showing superior specificity over baselines; it also demonstrates that distilling CoS-generated responses into smaller models improves their constraint adherence. The authors release ConstrainSPEC and related tools to the community, highlighting practical gains in knowledge-rich, constraint-driven tasks. Overall, CoS enhances the capability of LLMs to respect multi-faceted constraints and offers a viable path to transferring such capabilities to smaller models via distillation.

Abstract

Large Language Models (LLMs) exhibit remarkable generative capabilities, enabling the generation of valuable information. Despite these advancements, previous research found that LLMs sometimes struggle with adhering to specific constraints (e.g., in specific place or at specific time), at times even overlooking them, which leads to responses that are either too generic or not fully satisfactory. Existing approaches attempted to address this issue by decomposing or rewriting input instructions, yet they fall short in adequately emphasizing specific constraints and in unlocking the underlying knowledge (e.g., programming within the context of software development). In response, this paper proposes a simple yet effective method named Chain-of-Specificity (CoS). Specifically, CoS iteratively emphasizes the specific constraints in the input instructions, unlocks knowledge within LLMs, and refines responses. Experiments conducted on publicly available and self-build complex datasets demonstrate that CoS outperforms existing methods in enhancing generated content especially for the specificity. Besides, as the number of specific constraints increase, other baselines falter, while CoS still performs well. Moreover, we show that distilling responses generated by CoS effectively enhances the ability of smaller models to follow the constrained instructions. Resources of this paper will be released for further research.

Figures (6)

• Figure 1: The GPT-4 generation comparison between direct prompt method and Chain-of-Specificity (CoS). The direct prompt generate many generic responses, which could be broadly utilized in many other domain. In comparison, CoS generates more responses related to the specific constraint "software development team".
• Figure 2: The overview of the proposed Chain-of-Specificity (CoS).
• Figure 3: The initial words of the added specific constraints in ConstrainSPEC test set.
• Figure 4: The pairwise automatic evaluation results on ConstrainSPEC test set.
• Figure 5: The automatic evaluated general scores with different specific constraint numbers.