AbsInstruct: Eliciting Abstraction Ability from LLMs through Explanation Tuning with Plausibility Estimation

TL;DR

AbsInstruct tackles eliciting abstraction ability in LLMs by pairing instruction tuning with detailed explanation traces and a plausibility estimator to curate data aligned with pre-trained model knowledge. It builds a hybrid dataset by combining abstraction-centric instructions from AbsPyramid with general-domain prompts (Alpaca) and fine-tunes models to perform abstraction detection across noun, verb, and event entailments. Empirical results show that AbsInstruct yields substantial gains over baselines and maintains general instruction-following capabilities, with evidence from in-domain and out-of-domain evaluations (AbstractATOMIC and Levy/Holt) and comprehensive ablations. Analyses highlight the critical roles of explanation traces, quality/diversity filters, and plausibility estimator in robust data curation and generalization, suggesting practical potential for broader reasoning tasks.

Abstract

Abstraction ability is crucial in human intelligence, which can also benefit various tasks in NLP study. Existing work shows that LLMs are deficient in abstract ability, and how to improve it remains unexplored. In this work, we design the framework AbsInstruct to enhance LLMs' abstraction ability through instruction tuning. The framework builds instructions with in-depth explanations to assist LLMs in capturing the underlying rationale of abstraction. Meanwhile, we introduce a plausibility estimator to select instructions that are more consistent with the abstraction knowledge of LLMs to be aligned. Then, our framework combines abstraction instructions with general-purpose ones to build a hybrid dataset. Extensive experiments and analyses demonstrate that our framework can considerably enhance LLMs' abstraction ability with strong generalization performance while maintaining their general instruction-following abilities.

Figures (8)

• Figure 1: An illustration of our AbsInstruct framework. We collect explanation traces for each example and design a plausibility estimator to select data that match the knowledge of an LLM to be aligned.
• Figure 2: The overview of our AbsInstruct framework. We sample examples from AbsPyramid and collect explanation traces by prompting an LLM. Then, we design a plausibility estimator to choose examples that are more consistent with the knowledge of a model to be aligned. The framework combines abstraction instructions with general-domain ones (e.g., Alpaca) and instruction-tunes the model.
• Figure 3: The 15 most common verbs (inner circle) and their top 3 direct nominal objects (outer circle) in head events of collected examples.
• Figure 4: Distribution of the ROUGE-L scores between collected examples. For each example, we compute the highest similarity with other examples we gathered.
• Figure 5: The out-of-domain performance on the abstractATOMIC dataset. We provide results across all metrics in \ref{['app:full_ood_results']}.