Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Abstraction-of-Thought Makes Language Models Better Reasoners

Ruixin Hong, Hongming Zhang, Xiaoman Pan, Dong Yu, Changshui Zhang

TL;DR

Experimental results indicate that models aligned to AoT reasoning format substantially outperform those aligned to CoT in many reasoning tasks.

Abstract

Abstract reasoning, the ability to reason from the abstract essence of a problem, serves as a key to generalization in human reasoning. However, eliciting language models to perform reasoning with abstraction remains unexplored. This paper seeks to bridge this gap by introducing a novel structured reasoning format called Abstraction-of-Thought (AoT). The uniqueness of AoT lies in its explicit requirement for varying levels of abstraction within the reasoning process. This approach could elicit language models to first contemplate on the abstract level before incorporating concrete details, which is overlooked by the prevailing step-by-step Chain-of-Thought (CoT) method. To align models with the AoT format, we present AoT Collection, a generic finetuning dataset consisting of 348k high-quality samples with AoT reasoning processes, collected via an automated and scalable pipeline. We finetune a wide range of language models with AoT Collection and conduct extensive evaluations on 23 unseen tasks from the challenging benchmark Big-Bench Hard. Experimental results indicate that models aligned to AoT reasoning format substantially outperform those aligned to CoT in many reasoning tasks.

Abstraction-of-Thought Makes Language Models Better Reasoners

TL;DR

Experimental results indicate that models aligned to AoT reasoning format substantially outperform those aligned to CoT in many reasoning tasks.

Abstract

Abstract reasoning, the ability to reason from the abstract essence of a problem, serves as a key to generalization in human reasoning. However, eliciting language models to perform reasoning with abstraction remains unexplored. This paper seeks to bridge this gap by introducing a novel structured reasoning format called Abstraction-of-Thought (AoT). The uniqueness of AoT lies in its explicit requirement for varying levels of abstraction within the reasoning process. This approach could elicit language models to first contemplate on the abstract level before incorporating concrete details, which is overlooked by the prevailing step-by-step Chain-of-Thought (CoT) method. To align models with the AoT format, we present AoT Collection, a generic finetuning dataset consisting of 348k high-quality samples with AoT reasoning processes, collected via an automated and scalable pipeline. We finetune a wide range of language models with AoT Collection and conduct extensive evaluations on 23 unseen tasks from the challenging benchmark Big-Bench Hard. Experimental results indicate that models aligned to AoT reasoning format substantially outperform those aligned to CoT in many reasoning tasks.
Paper Structure (32 sections, 3 figures, 10 tables)

This paper contains 32 sections, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Chain-of-Thought Prompting
  4. Training Language Models for Reasoning
  5. Reasoning with Abstraction
  6. Abstraction-of-Thought (AoT)
  7. The AoT Collection
  8. Overview
  9. Source Dataset Selection
  10. AoT Response Generation
  11. Data Validation and Filtering
  12. Experiments
  13. Zero-Shot Performance
  14. Few-Shot Performance
  15. Ablation Study on Reasoning Format
  16. ...and 17 more sections

Figures (3)

  • Figure 1: Reasoning with abstraction attempts to answer questions from the perspective of abstract essences, which may be overlooked by step-by-step Chain-of-Thought (CoT) reasoning. The reasoning process with abstraction contains multiple levels of abstract information. The lower levels (blue nodes) are responsible for performing concrete reasoning and are typically rich in detail. Conversely, the higher levels (red nodes) are abstractions of lower levels, organizing the entire reasoning process.
  • Figure 2: Illustration of Abstraction-of-Thought (AoT) format with natural language (upper half) and programming language (lower half). Unlike the unconstrained CoT, AoT explicitly requires that different levels of abstraction be presented in the reasoning process. Here are examples of two-level abstraction AoT. In AoT, the high-level parts (represented in bold red, i.e., $\bm{a_{*}^{1}}$) plan and organize the entire reasoning process from an abstract perspective, while low-level parts (i.e., $a_{*,*}^{2}$) carry out concrete and detailed reasoning steps. The high-level parts are abstractions of the low-level parts, clarifying their functionality and objectives. For clarity, we omit some code snippets in AoT with programming language.
  • Figure 3: Zero-shot BBH performance of models trained with different numbers of training samples.