Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance

Kai Xiong, Xiao Ding, Ting Liu, Bing Qin, Dongliang Xu, Qing Yang, Hongtao Liu, Yixin Cao

TL;DR

This work tailor an abstract reasoning dataset (AbsR) together with a meaningful learning paradigm to teach LLMs how to leverage generic facts for reasoning purposes, and shows that this approach not only boosts the general reasoning performance of LLMs but also makes considerable strides towards their capacity for abstract reasoning.

Abstract

Large language models (LLMs) have developed impressive performance and strong explainability across various reasoning scenarios, marking a significant stride towards mimicking human-like intelligence. Despite this, when tasked with several simple questions supported by a generic fact, LLMs often struggle to abstract and apply the generic fact to provide consistent and precise answers, revealing a deficiency in abstract reasoning abilities. This has sparked a vigorous debate about whether LLMs are genuinely reasoning or merely memorizing. In light of this, we design a preliminary study to quantify and delve into the abstract reasoning abilities of existing LLMs. Our findings reveal a substantial discrepancy between their general reasoning and abstract reasoning performances. To relieve this problem, we tailor an abstract reasoning dataset (AbsR) together with a meaningful learning paradigm to teach LLMs how to leverage generic facts for reasoning purposes. The results show that our approach not only boosts the general reasoning performance of LLMs but also makes considerable strides towards their capacity for abstract reasoning, moving beyond simple memorization or imitation to a more nuanced understanding and application of generic facts. The code is available at https://github.com/Waste-Wood/MeanLearn.

Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance

TL;DR

This work tailor an abstract reasoning dataset (AbsR) together with a meaningful learning paradigm to teach LLMs how to leverage generic facts for reasoning purposes, and shows that this approach not only boosts the general reasoning performance of LLMs but also makes considerable strides towards their capacity for abstract reasoning.

Abstract

Large language models (LLMs) have developed impressive performance and strong explainability across various reasoning scenarios, marking a significant stride towards mimicking human-like intelligence. Despite this, when tasked with several simple questions supported by a generic fact, LLMs often struggle to abstract and apply the generic fact to provide consistent and precise answers, revealing a deficiency in abstract reasoning abilities. This has sparked a vigorous debate about whether LLMs are genuinely reasoning or merely memorizing. In light of this, we design a preliminary study to quantify and delve into the abstract reasoning abilities of existing LLMs. Our findings reveal a substantial discrepancy between their general reasoning and abstract reasoning performances. To relieve this problem, we tailor an abstract reasoning dataset (AbsR) together with a meaningful learning paradigm to teach LLMs how to leverage generic facts for reasoning purposes. The results show that our approach not only boosts the general reasoning performance of LLMs but also makes considerable strides towards their capacity for abstract reasoning, moving beyond simple memorization or imitation to a more nuanced understanding and application of generic facts. The code is available at https://github.com/Waste-Wood/MeanLearn.
Paper Structure (47 sections, 3 equations, 4 figures, 8 tables)

This paper contains 47 sections, 3 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Abstract reasoning study
  3. Abstract reasoning metric
  4. Large language models for evaluation
  5. Experiment I: abstract reasoning
  6. Experiment II: generic fact probing
  7. AbsR: abstract reasoning dataset
  8. Generic fact collection
  9. Dataset construction
  10. The quality of AbsR
  11. Method: meaningful learning
  12. Experiments
  13. Experimental details
  14. Results: vanilla accuracy
  15. Results: abstract reasoning accuracy
  16. ...and 32 more sections

Figures (4)

  • Figure 1: The responses of (a) Humans and (b) LLMs when facing two questions which are supported by the same generic fact.
  • Figure 2: Computation of abstract reasoning metric.
  • Figure 3: Two samples of AbsR guided by the generic fact about "unusual conditions".
  • Figure 4: Visualization of reasoning capabilities on MMLU, which is categorized by task domians.