Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Meta-Cognitive Analysis: Evaluating Declarative and Procedural Knowledge in Datasets and Large Language Models

Zhuoqun Li, Hongyu Lin, Yaojie Lu, Hao Xiang, Xianpei Han, Le Sun

TL;DR

This paper provides ground-truth knowledge for LLMs and evaluates the effective score, finding that in most tasks, benefits from declarative knowledge are greater than those from procedural knowledge.

Abstract

Declarative knowledge and procedural knowledge are two key parts in meta-cognitive theory, and these two hold significant importance in pre-training and inference of LLMs. However, a comprehensive analysis comparing these two types of knowledge is lacking, primarily due to challenges in definition, probing and quantitative assessment. In this paper, we explore from a new perspective by providing ground-truth knowledge for LLMs and evaluating the effective score. Through extensive experiments with widely-used datasets and models, we get conclusions: (1) In most tasks, benefits from declarative knowledge are greater than those from procedural knowledge. (2) Profits of procedural knowledge are larger than declarative knowledge only in reasoning tasks with simple logic. (3) As pre-training progresses and size increases, model ability to utilize both kinds of knowledge significantly improves, but in different speed. We do detailed analysis for the findings and this can provide primary guidance for evaluation and enhancement of large language models.

Meta-Cognitive Analysis: Evaluating Declarative and Procedural Knowledge in Datasets and Large Language Models

TL;DR

This paper provides ground-truth knowledge for LLMs and evaluates the effective score, finding that in most tasks, benefits from declarative knowledge are greater than those from procedural knowledge.

Abstract

Declarative knowledge and procedural knowledge are two key parts in meta-cognitive theory, and these two hold significant importance in pre-training and inference of LLMs. However, a comprehensive analysis comparing these two types of knowledge is lacking, primarily due to challenges in definition, probing and quantitative assessment. In this paper, we explore from a new perspective by providing ground-truth knowledge for LLMs and evaluating the effective score. Through extensive experiments with widely-used datasets and models, we get conclusions: (1) In most tasks, benefits from declarative knowledge are greater than those from procedural knowledge. (2) Profits of procedural knowledge are larger than declarative knowledge only in reasoning tasks with simple logic. (3) As pre-training progresses and size increases, model ability to utilize both kinds of knowledge significantly improves, but in different speed. We do detailed analysis for the findings and this can provide primary guidance for evaluation and enhancement of large language models.
Paper Structure (11 sections, 1 equation, 3 figures, 1 table)

This paper contains 11 sections, 1 equation, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Meta-Cognitive Analysis Method
  4. Decomposition of Declarative and Procedural Knowledge
  5. Evaluation via In-context Hint Knowledge Injection
  6. Datasets and Models
  7. Findings
  8. Finding 1.
  9. Finding 2.
  10. Finding 3.
  11. Conclusion

Figures (3)

  • Figure 1: The illustration of overall processing. First decompose original reasoning of the question to procedural knowledge and declarative knowledge, then evaluate models by providing one or both type of knowledge after the question.
  • Figure 2: Procedural, declarative and combined score of different size models. The black line is the difference between procedural and declarative score. Above figure shows that the ability to utilize both knowledge becomes stronger as the model size increases, with different improvement rate.
  • Figure 3: Scores of different checkpoints. The 220 means Baichuan-2-7B-00220, the model after 220 steps pre-training. It shows that the ability to utilize both type of knowledge becomes stronger as the pre-training step increases.