Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMs

Daking Rai, Ziyu Yao

TL;DR

It is revealed that the activation of reasoning neurons in the feed-forward layers of an LLM can explain the importance of various components in a CoT prompt, and future research can extend it for a more complete understanding.

Abstract

Large language models (LLMs) have shown strong arithmetic reasoning capabilities when prompted with Chain-of-Thought (CoT) prompts. However, we have only a limited understanding of how they are processed by LLMs. To demystify it, prior work has primarily focused on ablating different components in the CoT prompt and empirically observing their resulting LLM performance change. Yet, the reason why these components are important to LLM reasoning is not explored. To fill this gap, in this work, we investigate ``neuron activation'' as a lens to provide a unified explanation to observations made by prior work. Specifically, we look into neurons within the feed-forward layers of LLMs that may have activated their arithmetic reasoning capabilities, using Llama2 as an example. To facilitate this investigation, we also propose an approach based on GPT-4 to automatically identify neurons that imply arithmetic reasoning. Our analyses revealed that the activation of reasoning neurons in the feed-forward layers of an LLM can explain the importance of various components in a CoT prompt, and future research can extend it for a more complete understanding.

An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMs

TL;DR

It is revealed that the activation of reasoning neurons in the feed-forward layers of an LLM can explain the importance of various components in a CoT prompt, and future research can extend it for a more complete understanding.

Abstract

Large language models (LLMs) have shown strong arithmetic reasoning capabilities when prompted with Chain-of-Thought (CoT) prompts. However, we have only a limited understanding of how they are processed by LLMs. To demystify it, prior work has primarily focused on ablating different components in the CoT prompt and empirically observing their resulting LLM performance change. Yet, the reason why these components are important to LLM reasoning is not explored. To fill this gap, in this work, we investigate ``neuron activation'' as a lens to provide a unified explanation to observations made by prior work. Specifically, we look into neurons within the feed-forward layers of LLMs that may have activated their arithmetic reasoning capabilities, using Llama2 as an example. To facilitate this investigation, we also propose an approach based on GPT-4 to automatically identify neurons that imply arithmetic reasoning. Our analyses revealed that the activation of reasoning neurons in the feed-forward layers of an LLM can explain the importance of various components in a CoT prompt, and future research can extend it for a more complete understanding.
Paper Structure (29 sections, 3 equations, 4 figures, 14 tables, 2 algorithms)

This paper contains 29 sections, 3 equations, 4 figures, 14 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Prior Work towards Understanding the CoT Reasoning of LLMs
  4. Interpreting Neurons of LLMs
  5. Concept Promotion via Neuron Activation of geva2022transformer
  6. Neuron Discovery using GPT-4
  7. Experimental Setup
  8. Understanding the Mechanism of Reasoning in LLMs
  9. RQ1: Are there neurons or sub-updates related to the concept of "reasoning"?
  10. RQ2: Are the discovered neurons important for eliciting the reasoning capability of LLMs?
  11. Correlation between the reasoning performance of LLMs and the activation of their reasoning neurons
  12. Understanding Prior Observations via the Lens of Neuron Activation
  13. RQ3: Why do equations matter?
  14. RQ4: Why do text explanations matter?
  15. RQ5: Why does arithmetic diversity in exemplars matter?
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Correlation between prompt accuracy and the LLM's average coefficient on the discovered reasoning neurons (blue stars) and its average count of reasoning tokens (orange diamonds).
  • Figure 2: Activation pattern of logical connector neurons for a randomly sampled example. The horizontal dotted line represents the average coefficient of randomly sampled neurons for the same set of examples.
  • Figure 3: Activation pattern of neurons related to arithmetic neurons (add, subtraction, multiplication, division) for a randomly sampled example. The horizontal dotted line represents the average coefficient of randomly sampled neurons for the same set of examples.
  • Figure 4: Overlap of neuron activation when the LLM is prompted with CoT and "Incorrect Labels" or "OOD Labels".