Focus on Your Question! Interpreting and Mitigating Toxic CoT Problems in Commonsense Reasoning

TL;DR

A novel method is designed, called RIDERS (Residual decodIng and sERial-position Swap), which compensates for the information deficit in the model from both decoding and serial-position perspectives and effectively improves the model's overall commonsense reasoning performance.

Abstract

Large language models exhibit high-level commonsense reasoning abilities, especially with enhancement methods like Chain-of-Thought (CoT). However, we find these CoT-like methods lead to a considerable number of originally correct answers turning wrong, which we define as the Toxic CoT problem. To interpret and mitigate this problem, we first utilize attribution tracing and causal tracing methods to probe the internal working mechanism of the LLM during CoT reasoning. Through comparisons, we prove that the model exhibits information loss from the question over the shallow attention layers when generating rationales or answers. Based on the probing findings, we design a novel method called RIDERS (Residual decodIng and sERial-position Swap), which compensates for the information deficit in the model from both decoding and serial-position perspectives. Through extensive experiments on multiple commonsense reasoning benchmarks, we validate that this method not only significantly eliminates Toxic CoT problems (decreased by 23.6%), but also effectively improves the model's overall commonsense reasoning performance (increased by 5.5%).

Figures (22)

• Figure 1: Two examples for the Toxic CoT problem.
• Figure 2: Attribution tracing results on Winogrande.
• Figure 3: Information flow divergence comparison on Winogrande.
• Figure 4: Attention tracing results across different attention heads on Llama2-13B.
• Figure 5: Intervention tracing results on Winogrande in correct and drifting answering cases.

Theorems & Definitions (3)

• Definition 2.1: Toxic CoT
• Definition 2.2: Rationale Drift
• Definition 2.3: Answer Drift