Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!

Subbarao Kambhampati, Kaya Stechly, Karthik Valmeekam, Lucas Saldyt, Siddhant Bhambri, Vardhan Palod, Atharva Gundawar, Soumya Rani Samineni, Durgesh Kalwar, Upasana Biswas

TL;DR

The paper critiques the common practice of treating intermediate tokens as human-like reasoning traces, arguing that this anthropomorphization is misleading and potentially harmful. It surveys test-time inference and post-training strategies that leverage derivational traces, but presents evidence that the traces often lack stable semantics and do not reliably reflect internal reasoning. It offers alternative viewpoints, such as viewing reasoning as the incremental internalization of verifier signals and promoting prompt augmentation (Skolem function) to improve performance without semantic traces. The work urges the community to pursue more robust explanations for LRMs' capabilities and to pursue research directions that do not rely on anthropomorphizing intermediate tokens.

Abstract

Intermediate token generation (ITG), where a model produces output before the solution, has been proposed as a method to improve the performance of language models on reasoning tasks. These intermediate tokens have been called "reasoning traces" or even "thoughts" -- implicitly anthropomorphizing the model, implying these tokens resemble steps a human might take when solving a challenging problem.In this paper, we present evidence that this anthropomorphization isn't a harmless metaphor, and instead is quite dangerous -- it confuses the nature of these models and how to use them effectively, and leads to questionable research.

Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!

TL;DR

The paper critiques the common practice of treating intermediate tokens as human-like reasoning traces, arguing that this anthropomorphization is misleading and potentially harmful. It surveys test-time inference and post-training strategies that leverage derivational traces, but presents evidence that the traces often lack stable semantics and do not reliably reflect internal reasoning. It offers alternative viewpoints, such as viewing reasoning as the incremental internalization of verifier signals and promoting prompt augmentation (Skolem function) to improve performance without semantic traces. The work urges the community to pursue more robust explanations for LRMs' capabilities and to pursue research directions that do not rely on anthropomorphizing intermediate tokens.

Abstract

Intermediate token generation (ITG), where a model produces output before the solution, has been proposed as a method to improve the performance of language models on reasoning tasks. These intermediate tokens have been called "reasoning traces" or even "thoughts" -- implicitly anthropomorphizing the model, implying these tokens resemble steps a human might take when solving a challenging problem.In this paper, we present evidence that this anthropomorphization isn't a harmless metaphor, and instead is quite dangerous -- it confuses the nature of these models and how to use them effectively, and leads to questionable research.

Paper Structure

This paper contains 15 sections, 1 equation, 4 figures.

Table of Contents

  1. Introduction
  2. Background: Test Time Inference & Post-Training in Reasoning Models
  3. Test-time Inference
  4. Post-Training on Derivational Traces
  5. Generating Candidate Derivational Traces:
  6. Filtering Traces:
  7. Improving LLMs Using Derivational Traces:
  8. Consequences of Anthropomorphizing Intermediate Tokens
  9. On the Amorphous Semantics of Intermediate Tokens
  10. Intermediate Token Production and Problem Adaptive Computation
  11. Understanding LRMs without Anthropomorphizing Intermediate Tokens
  12. Reasoning as Incremental Learning of Verifier Signal
  13. Embracing Reason-less Intermediate Tokens
  14. Summary
  15. Acknowledgements

Figures (4)

  • Figure 1: Test-time scaling approaches for teasing out reasoning
  • Figure 2: Post-training Approaches for teasing out reasoning
  • Figure 3: Understanding LRM Improvement as Incremental Compilation of Verifier Signal
  • Figure 4: Augmenting a task prompt with additional tokens often seems to improve the accuracy of LLM completion even if the tokens don't have human-parseable meaning.