Rethinking Thinking Tokens: Understanding Why They Underperform in Practice

TL;DR

A comprehensive empirical analysis is provided to validate the reliance on a single embedding for TTs, which results in inconsistent learning signals and introduces noisy gradients and the implications for future research on unsupervised reasoning in LLMs are discussed.

Abstract

Thinking Tokens (TT) have been proposed as an unsupervised method to facilitate reasoning in language models. However, despite their conceptual appeal, our findings show that TTs marginally improves performance and consistently underperforms compared to Chain-of-Thought (CoT) reasoning across multiple benchmarks. We hypothesize that this underperformance stems from the reliance on a single embedding for TTs, which results in inconsistent learning signals and introduces noisy gradients. This paper provides a comprehensive empirical analysis to validate this hypothesis and discusses the implications for future research on unsupervised reasoning in LLMs.

Figures (5)

• Figure 1: Chain of thought compared to thinking tokens. These approaches show striking similarity despite their differences.
• Figure 2: One TT embedding hardly moves from the initialized value.
• Figure 3: One TT embedding receives insufficient cumulative gradient.
• Figure 4: Two TT embeddings receive clear large cumulative gradients.
• Figure 5: Two TT embeddings show clear deviation from initialization.