Chain-of-Thought Tokens are Computer Program Variables

TL;DR

This paper investigates whether chain-of-thought (CoT) tokens act as computer program variables in large language models. Through empirical studies on digit-wise multi-digit multiplication and grid-based dynamic programming, it shows CoT is necessary for these serial tasks, but performance is largely carried by tokens that store intermediate results, which can be represented in latent forms without harming ability up to a capacity limit. Intervening CoT values causally changes subsequent steps and the final answer, supporting the view that CoT tokens function as mutable variables. The findings reveal a computability/complexity limit between variable tokens and suggest design principles for more concise or latent CoTs with practical implications for instruction design and model analysis.

Abstract

Chain-of-thoughts (CoT) requires large language models (LLMs) to generate intermediate steps before reaching the final answer, and has been proven effective to help LLMs solve complex reasoning tasks. However, the inner mechanism of CoT still remains largely unclear. In this paper, we empirically study the role of CoT tokens in LLMs on two compositional tasks: multi-digit multiplication and dynamic programming. While CoT is essential for solving these problems, we find that preserving only tokens that store intermediate results would achieve comparable performance. Furthermore, we observe that storing intermediate results in an alternative latent form will not affect model performance. We also randomly intervene some values in CoT, and notice that subsequent CoT tokens and the final answer would change correspondingly. These findings suggest that CoT tokens may function like variables in computer programs but with potential drawbacks like unintended shortcuts and computational complexity limits between tokens. The code and data are available at https://github.com/solitaryzero/CoTs_are_Variables.

Figures (16)

• Figure 1: Comparison on model accuracy between plain prompting and chain-of-thought prompting.
• Figure 2: Model performance when non-result tokens are removed from CoT in multi-digit multiplication. Removing these tokens has little impact.
• Figure 3: The model structure used to reason with latent tokens. We use one-hot vectors as the latent embedding of latent tokens <LAT>. When the input token is a latent token, we use its projected latent embedding to replace the original input embedding. Correspondingly, a latent output head is added to predict the latent embedding of the next token from the last hidden state.
• Figure 4: Model performances when merging intermediate results into latent tokens.
• Figure 5: Examples of a successful intervention (left) and an intervention with a shortcut error (right). Blue numbers refer to relevant values in the original CoT, red numbers refer to the intervention, green numbers refer to values that change as expected, but purple numbers do not change due to a shortcut error.