Autoregressive + Chain of Thought = Recurrent: Recurrence's Role in Language Models' Computability and a Revisit of Recurrent Transformer

TL;DR

This work thoroughly investigates the influence of recurrent structures in neural models on their reasoning abilities and computability, contrasting the role autoregression plays in the neural models' computational power, and sheds light on how the CoT approach can mimic recurrent computation and act as a bridge between autoregression and recurrence in the context of language models.

Abstract

The Transformer architecture excels in a variety of language modeling tasks, outperforming traditional neural architectures such as RNN and LSTM. This is partially due to its elimination of recurrent connections, which allows for parallel training and a smoother flow of gradients. However, this move away from recurrent structures places the Transformer model at the lower end of Chomsky's computational hierarchy, imposing limitations on its computational abilities. Consequently, even advanced Transformer-based models face considerable difficulties in tasks like counting, string reversal, and multiplication. These tasks, though seemingly elementary, require a level of computational complexity that exceeds the capabilities of the Transformer architecture. Concurrently, the emergence of ``Chain of Thought" (CoT) prompting has enabled Transformer-based language models to tackle tasks that were previously impossible or poorly executed. In this work, we thoroughly investigate the influence of recurrent structures in neural models on their reasoning abilities and computability, contrasting the role autoregression plays in the neural models' computational power. We then shed light on how the CoT approach can mimic recurrent computation and act as a bridge between autoregression and recurrence in the context of language models. It is this approximated recurrence that notably improves the model's performance and computational capacity. Moreover, we revisit recent recurrent-based Transformer model designs, focusing on their computational abilities through our proposed concept of ``recurrence-completeness" and identify key theoretical limitations in models like Linear Transformer and RWKV. Through this, we aim to provide insight into the neural model architectures and prompt better model design.

Figures (12)

• Figure 1: Computability hierarchy with each neural network architecture according to experimental results.
• Figure 2: A comparison between using state machine to recognize the language of question answering and language of all the even numbers. Natural language tends to have wide branches but shallow depth whereas logical reasoning can be very deep as input sequence can be arbitrarily long.
• Figure 3: Visualization of how computational information is passed along sequentially. Information between red colors is sequential (Between layers for transformers and across steps for RNN). Transformer without CoT can only pass the information through layers sequentially and therefore its depth is limited to layer numbers. RNN is recurrent over time therefore can pass the hidden information as many times as input length. CoT converts the hidden information from vectors into strings and then converts it back to vectors, therefore achieving approximate recurrence.
• Figure 4: Architecture diagrams of all discussed models.
• Figure 5: A comparison between RC and RI.