Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Think, But Don't Overthink: Reproducing Recursive Language Models

Daren Wang

TL;DR

This project reproduces and extends the recently proposed ``Recursive Language Models''(RLMs) framework and investigates the impact of scaling the recursion depth.

Abstract

This project reproduces and extends the recently proposed ``Recursive Language Models'' (RLMs) framework by Zhang et al. (2026). This framework enables Large Language Models (LLMs) to process near-infinite contexts by offloading the prompt into an external REPL environment. While the original paper relies on a default recursion depth of 1 and suggests deeper recursion as a future direction, this study specifically investigates the impact of scaling the recursion depth. Using state-of-the-art open-source agentic models (DeepSeek v3.2 and Kimi K2), I evaluated pure LLM, RLM (depth=1), and RLM (depth=2) on the S-NIAH and OOLONG benchmarks. The findings reveal a compelling phenomenon: Deeper recursion causes models to ``overthink''. While depth-1 RLMs effectively boost accuracy on complex reasoning tasks, applying deeper recursion (depth=2) or using RLMs on simple retrieval tasks paradoxically degrades performance and exponentially inflates execution time (e.g., from 3.6s to 344.5s) and token costs. Code and data are available at: https://github.com/drbillwang/rlm-reproduction

Think, But Don't Overthink: Reproducing Recursive Language Models

TL;DR

This project reproduces and extends the recently proposed ``Recursive Language Models''(RLMs) framework and investigates the impact of scaling the recursion depth.

Abstract

This project reproduces and extends the recently proposed ``Recursive Language Models'' (RLMs) framework by Zhang et al. (2026). This framework enables Large Language Models (LLMs) to process near-infinite contexts by offloading the prompt into an external REPL environment. While the original paper relies on a default recursion depth of 1 and suggests deeper recursion as a future direction, this study specifically investigates the impact of scaling the recursion depth. Using state-of-the-art open-source agentic models (DeepSeek v3.2 and Kimi K2), I evaluated pure LLM, RLM (depth=1), and RLM (depth=2) on the S-NIAH and OOLONG benchmarks. The findings reveal a compelling phenomenon: Deeper recursion causes models to ``overthink''. While depth-1 RLMs effectively boost accuracy on complex reasoning tasks, applying deeper recursion (depth=2) or using RLMs on simple retrieval tasks paradoxically degrades performance and exponentially inflates execution time (e.g., from 3.6s to 344.5s) and token costs. Code and data are available at: https://github.com/drbillwang/rlm-reproduction
Paper Structure (19 sections, 5 figures)

This paper contains 19 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Setup Notes
  3. Environment and Core Libraries
  4. Data
  5. API Keys and Configuration
  6. Compute
  7. Reproduction Targets & Metric Definition
  8. Results and Analysis
  9. Paradoxical Degradation on Simple Retrieval ($O(1)$ Tasks)
  10. The "Overthinking" Effect on Complex Reasoning ($O(N)$ Tasks)
  11. Barriers to Industrial Deployment: Time, Tokens, and Cost
  12. Qualitative Analysis: How Deep Recursion Breaks Models
  13. Limitation: Single-Run Results
  14. Conclusions and Future Directions
  15. Debug Diary
  16. ...and 4 more sections

Figures (5)

  • Figure 1: Performance comparison of Base LLM, RLM (Depth=1), and RLM (Depth=2) against the original paper's benchmarks.
  • Figure 2: Average Execution Time (seconds) across different models and recursion depths.
  • Figure 3: Average Token Usage (thousands) across different models and recursion depths.
  • Figure 4: Average Token Cost (US$ cents) across different models and recursion depths.
  • Figure 5: Qualitative examples of RLM trajectory failures. Deeper recursion (Depth=2) often induces parametric hallucinations, role-playing confusion within the REPL, and performative over-explanation.