Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Recursive Models for Long-Horizon Reasoning

Chenxiao Yang, Nathan Srebro, Zhiyuan Li

TL;DR

It is proved that any computable problem admits a recursive decomposition in which each subtask requires only exponentially smaller active context than standard autoregressive models; this strictly surpasses any context management approach confined to a single sequence, such as summarization.

Abstract

Modern language models reason within bounded context, an inherent constraint that poses a fundamental barrier to long-horizon reasoning. We identify recursion as a core principle for overcoming this barrier, and propose recursive models as a minimal realization, where the model can recursively invoke itself to solve subtasks in isolated contexts. We prove that any computable problem admits a recursive decomposition in which each subtask requires only exponentially smaller active context than standard autoregressive models; this strictly surpasses any context management approach confined to a single sequence, such as summarization. We further generalize our framework to modern agentic systems with arbitrary context processing and control flows, and prove that recursive models can achieve optimal power within this broader class. Experimentally, we train a 3B model to reason recursively and evaluate on Boolean satisfiability, a task requiring long-horizon combinatorial search, where it significantly outperforms frontier LLMs.

Recursive Models for Long-Horizon Reasoning

TL;DR

It is proved that any computable problem admits a recursive decomposition in which each subtask requires only exponentially smaller active context than standard autoregressive models; this strictly surpasses any context management approach confined to a single sequence, such as summarization.

Abstract

Modern language models reason within bounded context, an inherent constraint that poses a fundamental barrier to long-horizon reasoning. We identify recursion as a core principle for overcoming this barrier, and propose recursive models as a minimal realization, where the model can recursively invoke itself to solve subtasks in isolated contexts. We prove that any computable problem admits a recursive decomposition in which each subtask requires only exponentially smaller active context than standard autoregressive models; this strictly surpasses any context management approach confined to a single sequence, such as summarization. We further generalize our framework to modern agentic systems with arbitrary context processing and control flows, and prove that recursive models can achieve optimal power within this broader class. Experimentally, we train a 3B model to reason recursively and evaluate on Boolean satisfiability, a task requiring long-horizon combinatorial search, where it significantly outperforms frontier LLMs.
Paper Structure (110 sections, 10 theorems, 38 equations, 3 figures, 1 table, 11 algorithms)

This paper contains 110 sections, 10 theorems, 38 equations, 3 figures, 1 table, 11 algorithms.

Table of Contents

  1. Introduction
  2. Recursive Models
  3. Variants and Extensions
  4. Power of Deep Recursion
  5. Separation of Global and Local Spaces
  6. Power of Deep Recursion (Main Result)
  7. No Recursion and Shallow Recursion
  8. Generalization to Agentic Systems
  9. Formalizing Recursive Systems
  10. Optimality of Recursive Models
  11. Experiments
  12. Experimental Setup
  13. Results
  14. Discussion
  15. Inference Efficiency
  16. ...and 95 more sections

Key Result

Theorem 1

For any $S(n) \ge n$, recursive models can solve any problem in $\mathsf{TIME}(2^{\mathcal{O}(S(n))})$ under local space constraint $\mathcal{O}(S(n))$:

Figures (3)

  • Figure 1: Overview of different context management strategies.
  • Figure 2: Summarization
  • Figure 3: Trajectory length vs. active context length.

Theorems & Definitions (21)

  • Definition 1: Global and Local Space
  • Definition 2: Recursive Model Complexity Class
  • Theorem 1: Deep Recursive Models
  • Theorem 2: Standard Autoregressive Models / CoT
  • Theorem 3: Constant-Depth Recursive Models
  • Definition 3: Recursive Agentic System
  • Definition 4: $L$-bounded execution
  • Theorem 4: Upper bounds under $L$-bounded executions (unbounded recursion depth)
  • Theorem 5: Upper bounds under $L$-bounded executions (constant recursion depth)
  • Theorem 6: Deep Recursive Models, Formal
  • ...and 11 more