On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Length

Abstract

Large language models (LLMs) have shown promise as interactive agents that solve tasks through extended sequences of environment interactions. While prior work has primarily focused on system-level optimizations or algorithmic improvements, the role of task horizon length in shaping training dynamics remains poorly understood. In this work, we present a systematic empirical study that examines horizon length through controlled task constructions. Specifically, we construct controlled tasks in which agents face identical decision rules and reasoning structures, but differ only in the length of action sequences required for successful completion. Our results reveal that increasing horizon length alone constitutes a training bottleneck, inducing severe training instability driven by exploration difficulties and credit assignment challenges. We demonstrate that horizon reduction is a key principle to address this limitation, stabilizing training and achieving better performance in long-horizon tasks. Moreover, we find that horizon reduction is related to stronger generalization across horizon lengths: models trained under reduced horizons generalize more effectively to longer-horizon variants at inference time, a phenomenon we refer to as horizon generalization.

Figures (19)

• Figure 1: A summary of our contributions. In this work, we study the training of long-horizon LLM agents from a horizon-centric perspective and identify horizon length as a fundamental bottleneck. We show that horizon reduction stabilizes RL and strengthens the tendency toward horizon generalization on longer tasks with similar reasoning difficulty.
• Figure 2: Training dynamics on different goal distance. While RL training is stable on short goal distance (L1--L2), it exhibits severe instability as the goal distance increases (L3--L4).
• Figure 3: Horizon reduction improves RL on long-horizon tasks. Training and test success rate on Sudoku and Rush Hour with atomic actions versus macro actions across different goal distance regimes. Across both environments, using macro actions for horizon reduction leads to more stable and effective RL, particularly in a long goal distance setting.
• Figure 4: RL stability depends on effective horizon. We compare two settings with macro-action policy: (A) reduced effective horizon via macro actions, and (B) an artificially restored long-horizon setting by restricting execution to single atomic actions.
• Figure 5: Results of macro action design. Flexible macro actions ($n \leq 5$ and macro ) show better performance than both atomic and fixed-length ($n=5$) designs across models on Sudoku. Additional ablation results for $n$ are provided in Figure \ref{['fig:macro_action_design_all']}.