Real-Time Reasoning Agents in Evolving Environments

TL;DR

The paper tackles the challenge of making logical, timely decisions in continuously evolving environments by formalizing real-time reasoning and introducing Real-Time Reasoning Gym, which houses three varied games to stress-test dynamic decision making. It compares reactive and planning paradigms and proposes AgileThinker, a dual-thread architecture that runs a planning thread for extended reasoning and a reactive thread for fast action, coordinated through a token-based time proxy and a time-sharing protocol. Empirical results show that single-paradigm agents underperform as cognitive load or wall-clock constraints tighten, while AgileThinker delivers robust gains across all games and settings, with wall-time experiments confirming practical applicability. Overall, this work establishes a practical testbed and architectural blueprint for real-time capable language-model agents, laying groundwork for temporally constrained AI systems and real-time decision making in dynamic domains.

Abstract

Agents in the real world must make not only logical but also timely judgments. This requires continuous awareness of the dynamic environment: hazards emerge, opportunities arise, and other agents act, while the agent's reasoning is still unfolding. Despite advances in language model reasoning, existing approaches fail to account for this dynamic nature. We introduce real-time reasoning as a new problem formulation for agents in evolving environments and build Real-Time Reasoning Gym to demonstrate it. We study two paradigms for deploying language models in agents: (1) reactive agents, which employ language models with bounded reasoning computation for rapid responses, and (2) planning agents, which allow extended reasoning computation for complex problems. Our experiments show that even state-of-the-art models struggle with making logical and timely judgments in either paradigm. To address this limitation, we propose AgileThinker, which simultaneously engages both reasoning paradigms. AgileThinker consistently outperforms agents engaging only one reasoning paradigm as the task difficulty and time pressure rise, effectively balancing reasoning depth and response latency. Our work establishes real-time reasoning as a critical testbed for developing practical agents and provides a foundation for research in temporally constrained AI systems, highlighting a path toward real-time capable agents.

Figures (10)

• Figure 1: Upper: Three real-time games, Freeway, Snake, and Overcooked. Lower: Under cognitive load and time pressure, AgileThinker, engaging both reactive and planning reasoning, consistently outperforms agents that engage either of them. Scores are averaged across different games.
• Figure 2: Agent loops in OpenAI Gym openaigym and Real-Time Reasoning Gym. Constants T_E and DEFAULT_ACTION will be explained in the following 'time pressure' paragraph.
• Figure 3: Existing evaluation setups for LLM Agents often assume a static setting, where the environment halts while the agent completes reasoning with unlimited computation. In Real-Time Reasoning Gym, environments are dynamic, evolving regardless of agents' computation state. As illustrated in the Freeway setting, Planning Agent, which performs extended reasoning without interruption, fails to act timely while Reactive Agent, which performs reasoning strictly within environment update period, lacks foresight and collides. AgileThinker combines both timely reaction and long-term planning to navigate such environments effectively.
• Figure 4: Two parallel threads in AgileThinker
• Figure 5: Performance of reasoning agents in Real-Time Reasoning Gym under varying cognitive loads and time pressures. Upper: we fix time pressure at 8k tokens per step and vary cognitive load. Lower: we fix cognitive load at medium level and vary time pressure. Full data and significance test at App. § \ref{['sec:main_results']} and § \ref{['sec:significance_test']}.