EvoFSM: Controllable Self-Evolution for Deep Research with Finite State Machines

TL;DR

EvoFSM addresses instability and inflexibility in unconstrained self-evolution by introducing a structured framework that models deep-research tasks as an explicit FSM. It decouples optimization into Flow (topology) and Skill (node-level prompts), and uses a critic to trigger constrained, atomic evolutions via two operator families, $ \mathcal{O}_{flow} $ and $ \mathcal{O}_{skill} $, with $ \mathcal{M}_{t+1} = \mathcal{M}_t \oplus \textit{op} $. A self-evolving memory stores successful trajectories as priors and failure patterns as constraints to enable continual learning across tasks. Empirical results on five multi-hop QA benchmarks and two interactive domains show EvoFSM achieving strong, generalizable improvements over standard baselines, including a notable 58.0% DeepSearch accuracy, demonstrating the practicality and robustness of structured, controllable self-evolution for open-ended research tasks.

Abstract

While LLM-based agents have shown promise for deep research, most existing approaches rely on fixed workflows that struggle to adapt to real-world, open-ended queries. Recent work therefore explores self-evolution by allowing agents to rewrite their own code or prompts to improve problem-solving ability, but unconstrained optimization often triggers instability, hallucinations, and instruction drift. We propose EvoFSM, a structured self-evolving framework that achieves both adaptability and control by evolving an explicit Finite State Machine (FSM) instead of relying on free-form rewriting. EvoFSM decouples the optimization space into macroscopic Flow (state-transition logic) and microscopic Skill (state-specific behaviors), enabling targeted improvements under clear behavioral boundaries. Guided by a critic mechanism, EvoFSM refines the FSM through a small set of constrained operations, and further incorporates a self-evolving memory that distills successful trajectories as reusable priors and failure patterns as constraints for future queries. Extensive evaluations on five multi-hop QA benchmarks demonstrate the effectiveness of EvoFSM. In particular, EvoFSM reaches 58.0% accuracy on the DeepSearch benchmark. Additional results on interactive decision-making tasks further validate its generalization.

Figures (7)

• Figure 1: Comparison of unconstrained self-evolution and our structured self-evolution.
• Figure 2: Overview of the EvoFSM framework. Our approach consists of three core components: (1) FSM Initialization, which formalizes the research process as a dynamic finite state machine initialized from prior experiences; (2) Structured Self-Evolution, which employs atomic operations to precisely optimize both the system's skill operators ($\mathcal{O}_{skill}$) and flow operators ($\mathcal{O}_{flow}$) based on critic feedback; and (3) Self-Evolving Memory Mechanism, which distills successful and failure trajectories into an experience pool to facilitate continuous learning and warm-starting for future tasks.
• Figure 3: Transferability study on the ALFWorld and WebShop benchmarks. We compare the success rate (a) and average reasoning steps (b) of each method.
• Figure 4: Effect of number of iterations on accuracy in the Bamboogle and DeepSearch benchmarks.
• Figure 5: An example of Flow Evolution. EvoFSM identifies a reasoning deadlock and structurally intervenes by injecting a new Verifier state, enabling the system to break out of a low-quality search loop.