Evolving, Not Training: Zero-Shot Reasoning Segmentation via Evolutionary Prompting

TL;DR

Reasoning Segmentation demands complex text-conditioned pixel localization, a capability traditionally constrained by costly training or rigid inference. EVOL-SAM3 reframes this task as an inference-time evolutionary search that maintains a population of prompt hypotheses and iteratively refines them via a Generate-Evaluate-Evolve loop, guided by a Visual Arena and Semantic Mutation, with a final arbitration through a Heterogeneous Arena. The approach leverages a frozen multi-modal language model for semantic planning and SAM 3 for precise segmentation, avoiding parameter updates. Empirical results on ReasonSeg and RefCOCO-derived benchmarks show substantial zero-shot gains, often surpassing fully supervised state-of-the-art methods, demonstrating that dynamic search can unlock latent reasoning capabilities in frozen models. The work provides practical implications for building robust, training-free reasoning systems in vision-language tasks and releases code for reproducibility.

Abstract

Reasoning Segmentation requires models to interpret complex, context-dependent linguistic queries to achieve pixel-level localization. Current dominant approaches rely heavily on Supervised Fine-Tuning (SFT) or Reinforcement Learning (RL). However, SFT suffers from catastrophic forgetting and domain dependency, while RL is often hindered by training instability and rigid reliance on predefined reward functions. Although recent training-free methods circumvent these training burdens, they are fundamentally limited by a static inference paradigm. These methods typically rely on a single-pass "generate-then-segment" chain, which suffers from insufficient reasoning depth and lacks the capability to self-correct linguistic hallucinations or spatial misinterpretations. In this paper, we challenge these limitations and propose EVOL-SAM3, a novel zero-shot framework that reformulates reasoning segmentation as an inference-time evolutionary search process. Instead of relying on a fixed prompt, EVOL-SAM3 maintains a population of prompt hypotheses and iteratively refines them through a "Generate-Evaluate-Evolve" loop. We introduce a Visual Arena to assess prompt fitness via reference-free pairwise tournaments, and a Semantic Mutation operator to inject diversity and correct semantic errors. Furthermore, a Heterogeneous Arena module integrates geometric priors with semantic reasoning to ensure robust final selection. Extensive experiments demonstrate that EVOL-SAM3 not only substantially outperforms static baselines but also significantly surpasses fully supervised state-of-the-art methods on the challenging ReasonSeg benchmark in a zero-shot setting. The code is available at https://github.com/AHideoKuzeA/Evol-SAM3.

Figures (4)

• Figure 1: Core differance and performance comparison with existing methods
• Figure 2: The overall framework of EVOL-SAM3. The pipeline transforms static reasoning segmentation into a dynamic evolutionary search process, consisting of three phases: (1) Initialization: An MLLM acts as a Meta-Generator to expand the initial query into a diverse population of prompt hypotheses. (2) Evolutionary Reasoning Loop: This core phase iteratively refines the prompts. It employs a Visual Arena for pairwise competition to select elite individuals and uses Semantic Mutation to generate better prompt offspring, ensuring the semantic alignment of the generated masks. (3) Final Arbitration: A safeguard mechanism that leverages a double-blind swap strategy to arbitrate between the evolved text-based mask and a geometric intuition-based mask, ensuring robust segmentation results.
• Figure 3: Ablation study on the number of evolutionary generations ($T_{max}$).
• Figure 4: Qualitative comparison on the ReasonSeg benchmark.