GenCellAgent: Generalizable, Training-Free Cellular Image Segmentation via Large Language Model Agents

TL;DR

GenCellAgent tackles the challenge of robust cellular image segmentation across heterogeneous modalities by deploying a training-free, multi-agent framework that orchestrates specialist tools and generalist vision–language models through a planner–executor–evaluator loop with memory. It automatically routes images to the most appropriate tools, adapts on-the-fly with reference examples, and enables text-guided segmentation of novel organelles, all while incorporating human corrections into a growing memory that supports self-evolution and personalization. Across benchmarks, the approach yields substantial gains: a 15.7 percentage point improvement in mean accuracy over state-of-the-art baselines and up to 37.6% IoU gains on unseen ER and mitochondrial datasets, plus the ability to segment new targets like Golgi via iterative refinement. This training-free paradigm reduces annotation and retraining needs, offering a practical, adaptable path for robust bioimage analysis aligned with user preferences and workflows.

Abstract

Cellular image segmentation is essential for quantitative biology yet remains difficult due to heterogeneous modalities, morphological variability, and limited annotations. We present GenCellAgent, a training-free multi-agent framework that orchestrates specialist segmenters and generalist vision-language models via a planner-executor-evaluator loop (choose tool $\rightarrow$ run $\rightarrow$ quality-check) with long-term memory. The system (i) automatically routes images to the best tool, (ii) adapts on the fly using a few reference images when imaging conditions differ from what a tool expects, (iii) supports text-guided segmentation of organelles not covered by existing models, and (iv) commits expert edits to memory, enabling self-evolution and personalized workflows. Across four cell-segmentation benchmarks, this routing yields a 15.7\% mean accuracy gain over state-of-the-art baselines. On endoplasmic reticulum and mitochondria from new datasets, GenCellAgent improves average IoU by 37.6\% over specialist models. It also segments novel objects such as the Golgi apparatus via iterative text-guided refinement, with light human correction further boosting performance. Together, these capabilities provide a practical path to robust, adaptable cellular image segmentation without retraining, while reducing annotation burden and matching user preferences.

Figures (10)

• Figure 1: Overview of the GenCellAgent framework. The system integrates a Planning Agent, Execution Agent, and Evaluation Agent within a multi-agent interaction loop, supported by memory and human-in-the-loop (HITL) interaction. Planning Agent interprets user queries, retrieves prior knowledge, and designs workflows. Execution Agent runs specialist and generalist tools to produce candidate segmentations. Evaluation Agent scores results and provides feedback for refinement. All runs are stored in memory, enabling retrieval-augmented generalization, self-evolving improvement, and personalization in future tasks.
• Figure 2: Intelligent tool selection and performance evaluation. a, Test image samples from four public cell segmentation benchmarks, i.e., LIVECell (incucyte HD phase-contrast microscopy images), TissueNet (fluorescence images), PlantSeg (confocal laser scanning microscopy images), Lizard Dataset (H$\&$E stained histology images). b, Test vs. Sample image style similarity heatmap. c, Model performance comparison.
• Figure 3: Enhancing specialist tools with in-context learning for out-of-distribution segmentation. a, Workflow of the proposed enhancement pipeline. b, Qualitative results on unseen datasets. c, Quantitative comparison.
• Figure 4: Fully automatic segmentation of novel objects. a, Iterative refinement on Golgi segmentation shows progressive improvements. b, Average IoU across iterations demonstrates performance gains through refinement. c, Test-time scaling with multiple trials per iteration yields more stable improvements.
• Figure 5: Human correction, capability emergence, progressive reference learning, and personalized HITL recommendation in GenCellAgent. a, Human correction interface with four prompt types. b, Segmentation comparison with human correction and ground truth. c, Novel capability emergence across reference scenarios.d, Personalized user profiling and recommendation. e, Personalized user profiling and recommendation.