AgentPolyp: Accurate Polyp Segmentation via Image Enhancement Agent

TL;DR

AgentPolyp addresses degraded endoscopic polyp images by integrating CLIP-based semantic guidance with a dynamic, reinforcement-learning-driven image enhancement module, followed by lightweight segmentation. The CLIP-based perception produces semantic degradation descriptors that enable context-aware selection of denoising, contrast adjustment, and artifact reduction, with a feedback loop aligning enhancement quality to segmentation accuracy. The resulting end-to-end enhance-then-segment pipeline with plug-and-play modularity achieves real-time performance and strong gains across five public datasets, improving robustness on unseen data. This work advances deployable, robust polyp segmentation for endoscopic devices by improving edge fidelity and lesion delineation under challenging imaging conditions.

Abstract

Since human and environmental factors interfere, captured polyp images usually suffer from issues such as dim lighting, blur, and overexposure, which pose challenges for downstream polyp segmentation tasks. To address the challenges of noise-induced degradation in polyp images, we present AgentPolyp, a novel framework integrating CLIP-based semantic guidance and dynamic image enhancement with a lightweight neural network for segmentation. The agent first evaluates image quality using CLIP-driven semantic analysis (e.g., identifying ``low-contrast polyps with vascular textures") and adapts reinforcement learning strategies to dynamically apply multi-modal enhancement operations (e.g., denoising, contrast adjustment). A quality assessment feedback loop optimizes pixel-level enhancement and segmentation focus in a collaborative manner, ensuring robust preprocessing before neural network segmentation. This modular architecture supports plug-and-play extensions for various enhancement algorithms and segmentation networks, meeting deployment requirements for endoscopic devices.

Figures (4)

• Figure 1: This figure illustrates various degraded forms existing in polyp images. Such degradation is usually of a composite nature. We employ an intelligent agent to select an image enhancement algorithm, aiming to provide high-quality information for the downstream image segmentation network.
• Figure 2: This is our framework (AgentPolyp), which consists of three parts: perception, feedback, and execution. It is capable of providing different image enhancement strategies based on the quality description of the input images.
• Figure 3: Performance comparison of segmentation models, highlight the predicted area with a red overlay in Image.
• Figure 4: Visualization results of ablation experiment.