Memory-SAM: Human-Prompt-Free Tongue Segmentation via Retrieval-to-Prompt

TL;DR

Memory-SAM addresses the data-dependency and human-prompt limitations of tongue segmentation by turning a small memory of annotated cases into actionable prompts for SAM2 through a retrieval-to-prompt pipeline. It leverages dense DINOv3 features and FAISS retrieval to perform mask-constrained, patch-level matching and generates one foreground point plus dense background negatives as prompts, all without training or manual prompts. The approach achieves near-perfect segmentation on mixed real-world data (mIoU $=0.9863$, mPA $=0.9951$, Acc $=0.9996$) and shows robustness under challenging conditions, outperforming supervised models and detector-to-box SAM baselines, with ablations highlighting the importance of background negatives and memory-free design. This data-efficient, automation-friendly framework enables robust tongue segmentation in varied clinical imagery and suggests broader applicability to medical imaging tasks beyond tongue analysis.

Abstract

Accurate tongue segmentation is crucial for reliable TCM analysis. Supervised models require large annotated datasets, while SAM-family models remain prompt-driven. We present Memory-SAM, a training-free, human-prompt-free pipeline that automatically generates effective prompts from a small memory of prior cases via dense DINOv3 features and FAISS retrieval. Given a query image, mask-constrained correspondences to the retrieved exemplar are distilled into foreground/background point prompts that guide SAM2 without manual clicks or model fine-tuning. We evaluate on 600 expert-annotated images (300 controlled, 300 in-the-wild). On the mixed test split, Memory-SAM achieves mIoU 0.9863, surpassing FCN (0.8188) and a detector-to-box SAM baseline (0.1839). On controlled data, ceiling effects above 0.98 make small differences less meaningful given annotation variability, while our method shows clear gains under real-world conditions. Results indicate that retrieval-to-prompt enables data-efficient, robust segmentation of irregular boundaries in tongue imaging. The code is publicly available at https://github.com/jw-chae/memory-sam.

Figures (4)

• Figure 1: Conceptual comparison of different tongue segmentation paradigms. (a) Traditional supervised methods require extensive training data. (b) Detector-based SAM pipelines are sensitive to detection errors. (c) Our Memory-SAM replaces these components with a training-free, retrieval-based prompt generation module.
• Figure 2: The overall framework of Memory-SAM. Given a query image, we retrieve the most similar case from a memory bank. Dense correspondences between the query and the retrieved example are then distilled into a set of foreground/background point prompts, which are used to guide SAM2 for precise segmentation.
• Figure 3: Qualitative comparison in challenging in-the-wild scenarios. SAM-based results (Tongue-SAM, Memory-SAM) use a blue overlay; supervised baselines use a translucent white overlay.
• Figure 4: Qualitative comparison on the simpler HIT-Tongue-Image dataset. Color coding is consistent with Fig. \ref{['fig:qualitative_wild']}. Most models perform reasonably well in this controlled setting.