Let Human Sketches Help: Empowering Challenging Image Segmentation Task with Freehand Sketches

TL;DR

The paper tackles challenging camouflaged object detection by introducing a sketch-guided interactive segmentation framework that replaces traditional prompts with freehand sketches. Built on a SAM backbone, the approach (DeepSketchCamo) uses a dedicated sketch encoder, a cross-attention fusion mechanism, and domain-specific adapters, augmented by Bezier-curve sketch augmentation, boundary refinement, and Adaptive Focal Loss to boost COD accuracy. It also introduces KOSCamo+, the first large freehand sketch dataset for COD, and shows that outputs can function as high-quality pseudo-labels for training other networks, dramatically reducing annotation time (up to ~120x). Overall, the method delivers improved segmentation performance over existing prompting modalities, enables efficient data annotation, and provides practical tools and datasets to advance sketch-assisted segmentation research.

Abstract

Sketches, with their expressive potential, allow humans to convey the essence of an object through even a rough contour. For the first time, we harness this expressive potential to improve segmentation performance in challenging tasks like camouflaged object detection (COD). Our approach introduces an innovative sketch-guided interactive segmentation framework, allowing users to intuitively annotate objects with freehand sketches (drawing a rough contour of the object) instead of the traditional bounding boxes or points used in classic interactive segmentation models like SAM. We demonstrate that sketch input can significantly improve performance in existing iterative segmentation methods, outperforming text or bounding box annotations. Additionally, we introduce key modifications to network architectures and a novel sketch augmentation technique to fully harness the power of sketch input and further boost segmentation accuracy. Remarkably, our model' s output can be directly used to train other neural networks, achieving results comparable to pixel-by-pixel annotations--while reducing annotation time by up to 120 times, which shows great potential in democratizing the annotation process and enabling model training with less reliance on resource-intensive, laborious pixel-level annotations. We also present KOSCamo+, the first freehand sketch dataset for camouflaged object detection. The dataset, code, and the labeling tool will be open sourced.

Figures (14)

• Figure 1: In this paper, we propose using human freehand sketches to improve image segmentation in challenging scenes—camouflaged object detection. (a) Comparison of SAM and our method with different inputs. While SAM struggles with point, box, and scribble inputs, sketch input (drawing a rough contour of the object) improves performance with extra annotation time. The blue area shows user response time, and green lines represent prompt types. (b) Practical applications of our method. Our method as an alternative to pixel-by-pixel annotation, producing comparable results, whereas SAM struggles with accuracy using point or box prompts.
• Figure 1: Comparison of freehand sketch and scribble.
• Figure 2: (a) Overall structure of DeepSketchCamo. Sketch augmentation, boundary refinement, and adaptive focal loss are introduced in this sketch-based task and are used in training to get elevated performance. The GT mask is used for supervision. (b) The application of our model. After the model has been trained, we input an image and a rough sketch into the model, the model can generate a predicted mask which can be directly used to train other deep-learning models.
• Figure 2: The difference between SAM and our network structure.
• Figure 3: Visualization for sketches augmentation. Sketches generated by different amplitudes of variation of the Bezier curve.