Tiny-YOLOSAM: Fast Hybrid Image Segmentation

TL;DR

This work tackles the latency challenge of universal segmentation with SAM by proposing Tiny-YOLOSAM, a fast hybrid pipeline that leverages YOLOv12 for foreground box prompts and TinySAM for high-quality masks, supplemented by sparse point prompts in uncovered regions to achieve near full-scene coverage. The approach blends detector-guided prompting with targeted refinement, yielding substantial gains in class-agnostic coverage (AR 16.4% to 77.1%, mIoU 19.2% to 67.8%) and a dramatic reduction in end-to-end runtime (49.20s/image to 10.39s/image on an Apple M1 Pro). In detector-based metrics, the box-prompted hybrid achieves 40.7% AP (vs 46.7% for a stronger ViTDet baseline), highlighting a precision ceiling tied to detector recall, especially for small objects. Overall, Tiny-YOLOSAM demonstrates a practical, speed-coverage trade-off for full-scene segmentation, with future work focused on improving proposal recall, adaptive sampling, and category-aware post-processing to narrow the gap to category-specific baselines.

Abstract

The Segment Anything Model (SAM) enables promptable, high-quality segmentation but is often too computationally expensive for latency-critical settings. TinySAM is a lightweight, distilled SAM variant that preserves strong zero-shot mask quality, yet its "segment-everything" mode still requires hundreds of prompts and remains slow in practice. We first replicate TinySAM on COCO val2017 using official checkpoints, matching the reported AP within 0.03%, establishing a reliable experimental baseline. Building on this, we propose Tiny-YOLOSAM, a fast hybrid pipeline that uses a recent YOLO detector (YOLOv12) to generate box prompts for TinySAM on salient foreground objects, and supplements uncovered regions with sparse point prompts sampled only where YOLO-guided masks provide no coverage. On COCO val2017, the hybrid system substantially improves class-agnostic coverage (AR from 16.4% to 77.1%, mIoU from 19.2% to 67.8%) while reducing end-to-end runtime from 49.20s/image to 10.39s/image (4.7x) on an Apple M1 Pro CPU. These results suggest detector-guided prompting combined with targeted sparse sampling as an effective alternative to dense "segment-everything" prompting for practical full-scene segmentation.

Figures (6)

• Figure 1: Pipeline for the naive YOLO$\rightarrow$TinySAM extension. YOLOv12 first detects objects and outputs bounding boxes. Each bounding box is then used as a box prompt for TinySAM, which generates instance segmentation masks for detected objects.
• Figure 2: Pipeline for the hybrid YOLO + sparse points extension. YOLOv12 first detects salient foreground objects, which are segmented by TinySAM using box prompts. A coverage mask is then constructed to mark regions already segmented. Sparse points are sampled only in the remaining uncovered regions and passed to TinySAM as point prompts to recover background and small objects. All masks are finally merged using IoU-based post-processing to form full-scene segmentation.
• Figure 3: Visual comparison of segmentation quality across the three systems evaluated in our study. Left: Hierarchical TinySAM produces coarse and fragmented masks due to dense but low-confidence sampling. Middle: YOLO-only captures prominent foreground objects but fails to segment background regions or smaller structures. Right: Hybrid (YOLO + Sparse Points) recovers both foreground and background content, producing substantially more complete scene coverage. These examples highlight the main advantage of our extension: combining fast detector-guided prompts with sparse, targeted sampling yields both higher coverage and more coherent full-scene segmentation.
• Figure 4: Comparison of hierarchical segment-everything prompting and Tiny-YOLOSAM prompting.
• Figure 5: Performance comparison.