RepViT-SAM: Towards Real-Time Segmenting Anything
Ao Wang, Hui Chen, Zijia Lin, Jungong Han, Guiguang Ding
TL;DR
Segment Anything Model (SAM) achieves strong segmentation but is impractical on mobile due to its heavy image encoder. This work substitutes SAM's image encoder with RepViT-M2.3 and trains via direct distillation from ViT-H, with stride adjustments to match the decoder, yielding RepViT-SAM. Across zero-shot edge, instance, video, salient object, SegInW, and anomaly detection benchmarks, RepViT-SAM achieves superior transfer performance relative to MobileSAM and ViT-B-SAM while delivering substantially lower latency (nearly 10x faster on MacBook). This demonstrates practical, real-time segmentation on mobile devices and positions RepViT-SAM as a strong edge deployment baseline for the SAM framework.
Abstract
Segment Anything Model (SAM) has shown impressive zero-shot transfer performance for various computer vision tasks recently. However, its heavy computation costs remain daunting for practical applications. MobileSAM proposes to replace the heavyweight image encoder in SAM with TinyViT by employing distillation, which results in a significant reduction in computational requirements. However, its deployment on resource-constrained mobile devices still encounters challenges due to the substantial memory and computational overhead caused by self-attention mechanisms. Recently, RepViT achieves the state-of-the-art performance and latency trade-off on mobile devices by incorporating efficient architectural designs of ViTs into CNNs. Here, to achieve real-time segmenting anything on mobile devices, following MobileSAM, we replace the heavyweight image encoder in SAM with RepViT model, ending up with the RepViT-SAM model. Extensive experiments show that RepViT-SAM can enjoy significantly better zero-shot transfer capability than MobileSAM, along with nearly $10\times$ faster inference speed. The code and models are available at \url{https://github.com/THU-MIG/RepViT}.