An Analysis of Data Transformation Effects on Segment Anything 2
Clayton Bromley, Alexander Moore, Amar Saini, Doug Poland, Carmen Carrano
TL;DR
This paper investigates how SAM 2 perceives and tracks objects in video under challenging transformations. It introduces five complex data transformations on DAVIS-derived videos and analyzes five observational positions from image embeddings to memory features. It uses a Regularized L2 Distance metric to quantify frame-to-memory changes and reveals how object representations emerge across stages. The work provides datasets, visualizations, and a decoding of the object pointer as a spatio-temporal object representation to guide robust VOS in cluttered real-world scenes.
Abstract
Video object segmentation (VOS) is a critical task in the development of video perception and understanding. The Segment-Anything Model 2 (SAM 2), released by Meta AI, is the current state-of-the-art architecture for end-to-end VOS. SAM 2 performs very well on both clean video data and augmented data, and completely intelligent video perception requires an understanding of how this architecture is capable of achieving such quality results. To better understand how each step within the SAM 2 architecture permits high-quality video segmentation, a variety of complex video transformations are passed through the architecture, and the impact at each stage of the process is measured. It is observed that each progressive stage enables the filtering of complex transformation noise and the emphasis of the object of interest. Contributions include the creation of complex transformation video datasets, an analysis of how each stage of the SAM 2 architecture interprets these transformations, and visualizations of segmented objects through each stage. By better understanding how each model structure impacts overall video understanding, VOS development can work to improve real-world applicability and performance tracking, localizing, and segmenting objects despite complex cluttered scenes and obscurations.