Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SAMannot: A Memory-Efficient, Local, Open-source Framework for Interactive Video Instance Segmentation based on SAM2

Gergely Dinya, András Gelencsér, Krisztina Kupán, Clemens Küpper, Kristóf Karacs, Anna Gelencsér-Horváth

Abstract

Current research workflows for precise video segmentation are often forced into a compromise between labor-intensive manual curation, costly commercial platforms, and/or privacy-compromising cloud-based services. The demand for high-fidelity video instance segmentation in research is often hindered by the bottleneck of manual annotation and the privacy concerns of cloud-based tools. We present SAMannot, an open-source, local framework that integrates the Segment Anything Model 2 (SAM2) into a human-in-the-loop workflow. To address the high resource requirements of foundation models, we modified the SAM2 dependency and implemented a processing layer that minimizes computational overhead and maximizes throughput, ensuring a highly responsive user interface. Key features include persistent instance identity management, an automated ``lock-and-refine'' workflow with barrier frames, and a mask-skeletonization-based auto-prompting mechanism. SAMannot facilitates the generation of research-ready datasets in YOLO and PNG formats alongside structured interaction logs. Verified through animal behavior tracking use-cases and subsets of the LVOS and DAVIS benchmark datasets, the tool provides a scalable, private, and cost-effective alternative to commercial platforms for complex video annotation tasks.

SAMannot: A Memory-Efficient, Local, Open-source Framework for Interactive Video Instance Segmentation based on SAM2

Abstract

Current research workflows for precise video segmentation are often forced into a compromise between labor-intensive manual curation, costly commercial platforms, and/or privacy-compromising cloud-based services. The demand for high-fidelity video instance segmentation in research is often hindered by the bottleneck of manual annotation and the privacy concerns of cloud-based tools. We present SAMannot, an open-source, local framework that integrates the Segment Anything Model 2 (SAM2) into a human-in-the-loop workflow. To address the high resource requirements of foundation models, we modified the SAM2 dependency and implemented a processing layer that minimizes computational overhead and maximizes throughput, ensuring a highly responsive user interface. Key features include persistent instance identity management, an automated ``lock-and-refine'' workflow with barrier frames, and a mask-skeletonization-based auto-prompting mechanism. SAMannot facilitates the generation of research-ready datasets in YOLO and PNG formats alongside structured interaction logs. Verified through animal behavior tracking use-cases and subsets of the LVOS and DAVIS benchmark datasets, the tool provides a scalable, private, and cost-effective alternative to commercial platforms for complex video annotation tasks.
Paper Structure (2 sections, 2 equations, 12 figures, 4 tables)

This paper contains 2 sections, 2 equations, 12 figures, 4 tables.

Table of Contents

  1. Pop-up windows
  2. Visualization of annotations

Figures (12)

  • Figure 1: This figure provides a high-level overview of the software architecture, organized by module functionality. Arrows denote invocation direction. Green boxes indicate core pipeline components responsible for a broad range of tasks. Blue boxes represent subsidiary subsystems with well-defined roles within the pipeline. Yellow boxes correspond to utility classes, including data-structure components and auxiliary widgets extending Tkinter functionality. The red box denotes the underlying tracking dependency (SAM2).
  • Figure 2: The graphical user interface is built up from the control panel (left), the canvas (right), and the slider (below the canvas). The control panel has the following modules: (A) loading and saving sessions (B) settings and information buttons (C) loading input media (D) label management (E) feature management (F) annotation propagation (G) toggle for visualization (H) frame slider.
  • Figure 3: Control flow for annotating a single block: the input video is processed in blocks to enable efficient memory usage. Within each block, the user provides prompts on selected frames. We apply SAM2 to extend prompts within frames to masks, and propagate the masks to the remaining frames in the block. Finally, the system allows for saving the annotation configuration, log and the annotations.
  • Figure 4: Qualitative examples of segmentation results on images from the DAVIS 2017 dataset. The columns display the original video frame (left), the ground truth (middle), and the masks predicted by SAMannot (right).
  • Figure 5: Illustrative frames from the analyzed DAVIS sequences for the performance metrics.
  • ...and 7 more figures