Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Motion-Compensated Latent Semantic Canvases for Visual Situational Awareness on Edge

Igor Lodin, Sergii Filatov, Vira Filatova, Dmytro Filatov

TL;DR

This paper tackles persistent semantic memory for visual situational awareness on resource-constrained edge devices under camera motion. It introduces Motion-Compensated Latent Semantic Canvases (MCLSC), which combines baseline-anchored stabilization, a 2× canvas warp, and motion-gated asynchronous segmentation to maintain two latent canvases (static and dynamic) that store persistent and current semantic information, respectively. Compared with naive per-frame segmentation, MCLSC achieves substantial compute savings on 480p video while preserving coherent semantic overlays, by writing only when motion indicates new information and by stabilizing coordinates through a lightweight transform. The approach is model-agnostic and designed for edge deployment, though it inherits limitations from affine motion models and segmentation errors, motivating future work on deeper gates, confidence-weighted writes, and lightweight geometric enhancements for depth variation.

Abstract

We propose Motion-Compensated Latent Semantic Canvases (MCLSC) for visual situational awareness on resource-constrained edge devices. The core idea is to maintain persistent semantic metadata in two latent canvases - a slowly accumulating static layer and a rapidly updating dynamic layer - defined in a baseline coordinate frame stabilized from the video stream. Expensive panoptic segmentation (Mask2Former) runs asynchronously and is motion-gated: inference is triggered only when motion indicates new information, while stabilization/motion compensation preserves a consistent coordinate system for latent semantic memory. On prerecorded 480p clips, our prototype reduces segmentation calls by >30x and lowers mean end-to-end processing time by >20x compared to naive per-frame segmentation, while maintaining coherent static/dynamic semantic overlays.

Motion-Compensated Latent Semantic Canvases for Visual Situational Awareness on Edge

TL;DR

This paper tackles persistent semantic memory for visual situational awareness on resource-constrained edge devices under camera motion. It introduces Motion-Compensated Latent Semantic Canvases (MCLSC), which combines baseline-anchored stabilization, a 2× canvas warp, and motion-gated asynchronous segmentation to maintain two latent canvases (static and dynamic) that store persistent and current semantic information, respectively. Compared with naive per-frame segmentation, MCLSC achieves substantial compute savings on 480p video while preserving coherent semantic overlays, by writing only when motion indicates new information and by stabilizing coordinates through a lightweight transform. The approach is model-agnostic and designed for edge deployment, though it inherits limitations from affine motion models and segmentation errors, motivating future work on deeper gates, confidence-weighted writes, and lightweight geometric enhancements for depth variation.

Abstract

We propose Motion-Compensated Latent Semantic Canvases (MCLSC) for visual situational awareness on resource-constrained edge devices. The core idea is to maintain persistent semantic metadata in two latent canvases - a slowly accumulating static layer and a rapidly updating dynamic layer - defined in a baseline coordinate frame stabilized from the video stream. Expensive panoptic segmentation (Mask2Former) runs asynchronously and is motion-gated: inference is triggered only when motion indicates new information, while stabilization/motion compensation preserves a consistent coordinate system for latent semantic memory. On prerecorded 480p clips, our prototype reduces segmentation calls by >30x and lowers mean end-to-end processing time by >20x compared to naive per-frame segmentation, while maintaining coherent static/dynamic semantic overlays.
Paper Structure (42 sections, 6 equations, 5 figures, 1 table)

This paper contains 42 sections, 6 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Panoptic and universal segmentation.
  4. Semantic mapping under motion.
  5. Video stabilization and motion compensation.
  6. Adaptive inference and conditional computation.
  7. Problem Statement and Design Goals
  8. Problem.
  9. Design goals.
  10. Method
  11. Baseline-Anchored Stabilization
  12. Trust gating.
  13. 2$\times$ Canvas Warp
  14. Latent Static and Dynamic Semantic Canvases
  15. Motion-Gated Asynchronous Segmentation
  16. ...and 27 more sections

Figures (5)

  • Figure 1: System Architecture of the Motion-Compensated Latent Semantic Canvases pipeline.
  • Figure 2: Aggregate runtime comparison between NAIVE and GATED (from evaluation logs).
  • Figure 3: Segmentation submit events over time. NAIVE submits on (almost) every frame; GATED submits only when motion triggers it.
  • Figure 4: End-to-end processing time per frame. Spikes typically align with segmentation events.
  • Figure 5: Runtime visualization (6 tiles): original input, edge map, baseline, stabilized viewport, static canvas overlay, and dynamic canvas overlay.