Transforming Static Images Using Generative Models for Video Salient Object Detection

TL;DR

This study shows that image-to-video diffusion models can generate realistic transformations of static images while understanding the contextual relationships between image components, which allows the model to generate plausible optical flows, preserving semantic integrity while reflecting the independent motion of scene elements.

Abstract

In many video processing tasks, leveraging large-scale image datasets is a common strategy, as image data is more abundant and facilitates comprehensive knowledge transfer. A typical approach for simulating video from static images involves applying spatial transformations, such as affine transformations and spline warping, to create sequences that mimic temporal progression. However, in tasks like video salient object detection, where both appearance and motion cues are critical, these basic image-to-video techniques fail to produce realistic optical flows that capture the independent motion properties of each object. In this study, we show that image-to-video diffusion models can generate realistic transformations of static images while understanding the contextual relationships between image components. This ability allows the model to generate plausible optical flows, preserving semantic integrity while reflecting the independent motion of scene elements. By augmenting individual images in this way, we create large-scale image-flow pairs that significantly enhance model training. Our approach achieves state-of-the-art performance across all public benchmark datasets, outperforming existing approaches.

Figures (8)

• Figure 1: Visualization of the optical flow generation process from a static image using two different target image generation methods.
• Figure 2: Overview of our training sample simulation process. Starting with a static source image, target images are generated using an image-to-video generation model. Optical flow maps are then estimated between the source and target images, with each map paired with the source image to enrich the training dataset.
• Figure 3: Visualization of our two-stream network architecture, which uses both RGB images and optical flow maps as input for primary object mask prediction.
• Figure 4: Qualitative comparison between state-of-the-art methods and the proposed RealFlow.
• Figure 5: Aligned qualitative comparison between real video data and our simulated data.