Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SalFoM: Dynamic Saliency Prediction with Video Foundation Models

Morteza Moradi, Mohammad Moradi, Francesco Rundo, Concetto Spampinato, Ali Borji, Simone Palazzo

TL;DR

SalFoM proposes the first video saliency predictor powered by a pure video foundation model (UMT) encoder to better capture temporal dynamics in dynamic scenes. The method uses a multiperspective heterogeneous decoder to fuse long-range spatio-temporal, local spatio-temporal, and spatial cues into high-fidelity saliency maps. Trained with a KL-divergence and correlation-based objective, SalFoM achieves state-of-the-art results on the challenging DHF1K benchmark and competitive performance on Hollywood-2 and UCF-Sports, demonstrating stronger generalization. This work highlights the potential of video foundation models to advance video understanding tasks by integrating robust temporal representations with flexible multi-branch decoding.

Abstract

Recent advancements in video saliency prediction (VSP) have shown promising performance compared to the human visual system, whose emulation is the primary goal of VSP. However, current state-of-the-art models employ spatio-temporal transformers trained on limited amounts of data, hindering generalizability adaptation to downstream tasks. The benefits of vision foundation models present a potential solution to improve the VSP process. However, adapting image foundation models to the video domain presents significant challenges in modeling scene dynamics and capturing temporal information. To address these challenges, and as the first initiative to design a VSP model based on video foundation models, we introduce SalFoM, a novel encoder-decoder video transformer architecture. Our model employs UnMasked Teacher (UMT) as feature extractor and presents a heterogeneous decoder which features a locality-aware spatio-temporal transformer and integrates local and global spatio-temporal information from various perspectives to produce the final saliency map. Our qualitative and quantitative experiments on the challenging VSP benchmark datasets of DHF1K, Hollywood-2 and UCF-Sports demonstrate the superiority of our proposed model in comparison with the state-of-the-art methods.

SalFoM: Dynamic Saliency Prediction with Video Foundation Models

TL;DR

SalFoM proposes the first video saliency predictor powered by a pure video foundation model (UMT) encoder to better capture temporal dynamics in dynamic scenes. The method uses a multiperspective heterogeneous decoder to fuse long-range spatio-temporal, local spatio-temporal, and spatial cues into high-fidelity saliency maps. Trained with a KL-divergence and correlation-based objective, SalFoM achieves state-of-the-art results on the challenging DHF1K benchmark and competitive performance on Hollywood-2 and UCF-Sports, demonstrating stronger generalization. This work highlights the potential of video foundation models to advance video understanding tasks by integrating robust temporal representations with flexible multi-branch decoding.

Abstract

Recent advancements in video saliency prediction (VSP) have shown promising performance compared to the human visual system, whose emulation is the primary goal of VSP. However, current state-of-the-art models employ spatio-temporal transformers trained on limited amounts of data, hindering generalizability adaptation to downstream tasks. The benefits of vision foundation models present a potential solution to improve the VSP process. However, adapting image foundation models to the video domain presents significant challenges in modeling scene dynamics and capturing temporal information. To address these challenges, and as the first initiative to design a VSP model based on video foundation models, we introduce SalFoM, a novel encoder-decoder video transformer architecture. Our model employs UnMasked Teacher (UMT) as feature extractor and presents a heterogeneous decoder which features a locality-aware spatio-temporal transformer and integrates local and global spatio-temporal information from various perspectives to produce the final saliency map. Our qualitative and quantitative experiments on the challenging VSP benchmark datasets of DHF1K, Hollywood-2 and UCF-Sports demonstrate the superiority of our proposed model in comparison with the state-of-the-art methods.
Paper Structure (14 sections, 6 equations, 4 figures, 3 tables)

This paper contains 14 sections, 6 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video Saliency Prediction
  4. Video Foundation Models
  5. Methodology
  6. Video Foundation Model--based Feature Encoder
  7. Multiperspective Heterogeneous Decoder
  8. Training Objective
  9. Experiments
  10. Datasets
  11. Experimental Setup
  12. Result Analysis
  13. Ablation Study
  14. Conclusion

Figures (4)

  • Figure 1: Summary of the proposed dynamic saliency prediction framework. The model utilizes a video foundation model called UMT-L as its encoder, while the decoding phase comprises three different intermediate branches aimed at progressively refining and reconstructing essential features for producing the ultimate saliency map. In the end, all intermediate features are combined to generate the final saliency map.
  • Figure 2: Qualitative comparison of the performance of different VSP models on DHF1K.
  • Figure 3: Qualitative evaluation of the performance of SalFoM for predicting saliency of faces against ground truths, on DHF1K.
  • Figure 4: A sample of failure case on UCF-Sports dataset, due to the task-driven annotation methodology.