Modeling Saliency Dataset Bias

TL;DR

The paper addresses the challenge of cross-dataset generalization in image-based saliency prediction by identifying dataset biases that cause a substantial inter-dataset gap. It introduces a bias-aware, multiscale encoder-decoder model with fewer than 20 interpretable dataset-specific parameters to capture center bias, multiscale structure, and fixation spread, enabling data-efficient adaptation to new datasets. Empirically, the approach achieves state-of-the-art performance on MIT300, CAT2000, and COCO-Freeview across generalization, adaptation, and full training regimes, while also providing insights into how saliency varies across datasets. The work demonstrates that explicit bias modeling is crucial for robust, cross-dataset saliency representations and suggests directions toward aggregated, continual evaluation and richer, multi-dataset benchmarks.

Abstract

Recent advances in image-based saliency prediction are approaching gold standard performance levels on existing benchmarks. Despite this success, we show that predicting fixations across multiple saliency datasets remains challenging due to dataset bias. We find a significant performance drop (around 40%) when models trained on one dataset are applied to another. Surprisingly, increasing dataset diversity does not resolve this inter-dataset gap, with close to 60% attributed to dataset-specific biases. To address this remaining generalization gap, we propose a novel architecture extending a mostly dataset-agnostic encoder-decoder structure with fewer than 20 dataset-specific parameters that govern interpretable mechanisms such as multi-scale structure, center bias, and fixation spread. Adapting only these parameters to new data accounts for more than 75% of the generalization gap, with a large fraction of the improvement achieved with as few as 50 samples. Our model sets a new state-of-the-art on all three datasets of the MIT/Tuebingen Saliency Benchmark (MIT300, CAT2000, and COCO-Freeview), even when purely generalizing from unrelated datasets, but with a substantial boost when adapting to the respective training datasets. The model also provides valuable insights into spatial saliency properties, revealing complex multi-scale effects that combine both absolute and relative sizes.

Figures (18)

• Figure 1: Model Architecture: An input image is rescaled into different resolutions, some defined in total image size in pixels, others in pixels per degree of visual angle. For each image, deep activations from CLIP and DINOv2 encoders are extracted and averaged across scales, from which a priority map is decoded which is then postprocessed with Blur, priority scaling and centerbias. See Appendix Figure \ref{['fig:app:architecture']} for a larger version.
• Figure 2: Performing well on unseen datasets is hard due to dataset biases: we show model performances averaged across all five datasets under different training conditions. Generalizing from one dataset to another incurs a substantial performance penalty ("inter-dataset gap"), which largely cannot be fixed by simply training on more other datasets (remaining "generalization gap"). However, accounting for a few dataset biases parameters and adapting them can mostly close the generalization gap. Performances are mean dataset performances averaged across the five datasets, error bars for paired comparisons are according to cousineauConfidenceIntervalsWithinsubject2005 and moreyConfidenceIntervalsNormalized2008.
• Figure 3: Performing well on multiple datasets is hard without taking dataset biases into account: We compare the performance of our full jointly trained model with dataset biases to the performance of a naively trained model and see that the naive model performs worse.
• Figure 4: Low data apdatation: adaptation performance depending on the number of images used for adaptation. We outperform generalization with average dataset biases with as little as 5-10 images and reach close to full adaptation performance with around 50 images (vertical lines). Errorbars indicate variance over multiple runs with different random subsets.
• Figure 5: Contribution of different biases to closing the generalization gap. Performances are averages across the five datasets. Error bars cousineauConfidenceIntervalsWithinsubject2005moreyConfidenceIntervalsNormalized2008 are quite large because contributions differ across different datasets (Appendix, \ref{['fig:loo_bias_per_dataset']})