A Psychophysically Oriented Saliency Map Prediction Model

Abstract

Visual attention is one of the most significant characteristics for selecting and understanding the outside redundancy world. The human vision system cannot process all information simultaneously due to the visual information bottleneck. In order to reduce the redundant input of visual information, the human visual system mainly focuses on dominant parts of scenes. This is commonly known as visual saliency map prediction. This paper proposed a new psychophysical saliency prediction architecture, WECSF, inspired by multi-channel model of visual cortex functioning in humans. The model consists of opponent color channels, wavelet transform, wavelet energy map, and contrast sensitivity function for extracting low-level image features and providing a maximum approximation to the human visual system. The proposed model is evaluated using several datasets, including the MIT1003, MIT300, TORONTO, SID4VAM, and UCF Sports datasets. We also quantitatively and qualitatively compare the saliency prediction performance with that of other state-of-the-art models. Our model achieved strongly stable and better performance with different metrics on natural images, psychophysical synthetic images and dynamic videos. Additionally, we found that Fourier and spectral-inspired saliency prediction models outperformed other state-of-the-art non-neural network and even deep neural network models on psychophysical synthetic images. It can be explained and supported by the Fourier Vision Hypothesis. In the meantime, we suggest that deep neural networks need specific architectures and goals to be able to predict salient performance on psychophysical synthetic images better and more reliably. Finally, the proposed model could be used as a computational model of primate vision system and help us understand mechanism of primate vision system.

Figures (12)

• Figure 1: Architecture of the proposed saliency prediction model. The left panel image was selected from the MIT1003 dataset. The flow chart shows the framework of the proposed model, containing the chromatic response in the retina and spatial feature processing in the visual cortex. The natural image is first adapted, before decomposing it into white-black, red-green, and yellow-blue opponent neural channels. In the spatial component, a discrete wavelet transform is applied to each opponent color channel, then the wavelet energy map is measured. In the last step of the proposed model, the CSF is applied to each opponent wavelet energy channel and combined with each opponent's feature. $i$ and $\theta$ indicate image and model parameters, respectively. The details of each component are described in the following section. The graph on the right refers to the map of the left panel image's saliency on the inflated visual cortex using the proposed model.
• Figure 2: Opponent color processing. The first column represents the raw RGB color space, followed by the white--black (WB) channel, red--green (RG) channel, and yellow--blue (YB) channel, each with a gray colormap. The final three columns depict the WB, RG, and YB channels in artificial color, in order to better visualize the opponent color processing in the visual system.
• Figure 3: The modeling of V1 simple and complex cells in each opponent channel. The red rectangle indicates hypercolumns in the visual cortex. From left to right, the graph depicts WB opponent neurons with different orientations and scales. The following zoomed out top/bottom graphs with artificial color, for better visualization of features, in each hypercolumn indicate RG/YB opponent neurons across different orientations and scales. The V1 complex cells can be obtained from the sum of squares of wavelet transform features across scales and orientations in the simple cells.
• Figure 4: The different decomposition levels of the DWT (e.g., first, second, and third levels): "a" indicates the original image, "h" indicates the horizontal feature, "v" refers to the vertical feature, and "d" represents the diagonal feature. The bottom-left image is the original image, and the following images represent the first-, second-, and third-level decomposition features from the original image.
• Figure 5: Each channel's DWT map and the wavelet energy maps corresponding to it. The first column shows the DWT maps for achromatic (WB) and chromatic (RG, YB) channels. The second column is the wavelet energy map, obtained by summing across scales and orientation features for WB, RG, and YB opponent channels, respectively. The last column shows the sum of squares energy maps in each opponent channel.