ESIQA: Perceptual Quality Assessment of Vision-Pro-based Egocentric Spatial Images

TL;DR

This work tackles perceptual quality assessment for egocentric spatial images in XR, identifying a gap in IQA resources for first-person stereo content. It introduces ESIQAD, a 500-image database with MOS under 2D, 3D-window, and 3D-immersive displays, and presents ESIQAnet, a mamba2-based architecture that fuses left/right views via cross attention and refines features with transposed attention within a multi-stage Visual State Space Duality (VSSD) backbone. ESIQAnet significantly outperforms 22 state-of-the-art NR IQA models across all three display modes and on image subsets, demonstrating strong generalization and robustness to egocentric content and immersive viewing conditions. The results highlight the importance of stereo fusion, channel-wise interactions, and deep semantic feature extraction for accurate QoE prediction in XR pipelines, enabling better quality control, adaptive streaming, and content optimization. The work provides a valuable resource for researchers and practitioners aiming to optimize perceptual QoE in egocentric VR/AR applications, while acknowledging device diversity and motion sickness considerations as avenues for future work.

Abstract

With the development of eXtended Reality (XR), photo capturing and display technology based on head-mounted displays (HMDs) have experienced significant advancements and gained considerable attention. Egocentric spatial images and videos are emerging as a compelling form of stereoscopic XR content. The assessment for the Quality of Experience (QoE) of XR content is important to ensure a high-quality viewing experience. Different from traditional 2D images, egocentric spatial images present challenges for perceptual quality assessment due to their special shooting, processing methods, and stereoscopic characteristics. However, the corresponding image quality assessment (IQA) research for egocentric spatial images is still lacking. In this paper, we establish the Egocentric Spatial Images Quality Assessment Database (ESIQAD), the first IQA database dedicated for egocentric spatial images as far as we know. Our ESIQAD includes 500 egocentric spatial images and the corresponding mean opinion scores (MOSs) under three display modes, including 2D display, 3D-window display, and 3D-immersive display. Based on our ESIQAD, we propose a novel mamba2-based multi-stage feature fusion model, termed ESIQAnet, which predicts the perceptual quality of egocentric spatial images under the three display modes. Specifically, we first extract features from multiple visual state space duality (VSSD) blocks, then apply cross attention to fuse binocular view information and use transposed attention to further refine the features. The multi-stage features are finally concatenated and fed into a quality regression network to predict the quality score. Extensive experimental results demonstrate that the ESIQAnet outperforms 22 state-of-the-art IQA models on the ESIQAD under all three display modes. The database and code are available at https://github.com/IntMeGroup/ESIQA.

Figures (11)

• Figure 1: Sample egocentric spatial images from our ESIQAD, where all samples are illustrated in their left view.
• Figure 2: Three display modes in our subjective experiments.
• Figure 3: MOS discriminability and mean CI evolution with participants’ number in ESIQAD.
• Figure 4: Distribution of MOSs of egocentric spatial images across three display modes.
• Figure 5: Distribution of MOS difference of spatial images across three modes. (a) $\text{MOS}_\text{3D-window}-\text{MOS}_\text{3D-immersive}$. (b) $\text{MOS}_\text{3D-immersive}-\text{MOS}_\text{2D}$. (c) $\text{MOS}_\text{3D-window}-\text{MOS}_\text{2D}$.