Cross-Modal Scene Semantic Alignment for Image Complexity Assessment

TL;DR

This work tackles ICA by exploiting cross-modal scene semantics to better reflect human perception. It introduces CM-SSA, a dual-branch framework that combines a complexity regression branch with a scene semantic alignment branch, both operating in a CLIP-based embedding space and guided by scene descriptions generated by InstructBLIP. The model optimizes a joint loss $L = \alpha L_A + \beta L_C$, where $L_C$ uses five complexity prompts and MOS ground truth, and $L_A$ enforces image-text alignment with scene semantics. Empirical results on IC9600, VISC-C, and Savoias show state-of-the-art performance and strong cross-dataset generalization, with ablations confirming the complementary value of the two branches and prompt strategies. This approach advances perceptually aligned ICA and demonstrates the practical potential of cross-modal, text-guided semantic cues for perceptual tasks.

Abstract

Image complexity assessment (ICA) is a challenging task in perceptual evaluation due to the subjective nature of human perception and the inherent semantic diversity in real-world images. Existing ICA methods predominantly rely on hand-crafted or shallow convolutional neural network-based features of a single visual modality, which are insufficient to fully capture the perceived representations closely related to image complexity. Recently, cross-modal scene semantic information has been shown to play a crucial role in various computer vision tasks, particularly those involving perceptual understanding. However, the exploration of cross-modal scene semantic information in the context of ICA remains unaddressed. Therefore, in this paper, we propose a novel ICA method called Cross-Modal Scene Semantic Alignment (CM-SSA), which leverages scene semantic alignment from a cross-modal perspective to enhance ICA performance, enabling complexity predictions to be more consistent with subjective human perception. Specifically, the proposed CM-SSA consists of a complexity regression branch and a scene semantic alignment branch. The complexity regression branch estimates image complexity levels under the guidance of the scene semantic alignment branch, while the scene semantic alignment branch is used to align images with corresponding text prompts that convey rich scene semantic information by pair-wise learning. Extensive experiments on several ICA datasets demonstrate that the proposed CM-SSA significantly outperforms state-of-the-art approaches. Codes are available at https://github.com/XQ2K/First-Cross-Model-ICA.

Figures (2)

• Figure 1: Examples of image complexity predictions with and without the guidance of our proposed cross-modal scene semantic alignment. Here, we adopt scene semantic information such as motion, culture, mood, spatial position, and quantity, which capture the relationships among the key visual entities (e.g., people, objects) within an image. The scatter plot visualizes the predictions of complexity scores on the IC9600 test set.
• Figure 2: The framework of the proposed Cross-Modal Scene Semantic Alignment (CM-SSA). The hyperparameters $\alpha$ and $\beta$ control the relative weights of the complexity regression branch and scene semantic alignment branch. The alignment loss $L_A$ encourages the image features to align with the scene semantic text features, using a ground truth of a vector of all ones (i.e., $g^{align}$) to ensure maximum alignment. The complexity loss $L_C$ predicts image complexity levels, where the ground truth $g^{complexity}$ corresponds to the MOS (Mean Opinion Score) of image complexity.