Language-Guided Visual Perception Disentanglement for Image Quality Assessment and Conditional Image Generation

TL;DR

The paper tackles the limitation of CLIP-style I&1T representations, which entangle semantic and perceptual cues and hinder perceptual tasks like IQA and CIG. It introduces the I&2T dataset of $(I, T_p, T_s)$ and a decoupled-text framework, DeCLIP, that uses two ResMLP projectors to create separate perceptual and semantic visual embeddings aligned with their respective texts via cross-modal losses. The approach yields two disentangled vision-language spaces and enables a DP-Adapter that provides fine-grained perceptual/semantic control for diffusion-based generation, while achieving strong zero-shot performance on image quality assessment and conditional image generation, including cross-dataset generalization. These results demonstrate robust, interpretable control over perceptual and semantic factors in multimodal tasks and open avenues for applying decoupled representations to other perception-related problems.

Abstract

Contrastive vision-language models, such as CLIP, have demonstrated excellent zero-shot capability across semantic recognition tasks, mainly attributed to the training on a large-scale I&1T (one Image with one Text) dataset. This kind of multimodal representations often blend semantic and perceptual elements, placing a particular emphasis on semantics. However, this could be problematic for popular tasks like image quality assessment (IQA) and conditional image generation (CIG), which typically need to have fine control on perceptual and semantic features. Motivated by the above facts, this paper presents a new multimodal disentangled representation learning framework, which leverages disentangled text to guide image disentanglement. To this end, we first build an I&2T (one Image with a perceptual Text and a semantic Text) dataset, which consists of disentangled perceptual and semantic text descriptions for an image. Then, the disentangled text descriptions are utilized as supervisory signals to disentangle pure perceptual representations from CLIP's original `coarse' feature space, dubbed DeCLIP. Finally, the decoupled feature representations are used for both image quality assessment (technical quality and aesthetic quality) and conditional image generation. Extensive experiments and comparisons have demonstrated the advantages of the proposed method on the two popular tasks. The dataset, code, and model will be available.

Figures (5)

• Figure 1: Conceptual difference between I&1T and I&2T. Upper: 'I&1T' features an image accompanied by one description that blends semantic and perceptual elements, with focus on semantics. Bottom: 'I&2T' features an image paired with two descriptions, offering rich perceptual insights while clearly differentiating between perception and semantics.
• Figure 2: The overall framework of the proposed method. First, the I&2T dataset, consisting of 112,769 image-text pairs formatted as $\left ( I, T_{p}, T_{s} \right )$, is established. Building upon this, decoupled text is utilized as supervision to disentangle pure semantic and perceptual features. Finally, the resulting decoupled representations are employed for image quality assessment and conditional image generation.
• Figure 3: The I&2T database construction pipeline and the prompt design. The perceptual abilities of humans, in conjunction with the semantic understanding and reasoning capabilities of MLLMs, collaboratively contribute to data generation.
• Figure 4: DP-Adapter vs. IP-Adapter. The former utilizes the decoupled vision-language representations obtained from DeCLIP to control Stable Diffusion, while the latter directly employs CLIP's vision-language representations for generative control.
• Figure 5: The visual comparison of the DP-Adapter with other methods conditioned on different perceptual and semantic prompts.