Exploring Image Representation with Decoupled Classical Visual Descriptors

TL;DR

The paper addresses the interpretability gap in deep image representations by introducing VisualSplit, which decouples images into classical descriptors (edges, colour segmentation, grey-level histograms) and learns to reconstruct images from these cues in a self-supervised, mask-free manner. It uses a ViT-based multi-modal encoder conditioned on the descriptors and a shallow decoder, with descriptor-consistency losses guiding learning. Empirically, VisualSplit delivers strong linear-probing and transfer performance, and enables high-quality, descriptor-guided image restoration and editing without retraining, demonstrating that classical cues can yield interpretable, controllable representations. This approach opens new avenues for decoupling-based learning and practical image manipulation across tasks.

Abstract

Exploring and understanding efficient image representations is a long-standing challenge in computer vision. While deep learning has achieved remarkable progress across image understanding tasks, its internal representations are often opaque, making it difficult to interpret how visual information is processed. In contrast, classical visual descriptors (e.g. edge, colour, and intensity distribution) have long been fundamental to image analysis and remain intuitively understandable to humans. Motivated by this gap, we ask a central question: Can modern learning benefit from these classical cues? In this paper, we answer it with VisualSplit, a framework that explicitly decomposes images into decoupled classical descriptors, treating each as an independent but complementary component of visual knowledge. Through a reconstruction-driven pre-training scheme, VisualSplit learns to capture the essence of each visual descriptor while preserving their interpretability. By explicitly decomposing visual attributes, our method inherently facilitates effective attribute control in various advanced visual tasks, including image generation and editing, extending beyond conventional classification and segmentation, suggesting the effectiveness of this new learning approach for visual understanding. Project page: https://chenyuanqu.com/VisualSplit/.

Figures (9)

• Figure 1: Qualitative illustration of the key idea.(a) Original image. (b) Visualisation of the threefold visual descriptors (grey-level histogram displayed in the top-right). (c) Image drawn by a human artist using (by looking at) only the combined descriptor set in (b). (d) Recovery from the proposed VisualSplit, only using the same combined descriptor set.
• Figure 2: VisualSplit framework. Initially, the RGB image is processed through the traditional Descriptor Extraction module to obtain segmented colour, edge, and intensity histograms. The colour and edge data will be patchtified and fed into the transformer, whereas the intensity information will serve as a condition for Cross-attention and to learn scale and shift parameters ($r, \beta$) for AdaLN-Zero. The loss is calculated on an image level as well as on the traditional descriptor level. It is worth mentioning that the acquired model can output both global and local representations, which can be applied to downstream visual tasks.
• Figure 3: Visualisation of the pre-training reconstruction task. Top: original images, bottom: our restoration results.
• Figure 4: Clustering visualisation with $K$ clusters. Each set shows the colour-clustered LAB image (top), alongside the isolated A-channel and B-channel (bottom left/right). Acc. indicates the classification accuracy of linear probing.
• Figure 5: Reconstruction with grey-level histograms at different brightness levels; edge and segmented colour map keep the same.