"I Know It When I See It": Mood Spaces for Connecting and Expressing Visual Concepts

TL;DR

Mood Space introduces a context-sensitive, compact latent space learned from a Mood Board of exemplars to express hard-to-describe visual concepts. By mapping DINO tokens to a small base space $M$ and back to CLIP space via learned encoders/decoders, and by enforcing spectral graph and manifold regularization driven by top eigenvectors, the method preserves relevant affinities while discarding irrelevant variations. This enables path-based operations such as object averaging, visual analogy, and pose transfer through simple vector arithmetic, with final images rendered by a diffusion model conditioned on Mood Space tokens. The approach achieves smooth concept interpolation and consistent composition with few exemplars, and demonstrates robust, interpretable manipulation suitable for intuitive visual editing and design tools.

Abstract

Expressing complex concepts is easy when they can be labeled or quantified, but many ideas are hard to define yet instantly recognizable. We propose a Mood Board, where users convey abstract concepts with examples that hint at the intended direction of attribute changes. We compute an underlying Mood Space that 1) factors out irrelevant features and 2) finds the connections between images, thus bringing relevant concepts closer. We invent a fibration computation to compress/decompress pre-trained features into/from a compact space, 50-100x smaller. The main innovation is learning to mimic the pairwise affinity relationship of the image tokens across exemplars. To focus on the coarse-to-fine hierarchical structures in the Mood Space, we compute the top eigenvector structure from the affinity matrix and define a loss in the eigenvector space. The resulting Mood Space is locally linear and compact, allowing image-level operations, such as object averaging, visual analogy, and pose transfer, to be performed as a simple vector operation in Mood Space. Our learning is efficient in computation without any fine-tuning, needs only a few (2-20) exemplars, and takes less than a minute to learn.

Figures (27)

• Figure 1: Imagine a fictional animal that is a crossbreed of a sheepdog and goldfish. No one has seen this animal, but we can recognize it 'when I see it'. We express such concepts via a Mood Board by curating exemplars that hint at our interest. From a pre-trained feature (i.e., DINO), a latent Mood Space that is 50-100x smaller learns to squeeze out irrelevant features, find connections between the curated reference images, and decode back to any pre-trained feature space (i.e., CLIP). The Mood Space is almost locally linear, thus supporting semantic image operations via simple vector algebra. We demonstrate two Mood Space operations "connect" (:) and "lift" (::). The "connect" (:) operation makes a straight line in the Mood Space connecting two examples and decodes to a nonlinear curve in the image space. It supports the image operation of object averaging. The "lift" (::) operation shifts the Mood Space curve by seeding it at a different reference sample. It supports visual analogy and pose transfer. Multiple samples along the Mood Space curve provides diverse outputs tailoring to our individual 'I know it when I see it' preferences.
• Figure 2: We propose a context-specific DINO-to-CLIP mapping via a compressed Mood Space. First, we compute per-patch image token embedding space, $V=\Bbb{R}^D$ using DINO. We construct a low dimensional Mood Space $M$, with one map $\pi_V: V\to M$, and a second map $\sigma_W: M \to W$. Intuitively, we think of $\pi_V: V \to M$ as a fiber bundle, or more generally, a fibration, that removes irrelevant feature spaces and brings closer the connection between the relevant tokens. The base space $M$ parametrizes the variation we want to control. The fibers, in other words, the pre-image $\pi_V^{-1}(m)$ of points $m\in M$, represent those aspects of the image tokens that we want to remain fixed. Finally, we use a CLIP-conditioned image synthesis to render an output image.
• Figure 3: We collect image patch tokens across the context images and compute affinity relationships based on the pre-trained features in $V$ (DINO). Our insight is that the token-pairwise affinity and its top eigenstructure contain part-whole hierarchical substructures that we want to preserve in $M$ (Mood Space). A crucial observation is that the affinity relationship is representation agnostic, thus allowing us to define a loss function on affinity and backprop to train the representation in M, without feature alignment. Furthermore, by row-normalization of the affinity matrix and the top eigenvectors, we bring up the relevant concepts across images, making the connection easier to find. We train a token-wise MLP to decode from $M$ to $W$ (CLIP).
• Figure 4: For the two images shown left, we collect all the image and class tokens, compute their DINO features, and construct a row-normalized token affinity matrix $S_V \in {\Bbb{R}}^{(2\times 256) \times (2\times 256)}$. We compute the top 5 graph eigenvectors: $E(S_V) \in {\Bbb{R}}^{ (2\times 256) \times 5}$. Each column of $E(S_V)$ can be reshaped into images, and visualized as shown in the middle. $E(S_V)$ are orthogonal to each other and encode hierarchical part-whole relationships, focusing on the relevant foreground objects. Furthermore, we obtained part-level correspondence for free because we used DINO features to construct the affinity. We decode the compact space $M$ to the CLIP space $W$, shown on the right.
• Figure 5: Connecting concepts with inputs A and B. We compare our interpolation in Mood Space with the baseline linear interpolation in CLIP embedding space. We selected two samples using a interpolation weight $t$ of 0.5 and 0.6, intending to capture the hybrid object. Our method brings the two concepts closer, creating a hybrid version, instead of merely copying one object or the other. We succeed in both within-domain examples, like connecting a dog and a tiger, as well as different-domain examples, such as a bird and a paper roll.