Domain Expansion: A Latent Space Construction Framework for Multi-Task Learning

TL;DR

This work tackles latent representation collapse in multi-task learning by structurally decoupling task signals through Domain Expansion, which constructs a latent space of mutually orthogonal subspaces aligned to the top M eigenvectors of the latent distribution. By projecting latent features onto dedicated axes and applying task-specific decoders, the method decouples gradients at the representation level, preventing interference and yielding an explicit, interpretable, and compositional latent space. The approach is validated on ShapeNet, MPIIGaze, and Rotated MNIST, showing improved representation quality and predictive performance over baselines, and enabling algebraic concept manipulation via well-defined operators. The framework demonstrates robustness to continual learning and supports future integration with generative models for human-interpretable latent compositions.

Abstract

Training a single network with multiple objectives often leads to conflicting gradients that degrade shared representations, forcing them into a compromised state that is suboptimal for any single task--a problem we term latent representation collapse. We introduce Domain Expansion, a framework that prevents these conflicts by restructuring the latent space itself. Our framework uses a novel orthogonal pooling mechanism to construct a latent space where each objective is assigned to a mutually orthogonal subspace. We validate our approach across diverse benchmarks--including ShapeNet, MPIIGaze, and Rotated MNIST--on challenging multi-objective problems combining classification with pose and gaze estimation. Our experiments demonstrate that this structure not only prevents collapse but also yields an explicit, interpretable, and compositional latent space where concepts can be directly manipulated.

Figures (8)

• Figure 1: (a) Latent representation collapse. In standard multi-task learning, competing objectives lead to latent representation collapse, where the solution spaces for different concepts (colored ellipses) overlap in only a small, compromised region. (b) Domain Expansion. In contrast, our method assigns each concept to an orthogonal basis vector in the latent space, preventing interference and creating a structured, interpretable representation where features for each concept are clearly separated.
• Figure 2: Problem statement. (Left) Real-world inputs contain multiple concepts simultaneously. (Center) Standard multi-objective training leads to latent representation collapse, where concepts interfere and the latent space becomes entangled. (Right) Our Domain Expansion resolves this by assigning each concept to a mutually orthogonal subspace, yielding an explicit, interpretable, and compositional latent space.
• Figure 3: A single latent vector represents multiple concepts through orthogonal projections, inspired by anamorphic art. (a) An anamorphic object, such as a cylinder, reveals different primitive concepts (a circle vs. a rectangle) when viewed from orthogonal directions. (b) Analogously, our method treats a single latent feature as a rich object that encodes multiple concepts simultaneously. The specific value for each concept is determined by its projection onto a corresponding orthogonal axis in the latent space.
• Figure 4: Concept space vs. latent space: (Left) Real-world inputs possess multiple abstract concepts simultaneously, such as color and pose. (Center) For training, we define a numerical concept space by assigning coordinates to these attributes. (Right) Our model then learns a mapping between this numerical space and its own internal, orthogonal latent space. This process creates an explicit structure where each axis corresponds to a single concept, allowing the model to robustly represent and manipulate the independent attributes of the input.
• Figure 5: System overview: An input image is first passed through an encoder to produce a latent feature. From the distribution of these features across the dataset, we compute the mean ($\mu$) and an orthogonal basis of eigenvectors ($V_M$). To ensure consistency during training, this basis is stabilized across epochs using Hungarian alignment. The orthogonal pooling then projects each latent feature onto these basis vectors. Finally, these projected, non-interfering representations are fed into concept-specific decoders to produce the final outputs.