Geometric Understanding of Discriminability and Transferability for Visual Domain Adaptation

TL;DR

This work formulates transferability and discriminability in visual domain adaptation as geometric relations between subspaces, defining transferability via domain subspace equivalence and discriminability via cluster subspace orthogonality. It introduces GOAL, a two-part model combining a task loss with a geometry-oriented constraint that uses nuclear-norm proxies to enforce subspace equivalence and orthogonality, with a co-regularization parameter guiding balance. The authors prove a feasible parameter regime where both abilities can be learned simultaneously and provide a practical, optimization-friendly implementation with pseudo-labels and a warm-up stage. Extensive experiments across major UDA benchmarks demonstrate that GOAL achieves strong transferability and discriminability, often improving over state-of-the-art methods and validating the theory-driven geometry approach. The work offers interpretable insights into invariant representations and suggests broader applicability to multi-domain and meta-learning problems.

Abstract

To overcome the restriction of identical distribution assumption, invariant representation learning for unsupervised domain adaptation (UDA) has made significant advances in computer vision and pattern recognition communities. In UDA scenario, the training and test data belong to different domains while the task model is learned to be invariant. Recently, empirical connections between transferability and discriminability have received increasing attention, which is the key to understanding the invariant representations. However, theoretical study of these abilities and in-depth analysis of the learned feature structures are unexplored yet. In this work, we systematically analyze the essentials of transferability and discriminability from the geometric perspective. Our theoretical results provide insights into understanding the co-regularization relation and prove the possibility of learning these abilities. From methodology aspect, the abilities are formulated as geometric properties between domain/cluster subspaces (i.e., orthogonality and equivalence) and characterized as the relation between the norms/ranks of multiple matrices. Two optimization-friendly learning principles are derived, which also ensure some intuitive explanations. Moreover, a feasible range for the co-regularization parameters is deduced to balance the learning of geometric structures. Based on the theoretical results, a geometry-oriented model is proposed for enhancing the transferability and discriminability via nuclear norm optimization. Extensive experiment results validate the effectiveness of the proposed model in empirical applications, and verify that the geometric abilities can be sufficiently learned in the derived feasible range.

Figures (10)

• Figure 1: Problem illustration. (a)-(b): A geometric view of discriminability and transferability. The bases in different classes are orthogonal, and the domains are linearly dependent. (c): Invariant representation learning with simultaneously enhanced transferability and discriminability can reduce domain gap between changing environments. However, the possibility is unknown yet.
• Figure 2: A geometric view of transferability and discriminability. (a): The left figure depicts that domains are overlapped to maximize the transferability; the right figure shows that the dimensions and dependence of subspaces are maximized when $\mathcal{L}_{\text{TB}}$ is optimized. (b): The orthogonality between different clusters enhances the discriminability. Under $\mathcal{L}_{\text{DB}}$, the subspaces of different clusters are orthogonal which maximizes the angles between hyperplanes.
• Figure 3: Illustration of the co-regularization, including trade-off state (a)/(c) and balance state (b), between transferability and discriminability by varying the parameter $\lambda = \lambda_{\text{TB}}/\lambda_{\text{DB}}$. (a): When $\lambda = 0$, only the discriminability objective is learned. The model may learn a trivial solution with most subspaces collapse to a point as zero element, i.e., Theorem \ref{['thm:TBDB_CoRegular']}(i). (b): When $\lambda \in (0,1+ \sqrt{2}]$, the co-regularization between transferability and discriminability reaches a balance. The model is possible to learn the geometric properties simultaneously as Theorem \ref{['thm:TBDB_CoRegular']}(iii). (c): When $\lambda \in (1+ \sqrt{2},\infty]$, the model will over-learn the transferability. Then the discriminability will be degraded as Theorem \ref{['thm:TBDB_CoRegular']}(ii). Especially, when $\lambda=\infty$, the model will learn transferability with totally overlapping clusters, i.e., the clusters are indistinguishable with minimum discriminability.
• Figure 4: (a)-(b): Visualization of the class orthogonality and the class-level domain equivalence. (c)-(d): Visualization of the co-regularization relation by varying the regularization parameters.
• Figure 5: Visualization of cosine values of pair-wise principal angles. Larger angle values imply more correlated domain subspaces.

Theorems & Definitions (6)

• Definition 1
• Definition 2
• Remark 1
• Theorem 1
• Theorem 2: qiu2015learning
• Theorem 3