Deep Regression Representation Learning with Topology

TL;DR

This work bridges the Information Bottleneck framework with the topology of regression representations, showing that minimizing ${\mathcal{H}}({\bf Z}|{\bf Y})$ is linked to the intrinsic dimension of the feature space and that topological similarity to the target space improves IB alignment. It introduces PH-Reg, a regression-specific regularizer consisting of an intrinsic-dimension term ${\mathcal L}_d$ and a topology term ${\mathcal L}_t$, implemented via a topology autoencoder and persistent-homology-based distances. Through synthetic and real-world experiments (depth estimation, super-resolution, and age estimation), PH-Reg consistently improves regression performance, especially when both regularizers are used together, and provides ablations that validate the roles of ID control and topology preservation. The results highlight the practical value of incorporating topology-aware regularization into regression models to align feature spaces with target topologies and to reduce representation complexity.

Abstract

Most works studying representation learning focus only on classification and neglect regression. Yet, the learning objectives and, therefore, the representation topologies of the two tasks are fundamentally different: classification targets class separation, leading to disconnected representations, whereas regression requires ordinality with respect to the target, leading to continuous representations. We thus wonder how the effectiveness of a regression representation is influenced by its topology, with evaluation based on the Information Bottleneck (IB) principle. The IB principle is an important framework that provides principles for learning effective representations. We establish two connections between it and the topology of regression representations. The first connection reveals that a lower intrinsic dimension of the feature space implies a reduced complexity of the representation Z. This complexity can be quantified as the conditional entropy of Z on the target Y, and serves as an upper bound on the generalization error. The second connection suggests a feature space that is topologically similar to the target space will better align with the IB principle. Based on these two connections, we introduce PH-Reg, a regularizer specific to regression that matches the intrinsic dimension and topology of the feature space with the target space. Experiments on synthetic and real-world regression tasks demonstrate the benefits of PH-Reg. Code: https://github.com/needylove/PH-Reg.

Figures (5)

• Figure 1: (a) Visualization of the feature space from depth estimation task. Enforcing an ID equal to the target space ($1$ dimensional) will squeeze the feature space into a line, reducing the unnecessary ${\mathcal{H}}({\bf{Z}}|{\bf Y}={\bf y}_i)$ corresponding to the solution space of $f({\bf z}) = \hat{{\bf y}}_i$ (the gray quadrilateral) for all $i$ and implying a lower ${\mathcal{H}}({\bf{Z}}|{\bf Y})$. (b) Visualization of the feature space (right) and the 'Mammoth' shape target space (left), see Sec. \ref{['subsection:synthetic']} for details. The feature space is topologically similar to the target space .
• Figure 2: Visualization of the last hidden layer's feature space from the depth estimation task. The representations are obtained through a modified ResNet-50, with the last hidden layer changed to dimension 3 for visualization. The target space is a $1$-dimensional line, and colors represent the ground truth depth. (b) ${\mathcal{L}}'_d$ encourages a lower intrinsic dimension yet fails to preserve the topology of the target space. (c) ${\mathcal{L}}_d$ takes the target space into consideration and can further preserve its topology. (d) ${\mathcal{L}}_t$ can enforce the topological similarity between the feature and target spaces. (e) Adding the ${\mathcal{L}}_t$ to ${\mathcal{L}}_d$ better preserves the topology of the target space.
• Figure 3: t-sne visualization of the feature spaces ($100$ dimensions $\rightarrow 3$ dimensions) with topological different target spaces.
• Figure 4: Ablation study based on (a-c) the Swiss roll coordinate prediction task and (d) the depth estimation task.
• Figure 5: Illustration of the (a) the use of PH-Reg for regression, and (b) calculating of $\text{PH}_{0}(\text{VR({\bf S})})$. Here ${\bf S}=\{s_1, s_2, s_3\}$. We say three connected components, i.e. $\beta_0$, $(\{\{s_1\}, \{s_2\}, \{s_3\}\})$ 'birth' when $\alpha=0$, one 'death' (two left $(\{\{s_1, s_3\}, s_2\})$) when $\alpha=\alpha_1$, and another one 'death' (one left $(\{\{s_1, s_3, s_2\})$) when $\alpha=\alpha_2$. Thus $\text{PH}_{0}(\text{VR({\bf S})})=\{[0, \alpha_1], [0, \alpha_2] \}$.

Theorems & Definitions (9)

• Theorem 1
• Theorem 2
• Proposition 1
• Definition 1
• Theorem 3
• Definition 2
• Proposition 2
• Lemma 1
• Definition 3