Generalization analysis with deep ReLU networks for metric and similarity learning
Junyu Zhou, Puyu Wang, Ding-Xuan Zhou
TL;DR
This work addresses the generalization behavior of metric and similarity learning under the hinge loss by deriving an explicit form for the true metric, $d_\rho(x,x') = \mathrm{sgn}(1 - 2 \eta(x,x'))$ with $\eta(x,x') = \langle P_x, P_{x'}\rangle$. It then constructs a structured deep ReLU network, $d_\mathcal{H}(x,x') = F_a\left(1 - 2 \sum_{i=1}^m \phi(h_i(x), h_i(x'))\right)$, to approximate $d_\rho$ by training sub-networks $h_i$ that estimate conditional probabilities $p_i(x)$, a product-approximation network $\phi$, and a sign-approximation layer $F_a$. The paper proves both approximation and estimation error bounds, culminating in an optimal excess generalization rate of $\mathcal{E}(\hat{d}_z) - \mathcal{E}(d_\rho) = O\big(n^{-{(\theta+1) r}/{(p + (\theta+2) r)}}\big)$ up to logarithmic terms under Tsybakov's noise condition and Sobolev smoothness, providing a theoretical foundation for deep metric learning with hinge loss. It also analyzes regular properties of the true metric under general losses (e.g., removal of bias, symmetry, finite label spaces) and discusses implications for when symmetric distance measures, such as Mahalanobis-type metrics, are justified. The results offer guidance on balancing network capacity to achieve favorable generalization and point to extensions to other losses like the logistic loss.
Abstract
While considerable theoretical progress has been devoted to the study of metric and similarity learning, the generalization mystery is still missing. In this paper, we study the generalization performance of metric and similarity learning by leveraging the specific structure of the true metric (the target function). Specifically, by deriving the explicit form of the true metric for metric and similarity learning with the hinge loss, we construct a structured deep ReLU neural network as an approximation of the true metric, whose approximation ability relies on the network complexity. Here, the network complexity corresponds to the depth, the number of nonzero weights and the computation units of the network. Consider the hypothesis space which consists of the structured deep ReLU networks, we develop the excess generalization error bounds for a metric and similarity learning problem by estimating the approximation error and the estimation error carefully. An optimal excess risk rate is derived by choosing the proper capacity of the constructed hypothesis space. To the best of our knowledge, this is the first-ever-known generalization analysis providing the excess generalization error for metric and similarity learning. In addition, we investigate the properties of the true metric of metric and similarity learning with general losses.