Locality Regularized Reconstruction: Structured Sparsity and Delaunay Triangulations

TL;DR

This work studies locality-regularized linear reconstruction for sparse coding: given a dictionary $oldsymbol{X} In \,\mathbb{R}^{d\times n}$ and a target $oldsymbol{y}\in\mathbb{R}^d$, it seeks $oldsymbol{w}\in\Delta^n$ that minimizes a least-squares objective with a locality penalty that encourages using atoms near $oldsymbol{y}$. Under mild general-position assumptions yielding a unique Delaunay triangulation, the relaxed problem in $ ho$ promotes locality while preserving sparsity, ensuring that for $oldsymbol{y}\in CH(\boldsymbol{X})$ the solution becomes at most $d+1$ sparse and aligned with the vertices of the containing Delaunay simplex; as $ ho\to 0$, $oldsymbol{X}\boldsymbol{w}_{\rho}$ converges linearly to $oldsymbol{y}$, and for $oldsymbol{y}\notin CH(\boldsymbol{X})$ the limit is the projection of $oldsymbol{y}$ onto $CH(\boldsymbol{X})$. The paper also connects the optimal sparse solution to the simplex containing $oldsymbol{y}$, analyzes stability under noise, and demonstrates comparable computational performance to existing Delaunay-simplex methods, with experiments illustrating the solution path as a function of $ ho$ and scalability. These findings offer locality-based features with clear geometric interpretation for unsupervised and semi-supervised learning, and suggest avenues for extensions to non-linear codes and Wasserstein spaces.

Abstract

Linear representation learning is widely studied due to its conceptual simplicity and empirical utility in tasks such as compression, classification, and feature extraction. Given a set of points $[\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_n] = \mathbf{X} \in \mathbb{R}^{d \times n}$ and a vector $\mathbf{y} \in \mathbb{R}^d$, the goal is to find coefficients $\mathbf{w} \in \mathbb{R}^n$ so that $\mathbf{X} \mathbf{w} \approx \mathbf{y}$, subject to some desired structure on $\mathbf{w}$. In this work we seek $\mathbf{w}$ that forms a local reconstruction of $\mathbf{y}$ by solving a regularized least squares regression problem. We obtain local solutions through a locality function that promotes the use of columns of $\mathbf{X}$ that are close to $\mathbf{y}$ when used as a regularization term. We prove that, for all levels of regularization and under a mild condition that the columns of $\mathbf{X}$ have a unique Delaunay triangulation, the optimal coefficients' number of non-zero entries is upper bounded by $d+1$, thereby providing local sparse solutions when $d \ll n$. Under the same condition we also show that for any $\mathbf{y}$ contained in the convex hull of $\mathbf{X}$ there exists a regime of regularization parameter such that the optimal coefficients are supported on the vertices of the Delaunay simplex containing $\mathbf{y}$. This provides an interpretation of the sparsity as having structure obtained implicitly from the Delaunay triangulation of $\mathbf{X}$. We demonstrate that our locality regularized problem can be solved in comparable time to other methods that identify the containing Delaunay simplex.

Figures (8)

• Figure 1: Left: An example of a Delaunay triangulation in $\mathbb{R}^2$ with the empty circumscribing hypersphere condition satisfied for a queried triangle containing the green dot. Right: An example of a point configuration in $\mathbb{R}^{2}$ with non-unique Delaunay triangulation. Either the red or blue edge together with the four outer edges generates a Delaunay triangulation. Note that the $4=d+2$ generating points are not in general position; they all lie on a common circle.
• Figure 2: Left: A visualization of the linear program described in Section \ref{['sec:CHLP']}. Both the 2D Delaunay triangulation and lower faces of the convex hull of the lifted points, coinciding with the triangulation, are shown. Points are connected to their lifted versions by dashed red lines. The red and green hyperplanes are interpretable as two solutions to \ref{['eqn:convexHullLP']}; the green one represents an optimal solution while the red one represents a suboptimal solution. The objective measures the length of the ray, drawn as a black line, from $\mathbf{y}$ on the plane to the hyperplane. The constraints prohibit the ray from passing through a lower face and hence the hyperplane intersecting the lower face is optimal. Right: A viable visibility walk for DelaunaySparse. The colored triangles denote those on the path with $S$ labeling the initial triangle. A line is drawn through each triangle on the walk and the faces from which $y$ is visible are intersected and colored blue.
• Figure 3: The point $\mathbf{y}$ is perturbed to $\tilde{\mathbf{y}}$. Our analysis relies on estimating the size of the dashed line using triangle inequality given the size of the solid lines.
• Figure 4: Sampled points within the convex hull colored by: left: the bound from Theorem \ref{['thm:main_R']} and right: the approximate $\rho$ needed empirically to determine the true vertices quantized to the nearest power of 2 (hence the distinct color bands). Points are colored according to $\log_{10}(\rho)$. In general the bound is pessimistic as to how small $\rho$ needs to be. In particular, we note that several triangles have points at the boundaries obtaining a true solution with relatively high $\rho$, where the theory suggests that points near the boundary of a triangle could need very small $\rho$.
• Figure 5: Each of these plots shows the representations formed by the solution path of $\mathbf{w}_\rho$ for a $d=2$ triangulation. Each path is colored as the representation progresses from the nearest neighbor (large $\rho$) to $\mathbf{y}$ (small $\rho)$. As $\rho$ increases, the solution passes from being dense (3-sparse), to approximately 2-sparse to approximately 1-sparse.

Theorems & Definitions (32)

• Definition 1
• Lemma 1
• proof
• Lemma 2: Lemma 2 restated from tasissa2023k
• proof
• Theorem 1: Generalization of Theorem 2 from tasissa2023k
• proof
• Remark 1
• Lemma 3
• proof