The Space Complexity of Approximating Logistic Loss

TL;DR

The paper addresses the space efficiency of data structures that approximate the logistic loss in logistic regression, introducing the dataset-dependent measure $μ_{oldsymbol{y}}(\mathbf{X})$ to capture compressibility. It proves strong lower bounds: a $\tilde{Ω}\left(\frac{d}{ε^2}\right)$ space requirement when $μ_{oldsymbol{y}}(\mathbf{X})=Θ(1)$ and a general $\tilde{Ω}\left(d\cdot μ_{oldsymbol{y}}(\mathbf{X})\right)$ bound for constant $ε$, demonstrating that the dependence on $μ_{oldsymbol{y}}(\mathbf{X})$ is intrinsic rather than an artifact of specific coreset constructions. The work also provides a polynomial-time linear-programming approach to compute $μ_{oldsymbol{y}}(\mathbf{X})$, refutes prior conjectures about its hardness, and includes empirical comparisons to prior methods. Overall, the results imply that existing coreset bounds are near-optimal in the typical $μ$-bounded regime while revealing fundamental limits to compressing logistic regression data beyond these bounds.

Abstract

We provide space complexity lower bounds for data structures that approximate logistic loss up to $ε$-relative error on a logistic regression problem with data $\mathbf{X} \in \mathbb{R}^{n \times d}$ and labels $\mathbf{y} \in \{-1,1\}^d$. The space complexity of existing coreset constructions depend on a natural complexity measure $μ_\mathbf{y}(\mathbf{X})$, first defined in (Munteanu, 2018). We give an $\tildeΩ(\frac{d}{ε^2})$ space complexity lower bound in the regime $μ_\mathbf{y}(\mathbf{X}) = O(1)$ that shows existing coresets are optimal in this regime up to lower order factors. We also prove a general $\tildeΩ(d\cdot μ_\mathbf{y}(\mathbf{X}))$ space lower bound when $ε$ is constant, showing that the dependency on $μ_\mathbf{y}(\mathbf{X})$ is not an artifact of mergeable coresets. Finally, we refute a prior conjecture that $μ_\mathbf{y}(\mathbf{X})$ is hard to compute by providing an efficient linear programming formulation, and we empirically compare our algorithm to prior approximate methods.

Figures (1)

• Figure 1: Simulated data results for exact computation of $\mu_\mathbf{y}(\mathbf{X})$ (Theorem \ref{['thm:compute_complexity_measure']}) using the full data (Exactfull), sketched data (ExactSketched) vs the approximate upper (ApprSketchedUpper) and lower bounds (ApprSketchedLower) as suggested by munteanu2018coresets (see Section \ref{['sec:appendixExperiments']}). The results clearly show that the upper and lower bounds can be very loose compared to our exact calculation of the complexity measure $\mu_\mathbf{y}(\mathbf{X})$

Theorems & Definitions (14)

• Definition 1
• Lemma 2.1
• Lemma 2.2
• Theorem 1
• Corollary 2
• Lemma 2.3
• Lemma 2.4
• Theorem 3
• Theorem 4
• Theorem 5: Theorem 2 in Hoeff63