Replicable Learning of Large-Margin Halfspaces
Alkis Kalavasis, Amin Karbasi, Kasper Green Larsen, Grigoris Velegkas, Felix Zhou
TL;DR
The paper tackles the problem of replicably learning large-margin halfspaces in $\mathbb{R}^d$ under margin $\tau$, aiming for dimension-independent guarantees and practical runtimes. It develops multiple replicable algorithms that combine Johnson-Lindenstrauss dimensionality reduction, the Alon-Klartag rounding scheme, and batch-based aggregation, including a principled SGD-based approach with boosting. The main contributions include a dimension-independent, polynomial-time algorithm with sample complexity $\tilde{O}(\epsilon^{-1} \tau^{-7} \rho^{-2})$ (Alg2), a second SGD-based variant with $\tilde{O}(\epsilon^{-2} \tau^{-6} \rho^{-2})$ sample complexity (Alg4), and a DP-to-replicability approach yielding an inefficient but $\tau$-biased method with $\tilde{O}(\epsilon^{-2} \tau^{-4} \rho^{-2})$ samples; plus an even more efficient but computationally heavier net-based method (Alg3-inefficient) achieving $\tilde{O}(\epsilon^{-1} \tau^{-4} \rho^{-2})$ samples. The work links replicability to stability and differential privacy, providing concrete algorithms with provable replicability guarantees and explicit trade-offs between accuracy, margin, replicability, and running time. Overall, it significantly advances replicable learning for margin-based classifiers by delivering dimension-free, efficient procedures and clarifying the computational-precision landscape.
Abstract
We provide efficient replicable algorithms for the problem of learning large-margin halfspaces. Our results improve upon the algorithms provided by Impagliazzo, Lei, Pitassi, and Sorrell [STOC, 2022]. We design the first dimension-independent replicable algorithms for this task which runs in polynomial time, is proper, and has strictly improved sample complexity compared to the one achieved by Impagliazzo et al. [2022] with respect to all the relevant parameters. Moreover, our first algorithm has sample complexity that is optimal with respect to the accuracy parameter $ε$. We also design an SGD-based replicable algorithm that, in some parameters' regimes, achieves better sample and time complexity than our first algorithm. Departing from the requirement of polynomial time algorithms, using the DP-to-Replicability reduction of Bun, Gaboardi, Hopkins, Impagliazzo, Lei, Pitassi, Sorrell, and Sivakumar [STOC, 2023], we show how to obtain a replicable algorithm for large-margin halfspaces with improved sample complexity with respect to the margin parameter $τ$, but running time doubly exponential in $1/τ^2$ and worse sample complexity dependence on $ε$ than one of our previous algorithms. We then design an improved algorithm with better sample complexity than all three of our previous algorithms and running time exponential in $1/τ^{2}$.