Exact Instance Compression for Convex Empirical Risk Minimization via Color Refinement
Bryan Zhu, Ziang Chen
TL;DR
The paper introduces an exact compression method for convex empirical risk minimization based on color refinement and equitable partitions, enabling lossless reduction of problem size beyond traditional permutation symmetry. It generalizes earlier LP/QP reductions to differentiable convex objectives and instantiates the framework for linear, logistic, elastic-net, and kernel-based models, with explicit reduction conditions. The authors provide a general Reduction Theorem and practical algorithms for constructing coarsest equitable partitions, accompanied by a detailed treatment of matrix refinements and kernel extensions. Empirical results on OpenML and LIBSVM datasets show meaningful reductions in problem size and tangible end-to-end training speedups without sacrificing solution quality. This framework offers both theoretical guarantees and practical benefits for scalable, interpretable convex ERM in diverse learning settings.
Abstract
Empirical risk minimization (ERM) can be computationally expensive, with standard solvers scaling poorly even in the convex setting. We propose a novel lossless compression framework for convex ERM based on color refinement, extending prior work from linear programs and convex quadratic programs to a broad class of differentiable convex optimization problems. We develop concrete algorithms for a range of models, including linear and polynomial regression, binary and multiclass logistic regression, regression with elastic-net regularization, and kernel methods such as kernel ridge regression and kernel logistic regression. Numerical experiments on representative datasets demonstrate the effectiveness of the proposed approach.