Difference of Convex (DC) approach for neural network approximation with uniform loss function
Vinesha Peiris, Nadezda Sukhorukova
TL;DR
The paper tackles minimax (uniform) neural network approximation using a single-hidden-layer network with ReLU activations. It formulates the uniform loss as a difference-of-convex (DC) program and solves it with the DC algorithm (DCA), turning the Step 2 update into linear programming problems. Empirical comparisons against ADAMAX on ECG datasets and synthetic functions show that DC-DCA can achieve tighter minimax objectives and interpolation on small data, at the cost of longer run times. This work demonstrates DC-programming as a viable exact optimization route for nonsmooth uniform losses in neural network approximation, with potential for speedups via line-search and other optimizations.
Abstract
Neural networks (NNs) can be viewed as approximation tools. Traditionally, NNs are relying on gradient and stochastic gradient (SG) methods. There are a number of available computational packages for constructing least squares approximations, while uniform (minimax) approximations are hard due to their nonsmooth nature. It was recently demonstrated that a difference convex (DC) programming approach is an efficient alternative optimiser for NNs. In this paper, we demonstrate that a DC programming approach is also efficient for minimax approximation. In our numerical experiments, we compare a DC-programming approach and ADAMAX, a commonly used method for minimax NN approximations.