Loss-aware Binarization of Deep Networks
Lu Hou, Quanming Yao, James T. Kwok
TL;DR
This work targets the inefficiency of deep networks by introducing Loss-Aware Binarization (LAB), a binarization method that directly accounts for the impact on the loss during binarization. LAB uses a proximal Newton framework with a diagonal Hessian, obtaining a closed-form update that leverages Adam-style second-moment estimates as the curvature proxy, enabling efficient training. The approach extends to recurrent networks and demonstrates superior or robust performance across feedforward, CNN, and LSTM tasks, often outperforming loss-agnostic binarization schemes and maintaining competitiveness with full-precision models. Overall, LAB provides a principled, scalable route to loss-aware binary networks suitable for wide and deep architectures.
Abstract
Deep neural network models, though very powerful and highly successful, are computationally expensive in terms of space and time. Recently, there have been a number of attempts on binarizing the network weights and activations. This greatly reduces the network size, and replaces the underlying multiplications to additions or even XNOR bit operations. However, existing binarization schemes are based on simple matrix approximation and ignore the effect of binarization on the loss. In this paper, we propose a proximal Newton algorithm with diagonal Hessian approximation that directly minimizes the loss w.r.t. the binarized weights. The underlying proximal step has an efficient closed-form solution, and the second-order information can be efficiently obtained from the second moments already computed by the Adam optimizer. Experiments on both feedforward and recurrent networks show that the proposed loss-aware binarization algorithm outperforms existing binarization schemes, and is also more robust for wide and deep networks.