Convergence for Discrete Parameter Update Schemes
Paul Wilson, Fabio Zanasi, George Constantinides
TL;DR
This work proposes a fully discrete update framework for training, eschewing quantised continuous updates in favor of integer-valued update rules. It establishes a general convergence theorem under mild Lipschitz-gradient assumptions and discrete-update bounds, showing that fixed learning-rate discrete updates converge with a quantified limit on the average gradient norm. The authors instantiate the framework with a zero-inflated multinomial update, deriving explicit moment- and convergence-results that scale favorably with dimension compared to prior fully discrete methods like BOLD. Empirical evaluation on MNIST demonstrates convergence of discrete updates in CNNs and ResNets with a small accuracy penalty, illustrating the practicality of discrete learning for potentially memory-efficient training. The work opens avenues for robust, fully discrete learning systems and applications to inherently discrete architectures.
Abstract
Modern deep learning models require immense computational resources, motivating research into low-precision training. Quantised training addresses this by representing training components in low-bit integers, but typically relies on discretising real-valued updates. We introduce an alternative approach where the update rule itself is discrete, avoiding the quantisation of continuous updates by design. We establish convergence guarantees for a general class of such discrete schemes, and present a multinomial update rule as a concrete example, supported by empirical evaluation. This perspective opens new avenues for efficient training, particularly for models with inherently discrete structure.