Orthogonalising gradients to speed up neural network optimisation
Mark Tuddenham, Adam Prügel-Bennett, Jonathan Hare
TL;DR
The paper introduces a gradient-orthogonalisation technique that biases learning toward diverse intermediate representations by projecting layer-component gradients onto an orthonormal basis before updating parameters. It replaces weight-space constraints with a gradient-projection step, enabling faster convergence and robust performance across CIFAR-10, ImageNet, and semi-supervised settings like Barlow Twins. Empirical results show accelerated learning, resilience to hyper-parameter choices, and competitive accuracy, with clear caveats around computational overhead and batch-size requirements. The approach offers a practical drop-in optimizer variant and motivates further work on efficient implementations and application to attention mechanisms.
Abstract
The optimisation of neural networks can be sped up by orthogonalising the gradients before the optimisation step, ensuring the diversification of the learned representations. We orthogonalise the gradients of the layer's components/filters with respect to each other to separate out the intermediate representations. Our method of orthogonalisation allows the weights to be used more flexibly, in contrast to restricting the weights to an orthogonalised sub-space. We tested this method on ImageNet and CIFAR-10 resulting in a large decrease in learning time, and also obtain a speed-up on the semi-supervised learning BarlowTwins. We obtain similar accuracy to SGD without fine-tuning and better accuracy for naïvely chosen hyper-parameters.