Projected Stochastic Gradient Descent with Quantum Annealed Binary Gradients

TL;DR

QP-SBGD is a novel layer-wise stochastic optimiser tailored towards training neural networks with binary weights, known as binary neural networks (BNNs), on quantum hardware, and outperforms or is on par with competitive and well-established baselines when optimising the Rosenbrock function.

Abstract

We present, QP-SBGD, a novel layer-wise stochastic optimiser tailored towards training neural networks with binary weights, known as binary neural networks (BNNs), on quantum hardware. BNNs reduce the computational requirements and energy consumption of deep learning models with minimal loss in accuracy. However, training them in practice remains to be an open challenge. Most known BNN-optimisers either rely on projected updates or binarise weights post-training. Instead, QP-SBGD approximately maps the gradient onto binary variables, by solving a quadratic constrained binary optimisation. Under practically reasonable assumptions, we show that this update rule converges with a rate of $\mathcal{O}(1 / \sqrt{T})$. Moreover, we show how the $\mathcal{NP}$-hard projection can be effectively executed on an adiabatic quantum annealer, harnessing recent advancements in quantum computation. We also introduce a projected version of this update rule and prove that if a fixed point exists in the binary variable space, the modified updates will converge to it. Last but not least, our algorithm is implemented layer-wise, making it suitable to train larger networks on resource-limited quantum hardware. Through extensive evaluations, we show that QP-SBGD outperforms or is on par with competitive and well-established baselines such as BinaryConnect, signSGD and ProxQuant when optimising the Rosenbrock function, training BNNs as well as binary graph neural networks.

Figures (15)

• Figure 1: We propose Quantum Projected Stochastic Binary-Gradient Descent (QP-SBGD), a provably convergent, layer-wise optimizer for training binary (graph) neural networks on adiabatic quantum annealers. Our hybrid approach iteratively optimizes each hidden layer. We first apply a forward-backward pass on the neural network computing the gradients on a classical computer (steps 1--2). We then update the weights in binary variables for each layer separately on the actual quantum hardware of D-Wave Boothby2020arXiv.
• Figure 2: Binary logistic regression.
• Figure 3: Mean loss over five runs. Left: two- layer setup; Right: 10-layer setup. We compare QP-SBGD against ProxQuant and BinaryConnect (BC) with signSGD and SGD optimisers.
• Figure 4: Graph classification: We report mean test accuracy over five runs for Karate club zachary1977information (left), Cora mccallum2000automating (middle) and Pubmed namata2012query (right) datasets.
• Figure 5: Label generation on MNIST lecun-mnisthandwrittendigit-2010 (example). Left: Initial image of the digit "2"; Middle: Extracted keypoint features with the superpixel approach monti2017geometric; Right: sampling four lines to determine binary features.

Theorems & Definitions (17)

• Definition 1: Binary map (BM)
• Proposition 1
• Definition 2: P-SBGD
• Remark 1
• Theorem 1: Fixed point of P-SBGD
• proof
• Proposition 2: BM as QUBO
• proof : Sketch of Proof
• Definition 3: QP-SBGD
• Definition 4: Intermediate gradient