Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quadratic Unconstrained Binary Optimisation for Training and Regularisation of Binary Neural Networks

Jonas Christoffer Villumsen, Yusuke Sugita

TL;DR

This paper extends QUBO-based training of binary neural networks (BNNs) to accommodate arbitrary network topologies, addressing the discrete optimisation challenge inherent to BNNs. It introduces a rigorous activation-encoding scheme that maps pre-activations to a binary expansion and ties activation to the most significant bit, enabling a quadratic unconstrained binary optimisation (QUBO) formulation. Two regularisation strategies are proposed: (i) a quadratic margin-maximising penalty that enhances robustness and generalisation, and (ii) a dropout-inspired iterative procedure that alternates training on reduced subnetworks to shape linear penalties on weights, with experiments showing substantial gains in test accuracy for margin-based regularisation and modest gains for dropout on a small four-class dataset. The work demonstrates feasible QUBO-based training on Ising-machine hardware, highlights improvements in generalisation tied to margin guidance, and discusses scalability and future directions such as multi-bit weights and cyclic topologies.

Abstract

Advances in artificial intelligence (AI) and deep learning have raised concerns about its increasing energy consumption, while demand for deploying AI in mobile devices and machines at the edge is growing. Binary neural networks (BNNs) have recently gained attention as energy and memory efficient models suitable for resource constrained environments; however, training BNNs exactly is computationally challenging because of its discrete characteristics. Recent work proposing a framework for training BNNs based on quadratic unconstrained binary optimisation (QUBO) and progress in the design of Ising machines for solving QUBO problems suggest a potential path to efficiently optimising discrete neural networks. In this work, we extend existing QUBO models for training BNNs to accommodate arbitrary network topologies and propose two novel methods for regularisation. The first method maximises neuron margins biasing the training process toward parameter configurations that yield larger pre-activation magnitudes. The second method employs a dropout-inspired iterative scheme in which reduced subnetworks are trained and used to adjust linear penalties on network parameters. We apply the proposed QUBO formulation to a small binary image classification problem and conduct computational experiments on a GPU-based Ising machine. The numerical results indicate that the proposed regularisation terms modify training behaviour and yield improvements in classification accuracy on data not present in the training set.

Quadratic Unconstrained Binary Optimisation for Training and Regularisation of Binary Neural Networks

TL;DR

This paper extends QUBO-based training of binary neural networks (BNNs) to accommodate arbitrary network topologies, addressing the discrete optimisation challenge inherent to BNNs. It introduces a rigorous activation-encoding scheme that maps pre-activations to a binary expansion and ties activation to the most significant bit, enabling a quadratic unconstrained binary optimisation (QUBO) formulation. Two regularisation strategies are proposed: (i) a quadratic margin-maximising penalty that enhances robustness and generalisation, and (ii) a dropout-inspired iterative procedure that alternates training on reduced subnetworks to shape linear penalties on weights, with experiments showing substantial gains in test accuracy for margin-based regularisation and modest gains for dropout on a small four-class dataset. The work demonstrates feasible QUBO-based training on Ising-machine hardware, highlights improvements in generalisation tied to margin guidance, and discusses scalability and future directions such as multi-bit weights and cyclic topologies.

Abstract

Advances in artificial intelligence (AI) and deep learning have raised concerns about its increasing energy consumption, while demand for deploying AI in mobile devices and machines at the edge is growing. Binary neural networks (BNNs) have recently gained attention as energy and memory efficient models suitable for resource constrained environments; however, training BNNs exactly is computationally challenging because of its discrete characteristics. Recent work proposing a framework for training BNNs based on quadratic unconstrained binary optimisation (QUBO) and progress in the design of Ising machines for solving QUBO problems suggest a potential path to efficiently optimising discrete neural networks. In this work, we extend existing QUBO models for training BNNs to accommodate arbitrary network topologies and propose two novel methods for regularisation. The first method maximises neuron margins biasing the training process toward parameter configurations that yield larger pre-activation magnitudes. The second method employs a dropout-inspired iterative scheme in which reduced subnetworks are trained and used to adjust linear penalties on network parameters. We apply the proposed QUBO formulation to a small binary image classification problem and conduct computational experiments on a GPU-based Ising machine. The numerical results indicate that the proposed regularisation terms modify training behaviour and yield improvements in classification accuracy on data not present in the training set.
Paper Structure (13 sections, 2 theorems, 34 equations, 12 figures, 7 tables, 1 algorithm)

This paper contains 13 sections, 2 theorems, 34 equations, 12 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Model formulation
  3. Activation function
  4. Training model
  5. Experimental set-up
  6. Analysis of network architectures
  7. Maximising sum of margins
  8. Preliminary analysis and model formulation
  9. Computational results
  10. A dropout algorithm
  11. Model and algorithm
  12. Computational results
  13. Conclusion

Key Result

Theorem 1

Given a non-negative integer $m$ and $x_i\in\{-1,1\}$ for $i=1,\dots,m$, define the non-negative integers where $\lfloor \cdot \rfloor$ denotes the floor function. Define bits $s_0,\dots,s_n\in\{0,1\}$ by the $(n{+}1)$-bit expansion Then the sum of the bipolar variables $x_i$ is positive if and only if the most significant bit $s_n$ is $1$, i.e.,

Figures (12)

  • Figure 1: Schematic diagram of a single node $j$ and its predecessors (\ref{['fig:single-node']}) and a small layered network with six nodes (\ref{['fig:six-node-network']}).
  • Figure 2: Schematic diagram of a neuron $j=3$ with two predecessor neurons in $P_j=\{1,2\}$ and bias $b_3$.
  • Figure 3: Data set comprising 44 monochrome images each labelled with their respective class (O, N, L, or X). Four images (\ref{['fig:train_images']}) are used for training, while 40 images (\ref{['fig:test_images']}) are used for testing.
  • Figure 4: Schematic diagram of Network 2 in Table \ref{['tab:networks']}, that is a convolutional neural network with $5\times5$ input nodes, $3\times3$ convolutional filter giving rise to a $3\times3$ node hidden convolutional layer, and a 2-node fully-connected output layer. Note, that for clarity, incoming connections are only shown for two of the nodes in the hidden layer.
  • Figure 5: Test and training accuracy statistics for the 18 network architectures. Each box represents the first quartile, median, and third quartile, while whiskers indicate the range of the distribution (minimum to maximum) of the test accuracy. The grey squares indicate the mean training accuracy, while the attached grey vertical lines indicate the range of training accuracies (minimum to maximum).
  • ...and 7 more figures

Theorems & Definitions (3)

  • Theorem 1
  • proof
  • Corollary 1