Energy-Efficient Deep Learning Without Backpropagation: A Rigorous Evaluation of Forward-Only Algorithms

TL;DR

The paper tackles the energy and memory bottlenecks of backpropagation by evaluating forward-only BP-free learning algorithms—Forward-Forward (FF), Cascaded Forward (CaFo), and Mono-Forward (MF)—under a rigorous, hardware-validated framework with universal hyperparameter tuning. It demonstrates MF can match or surpass fairly tuned BP baselines in accuracy on native MLPs while delivering substantial efficiency gains, including up to 41% energy reduction and up to 34% faster training on demanding tasks. The work provides a data-driven evolutionary view of BP-free methods, showing that MF's layer-wise projection matrices and local losses yield strong generalization and favorable hardware profiles. These findings highlight the potential for sustainable AI and motivate hardware-software co-design toward forward-only learning, while noting limitations to CNNs/Transformers and memory trade-offs observed in practice.

Abstract

The long-held assumption that backpropagation (BP) is essential for state-of-the-art performance is challenged by this work. We present rigorous, hardware-validated evidence that the Mono-Forward (MF) algorithm, a backpropagation-free method, consistently surpasses an optimally tuned BP baseline in classification accuracy on its native Multi-Layer Perceptron (MLP) architectures. This superior generalization is achieved with profound efficiency gains, including up to 41% less energy consumption and up to 34% faster training. Our analysis, which charts an evolutionary path from Geoffrey Hinton's Forward-Forward (FF) to the Cascaded Forward (CaFo) and finally to MF, is grounded in a fair comparative framework using identical architectures and universal hyperparameter optimization. We further provide a critical re-evaluation of memory efficiency in BP-free methods, empirically demonstrating that practical overhead can offset theoretical gains. Ultimately, this work establishes MF as a practical, high-performance, and sustainable alternative to BP for MLPs.

Figures (5)

• Figure 1: FF's slow convergence dynamics on the MNIST 4x2000 MLP, showing (a) significantly slower validation accuracy gains and (b) an erratic, high level training loss compared to BP's smooth optimization.
• Figure 2: The Hardware Autopsy of FF: (a) FF consumes slightly more peak memory than BP on this MLP. (b) Volatile GPU clock speeds reveal inefficient hardware utilization compared to BP's stable saturation.
• Figure 3: CaFo's Acute Trade-off on Fashion-MNIST: BP shows ideal convergence. CaFo-DFA is effective but incurs an upfront pretraining cost (flat line). CaFo-Rand is highly volatile, showing the necessity of quality features.
• Figure 4: Hardware monitoring plots for MF vs. BP on CIFAR-10. (a) MF's lower GPU utilization and (b) reduced thermal profile indicate superior energy efficiency. (c) The characteristic step like pattern in MF's memory usage serves as a unique signature of its layer wise training mechanism.
• Figure 5: Explaining MF's Superior Generalization: MF's greedy local optimizations converge to a more favorable (lower) final validation loss than BP's end-to-end global optimization on the CIFAR-10 3x2000 MLP, challenging the notion that global optimization is always superior.