SCPL: Enhancing Neural Network Training Throughput with Decoupled Local Losses and Model Parallelism

TL;DR

SCPL tackles the training-time bottleneck of deep networks by decoupling backpropagation into short, local supervised contrastive learning objectives distributed across model segments. This enables pipelined model parallelism and concurrent gradient updates, reducing backward locking and increasing hardware utilization. Across NLP and CV benchmarks, SCPL achieves substantial throughput gains (up to ~$1.92\times$) with accuracies that are competitive with or surpass BP, while highlighting architecture- and hyperparameter-dependent trade-offs. The work offers practical guidance for deploying large models in enterprise settings and outlines future directions for tooling and broader applicability.

Abstract

Adopting large-scale AI models in enterprise information systems is often hindered by high training costs and long development cycles, posing a significant managerial challenge. The standard end-to-end backpropagation (BP) algorithm is a primary driver of modern AI, but it is also the source of inefficiency in training deep networks. This paper introduces a new training methodology, Supervised Contrastive Parallel Learning (SCPL), that addresses this issue by decoupling BP and transforming a long gradient flow into multiple short ones. This design enables the simultaneous computation of parameter gradients in different layers, achieving superior model parallelism and enhancing training throughput. Detailed experiments are presented to demonstrate the efficiency and effectiveness of our model compared to BP, Early Exit, GPipe, and Associated Learning (AL), a state-of-the-art method for decoupling backpropagation. By mitigating a fundamental performance bottleneck, SCPL provides a practical pathway for organizations to develop and deploy advanced information systems more cost-effectively and with greater agility. The experimental code is released for reproducibility. https://github.com/minyaho/scpl/

Figures (10)

• Figure 1: An illustration of contrastive learning (CL) and supervised contrastive learning (SCL). CL regards an anchor image's augmented images as positive pairs (e.g., $\boldsymbol{x}_1$ and $\boldsymbol{x}_2$ above) and regards the anchor image's augmented image to all non-augmented images as negative pairs (e.g., $\boldsymbol{x}_1$ and $\boldsymbol{x}_1'$ on the left). SCL regards augmented images as positive pairs if they have the same label (e.g., $\boldsymbol{x}_1$ and $\boldsymbol{x}_1'$ on the right); a pair of augmented images is a negative pair if their labels are different (e.g., $\boldsymbol{x}_1$ and $\boldsymbol{x}_1"$).
• Figure 2: An example neural network with 3 hidden layers (left) and its corresponding SCPL network (right). Solid blue arrows correspond to forward paths, red dashed arrows correspond to backward paths, green boxes denote the parameters (functions), and orange boxes represent the loss functions. Gradient flows are blocked between neighboring blocks for SCPL.
• Figure 3: An illustrating example to compare the GPU usage of one iteration for standard BP, NMP, and SCPL. The GPU utilization in a real environment is shown in Figure \ref{['fig:lstm-imdb-profiler-cmp']}.
• Figure 4: A comparison of SCPL's and BP's test accuracies when using LSTM on AG's news with different batch sizes for training. The shaded area represents the mean plus and minus one standard deviation.
• Figure 5: A comparison of SCPL's and BP's test accuracies when using LSTM on IMDB with different batch sizes for training. The shaded area represents the mean plus and minus one standard deviation.