Mixtraining: A Better Trade-Off Between Compute and Performance

TL;DR

MixTraining tackles the data- and compute-efficiency gap in SSL+SL pipelines by inserting a middle mixtraining phase that jointly optimizes self-supervised and supervised objectives. By merging backbone forward/backward passes and constructing a mixed dataset, it achieves smoother optimization and reduced computation, yielding significant accuracy improvements (for example 8.81% absolute and 18.89% relative on TinyImageNet) with up to 1.29x speedups on ViT-Tiny. The approach proves effective across single-task and multi-task settings and remains robust when SSL and SL data differ, with stronger gains under data-limited regimes. Overall, MixTraining provides a practical, plug-and-play path to better compute-performance trade-offs in SSL-informed models with broad applicability to vision tasks and beyond.

Abstract

Incorporating self-supervised learning (SSL) before standard supervised learning (SL) has become a widely used strategy to enhance model performance, particularly in data-limited scenarios. However, this approach introduces a trade-off between computation and performance: while SSL helps with representation learning, it requires a separate, often time-consuming training phase, increasing computational overhead and limiting efficiency in resource-constrained settings. To address these challenges, we propose MixTraining, a novel framework that interleaves several SSL and SL epochs within a unified mixtraining training phase, featuring a smooth transition between two learning objectives. MixTraining enhances synergy between SSL and SL for improved accuracy and consolidates shared computation steps to reduce computation overhead. MixTraining is versatile and applicable to both single-task and multi-task learning scenarios. Extensive experiments demonstrate that MixTraining offers a superior compute-performance trade-off compared to conventional pipelines, achieving an 8.81% absolute accuracy gain (18.89% relative accuracy gain) on the TinyImageNet dataset while accelerating training by up to 1.29x with the ViT-Tiny model.

Figures (4)

• Figure 1: MixTraining demonstrates significant accuracy and computation gains over standard self-supervised learning plus supervised learning pipeline across various data limitations levels (10%, 50%, and 100%). Experiments are conducted on the TinyImageNet dataset with the ViT-Tiny model; Each set of three points on the same line represents results obtained with different training durations (50, 75, and 100 epochs).
• Figure 2: Comparison of MixTraining with the standard SSL+SL framework. (a) The standard SSL+SL paradigm featuring an abrupt transition from self-supervised objective ($\mathsf{obj}_{\mathsf{ssl}}$) to supervised objective ($\mathsf{obj}_{\mathsf{sl}}$). (b) Our MixTraining framework features an added mixtraining phase in the middle. The mixtraining phase optimizes towards a mixed objective ($\mathsf{obj}_{\mathsf{mix}}$), which enables a smooth transition from the self-supervised objective to the supervised objective.
• Figure 3: Comparison of MixTraining with the standard SSL+SL framework. MixTraining achieves computation gains over the standard SSL+SL framework. Top: Standard SSL+SL. Data first goes through a self-supervised learning pass ($F_{\mathsf{ssl,1}}\rightarrow F_{\mathsf{ssl,2}} \rightarrow B_{\mathsf{ssl, 2}} \rightarrow B_{\mathsf{ssl, 1}}$) and then goes through a supervised learning pass ($F_{\mathsf{sl,1}}\rightarrow F_{\mathsf{sl,2}} \rightarrow B_{\mathsf{sl, 2}} \rightarrow B_{\mathsf{sl, 1}}$). Bottom: MixTraining. We merge two forward passes ($F_{\mathsf{ssl,1}}$ and $F_{\mathsf{sl,1}}$) over the backbone model together into a single pass $F_{\mathsf{mix,1}}$ , and use its result for both self-supervised head and supervised head; the backward passes ($B_{\mathsf{ssl,1}}$ and $B_{\mathsf{sl,1}}$) are merged into $B_{\mathsf{mix,1}}$ (bottom left). Our modifications reduce computation requirements and allow better parallelization (bottom right).
• Figure 4: Reconstruction examples from TinyImageNet. The SSL+SL approach significantly deteriorates the reconstruction abilities (second row), yet MixTraining maintains competitive reconstruction abilities (third row).