Exploring Synergistic Ensemble Learning: Uniting CNNs, MLP-Mixers, and Vision Transformers to Enhance Image Classification

TL;DR

The study addresses whether cross-architecture ensembles of CNNs, Vision Transformers, and MLP-Mixers can outperform homogeneous ensembles in image classification. It introduces an isolated-model ensemble framework with Softmax fusion and analyzes complementarity using Partial Correlation, Grad-CAM, and Fourier-domain feature maps across ImageNet1k and CIFAR-10, formalized by $AccGain = SoftAcc - \max(Accmodel_1, Accmodel_2, Accmodel_3)$. The results show that CNN–Transformer ensembles provide the strongest gains and can achieve competitive or state-of-the-art performance with lower latency, while MLP-Mixers offer limited added benefit when mixed with the other architectures. The work offers a practical, systematic approach to building robust, efficient image classifiers and lays groundwork for explainable analysis of architectural complementarity across diverse backbone families.

Abstract

In recent years, Convolutional Neural Networks (CNNs), MLP-mixers, and Vision Transformers have risen to prominence as leading neural architectures in image classification. Prior research has underscored the distinct advantages of each architecture, and there is growing evidence that combining modules from different architectures can boost performance. In this study, we build upon and improve previous work exploring the complementarity between different architectures. Instead of heuristically merging modules from various architectures through trial and error, we preserve the integrity of each architecture and combine them using ensemble techniques. By maintaining the distinctiveness of each architecture, we aim to explore their inherent complementarity more deeply and with implicit isolation. This approach provides a more systematic understanding of their individual strengths. In addition to uncovering insights into architectural complementarity, we showcase the effectiveness of even basic ensemble methods that combine models from diverse architectures. These methods outperform ensembles comprised of similar architectures. Our straightforward ensemble framework serves as a foundational strategy for blending complementary architectures, offering a solid starting point for further investigations into the unique strengths and synergies among different architectures and their ensembles in image classification. A direct outcome of this work is the creation of an ensemble of classification networks that surpasses the accuracy of the previous state-of-the-art single classification network on ImageNet, setting a new benchmark, all while requiring less overall latency.

Figures (4)

• Figure 1: Accuracy Gain vs Partial Correlation for all ensembles. There is a trend of decreased accuracy gain with increased partial correlation in both datasets. The trend is more visible in the CIFAR-10 dataset
• Figure 2: Gradient Activation Maps of two sample images from Imagenet on the best ensembled combination (SeNet, Swin, ViT). All three models visually show distinct characteristics implying their complementarity. This aligns with our average frequency analysis across all models \ref{['fig:two']}
• Figure 3: Log Amplitude vs. Frequency graph of intermediate features in CNNs, MLP, and Transformers. The frequency characteristics of CNNs and MLP-Mixers are more similar or correlated than that of CNNs and Transformers.
• Figure 4: Softmax ensemble accuracy gain vs inference time for different types of ensembles. A common trend that can be observed is that combinations mixing CNNs and Transformers tend to outperform all other types of combinations among different inference times