Exploring Fusion Strategies for Multimodal Vision-Language Systems

TL;DR

The paper investigates how the timing of cross-modal fusion affects accuracy and latency in multimodal vision-language systems. It implements three fusion points (late, intermediate, early) using a BERT backbone with MobileNetV2 and ViT backbones on the CMU-MOSI dataset and measures latency on a Jetson Orin AGX. Results show late fusion achieves the highest accuracy, while early fusion offers the lowest latency, revealing a spectrum of trade-offs for edge deployment. The work also provides ViT-based variants and ablation studies, highlighting how architecture choices shape performance and latency.

Abstract

Modern machine learning models often combine multiple input streams of data to more accurately capture the information that informs their decisions. In multimodal machine learning, choosing the strategy for fusing data together requires careful consideration of the application's accuracy and latency requirements, as fusing the data at earlier or later stages in the model architecture can lead to performance changes in accuracy and latency. To demonstrate this tradeoff, we investigate different fusion strategies using a hybrid BERT and vision network framework that integrates image and text data. We explore two different vision networks: MobileNetV2 and ViT. We propose three models for each vision network, which fuse data at late, intermediate, and early stages in the architecture. We evaluate the proposed models on the CMU MOSI dataset and benchmark their latency on an NVIDIA Jetson Orin AGX. Our experimental results demonstrate that while late fusion yields the highest accuracy, early fusion offers the lowest inference latency. We describe the three proposed model architectures and discuss the accuracy and latency tradeoffs, concluding that data fusion earlier in the model architecture results in faster inference times at the cost of accuracy.

Figures (4)

• Figure 1: Accuracy vs. Inference Latency trade-off space for the proposed MobileNetV2-fused and ViT-fused models on the Jetson Orin AGX.
• Figure 2: Architecture of BERT and MobileNetV2 fused at the late stage. The simplified structures of BERT and MobileNetV2 are shown. Their individual classification heads are removed and instead the outputs from the pooling functions of models are passed directly to a unified classification head.
• Figure 3: Architecture diagram of BERT and MobileNetV2 fused at the intermediate level. Features are extracted from both unimodal models at separate points and concatenated before being passed to a final classification head.
• Figure 4: Architecture diagram of BERT and MobileNetV2 fused at the early stage. The base models are cut off after six feature extraction layers and their output is combined and passed to a unified model which uses attention blocks to extract more features before passing to a classification head.