Feature Entanglement-based Quantum Multimodal Fusion Neural Network

TL;DR

The paper tackles multimodal fusion in perception tasks by marrying high-accuracy feature-level integration with interpretable decision-level reasoning within a quantum framework. It introduces QCMM, a hybrid architecture consisting of classical unimodal processing, a quantum fusion block grounded in quantum evidence theory, and a QCNN for deep feature extraction. The approach achieves competitive classification performance with orders-of-magnitude fewer trainable parameters and provides a transparent interpretation of fusion through belief masses and conjunctions enacted by quantum gates. Experiments on remote sensing datasets demonstrate strong generalization, parameter efficiency, and robustness across multiple quantum kernel choices, underscoring the practical potential of interpretable quantum multimodal fusion. The work lays a foundation for scalable, explainable quantum fusion across additional modalities and tasks.

Abstract

Multimodal learning aims to enhance perceptual and decision-making capabilities by integrating information from diverse sources. However, classical deep learning approaches face a critical trade-off between the high accuracy of black-box feature-level fusion and the interpretability of less outstanding decision-level fusion, alongside the challenges of parameter explosion and complexity. This paper discusses the accuracy-interpretablity-complexity dilemma under the quantum computation framework and propose a feature entanglement-based quantum multimodal fusion neural network. The model is composed of three core components: a classical feed-forward module for unimodal processing, an interpretable quantum fusion block, and a quantum convolutional neural network (QCNN) for deep feature extraction. By leveraging the strong expressive power of quantum, we have reduced the complexity of multimodal fusion and post-processing to linear, and the fusion process also possesses the interpretability of decision-level fusion. The simulation results demonstrate that our model achieves classification accuracy comparable to classical networks with dozens of times of parameters, exhibiting notable stability and performance across multimodal image datasets.

Figures (8)

• Figure 1: Different topology of entangling fusion circuit: (a) All-to-All and (b) Circuit-Block. $|\psi_{1}\rangle$ and $|\psi_{2}\rangle$ represent different modalities.
• Figure 2: A quantum evidential fusion circuit for two modalities with 2 qubits.
• Figure 3: The overall architecture of the proposed QCMM framework. The pipeline consists of three key stages: (1) Unimodal feature alignment: Classical MLPs extract and align features. (2) Quantum embedding and fusion: Bit-wise angle encoding and trainable evidence fusion; (3) QCNN: Deep semantic extraction and compression by convolution and pooling.
• Figure 4: Detailed quantum circuit architecture of the QCMM framework. The left part shows the angle encoding and the bit-wise feature fusion. The right part details the hierarchical structure of the QCNN, which consists of two stacked convolutional-pooling layers, reducing the system from 8 qubits to 2 qubits for final measurement. The convolution and pooling kernels in the figure are just illustrative.
• Figure 5: Architectures of the three representative ansatzes for the local two-qubit convolution kernels.