Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QiNN-QJ: A Quantum-inspired Neural Network with Quantum Jump for Multimodal Sentiment Analysis

Yiwei Chen, Kehuan Yan, Yu Pan, Daoyi Dong

TL;DR

This work establishes a principled framework for entangled multimodal fusion and paves the way for quantum-inspired approaches in modelling complex cross-modal correlations in modelling complex cross-modal correlations.

Abstract

Quantum theory provides non-classical principles, such as superposition and entanglement, that inspires promising paradigms in machine learning. However, most existing quantum-inspired fusion models rely solely on unitary or unitary-like transformations to generate quantum entanglement. While theoretically expressive, such approaches often suffer from training instability and limited generalizability. In this work, we propose a Quantum-inspired Neural Network with Quantum Jump (QiNN-QJ) for multimodal entanglement modelling. Each modality is firstly encoded as a quantum pure state, after which a differentiable module simulating the QJ operator transforms the separable product state into the entangled representation. By jointly learning Hamiltonian and Lindblad operators, QiNN-QJ generates controllable cross-modal entanglement among modalities with dissipative dynamics, where structured stochasticity and steady-state attractor properties serve to stabilize training and constrain entanglement shaping. The resulting entangled states are projected onto trainable measurement vectors to produce predictions. In addition to achieving superior performance over the state-of-the-art models on benchmark datasets, including CMU-MOSI, CMU-MOSEI, and CH-SIMS, QiNN-QJ facilitates enhanced post-hoc interpretability through von-Neumann entanglement entropy. This work establishes a principled framework for entangled multimodal fusion and paves the way for quantum-inspired approaches in modelling complex cross-modal correlations.

QiNN-QJ: A Quantum-inspired Neural Network with Quantum Jump for Multimodal Sentiment Analysis

TL;DR

This work establishes a principled framework for entangled multimodal fusion and paves the way for quantum-inspired approaches in modelling complex cross-modal correlations in modelling complex cross-modal correlations.

Abstract

Quantum theory provides non-classical principles, such as superposition and entanglement, that inspires promising paradigms in machine learning. However, most existing quantum-inspired fusion models rely solely on unitary or unitary-like transformations to generate quantum entanglement. While theoretically expressive, such approaches often suffer from training instability and limited generalizability. In this work, we propose a Quantum-inspired Neural Network with Quantum Jump (QiNN-QJ) for multimodal entanglement modelling. Each modality is firstly encoded as a quantum pure state, after which a differentiable module simulating the QJ operator transforms the separable product state into the entangled representation. By jointly learning Hamiltonian and Lindblad operators, QiNN-QJ generates controllable cross-modal entanglement among modalities with dissipative dynamics, where structured stochasticity and steady-state attractor properties serve to stabilize training and constrain entanglement shaping. The resulting entangled states are projected onto trainable measurement vectors to produce predictions. In addition to achieving superior performance over the state-of-the-art models on benchmark datasets, including CMU-MOSI, CMU-MOSEI, and CH-SIMS, QiNN-QJ facilitates enhanced post-hoc interpretability through von-Neumann entanglement entropy. This work establishes a principled framework for entangled multimodal fusion and paves the way for quantum-inspired approaches in modelling complex cross-modal correlations.

Paper Structure

This paper contains 32 sections, 17 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Quantum State
  4. Pure State
  5. Mixed State
  6. Quantum Jump Method
  7. Quantum Measurement
  8. Related Work
  9. Classical Multimodal Fusion Models
  10. Quantum-Inspired Multimodal Models
  11. Method
  12. Overall Framework
  13. Quantum State Embedding
  14. Quantum Jump Module
  15. Quantum Measurement
  16. ...and 17 more sections

Figures (3)

  • Figure 1: Pipeline of QiNN-QJ. The blue $\otimes$ denotes the tensor product, while the gray $\otimes$ indicates the inner product between vectors. Trainable and non-trainable modules are explicitly distinguished using designated symbols.
  • Figure 2: Variation of von-Neumann entanglement entropy, Acc-7 and jump rate across time steps for the text and video modalities.
  • Figure 3: Visualization of Least and Most Entangled Pairs. The left columns present the selected least-entangled pairs, while the right columns show the most-entangled pairs. Each part includes the textual input alongside its corresponding image for comparison.