Generalized Multimodal Fusion via Poisson-Nernst-Planck Equation

TL;DR

A generalized multimodal fusion method via the Poisson-Nernst-Planck (PNP) equation, which adeptly addresses issues regarding the efficacy of feature extraction, data integrity, consistency of feature dimensions, and adaptability across various downstream tasks is proposed.

Abstract

Previous studies have highlighted significant advancements in multimodal fusion. Nevertheless, such methods often encounter challenges regarding the efficacy of feature extraction, data integrity, consistency of feature dimensions, and adaptability across various downstream tasks. This paper proposes a generalized multimodal fusion method (GMF) via the Poisson-Nernst-Planck (PNP) equation, which adeptly addresses the aforementioned issues. Theoretically, the optimization objective for traditional multimodal tasks is formulated and redefined by integrating information entropy and the flow of gradient backward step. Leveraging these theoretical insights, the PNP equation is applied to feature fusion, rethinking multimodal features through the framework of charged particles in physics and controlling their movement through dissociation, concentration, and reconstruction. Building on these theoretical foundations, GMF disassociated features which extracted by the unimodal feature extractor into modality-specific and modality-invariant subspaces, thereby reducing mutual information and subsequently lowering the entropy of downstream tasks. The identifiability of the feature's origin enables our approach to function independently as a frontend, seamlessly integrated with a simple concatenation backend, or serve as a prerequisite for other modules. Experimental results on multiple downstream tasks show that the proposed GMF achieves performance close to the state-of-the-art (SOTA) accuracy while utilizing fewer parameters and computational resources. Furthermore, by integrating GMF with advanced fusion methods, we surpass the SOTA results.

Figures (14)

• Figure 1: Stages of information entropy change. Where $Z_i$ might be a set of vectors ($\{Z_i^A, \dots,Z_i^M\}$) or a vector, depending on the fusion method $F(\cdot)$, and $C(\cdot)$ stands for classifier.
• Figure 2: Structure of GMF. The input is taken from $f(X_i,\theta)$ and the output is taken as $Z_i$. This is done in three steps: dissociation concentration, and reconstruction. As a front-end, the output can be directly used for classification or can be connected to other fusion modules. See Appendix \ref{['algo']}
• Figure 3: The gradient diagram extended from Figure \ref{['fig:struct']}, the notation system is consistent with Figure \ref{['fig:struct']}. The blue arrow represents the loss in the fusion stage ($\mathcal{L}_{fusion}$), and the red arrow represents the loss in the downstream task ($\mathcal{L}_{task}$). The green arrow is related to our redefined optimization objective, and the meaning is consistent with the green dashed arrow in Figure \ref{['fig:struct']}. Not all multimodal fusion methods have gradients with blue arrows and green arrows. These are not specific losses, nor are they necessarily individual losses.
• Figure 4: Structure of Residual in Networks.
• Figure 5: Schematic diagram of the electrolytic cell, + (orange) and - (black) represent the charged species (ions and electrodes). There is a boundary $b$ (black line) in the electrolytic cell, assuming that the positive potential is $U_0$, the negative potential is $-U_0$, and the boundary $b$ is the zero potential.