Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Incorporating brain-inspired mechanisms for multimodal learning in artificial intelligence

Xiang He, Dongcheng Zhao, Yang Li, Qingqun Kong, Xin Yang, Yi Zeng

TL;DR

The paper addresses the limitation of static fusion in multimodal learning by introducing inverse effectiveness driven multimodal fusion (IEMF), a brain-inspired mechanism that adaptively scales fusion updates based on unimodal reliability. By computing per-batch modality strength scores and applying a bounded inverse-effectiveness coefficient to fusion gradients, IEMF enables stronger cross-modal compensation when a single modality is weak, while stabilizing learning when unimodal evidence is strong. Across audiovisual classification, continual learning, and question answering, IEMF improves performance for both Artificial Neural Networks and Spiking Neural Networks and achieves up to 50% training-cost reductions, with ablations highlighting the optimal inverse gain at $\gamma=1$ and the role of the coefficient $\xi_t$ in guiding fusion updates. The method generalizes across architectures and tasks, suggesting a practical biologically inspired pathway to more robust and efficient multimodal AI, with potential implications for neuromorphic computing.

Abstract

Multimodal learning enhances the perceptual capabilities of cognitive systems by integrating information from different sensory modalities. However, existing multimodal fusion research typically assumes static integration, not fully incorporating key dynamic mechanisms found in the brain. Specifically, the brain exhibits an inverse effectiveness phenomenon, wherein weaker unimodal cues yield stronger multisensory integration benefits; conversely, when individual modal cues are stronger, the effect of fusion is diminished. This mechanism enables biological systems to achieve robust cognition even with scarce or noisy perceptual cues. Inspired by this biological mechanism, we explore the relationship between multimodal output and information from individual modalities, proposing an inverse effectiveness driven multimodal fusion (IEMF) strategy. By incorporating this strategy into neural networks, we achieve more efficient integration with improved model performance and computational efficiency, demonstrating up to 50% reduction in computational cost across diverse fusion methods. We conduct experiments on audio-visual classification, continual learning, and question answering tasks to validate our method. Results consistently demonstrate that our method performs excellently in these tasks. To verify universality and generalization, we also conduct experiments on Artificial Neural Networks (ANN) and Spiking Neural Networks (SNN), with results showing good adaptability to both network types. Our research emphasizes the potential of incorporating biologically inspired mechanisms into multimodal networks and provides promising directions for the future development of multimodal artificial intelligence. The code is available at https://github.com/Brain-Cog-Lab/IEMF.

Incorporating brain-inspired mechanisms for multimodal learning in artificial intelligence

TL;DR

The paper addresses the limitation of static fusion in multimodal learning by introducing inverse effectiveness driven multimodal fusion (IEMF), a brain-inspired mechanism that adaptively scales fusion updates based on unimodal reliability. By computing per-batch modality strength scores and applying a bounded inverse-effectiveness coefficient to fusion gradients, IEMF enables stronger cross-modal compensation when a single modality is weak, while stabilizing learning when unimodal evidence is strong. Across audiovisual classification, continual learning, and question answering, IEMF improves performance for both Artificial Neural Networks and Spiking Neural Networks and achieves up to 50% training-cost reductions, with ablations highlighting the optimal inverse gain at and the role of the coefficient in guiding fusion updates. The method generalizes across architectures and tasks, suggesting a practical biologically inspired pathway to more robust and efficient multimodal AI, with potential implications for neuromorphic computing.

Abstract

Multimodal learning enhances the perceptual capabilities of cognitive systems by integrating information from different sensory modalities. However, existing multimodal fusion research typically assumes static integration, not fully incorporating key dynamic mechanisms found in the brain. Specifically, the brain exhibits an inverse effectiveness phenomenon, wherein weaker unimodal cues yield stronger multisensory integration benefits; conversely, when individual modal cues are stronger, the effect of fusion is diminished. This mechanism enables biological systems to achieve robust cognition even with scarce or noisy perceptual cues. Inspired by this biological mechanism, we explore the relationship between multimodal output and information from individual modalities, proposing an inverse effectiveness driven multimodal fusion (IEMF) strategy. By incorporating this strategy into neural networks, we achieve more efficient integration with improved model performance and computational efficiency, demonstrating up to 50% reduction in computational cost across diverse fusion methods. We conduct experiments on audio-visual classification, continual learning, and question answering tasks to validate our method. Results consistently demonstrate that our method performs excellently in these tasks. To verify universality and generalization, we also conduct experiments on Artificial Neural Networks (ANN) and Spiking Neural Networks (SNN), with results showing good adaptability to both network types. Our research emphasizes the potential of incorporating biologically inspired mechanisms into multimodal networks and provides promising directions for the future development of multimodal artificial intelligence. The code is available at https://github.com/Brain-Cog-Lab/IEMF.
Paper Structure (10 sections, 1 theorem, 24 equations, 4 figures, 5 tables)

This paper contains 10 sections, 1 theorem, 24 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Results
  3. Discussion
  4. Materials and methods
  5. Datasets.
  6. Data processing and network backbones.
  7. Optimization details.
  8. Experimental platform.
  9. Acknowledgment
  10. appendix

Key Result

Theorem 1

Assume asm:smooth. Let $\mathbf W^{f*}$ be a local minimizer and write $\mathbf H^*=\mathbf H(\mathbf W^{f*})$ with eigen‑pairs $\{(\lambda_i,\mathbf e_i)\}_{i=1}^{d}$, $0<\lambda_1\le\dots\le\lambda_d$. The IEMF updates the fusion module parameters according to the equation eq:update. Define the de Then, As a result, with IEMF, unimodal‑dominated batches reduce sharp directions more than vanilla

Figures (4)

  • Figure 1: Illustration of multisensory integration and the role of inverse effectiveness in IEMF (Inverse Effectiveness driven Multimodal Fusion).(a) Comparison between unimodal sensory input and multisensory integration: integrating visual and auditory cues reduces ambiguity and uncertainty compared to relying on a single modality. (b) Neural basis of audiovisual integration in the human brain, focusing on the superior temporal sulcus (STS) where visual and auditory inputs converge onto multisensory neurons. (c) Biological principle of inverse effectiveness: multisensory integration is strengthened when unimodal signals are weak. Visual and auditory stimuli are processed through distinct sensory pathways and converge at multisensory synapses. The inset illustrates the inverse relationship between unimodal strength and integrative gain. (d) The proposed inverse effectiveness driven multimodal fusion strategy inspired by biological multisensory fusion mechanisms. Visual and auditory inputs are processed by respective encoders, fused via a dynamic fusion module regulated by inverse effectiveness principles, and evaluated using modality strength score estimation. The fusion module weights are dynamically adjusted according to the computed scores. (This figure was created with https://BioRender.com.)
  • Figure 2: Comprehensive evaluation of the proposed inverse effectiveness driven multimodal fusion (IEMF).(a) Overall performance on ANNs. Bar charts compare the vanilla method (blue) with the method augmented by IEMF (khaki) on three audiovisual classification benchmarks—CREMA-D (a1), Kinetics-Sounds (a2) and UrbanSound8K-AV (a3)—under four representative fusion schemes (Concat, MSLR, OGM_GE and LFM). (b) Overall performance on SNNs. Same layout as (a) but using spiking neural networks, demonstrating that IEMF consistently boosts accuracy across network paradigms and datasets (b1–b3). (c) Mechanism ablation on the Kinetics-Sounds dataset. (c1) Effect of the inverse gain coefficient $\gamma$: the baseline without IEMF (grey dashed line on the colour bar) scores below every IEMF setting; model accuracy peaks at $\gamma{=}1$. (c2) Removing the IEMF term ("Joint Learning only") leads to a clear performance drop, highlighting the essential role of the inverse effectiveness multimodal fusion component. (d) Unimodal gain analysis. Test accuracy for (i) a unimodal audio model trained alone ("Unimodal"), (ii) the audio branch extracted from a multimodal model without IEMF ("Unimodal branch (MM) w/o IEMF"), and (iii) the audio branch with IEMF ("Unimodal branch (MM) w/ IEMF"). IEMF yields a persistent relative gain for the unimodal branch. (e) Dynamics of the IEMF coefficient. Evolution of the learnt IMEF coefficient $\xi$ (dashed orange, right y-axis) alongside the test accuracy (solid blue, left y-axis) during training. At the early stage, the value of $\xi$ is large to accelerate the multimodal integration, while as the network converges, $\xi$ falls back and remains stable to maintain the fusion stability. (f) Loss landscape visualization. (f1) 3D loss surface comparison: vanilla method (left) versus IEMF-enhanced method (right). (f2-f3) 2D contour plots: without IEMF (f2) versus with IEMF (f3). The IEMF method leads to broader and flatter minima. (g) Computational cost analysis. Comparison of computational cost between standard models (blue) and IEMF-enhanced models (khaki). IEMF significantly reduces computational costs across all fusion methods, with reductions ranging from 15.2% to 50.0% (highlighted in yellow percentages).
  • Figure 3: Inverse effectiveness driven multimodal fusion boosts audio visual continual learning.(a) Task schematic. A single audiovisual model is incrementally updated as new classes arrive; the goal is to absorb the new knowledge while preserving performance on previously learned classes—achieving "learning without forgetting". (b) Results on AVE-CI, (c) K-S-CI and (d) VS100-CI. For three representative class incremental learning baselines—LwF, SSIL and AV-CIL—we compare the vanilla method (blue) with the method augmented by IEMF (khaki). Each sub-panel is split into top and bottom: the top bar chart reports the overall performance (mean accuracy across all tasks, error bars denote one standard deviation), while the bottom line plot traces the incremental accuracy after each successive task. Across all datasets and baselines, IEMF consistently increases mean accuracy and yields a flatter accuracy-decay curve, indicating the better knowledge transfer.
  • Figure 4: Quantitative and qualitative impact of IEMF on audio visual question answering task.(a) Baseline model. The three radar charts (top) report accuracy on audio-only, visual-only, and audio-visual questions, respectively. Orange = vanilla, Blue = w/ IEMF. The bottom row shows a representative sample—waveform, video frames, and question/answer—where the vanilla fusion miscounts the instruments ("Two"), whereas IEMF answers correctly ("One"). (b) ST-AVQA model. The same layout as (a), but using the stronger ST-AVQA model. IEMF again enlarges the radar area for every question type and corrects the localization query in the illustrated example (vanilla: "Left"; w/ IEMF: "Right"). Across both models, the blue polygons consistently enclose the orange ones, confirming that the inverse effectiveness driven multimodal fusion mechanism improves all question categories while providing intuitive per-sample gains.

Theorems & Definitions (2)

  • Theorem 1: Convergence properties of IEMF
  • proof