Multi-modal Collaborative Optimization and Expansion Network for Event-assisted Single-eye Expression Recognition

TL;DR

This work tackles robust single-eye expression recognition under challenging lighting by fusing RGB and event modalities. It introduces two novel components: MCO-Mamba for joint multi-modal optimization and MCIB for interactive fusion, plus HCE-MoE for heterogeneous, expert-based decision making. MJOS employs bidirectional state-space modeling to align cross-modal dynamics, while MCIB uses cross-modal attention with a simple gating mechanism to balance modality contributions, enabling task-driven fusion. On the SSE dataset, the approach achieves state-of-the-art WAR of $91.3\%$ and UAR of $91.9\%$, with strong robustness across Normal, Overexposure, Low-Light and HDR conditions, demonstrating the usefulness of combining high-temporal-resolution event data with RGB textures for privacy-preserving, robust expression recognition.

Abstract

In this paper, we proposed a Multi-modal Collaborative Optimization and Expansion Network (MCO-E Net), to use event modalities to resist challenges such as low light, high exposure, and high dynamic range in single-eye expression recognition tasks. The MCO-E Net introduces two innovative designs: Multi-modal Collaborative Optimization Mamba (MCO-Mamba) and Heterogeneous Collaborative and Expansion Mixture-of-Experts (HCE-MoE). MCO-Mamba, building upon Mamba, leverages dual-modal information to jointly optimize the model, facilitating collaborative interaction and fusion of modal semantics. This approach encourages the model to balance the learning of both modalities and harness their respective strengths. HCE-MoE, on the other hand, employs a dynamic routing mechanism to distribute structurally varied experts (deep, attention, and focal), fostering collaborative learning of complementary semantics. This heterogeneous architecture systematically integrates diverse feature extraction paradigms to comprehensively capture expression semantics. Extensive experiments demonstrate that our proposed network achieves competitive performance in the task of single-eye expression recognition, especially under poor lighting conditions.

Figures (5)

• Figure 1: Overall architecture of our proposed MCO-E Net. The Network contains two novel designs: Multi-modal Collaborative Optimization Mamba (MCO-Mamba) and Heterogeneous Collaborative and Expansion Mixture of Experts (HCE-MoE).
• Figure 2: Architectures of proposed MCO-Mamba. We first jointly optimize the model using Event and RGB modalities to drive the model to balance the learning of the two modal distributions; Next, we model the collaborative interaction between the two modalities to leverage their respective strengths and obtain high-quality expression descriptors.
• Figure 3: Detailed of the proposed Heterogeneous Collaborative and Expansion MoE (HCE-MoE). (a) Router with Attention. (b) Deep Expert. (c) Attention Expert. (d) Focal Expert.
• Figure 4: Heatmap visualization of the comparison between our proposed MCO-E Net and SOTA methods
• Figure 5: Heatmap visualization of the comparison between our proposed MCO-E Net and removing MCO-Mamba or HCE-MoE