Lyapunov-Stable Adaptive Control for Multimodal Concept Drift

TL;DR

The paper tackles concept drift in online multimodal learning by introducing LS-OGD, a Lyapunov-guided adaptive controller that concurrently adjusts the learning rate $\eta_t$ and fusion weight $\alpha_t$ based on the error signal $e_t$. Stability is established via a Lyapunov function $V(t)=\tfrac{1}{2}e_t^2$, with a bound on $\Delta V(t)$ ensuring uniform ultimate boundedness (UUB) and convergence when drift ceases. The authors prove two key results: (i) Lyapunov stability under bounded drift and (ii) robust modality adaptation under severe unilateral drift, supported by Lemmas 1 and 2 and Propositions 1 and Theorem 2. Empirical validation on drifting multimodal data demonstrates that LS-OGD maintains bounded error, actively reweights modalities to isolate faulty streams, and maintains performance through online adjustments. This framework offers a principled, real-time, and fault-tolerant approach for deploying stable multimodal AI in non-stationary environments with potential broad impact across autonomous, security, and misinformation-detection systems.

Abstract

Multimodal learning systems often struggle in non-stationary environments due to concept drift, where changing data distributions can degrade performance. Modality-specific drifts and the lack of mechanisms for continuous, stable adaptation compound this challenge. This paper introduces LS-OGD, a novel adaptive control framework for robust multimodal learning in the presence of concept drift. LS-OGD uses an online controller that dynamically adjusts the model's learning rate and the fusion weights between different data modalities in response to detected drift and evolving prediction errors. We prove that under bounded drift conditions, the LS-OGD system's prediction error is uniformly ultimately bounded and converges to zero if the drift ceases. Additionally, we demonstrate that the adaptive fusion strategy effectively isolates and mitigates the impact of severe modality-specific drift, thereby ensuring system resilience and fault tolerance. These theoretical guarantees establish a principled foundation for developing reliable and continuously adapting multimodal learning systems.

Figures (8)

• Figure 1: Illustrations of the proposed system architecture. Dashed and solid arrows represent the feedback signals and main data flow, respectively.
• Figure 2: LS-OGD Error Signal & Drift Estimation in Phase 2
• Figure 3: LS-OGD Controller Adaptation Signals in Phase 2
• Figure 4: Types of concept drifts: The circles in this figure represent instances, and the different colors represent different classes.
• Figure 5: The distribution of samples in the M3A dataset we used in experiments (per year per label)