Learning Multi-Modal Mobility Dynamics for Generalized Next Location Recommendation

TL;DR

The paper addresses the generalization gap in next location recommendation by introducing M^3ob, a multi-modal mobility framework that unifies static multimodal data with dynamic mobility through a Spatial-Temporal Relational Graph (STRG) built atop an LLM-enhanced Spatial-Temporal Knowledge Graph (STKG). It introduces gating-based fusion, STKG-guided cross-modal alignment, and multi-level user preferences to integrate textual, visual, and location modalities with mobility dynamics, optimized via hierarchical regularization and multitask objectives. Empirical results across six city datasets show that M^3ob consistently outperforms state-of-the-art baselines and demonstrates robust generalization in adverse weather and long-tail scenarios, while maintaining efficiency. The work advances practical location-based services by enabling more reliable recommendations under sparse data and diverse environmental conditions, and lays groundwork for dynamic multimodal mobility modeling and interpretable LLM-guided mobility reasoning.

Abstract

The precise prediction of human mobility has produced significant socioeconomic impacts, such as location recommendations and evacuation suggestions. However, existing methods suffer from limited generalization capability: unimodal approaches are constrained by data sparsity and inherent biases, while multi-modal methods struggle to effectively capture mobility dynamics caused by the semantic gap between static multi-modal representation and spatial-temporal dynamics. Therefore, we leverage multi-modal spatial-temporal knowledge to characterize mobility dynamics for the location recommendation task, dubbed as \textbf{M}ulti-\textbf{M}odal \textbf{Mob}ility (\textbf{M}$^3$\textbf{ob}). First, we construct a unified spatial-temporal relational graph (STRG) for multi-modal representation, by leveraging the functional semantics and spatial-temporal knowledge captured by the large language models (LLMs)-enhanced spatial-temporal knowledge graph (STKG). Second, we design a gating mechanism to fuse spatial-temporal graph representations of different modalities, and propose an STKG-guided cross-modal alignment to inject spatial-temporal dynamic knowledge into the static image modality. Extensive experiments on six public datasets show that our proposed method not only achieves consistent improvements in normal scenarios but also exhibits significant generalization ability in abnormal scenarios.

Figures (11)

• Figure 1: The illustration diagram of our proposed Multi-Modal Mobility representation (M$^3$ob) framework. Among these, spatial-temporal knowledge graph, spatial-temporal relational graph, and image relational graph are responsible for modeling multi-modal dynamics, while gated multi-modal fusion and STKG-guided cross-modal alignment enable multi-modal alignment.
• Figure 2: Urban Variations in Time Intervals and Distances Between Normal and Rainy Weather.
• Figure 3: Human Activity in Normal VS Cold Weather.
• Figure 4: Prediction accuracy on rainy weather.
• Figure 5: Prediction accuracy on cold weather.