MM-Path: Multi-modal, Multi-granularity Path Representation Learning -- Extended Version

TL;DR

MM-Path addresses the gap in path representation by integrating road-path topology with image-path geometry and context. It introduces a multi-granularity alignment mechanism and a graph-based cross-modal residual fusion module, using Transformer-based encoders and a cross-modal GCN to fuse modalities across fine to coarse granularities. The approach is trained with a combination of masking, multi-granularity alignment, and fusion losses, and demonstrates state-of-the-art performance on travel time estimation and path ranking across two large-city datasets, with ablations confirming the importance of each component. The framework provides a principled pre-training paradigm for generic path representations with potential for few-shot and zero-shot adaptation in downstream tasks.

Abstract

Developing effective path representations has become increasingly essential across various fields within intelligent transportation. Although pre-trained path representation learning models have shown improved performance, they predominantly focus on the topological structures from single modality data, i.e., road networks, overlooking the geometric and contextual features associated with path-related images, e.g., remote sensing images. Similar to human understanding, integrating information from multiple modalities can provide a more comprehensive view, enhancing both representation accuracy and generalization. However, variations in information granularity impede the semantic alignment of road network-based paths (road paths) and image-based paths (image paths), while the heterogeneity of multi-modal data poses substantial challenges for effective fusion and utilization. In this paper, we propose a novel Multi-modal, Multi-granularity Path Representation Learning Framework (MM-Path), which can learn a generic path representation by integrating modalities from both road paths and image paths. To enhance the alignment of multi-modal data, we develop a multi-granularity alignment strategy that systematically associates nodes, road sub-paths, and road paths with their corresponding image patches, ensuring the synchronization of both detailed local information and broader global contexts. To address the heterogeneity of multi-modal data effectively, we introduce a graph-based cross-modal residual fusion component designed to comprehensively fuse information across different modalities and granularities. Finally, we conduct extensive experiments on two large-scale real-world datasets under two downstream tasks, validating the effectiveness of the proposed MM-Path. The code is available at: https://github.com/decisionintelligence/MM-Path.

Figures (7)

• Figure 1: A path in different modalities
• Figure 2: An example of image path processing
• Figure 3: Overall framework of MM-Path
• Figure 4: An example of multi-modal graph construction
• Figure 5: Effect of pre-training