Pretrained Mobility Transformer: A Foundation Model for Human Mobility

TL;DR

This work presents Pretrained Mobility Transformer (PMT), a transformer-based foundation model trained on massive unlabeled location-based service trajectories to learn urban space representations. By tokenizing geographic areas as trainable spatial embeddings and integrating spatiotemporal encoding, PMT is pretrained with next-location prediction and mask imputation tasks to capture complex mobility patterns. Across three U.S. MSAs, PMT learns spatial embeddings that reflect geographic proximity and socio-demographic attributes, and larger PMT variants consistently outperform baselines on next-location prediction, trajectory imputation, and trajectory generation, suggesting scalable benefits from foundation-model pretraining in human mobility. The study highlights PMT's potential to inform urban planning and mobility analytics while acknowledging sampling bias and privacy considerations inherent to LBS data.

Abstract

Ubiquitous mobile devices are generating vast amounts of location-based service data that reveal how individuals navigate and utilize urban spaces in detail. In this study, we utilize these extensive, unlabeled sequences of user trajectories to develop a foundation model for understanding urban space and human mobility. We introduce the \textbf{P}retrained \textbf{M}obility \textbf{T}ransformer (PMT), which leverages the transformer architecture to process user trajectories in an autoregressive manner, converting geographical areas into tokens and embedding spatial and temporal information within these representations. Experiments conducted in three U.S. metropolitan areas over a two-month period demonstrate PMT's ability to capture underlying geographic and socio-demographic characteristics of regions. The proposed PMT excels across various downstream tasks, including next-location prediction, trajectory imputation, and trajectory generation. These results support PMT's capability and effectiveness in decoding complex patterns of human mobility, offering new insights into urban spatial functionality and individual mobility preferences.

Figures (6)

• Figure 1: An illustrative user trajectory sequence with $T=8$ time steps. Location information is available for five time steps, denoted by light-colored boxes. The remaining three time steps, which lack user location data, are represented by dark-colored boxes.
• Figure 2: Overall pretraining procedures for PMT. In the input layer, light-colored, dark-colored, and gray-shadowed boxes distinctly represent elements in the trajectory sequence that possess location information, are devoid of location information, and have obscured location information, respectively. Apart from the causal masking, the same architectures are used in both pretraining tasks.
• Figure 3: Spatial embedding similarity distribution of black-black comparison and black-white comparison (PMT-1.6M).
• Figure 4: Accuracy of imputation across varying sparsity levels. The removal ratio represents the proportion of locations randomly omitted from the trajectory sequences. Considering the pre-existing gaps in the sequences, the actual missingness rate is higher than indicated by the removal ratio alone. At a 90% removal ratio, the average temporal occupancy of sequences is approximately 7.5%.
• Figure 5: A synthetic example of LBS data. It is important to note that, in order to protect user privacy, the latitude and longitude of a user's primary addresses (such as home or workplace) will be shifted to the centroid of the corresponding CBG.