Building a Foundation Model for Trajectory from Scratch

TL;DR

The paper tackles the challenge of building mobility trajectory foundation models from scratch by offering a hands-on, code-driven tutorial that starts from GPT-2 and adapts it to spatiotemporal data represented as $$(\text{latitude}, \text{longitude}, t)$$ triplets. It details an end-to-end pipeline including custom data parsing, delta encoding, learnable projections, a compact two-block Transformer, and a masked variant for trajectory completion, situating this educational model among TrajFM, TrajGPT, and TimesFM. The work highlights architectural innovations such as RoPE for spatiotemporal data, Time2Vec/Space2Vec embeddings, region-level predictions, and masking strategies that enable multi-task mobility tasks, offering concrete code and open-source materials to accelerate adoption. By providing clear terminology, practical implementation steps, and benchmarking context, the authors aim to enhance research clarity and peer-review effectiveness within the SIGSPATIAL mobility AI community.

Abstract

Foundation models are transformative in artificial intelligence, but building them from scratch, especially for mobility trajectories, is not yet clear or documented. This tutorial bridges this gap by demonstrating the steps and code of a minimal implementation of a trajectory-focused foundation model starting from GPT-2. Through a concise, step-by-step, code-driven process, we demonstrate adapting GPT-2 for spatiotemporal data. We then review and compare representative trajectory foundation models, such as TrajFM and TrajGPT, highlighting their architectural innovations and differences. Additionally, we introduce complementary techniques from related domains, like TimesFM's patching approach. Targeted at researchers and practitioners, this tutorial aims to explain the concepts and terminology of foundation models, at the implementation level. We find it timely and indispensable to create this educational material in order to support the SIGSPATIAL community in building and evaluating mobility foundation models, enhancing both research clarity and peer-review effectiveness in mobility AI.