TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Data

TL;DR

TEOChat addresses the need for temporal reasoning over earth observation data by introducing the first vision-language model capable of handling temporal EO image sequences. It introduces TEOChatlas, a large, diverse instruction-following dataset spanning single-image and temporal tasks across multiple EO datasets, enabling robust temporal multimodal learning. Empirical results show that TEOChat outperforms prior VLMs trained on natural imagery or single EO images, rivals or surpasses specialist models on several temporal tasks, and demonstrates strong zero-shot generalization to new change-detection datasets and to proprietary models on multiple tasks. The work provides public data, code, and model weights, enabling reproducibility and offering a practical step toward scalable, general-purpose EO analysis for disaster response and urban monitoring.

Abstract

Large vision and language assistants have enabled new capabilities for interpreting natural images. These approaches have recently been adapted to earth observation data, but they are only able to handle single image inputs, limiting their use for many real-world tasks. In this work, we develop a new vision and language assistant called TEOChat that can engage in conversations about temporal sequences of earth observation data. To train TEOChat, we curate an instruction-following dataset composed of many single image and temporal tasks including building change and damage assessment, semantic change detection, and temporal scene classification. We show that TEOChat can perform a wide variety of spatial and temporal reasoning tasks, substantially outperforming previous vision and language assistants, and even achieving comparable or better performance than several specialist models trained to perform specific tasks. Furthermore, TEOChat achieves impressive zero-shot performance on a change detection and change question answering dataset, outperforms GPT-4o and Gemini 1.5 Pro on multiple temporal tasks, and exhibits stronger single image capabilities than a comparable single image instruction-following model on scene classification, visual question answering, and captioning. We publicly release our data, model, and code at https://github.com/ermongroup/TEOChat .

Figures (8)

• Figure 1: Left: VLM Capabilities. TEOChat is the first VLM for temporal EO data. Right: Example outputs of prior VLMs. We compare to a temporal VLM (Video-LLaVA) and a single EO image VLM (GeoChat).
• Figure 2: Examples of instruction-following tasks in TEOChatlas. We curate many instruction-following tasks for temporal EO data and group them into seven categories. The tasks require spatial and temporal reasoning capabilities, and span real-world applications including disaster relief and urban development monitoring.
• Figure 3: Overview of TEOChat. TEOChat inputs a temporal sequence of EO images and a user instruction, and outputs a natural language response. It can input and output regions specified through bounding boxes, and can also input and output references to specific timesteps using image identifiers. We show one example from each task category, including temporal scene classification (peach), change detection (yellow), spatial change referring expression (green), change question answering (cyan), region-based change question answering (blue), temporal referring expression (purple), and region-based temporal question answering (pink).
• Figure 4: Qualitative examples of task generalization exhibited by TEOChat on temporal EO images.
• Figure 5: Locations of the examples in the xBD and fMoW RGB, and fMoW Sentinel subsets of the TEOChatlas dataset. GeoChat, S2Looking, and QFabric do not have geographic locations in the released data. However, the datasets in GeoChat span many locations around the world, and both S2Looking and QFabric contain globally distributed locations as well.