AllSpark: A Multimodal Spatio-Temporal General Intelligence Model with Ten Modalities via Language as a Reference Framework

TL;DR

AllSpark introduces a Language as Reference Framework (LaRF) to unify ten spatio-temporal modalities by aligning them to a language space while preserving modality autonomy. The model uses modality-specific encoders, a modal bridge to map features into language tokens, a shared multimodal LLM, and task-specific heads guided by modality prompts, achieving strong few-shot performance and broad modality understanding. Empirical results across RGB, MSI, HSI, table, code, graph, trajectory, SAR, point cloud, language, and more demonstrate competitive accuracy and adaptability without extensive modality-specific expert training, with training/inference costs reported per modality. This work highlights the potential of language-centric alignment to scale multimodal general intelligence for geospatial objects, offering a path toward flexible, interpretable, and interactive multimodal reasoning in remote sensing and related domains.

Abstract

Leveraging multimodal data is an inherent requirement for comprehending geographic objects. However, due to the high heterogeneity in structure and semantics among various spatio-temporal modalities, the joint interpretation of multimodal spatio-temporal data has long been an extremely challenging problem. The primary challenge resides in striking a trade-off between the cohesion and autonomy of diverse modalities. This trade-off becomes progressively nonlinear as the number of modalities expands. Inspired by the human cognitive system and linguistic philosophy, where perceptual signals from the five senses converge into language, we introduce the Language as Reference Framework (LaRF), a fundamental principle for constructing a multimodal unified model. Building upon this, we propose AllSpark, a multimodal spatio-temporal general artificial intelligence model. Our model integrates ten different modalities into a unified framework. To achieve modal cohesion, AllSpark introduces a modal bridge and multimodal large language model (LLM) to map diverse modal features into the language feature space. To maintain modality autonomy, AllSpark uses modality-specific encoders to extract the tokens of various spatio-temporal modalities. Finally, observing a gap between the model's interpretability and downstream tasks, we designed modality-specific prompts and task heads, enhancing the model's generalization capability across specific tasks. Experiments indicate that the incorporation of language enables AllSpark to excel in few-shot classification tasks for RGB and point cloud modalities without additional training, surpassing baseline performance by up to 41.82\%. The source code is available at https://github.com/GeoX-Lab/AllSpark.

Figures (3)

• Figure 1: AllSpark demonstrates excellent adaptability across up to 10 heterogeneous modalities and shows outstanding few-shot learning capabilities in RGB and point cloud modalities.
• Figure 2: Guided by the LaRF principle, multimodal data are transformed into a token-context structure akin to language, based on their respective prior assumptions. This approach preserves the autonomy of each modality while achieving cohesion between them, enabling the interpretation of multimodal data within a unified language representation space.
• Figure 3: AllSpark Architecture. Multimodal data are extracted by their respective modal encoders into token sequences. Following dimension alignment with modality-specific text prompt tokens via a modal bridge, both the text prompt tokens and modality tokens are passed into a large language multimodal model for interpretation. The interpretation results are then aligned with downstream tasks through task-specific heads.