GeoAggregator: An Efficient Transformer Model for Geo-Spatial Tabular Data

TL;DR

GeoAggregator tackles scalability and flexibility challenges in geospatial tabular data modeling by introducing a lightweight transformer that embeds geographical priors via Gaussian-biased local attention and global 2D positional awareness. The model leverages a novel Multi-head Cartesian Product Attention to control parameter growth through inducing points in an encoder-processor-decoder framework, achieving linear-scale efficiency while maintaining expressive power. Empirical results across synthetic and real-world datasets show competitive to state-of-the-art performance with substantially fewer parameters and FLOPs, and ablations confirm the effectiveness of the Gaussian bias and Cartesian attention mechanisms. The approach offers a practical, scalable solution for geospatial regression tasks and suggests avenues for further improvements, including per-feature priors and support for categorical inputs.

Abstract

Modeling geospatial tabular data with deep learning has become a promising alternative to traditional statistical and machine learning approaches. However, existing deep learning models often face challenges related to scalability and flexibility as datasets grow. To this end, this paper introduces GeoAggregator, an efficient and lightweight algorithm based on transformer architecture designed specifically for geospatial tabular data modeling. GeoAggregators explicitly account for spatial autocorrelation and spatial heterogeneity through Gaussian-biased local attention and global positional awareness. Additionally, we introduce a new attention mechanism that uses the Cartesian product to manage the size of the model while maintaining strong expressive power. We benchmark GeoAggregator against spatial statistical models, XGBoost, and several state-of-the-art geospatial deep learning methods using both synthetic and empirical geospatial datasets. The results demonstrate that GeoAggregators achieve the best or second-best performance compared to their competitors on nearly all datasets. GeoAggregator's efficiency is underscored by its reduced model size, making it both scalable and lightweight. Moreover, ablation experiments offer insights into the effectiveness of the Gaussian bias and Cartesian attention mechanism, providing recommendations for further optimizing the GeoAggregator's performance.

Figures (11)

• Figure 1: An illustration of the geospatial regression workflow. Each data point has several covariates and a spatial location. The target variable (the housing price in this case) is observed only for part of the points. Using aggregated neighborhood information, we propose an encoder-processor-decoder architecture to predict unobserved target variables.
• Figure 2: Illustration of the GeoAggregator model and the Multi-head Cartesian Product Attention (MCPA) mechanism.
• Figure 3: Effect of the attention bias factor $\lambda$ of 3 variants of GeoAggregator, on the Housing dataset.
• Figure 4: Effect of the input sequence length $\ell_{max}$. We compare results of the GA-mini model with $\lambda = 1.0$, on the Housing dataset.
• Figure 5: Computational cost of one inference of different attention mechanisms, on the Housing dataset.