ChromaFormer: A Scalable and Accurate Transformer Architecture for Land Cover Classification

TL;DR

ChromaFormer introduces a Spectral Dependency Module (SDM) that enables spectral-band–level attention within a Swin Transformer backbone for multi-spectral land cover classification. By integrating SDM at early stages and employing patch-based gridding training, the approach systematically studies scaling laws on the large BVM dataset, showing substantial accuracy gains over conventional architectures and favorable scaling efficiency. The work demonstrates that hundreds of millions of parameters can deliver >95% Overall Accuracy on dense, multi-spectral RS data, underscoring the value of spectral-aware transformers for large-scale land use mapping. While validated on a Belgian region, the results suggest practical impact for dense, nation- or continent-scale labeling with high-resolution multispectral imagery and scalable architectures.

Abstract

Remote sensing imagery from systems such as Sentinel provides full coverage of the Earth's surface at around 10-meter resolution. The remote sensing community has transitioned to extensive use of deep learning models due to their high performance on benchmarks such as the UCMerced and ISPRS Vaihingen datasets. Convolutional models such as UNet and ResNet variations are commonly employed for remote sensing but typically only accept three channels, as they were developed for RGB imagery, while satellite systems provide more than ten. Recently, several transformer architectures have been proposed for remote sensing, but they have not been extensively benchmarked and are typically used on small datasets such as Salinas Valley. Meanwhile, it is becoming feasible to obtain dense spatial land-use labels for entire first-level administrative divisions of some countries. Scaling law observations suggest that substantially larger multi-spectral transformer models could provide a significant leap in remote sensing performance in these settings. In this work, we propose ChromaFormer, a family of multi-spectral transformer models, which we evaluate across orders of magnitude differences in model parameters to assess their performance and scaling effectiveness on a densely labeled imagery dataset of Flanders, Belgium, covering more than 13,500 km^2 and containing 15 classes. We propose a novel multi-spectral attention strategy and demonstrate its effectiveness through ablations. Furthermore, we show that models many orders of magnitude larger than conventional architectures, such as UNet, lead to substantial accuracy improvements: a UNet++ model with 23M parameters achieves less than 65% accuracy, while a multi-spectral transformer with 655M parameters achieves over 95% accuracy on the Biological Valuation Map of Flanders.

Figures (9)

• Figure 1: Comparison of the size of mainstream remote sensing datasets and the BVM dataset bvm (dataset size vs. model size). Y-axis represents the overall dataset size in the form of total pixel number per class in each dataset. X-axis is the averaged parameter size of models used on each of the datasets. The size of bubbles represents averaged accuracy scores. The BVM dataset is placed at the top-right corner, indicating both the model size and dataset size are well above the current norms of the latest research.
• Figure 2: A demonstration of map sheet cutouts (Source: Digital Flanders Agency): the Flanders region is confined in 43 blocks, each block is further partitioned into 16 smaller ones where 8 on the left is marked for training, 4 on the upper right marked for validation and 4 on the lower right marked for test
• Figure 3: SDM block mechanism, similar to transformer attention head, the query token number dimension is substituted by multi-spectral channel number. Embeddings from same channels are marked in the same color.
• Figure 4: The transformer block of a ChromaFormer model
• Figure 5: Validation accuracy curves of scaled networks, "m" in legend stands for million parameters, and "epochs" stands for how many times the entire dataset has passed through the model.