Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OReole-FM: successes and challenges toward billion-parameter foundation models for high-resolution satellite imagery

Philipe Dias, Aristeidis Tsaris, Jordan Bowman, Abhishek Potnis, Jacob Arndt, H. Lexie Yang, Dalton Lunga

TL;DR

This study assesses performance of different pretrained variants of vision Transformers across image classification, semantic segmentation and object detection benchmarks, which highlight the importance of data scaling for effective model scaling.

Abstract

While the pretraining of Foundation Models (FMs) for remote sensing (RS) imagery is on the rise, models remain restricted to a few hundred million parameters. Scaling models to billions of parameters has been shown to yield unprecedented benefits including emergent abilities, but requires data scaling and computing resources typically not available outside industry R&D labs. In this work, we pair high-performance computing resources including Frontier supercomputer, America's first exascale system, and high-resolution optical RS data to pretrain billion-scale FMs. Our study assesses performance of different pretrained variants of vision Transformers across image classification, semantic segmentation and object detection benchmarks, which highlight the importance of data scaling for effective model scaling. Moreover, we discuss construction of a novel TIU pretraining dataset, model initialization, with data and pretrained models intended for public release. By discussing technical challenges and details often lacking in the related literature, this work is intended to offer best practices to the geospatial community toward efficient training and benchmarking of larger FMs.

OReole-FM: successes and challenges toward billion-parameter foundation models for high-resolution satellite imagery

TL;DR

This study assesses performance of different pretrained variants of vision Transformers across image classification, semantic segmentation and object detection benchmarks, which highlight the importance of data scaling for effective model scaling.

Abstract

While the pretraining of Foundation Models (FMs) for remote sensing (RS) imagery is on the rise, models remain restricted to a few hundred million parameters. Scaling models to billions of parameters has been shown to yield unprecedented benefits including emergent abilities, but requires data scaling and computing resources typically not available outside industry R&D labs. In this work, we pair high-performance computing resources including Frontier supercomputer, America's first exascale system, and high-resolution optical RS data to pretrain billion-scale FMs. Our study assesses performance of different pretrained variants of vision Transformers across image classification, semantic segmentation and object detection benchmarks, which highlight the importance of data scaling for effective model scaling. Moreover, we discuss construction of a novel TIU pretraining dataset, model initialization, with data and pretrained models intended for public release. By discussing technical challenges and details often lacking in the related literature, this work is intended to offer best practices to the geospatial community toward efficient training and benchmarking of larger FMs.

Paper Structure

This paper contains 15 sections, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methods
  4. Pretraining datasets
  5. OReole-MR Experiments
  6. OReole-HR experiments
  7. Conclusion
  8. Appendix
  9. Pretraining implementation details
  10. Image classification experiments details
  11. Object detection experiments details
  12. Semantic segmentation experiments details
  13. OReole-HR experiments details
  14. OReole-MR - additional results
  15. OReole-HR additional results

Figures (5)

  • Figure 1: Illustration of characteristics of the TIU dataset.
  • Figure 2: Convergence curves for object detection configurations, and finetuning results for object detection and semantic segmentation experiments with varying training set sizes.
  • Figure 3: Examples of detections provided by our OReole-MR ViT-B(top) and ViT-G(1B) (bottom) model variants for composing the test set of the DIOR-R dataset.
  • Figure 4: Loss curves for OReole-HR models pretrained from scratch vs with OReole-MR pretrained weights inflated with an extra randomly initialized 4th band.
  • Figure 5: Qualitative example comparing BFE outputs by a ViT-H pretrained in TIU (in blue) compared to a ViT-B pretrained on ORB only (in pink).