Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Compressing Vision Transformers in Geospatial Transfer Learning with Manifold-Constrained Optimization

Thomas Snyder, H. Lexie Yang, Stefan Schnake, Steffen Schotthöfer

TL;DR

This work leverages manifold-constrained optimization framework DLRT to compress large vision transformer-based geospatial foundation models during transfer learning, and shows that the method outperforms of-the-shelf low-rank methods as LoRA.

Abstract

Deploying geospatial foundation models on resource-constrained edge devices demands compact architectures that maintain high downstream performance. However, their large parameter counts and the accuracy loss often induced by compression limit practical adoption. In this work, we leverage manifold-constrained optimization framework DLRT to compress large vision transformer-based geospatial foundation models during transfer learning. By enforcing structured low-dimensional parameterizations aligned with downstream objectives, this approach achieves strong compression while preserving task-specific accuracy. We show that the method outperforms of-the-shelf low-rank methods as LoRA. Experiments on diverse geospatial benchmarks confirm substantial parameter reduction with minimal accuracy loss, enabling high-performing, on-device geospatial models.

Compressing Vision Transformers in Geospatial Transfer Learning with Manifold-Constrained Optimization

TL;DR

This work leverages manifold-constrained optimization framework DLRT to compress large vision transformer-based geospatial foundation models during transfer learning, and shows that the method outperforms of-the-shelf low-rank methods as LoRA.

Abstract

Deploying geospatial foundation models on resource-constrained edge devices demands compact architectures that maintain high downstream performance. However, their large parameter counts and the accuracy loss often induced by compression limit practical adoption. In this work, we leverage manifold-constrained optimization framework DLRT to compress large vision transformer-based geospatial foundation models during transfer learning. By enforcing structured low-dimensional parameterizations aligned with downstream objectives, this approach achieves strong compression while preserving task-specific accuracy. We show that the method outperforms of-the-shelf low-rank methods as LoRA. Experiments on diverse geospatial benchmarks confirm substantial parameter reduction with minimal accuracy loss, enabling high-performing, on-device geospatial models.
Paper Structure (18 sections, 6 equations, 1 figure, 5 tables)

This paper contains 18 sections, 6 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Recap: Dynamical Low-Rank Training
  3. Basis Augmentation:
  4. Latent Space Training:
  5. Truncation:
  6. Computational cost:
  7. Convergence:
  8. Error control and robustness:
  9. Low-Rank Compressed Transfer Learning for Geospatial Vision Transformers
  10. Conclusion
  11. Test case descriptions
  12. UCM
  13. AID
  14. NWPU
  15. Data preprocessing
  16. ...and 3 more sections

Figures (1)

  • Figure 1: Geometric interpretation of DLRT. First, we compute the parametrization of the tangent plane $\mathcal{T}_{\mathcal{M}_r}$. Then we compute the projected gradient update with $\nabla_{\widehat{S}}\mathcal{L}$. Lastly, we retract the updated coefficients back onto the manifold $\mathcal{M}_r$. The regularizer $\mathcal{R}$ effectively changes the local curvature of $\mathcal{M}_r$.