Scaling Deep Learning Research with Kubernetes on the NRP Nautilus HyperCluster

TL;DR

This work explores utilizing the NRP Nautilus HyperCluster to automate and scale deep learning model training for three separate applications of DNNs, including overhead object detection, burned area segmentation, and deforestation detection.

Abstract

Throughout the scientific computing space, deep learning algorithms have shown excellent performance in a wide range of applications. As these deep neural networks (DNNs) continue to mature, the necessary compute required to train them has continued to grow. Today, modern DNNs require millions of FLOPs and days to weeks of training to generate a well-trained model. The training times required for DNNs are oftentimes a bottleneck in DNN research for a variety of deep learning applications, and as such, accelerating and scaling DNN training enables more robust and accelerated research. To that end, in this work, we explore utilizing the NRP Nautilus HyperCluster to automate and scale deep learning model training for three separate applications of DNNs, including overhead object detection, burned area segmentation, and deforestation detection. In total, 234 deep neural models are trained on Nautilus, for a total time of 4,040 hours

Figures (8)

• Figure 1: Sample Scenes from the Datasets selected for Overhead Detection with Transformers: RarePlanes, DOTA, and XView
• Figure 2: Example of ground-truth polygons for Burned Area Mapping: red bounding boxes drawn around the burned area polygons
• Figure 3: Examples of rasters to be downloaded for Burned Area Mapping. Each orange square represents a raster.
• Figure 4: Example chips used for training Burned Area Mapping models
• Figure 5: The Amazon Biome (top) is outlined by the yellow boundary and the conservation units are highlighted in red. Color-shifted infrared from chips for 2020 (left) and 2021 (right) highlight deforestation changes up close.