Effective Benchmarks for Optical Turbulence Modeling
Christopher Jellen, Charles Nelson, Cody Brownell, John Burkhardt
TL;DR
The paper addresses the lack of standardized benchmarks for modeling optical turbulence strength, quantified as $C_n^2$, particularly in boundary-layer propagation. It introduces otbench, a Python framework that links field data sets (MLO and USNA), regression and forecasting tasks, and evaluation metrics into reusable benchmarks, with built-in baselines such as Persistence, Climatology, macro-meteorological models, GBRT, and an RNN. Across regression and forecasting benchmarks, data-driven and deep learning baselines show competitive performance, though simple climatology baselines remain strong in some environments with pronounced diurnal cycles, underscoring the value of diverse data sets for robust generalization. The framework is extensible and openly available at the provided GitHub repository, offering a robust platform to compare methods, reproduce results, and extend benchmarks with new campaigns for essential operational applications in communications, directed energy, and imaging.
Abstract
Optical turbulence presents a significant challenge for communication, directed energy, and imaging systems, especially in the atmospheric boundary layer. Effective modeling of optical turbulence strength is critical for the development and deployment of these systems. The lack of standard evaluation tools, especially long-term data sets, modeling tasks, metrics, and baseline models, prevent effective comparisons between approaches and models. This reduces the ease of reproducing results and contributes to over-fitting on local micro-climates. Performance characterized using evaluation metrics provides some insight into the applicability of a model for predicting the strength of optical turbulence. However, these metrics are not sufficient for understanding the relative quality of a model. We introduce the \texttt{otbench} package, a Python package for rigorous development and evaluation of optical turbulence strength prediction models. The package provides a consistent interface for evaluating optical turbulence models on a variety of benchmark tasks and data sets. The \texttt{otbench} package includes a range of baseline models, including statistical, data-driven, and deep learning models, to provide a sense of relative model quality. \texttt{otbench} also provides support for adding new data sets, tasks, and evaluation metrics. The package is available at \url{https://github.com/cdjellen/otbench}.