HyperNOs: Automated and Parallel Library for Neural Operators Research

TL;DR

This work introduces HyperNOs, a PyTorch library that automates the exhaustive hyperparameter optimization of neural operators (NOs) using Ray Tune, enabling parallel, scalable exploration across architectures such as Fourier Neural Operators (FNOs) and Convolutional Neural Operators (CNOs). By supporting fixed-parameter training, multi-dataset concatenation, and multi-resolution training, HyperNOs provides a flexible framework for rapidly identifying high-performing NO configurations on PDE benchmarks, while offering physics-informed options and wrappers to enforce constraints. The authors demonstrate the library on Darcy and Poisson-type problems, various PDE benchmarks including Navier–Stokes, Wave, Euler, transport, and Allen–Cahn, and show substantial performance gains with optimized hyperparameters, sometimes under fixed model complexity. The platform, including default hyperparameters, model builders, and a web application, promotes reproducibility, fair comparisons, and broader adoption of neural operators in scientific computing. The work also outlines future directions to incorporate additional architectures and community contributions, aiming to keep HyperNOs at the forefront of automated NO research.

Abstract

This paper introduces HyperNOs, a PyTorch library designed to streamline and automate the process of exploring neural operators, with a special focus on hyperparameter optimization for comprehensive and exhaustive exploration. Indeed, HyperNOs takes advantage of state-of-the-art optimization algorithms and parallel computing implemented in the Ray-tune library to efficiently explore the hyperparameter space of neural operators. We also implement many useful functionalities for studying neural operators with a user-friendly interface, such as the possibility to train the model with a fixed number of parameters or to train the model with multiple datasets and different resolutions. We integrate Fourier neural operators and convolutional neural operators in our library, achieving state of the art results on many representative benchmarks, demonstrating the capabilities of HyperNOs to handle real datasets and modern architectures. The library is designed to be easy to use with the provided model and datasets, but also to be easily extended to use new datasets and custom neural operator architectures.

Figures (7)

• Figure 1: Representation of a MLP with $L=2$ hidden layers.
• Figure 2: Visual representation of a Fourier Neural Operator.
• Figure 3: Visual representation of a Convolutional Neural Operator with channel multiplier equal to $16$ and initial resolution equal to $64$.
• Figure 4: We plot the loss functions (relative $L^1$ error) with respect to the train set and the test set for both the default and the best hyperparameter configurations (Fig. \ref{['fig:loss_fun']}) and display the distribution of the relative $L^1$ error with histograms (Fig. \ref{['fig:distribution']}) and warm-plot (Fig. \ref{['fig:swarm']}). In Figure \ref{['fig:loss_fun']} the train and the test are almost overlapping, so we can see only one line.
• Figure 5: In this plot we can see the relative $L^1$ error of each hyperparameter configuration with respect to the passed time from the beginning of the simulation.