Towards Sobolev Pruning

TL;DR

This work addresses the need for fast, uncertainty-aware surrogates for high-dimensional stochastic models by combining Interval Adjoint Significance Analysis (IASA) pruning with Sobolev Training. A large neural network is pruned via IASA to preserve sensitivity information, then fine-tuned using Sobolev Training to recover first- and second-order derivative accuracy, formalized through the loss $L = ||y - f_{\vartheta}(x)||_2^2 + \lambda ||\nabla_x y - \nabla_x f_{\vartheta}(x)||_2^2$. The approach is validated on Gaussian basket option pricing (Bachelier model and correlated baskets), showing that Delta and Gamma accuracy can be recovered or even surpassed after Sobolev fine-tuning, while significantly reducing model size. The authors emphasize the generality of the method beyond finance, its use of smoothing to enable pathwise derivatives, and the potential for extending pruning to edge-level analyses and higher-order sensitivity recovery in future work.

Abstract

The increasing use of stochastic models for describing complex phenomena warrants surrogate models that capture the reference model characteristics at a fraction of the computational cost, foregoing potentially expensive Monte Carlo simulation. The predominant approach of fitting a large neural network and then pruning it to a reduced size has commonly neglected shortcomings. The produced surrogate models often will not capture the sensitivities and uncertainties inherent in the original model. In particular, (higher-order) derivative information of such surrogates could differ drastically. Given a large enough network, we expect this derivative information to match. However, the pruned model will almost certainly not share this behavior. In this paper, we propose to find surrogate models by using sensitivity information throughout the learning and pruning process. We build on work using Interval Adjoint Significance Analysis for pruning and combine it with the recent advancements in Sobolev Training to accurately model the original sensitivity information in the pruned neural network based surrogate model. We experimentally underpin the method on an example of pricing a multidimensional Basket option modelled through a stochastic differential equation with Brownian motion. The proposed method is, however, not limited to the domain of quantitative finance, which was chosen as a case study for intuitive interpretations of the sensitivities. It serves as a foundation for building further surrogate modelling techniques considering sensitivity information.

Figures (9)

• Figure 1: Visualization of Sobolev Training.
• Figure 2: Computational graphs of two layer NN to evaluate primal values and interval adjoints. The inputs, intermediate variables, hidden node values and output are shown inside the circles along with the mathematical operator or the activation funtion to be applied on incoming variables. Forward pass is shown with black arrows while blue arrows and the blue values show the reverse pass and the partial derivatives.
• Figure 3: Baseline results of ML with a basic MLP.
• Figure 4: Smoothing functions for payoff $(\cdot)^{+}$, where smoothing width $w=0.05$.
• Figure 5: Results of the pruned model.

Theorems & Definitions (9)

• Definition 1: Sobolev Loss
• Example 3.1
• Definition 2: Forward Price
• Definition 3: European call option payoff
• Definition 4: Option price
• Definition 5: Basket
• Definition 6
• Example 5.1
• Example 5.2