Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Jacobian-Enhanced Neural Networks

Steven H. Berguin

TL;DR

Jacobian-Enhanced Neural Networks (JENN) address the need for accurate gradient information in surrogates by modifying the neural network training objective to include Jacobian prediction errors, enabling high-accuracy derivatives with fewer data. The paper provides a complete vectorized mathematical derivation, including forward and backward propagation adjusted for partial derivatives, and demonstrates benefits across verification tests and an airfoil optimization application. Key findings show that JENN outperforms standard neural networks in derivative accuracy and surrogate-based optimization, with polishing and handling of incomplete partials further enhancing performance. The work offers a practical, open-source implementation framework that can accelerate CAD surrogate modeling and gradient-driven design optimization, while noting that dimensionality and data-generation costs remain important considerations.

Abstract

Jacobian-Enhanced Neural Networks (JENN) are densely connected multi-layer perceptrons, whose training process is modified to predict partial derivatives accurately. Their main benefit is better accuracy with fewer training points compared to standard neural networks. These attributes are particularly desirable in the field of computer-aided design, where there is often the need to replace computationally expensive, physics-based models with fast running approximations, known as surrogate models or meta-models. Since a surrogate emulates the original model accurately in near-real time, it yields a speed benefit that can be used to carry out orders of magnitude more function calls quickly. However, in the special case of gradient-enhanced methods, there is the additional value proposition that partial derivatives are accurate, which is a critical property for one important use-case: surrogate-based optimization. This work derives the complete theory and exemplifies its superiority over standard neural nets for surrogate-based optimization.

Jacobian-Enhanced Neural Networks

TL;DR

Jacobian-Enhanced Neural Networks (JENN) address the need for accurate gradient information in surrogates by modifying the neural network training objective to include Jacobian prediction errors, enabling high-accuracy derivatives with fewer data. The paper provides a complete vectorized mathematical derivation, including forward and backward propagation adjusted for partial derivatives, and demonstrates benefits across verification tests and an airfoil optimization application. Key findings show that JENN outperforms standard neural networks in derivative accuracy and surrogate-based optimization, with polishing and handling of incomplete partials further enhancing performance. The work offers a practical, open-source implementation framework that can accelerate CAD surrogate modeling and gradient-driven design optimization, while noting that dimensionality and data-generation costs remain important considerations.

Abstract

Jacobian-Enhanced Neural Networks (JENN) are densely connected multi-layer perceptrons, whose training process is modified to predict partial derivatives accurately. Their main benefit is better accuracy with fewer training points compared to standard neural networks. These attributes are particularly desirable in the field of computer-aided design, where there is often the need to replace computationally expensive, physics-based models with fast running approximations, known as surrogate models or meta-models. Since a surrogate emulates the original model accurately in near-real time, it yields a speed benefit that can be used to carry out orders of magnitude more function calls quickly. However, in the special case of gradient-enhanced methods, there is the additional value proposition that partial derivatives are accurate, which is a critical property for one important use-case: surrogate-based optimization. This work derives the complete theory and exemplifies its superiority over standard neural nets for surrogate-based optimization.
Paper Structure (31 sections, 39 equations, 9 figures, 1 table)

This paper contains 31 sections, 39 equations, 9 figures, 1 table.

Table of Contents

  1. Introduction
  2. Overview
  3. Intended Use-Case
  4. Limitations
  5. Previous Work
  6. Mathematical Derivation
  7. Notation
  8. Normalize Training Data
  9. Forward Propagation
  10. Parameter Update
  11. Backward Propagation
  12. Loss Function Derivatives w.r.t. Neural Net Parameters
  13. Loss Function Derivatives w.r.t. to Neural Net Activations
  14. Cost Function Derivatives w.r.t. Parameters
  15. Implementation
  16. ...and 16 more sections

Figures (9)

  • Figure 1: summary of neural net training process
  • Figure 2: Validation results
  • Figure 3: Runtime vs. sample size
  • Figure 4: Optimization history
  • Figure 5: Predictive quality as a function of noisy partials
  • ...and 4 more figures