Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FANoS: Friction-Adaptive Nosé--Hoover Symplectic Momentum for Stiff Objectives

Nalin Dhiman

TL;DR

FANoS addresses optimization in stiff landscapes by integrating a momentum-based update with a Nosé–Hoover–inspired friction thermostat and a semi-implicit (symplectic) discretization, optionally augmented by a diagonal RMS preconditioner. The core idea is to regulate kinetic energy through a learned scalar friction variable $\zeta$ updated via kinetic-energy feedback, aiming to stabilize oscillatory directions while adapting the effective step-size. Empirically, FANoS shows mixed results: it improves over some unclipped baselines on Rosenbrock-100D under a fixed budget but is outperformed by AdamW with gradient clipping and underperforms on ill-conditioned quadratics and PINN warm-starts, with thermostat tracking sensitive to hyperparameters. The study provides a reproducible evaluation framework and clarifies when thermostat-based, structure-preserving updates can be beneficial, highlighting limitations and guiding future tuning and design of optimizer components for stiff nonconvex regimes.

Abstract

We study a physics-inspired optimizer, \emph{FANoS} (Friction-Adaptive Nosé--Hoover Symplectic momentum), which combines (i) a momentum update written as a discretized second-order dynamical system, (ii) a Nosé--Hoover-like thermostat variable that adapts a scalar friction coefficient using kinetic-energy feedback, and (iii) a semi-implicit (symplectic-Euler) integrator, optionally with a diagonal RMS preconditioner. The method is motivated by structure-preserving integration and thermostat ideas from molecular dynamics, but is used here purely as an optimization heuristic. We provide the algorithm and limited theoretical observations in idealized settings. On the deterministic Rosenbrock-100D benchmark with 3000 gradient evaluations, FANoS-RMS attains a mean final objective value of $1.74\times 10^{-2}$, improving substantially over unclipped AdamW ($48.50$) and SGD+momentum ($90.76$) in this protocol. However, AdamW with gradient clipping is stronger, reaching $1.87\times 10^{-3}$, and L-BFGS reaches $\approx 4.4\times 10^{-10}$. On ill-conditioned convex quadratics and in a small PINN warm-start suite (Burgers and Allen--Cahn), the default FANoS configuration underperforms AdamW and can be unstable or high-variance. Overall, the evidence supports a conservative conclusion: FANoS is an interpretable synthesis of existing ideas that can help on some stiff nonconvex valleys, but it is not a generally superior replacement for modern baselines, and its behavior is sensitive to temperature-schedule and hyperparameter choices.

FANoS: Friction-Adaptive Nosé--Hoover Symplectic Momentum for Stiff Objectives

TL;DR

FANoS addresses optimization in stiff landscapes by integrating a momentum-based update with a Nosé–Hoover–inspired friction thermostat and a semi-implicit (symplectic) discretization, optionally augmented by a diagonal RMS preconditioner. The core idea is to regulate kinetic energy through a learned scalar friction variable updated via kinetic-energy feedback, aiming to stabilize oscillatory directions while adapting the effective step-size. Empirically, FANoS shows mixed results: it improves over some unclipped baselines on Rosenbrock-100D under a fixed budget but is outperformed by AdamW with gradient clipping and underperforms on ill-conditioned quadratics and PINN warm-starts, with thermostat tracking sensitive to hyperparameters. The study provides a reproducible evaluation framework and clarifies when thermostat-based, structure-preserving updates can be beneficial, highlighting limitations and guiding future tuning and design of optimizer components for stiff nonconvex regimes.

Abstract

We study a physics-inspired optimizer, \emph{FANoS} (Friction-Adaptive Nosé--Hoover Symplectic momentum), which combines (i) a momentum update written as a discretized second-order dynamical system, (ii) a Nosé--Hoover-like thermostat variable that adapts a scalar friction coefficient using kinetic-energy feedback, and (iii) a semi-implicit (symplectic-Euler) integrator, optionally with a diagonal RMS preconditioner. The method is motivated by structure-preserving integration and thermostat ideas from molecular dynamics, but is used here purely as an optimization heuristic. We provide the algorithm and limited theoretical observations in idealized settings. On the deterministic Rosenbrock-100D benchmark with 3000 gradient evaluations, FANoS-RMS attains a mean final objective value of , improving substantially over unclipped AdamW () and SGD+momentum () in this protocol. However, AdamW with gradient clipping is stronger, reaching , and L-BFGS reaches . On ill-conditioned convex quadratics and in a small PINN warm-start suite (Burgers and Allen--Cahn), the default FANoS configuration underperforms AdamW and can be unstable or high-variance. Overall, the evidence supports a conservative conclusion: FANoS is an interpretable synthesis of existing ideas that can help on some stiff nonconvex valleys, but it is not a generally superior replacement for modern baselines, and its behavior is sensitive to temperature-schedule and hyperparameter choices.
Paper Structure (37 sections, 1 theorem, 12 equations, 5 figures, 4 tables, 1 algorithm)

This paper contains 37 sections, 1 theorem, 12 equations, 5 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. What this paper is (and is not).
  3. Contributions.
  4. Related work
  5. Momentum and adaptive first-order methods.
  6. Second-order and quasi-Newton methods.
  7. Thermostats and stochastic-gradient MCMC.
  8. Geometric integration.
  9. FANoS: dynamics and discrete algorithm
  10. Continuous-time motivation (idealized)
  11. Discrete update (implemented)
  12. Hyperparameters in the released implementation.
  13. Gradient clipping.
  14. Algorithm
  15. Theory in a simplified setting
  16. ...and 22 more sections

Key Result

Theorem 1

Apply the semi-implicit update The update matrix has determinant $1$ and eigenvalues on the unit circle when $0 < h\omega < 2$. In contrast, explicit Euler applied to the same system yields an update matrix with spectral radius strictly greater than $1$ for any $h>0$.

Figures (5)

  • Figure 1: Rosenbrock-100D learning-rate sweep (mean final loss; 95% bootstrap CI; 10 seeds; 3000 gradient evaluations).
  • Figure 2: Ill-conditioned quadratic sweep (mean final loss; 95% bootstrap CI; 3 seeds; 3000 gradient evaluations). Learning rates are fixed per method (no tuning per condition number).
  • Figure 3: PINN warm-start suite: distribution of final loss after L-BFGS refinement (5 seeds). Lower is better.
  • Figure 4: Thermostat diagnostics: friction coefficient $\zeta$ (top) and temperature proxies (bottom) for two Rosenbrock regimes over 3000 steps.
  • Figure 5: Rosenbrock ablations (mean final loss; 95% bootstrap CI). Interpret cautiously due to hyperparameter interactions.

Theorems & Definitions (4)

  • Remark 1: Scope of the continuous-time view
  • Theorem 1: Linear stability of symplectic Euler on the harmonic oscillator
  • proof
  • Remark 2: Interpretation