Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamic Bayesian Optimization Framework for Instruction Tuning in Partial Differential Equation Discovery

Junqi Qu, Yan Zhang, Shangqian Gao, Shibo Li

TL;DR

The paper addresses instruction brittleness in LLM-based equation discovery by treating prompt engineering as a dynamic, discrete optimization problem. NeuroSym-BO uses a closed-loop three-agent system where a Bayesian Optimization agent selects among a bank of 100 reasoning strategies to assemble adaptive prompts for the LLM, guided by a numerical evaluator that fits coefficients and balances accuracy with parsimony. Empirically, the approach yields higher recovery rates and more parsimonious symbolic forms across five PDE benchmarks (e.g., Fisher approaching $R^2=0.9999$) compared to a fixed-prompt baseline, demonstrating improved sample efficiency and robustness. This framework significantly advances automated scientific discovery by integrating adaptive prompt design with rigorous numerical feedback, with potential extensions to higher-dimensional or more complex dynamical systems.

Abstract

Large Language Models (LLMs) show promise for equation discovery, yet their outputs are highly sensitive to prompt phrasing, a phenomenon we term instruction brittleness. Static prompts cannot adapt to the evolving state of a multi-step generation process, causing models to plateau at suboptimal solutions. To address this, we propose NeuroSymBO, which reframes prompt engineering as a sequential decision problem. Our method maintains a discrete library of reasoning strategies and uses Bayesian Optimization to select the optimal instruction at each step based on numerical feedback. Experiments on PDE discovery benchmarks show that adaptive instruction selection significantly outperforms fixed prompts, achieving higher recovery rates with more parsimonious solutions.

Dynamic Bayesian Optimization Framework for Instruction Tuning in Partial Differential Equation Discovery

TL;DR

The paper addresses instruction brittleness in LLM-based equation discovery by treating prompt engineering as a dynamic, discrete optimization problem. NeuroSym-BO uses a closed-loop three-agent system where a Bayesian Optimization agent selects among a bank of 100 reasoning strategies to assemble adaptive prompts for the LLM, guided by a numerical evaluator that fits coefficients and balances accuracy with parsimony. Empirically, the approach yields higher recovery rates and more parsimonious symbolic forms across five PDE benchmarks (e.g., Fisher approaching ) compared to a fixed-prompt baseline, demonstrating improved sample efficiency and robustness. This framework significantly advances automated scientific discovery by integrating adaptive prompt design with rigorous numerical feedback, with potential extensions to higher-dimensional or more complex dynamical systems.

Abstract

Large Language Models (LLMs) show promise for equation discovery, yet their outputs are highly sensitive to prompt phrasing, a phenomenon we term instruction brittleness. Static prompts cannot adapt to the evolving state of a multi-step generation process, causing models to plateau at suboptimal solutions. To address this, we propose NeuroSymBO, which reframes prompt engineering as a sequential decision problem. Our method maintains a discrete library of reasoning strategies and uses Bayesian Optimization to select the optimal instruction at each step based on numerical feedback. Experiments on PDE discovery benchmarks show that adaptive instruction selection significantly outperforms fixed prompts, achieving higher recovery rates with more parsimonious solutions.
Paper Structure (22 sections, 8 equations, 2 figures, 1 table, 1 algorithm)

This paper contains 22 sections, 8 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. PDE Discovery
  4. Bayesian Optimization(BO)
  5. Methodology
  6. Dynamic Prompt Construction
  7. Bayesian Optimization for Strategy Selection
  8. Numerical Evaluation with Parsimony Penalty
  9. Related Work
  10. Experiments
  11. Conclusion
  12. Algorithm Overview
  13. Dataset and Equation Details
  14. Benchmarks from LLM4ED
  15. Burgers' Equation.
  16. ...and 7 more sections

Figures (2)

  • Figure 1: Overview of NeuroSym-BO. A Bayesian Optimizer (1) selects instruction strategies that are assembled into prompts (2) with historical context. The LLM generates candidate equations for evaluation (3), and feedback updates both the history and the optimizer (4), forming a closed-loop system.
  • Figure 2: Optimization trajectories (test $R^2$) across five PDEs: red solid: NeuroSym-BO; blue dashed: Fixed Prompt baseline. Shaded regions indicate $\pm$SEM over 5 trials. Our method exhibits step-wise improvements from adaptive strategy switching, while the baseline plateaus early.