FIRE: Multi-fidelity Regression with Distribution-conditioned In-context Learning using Tabular Foundation Models

TL;DR

FIRE addresses the challenge of extreme high-fidelity data scarcity in multi-fidelity regression by leveraging frozen tabular foundation models to perform zero-shot Bayesian inference, producing low-fidelity predictive statistics that condition high-fidelity corrections. It decomposes MF regression into low-fidelity inference, distribution-conditioned residual learning, and additive uncertainty propagation, enabling robust corrections without retraining. Empirically, FIRE outperforms seven GP-based or deep-learning MF baselines across 31 benchmarks, with pronounced gains in 2–5% HF data regimes and favorable runtimes. The framework demonstrates the value of conditioning residuals on full distributional information (variance and quantiles) rather than mean alone, offering a scalable, uncertainty-aware solution for tabular, non-nested MF problems.

Abstract

Multi-fidelity (MF) regression often operates in regimes of extreme data imbalance, where the commonly-used Gaussian-process (GP) surrogates struggle with cubic scaling costs and overfit to sparse high-fidelity observations, limiting efficiency and generalization in real-world applications. We introduce FIRE, a training-free MF framework that couples tabular foundation models (TFMs) to perform zero-shot in-context Bayesian inference via a high-fidelity correction model conditioned on the low-fidelity model's posterior predictive distributions. This cross-fidelity information transfer via distributional summaries captures heteroscedastic errors, enabling robust residual learning without model retraining. Across 31 benchmark problems spanning synthetic and real-world tasks (e.g., DrivAerNet, LCBench), FIRE delivers a stronger performance-time trade-off than seven state-of-the-art GP-based or deep learning MF regression methods, ranking highest in accuracy and uncertainty quantification with runtime advantages. Limitations include context window constraints and dependence on the quality of the pre-trained TFM's.

Figures (30)

• Figure 1: Overview of FIRE. The method decomposes multi-fidelity regression into a low-fidelity base inference and a residual correction step, conditioned on the base model's statistical and distributional information.
• Figure 2: Overall performance. Elo rating and average rank for (a) accuracy performance (NRMSE) and (b) uncertainty calibration (NLL) across 31 problems. FIRE outperforms the other baselines in both accuracy and uncertainty.
• Figure 3: Pareto frontier analysis. Pareto plots of accuracy (Left) and uncertainty (Right) vs. runtime. FIRE dominates the Pareto frontier for both metrics, being the fastest and most accurate.
• Figure 4: Ablation study of FIRE components. We compare the Elo rating of various configurations against NARGP, the strongest GP baseline. The full FIRE framework (Top), which combines residual learning with statistical augmentation, significantly outperforms partial implementations.
• Figure 5: Performance (NRMSE and NLL) for representative synthetic (Currin, HD 50), physics-based (Car, Concrete), and HPO (Higgs) tasks. FIRE maintains top-ranked accuracy and uncertainty calibration in the extreme low-data regime (2–5% HF ratio), where baseline performance degrades.

Theorems & Definitions (17)

• Theorem 7.2: Bayes Risk Monotonicity
• Lemma 7.4: Quantile Embedding
• Proposition 7.6: Quantile Conditioning Reduces Risk Under Misspecification
• Theorem 3.1: PFN Posterior Approximation; muller2021transformers
• proof
• Proposition 3.2: Additive Variance
• proof
• Proposition 3.4: Data Processing Inequality for Residuals
• Proposition 3.5: Variance-Residual Coupling
• Corollary 3.6: Mean-Only Misspecification