Dynamic Meta-Learning for Adaptive XGBoost-Neural Ensembles
Arthur Sedek
TL;DR
This work presents Dynamic Meta-Learning (DML), an adaptive ensemble that combines XGBoost and a neural network using a meta-learner guided by uncertainty estimates and cross-model feature importances. The framework employs Monte Carlo Dropout for neural uncertainty, tree-variance for XGBoost confidence, and Integrated Gradients alongside native feature importances to enrich meta-features. A three-phase training procedure and a softmax-based meta-learner enable 3-way model selection (XGBoost, neural network, or hybrid) on a per-sample basis, achieving superior performance on the California Housing dataset with notable gains in RMSE and $R^2$. The approach improves predictive accuracy while enhancing interpretability through explicit meta-features and model selection rationales, though it introduces additional computational overhead primarily from uncertainty estimation. This work advances adaptive, uncertainty-aware ensemble methods with practical implications for domains requiring robust and flexible predictive modeling.
Abstract
This paper introduces a novel adaptive ensemble framework that synergistically combines XGBoost and neural networks through sophisticated meta-learning. The proposed method leverages advanced uncertainty quantification techniques and feature importance integration to dynamically orchestrate model selection and combination. Experimental results demonstrate superior predictive performance and enhanced interpretability across diverse datasets, contributing to the development of more intelligent and flexible machine learning systems.