Towards Stable Machine Learning Model Retraining via Slowly Varying Sequences
Dimitris Bertsimas, Vassilis Digalakis, Yu Ma, Phevos Paschalidis
TL;DR
This work tackles unstable ML retraining by proposing Slowly Varying ML Sequences (SVML) that explicitly balance predictive power and structural stability across data batches. It grounds the approach in a model-agnostic framework with a Pareto-optimality guarantee, supported by a tractable MIO-based algorithm and a polynomial-time restricted formulation that reduces to a shortest-path problem. The authors demonstrate theoretically and empirically that modest sacrifices in accuracy yield substantial gains in stability and interpretability, validated across regression, decision trees, boosted trees, and neural networks in healthcare, vision, and language settings, including a hospital deployment. The practical impact lies in enabling trustworthy, deployable retraining pipelines that preserve analytical insights while maintaining competitive predictive performance. Future work aims to handle anticipated distribution shifts and incorporate explicit cost considerations into the stability framework.
Abstract
We consider the problem of retraining machine learning (ML) models when new batches of data become available. Existing approaches greedily optimize for predictive power independently at each batch, without considering the stability of the model's structure or analytical insights across retraining iterations. We propose a model-agnostic framework for finding sequences of models that are stable across retraining iterations. We develop a mixed-integer optimization formulation that is guaranteed to recover Pareto optimal models (in terms of the predictive power-stability trade-off) with good generalization properties, as well as an efficient polynomial-time algorithm that performs well in practice. We focus on retaining consistent analytical insights-which is important to model interpretability, ease of implementation, and fostering trust with users-by using custom-defined distance metrics that can be directly incorporated into the optimization problem. We evaluate our framework across models (regression, decision trees, boosted trees, and neural networks) and application domains (healthcare, vision, and language), including deployment in a production pipeline at a major US hospital. We find that, on average, a 2% reduction in predictive power leads to a 30% improvement in stability.