Deadline-Aware Online Scheduling for LLM Fine-Tuning with Spot Market Predictions
Linggao Kong, Yuedong Xu, Lei Jiao, Chuan Xu
TL;DR
This work tackles the high cost of fine-tuning large language models by leveraging a hybrid mix of on-demand and spot GPUs under deadline constraints. It introduces a prediction-based allocation framework (AHAP) built on Committed Horizon Control, a non-predictive fallback (AHANP), and an online policy selector that learns the best policy from a rich pool of configurations. Theoretical guarantees show tighter performance with improved prediction accuracy and a sublinear regret bound for policy selection; empirically, the approach consistently outperforms baselines and adapts to varying market dynamics, achieving substantial cost and utility gains. The framework is demonstrated on LoRA-based LLM fine-tuning, highlighting practical impact for cost-efficient, deadline-aware cloud training in volatile spot markets.
Abstract
As foundation models grow in size, fine-tuning them becomes increasingly expensive. While GPU spot instances offer a low-cost alternative to on-demand resources, their volatile prices and availability make deadline-aware scheduling particularly challenging. We tackle this difficulty by using a mix of spot and on-demand instances. Distinctively, we show the predictability of prices and availability in a spot instance market, the power of prediction in enabling cost-efficient scheduling and its sensitivity to estimation errors. An integer programming problem is formulated to capture the use of mixed instances under both the price and availability dynamics. We propose an online allocation algorithm with prediction based on the committed horizon control approach that leverages a \emph{commitment level} to enforce the partial sequence of decisions. When this prediction becomes inaccurate, we further present a complementary online algorithm without predictions. An online policy selection algorithm is developed that learns the best policy from a pool constructed by varying the parameters of both algorithms. We prove that the prediction-based algorithm achieves tighter performance bounds as prediction error decreases, while the policy selection algorithm possesses a regret bound of $\mathcal{O}(\sqrt{T})$. Experimental results demonstrate that our online framework can adaptively select the best policy under varying spot market dynamics and prediction quality, consistently outperforming baselines and improving utility by up to 54.8\%.