Joint Continual Learning of Local Language Models and Cloud Offloading Decisions with Budget Constraints

TL;DR

This paper tackles the challenge of continual learning for resource-constrained local language models that must regulate cloud offloading under a fixed budget. It introduces DA-GRPO, a constraint-aware extension of Group Relative Policy Optimization, which embeds cloud usage into the per-sample advantage via a dual variable that updates to meet a target budget. By using separate group-relative rewards and costs to form a dual-weighted advantage, DA-GRPO enables the local model to jointly improve task competence and collaboration behavior without external routers or per-task reward tuning. Empirical results on math reasoning and code generation benchmarks show that DA-GRPO achieves higher post-switch accuracy, substantially reduces forgetting, and maintains stable cloud usage across task switches, highlighting its practicality for edge deployments with budgeted cloud resources.

Abstract

Locally deployed Small Language Models (SLMs) must continually support diverse tasks under strict memory and computation constraints, making selective reliance on cloud Large Language Models (LLMs) unavoidable. Regulating cloud assistance during continual learning is challenging, as naive reward-based reinforcement learning often yields unstable offloading behavior and exacerbates catastrophic forgetting as task distributions shift. We propose DA-GRPO, a dual-advantage extension of Group Relative Policy Optimization that incorporates cloud-usage constraints directly into advantage computation, avoiding fixed reward shaping and external routing models. This design enables the local model to jointly learn task competence and collaboration behavior, allowing cloud requests to emerge naturally during post-training while respecting a prescribed assistance budget. Experiments on mathematical reasoning and code generation benchmarks show that DA-GRPO improves post-switch accuracy, substantially reduces forgetting, and maintains stable cloud usage compared to prior collaborative and routing-based approaches.

Figures (10)

• Figure 1: Overview of DA-GRPO: the local SLM produces responses, requests cloud help only when needed, and updates via dual-weighted GRPO, reducing cloud cost and strengthening local performance, while preserving task capabilities.
• Figure 2: Catastrophic forgetting in local SLMs. Sequential fine-tuning results for Qwen2.5-1.5B, 3B, and 7B models evaluated on MATH-500 dataset. After switching to a new task, smaller models (1.5B, 3B) show significant drops in performance on the previous task, while the larger 7B model retains most of its ability. This illustrates the severe forgetting issue for compact LLMs deployed on resource-constrained local devices.
• Figure 3: Testing accuracy on the MATH-lighteval dataset under task switches. Our method exhibits the smallest performance drop after transitioning from math-task training to other task groups.
• Figure 4: Comparison of dual variable $\lambda$ trajectories and collaboration ratios for Qwen-2.5B and Llama-3.2B across training iterations for various initialization values.
• Figure 5: Adaptive dual advantage under varying collaboration target $\tau$. Background colors indicate different target $\tau$ values.

Theorems & Definitions (4)

• Proposition A.2: Unbiased dual-weighted group gradient estimator
• proof
• Proposition B.1: Fixed point of the dual update enforces target collaboration
• proof