Plasticity vs. Rigidity: The Impact of Low-Rank Adapters on Reasoning on a Micro-Budget

TL;DR

This study probes whether strong mathematical reasoning can emerge in small language models under extreme compute limits by applying reinforcement learning with verifiable rewards (RLVR) to parameter-efficient LoRA adapters on a micro-budget. A diverse set of models ≤$1.5\text{B}$ was trained for about $24$ hours on a single $NVIDIA\,A40$, varying LoRA rank $r$ across $\{8,64,256\}$ and using Group Relative Policy Optimization to learn reasoning strategies; results show a clear plasticity–rigidity dichotomy: high-rank adapters unlock substantial reasoning plasticity in generalist models (achieving state-of-the-art metrics such as $40.0\%$ Pass@1 on AIME24 and $70.0\%$ Pass@16 in some cases), while heavily math-aligned models can experience destructive interference and performance collapse. The findings suggest that maximizing latent reasoning in pre-existing generalist manifolds via high-rank adaptation is a cost-effective route to budgeting reasoning capabilities, with implications for scaling laws, warm-start strategies, and algorithmic choices in low-resource RL for reasoning. Future work should explore larger architectures, alternative optimization schemes, and brief warm-up phases to further stabilize and generalize micro-budget reasoning.

Abstract

Recent advances in mathematical reasoning typically rely on massive scale, yet the question remains: can strong reasoning capabilities be induced in small language models ($\leq1.5\text{B}$) under extreme constraints? We investigate this by training models on a single A40 GPU (48GB) for under 24 hours using Reinforcement Learning with Verifiable Rewards (RLVR) and Low-Rank Adaptation (LoRA). We find that the success of this ``micro-budget" regime depends critically on the interplay between adapter capacity and model initialization. While low-rank adapters ($r=8$) consistently fail to capture the complex optimization dynamics of reasoning, high-rank adapters ($r=256$) unlock significant plasticity in standard instruction-tuned models. Our best result achieved an impressive 40.0\% Pass@1 on AIME 24 (an 11.1\% absolute improvement over baseline) and pushed Pass@16 to 70.0\%, demonstrating robust exploration capabilities. However, this plasticity is not universal: while instruction-tuned models utilized the budget to elongate their chain-of-thought and maximize reward, heavily math-aligned models suffered performance collapse, suggesting that noisy, low-budget RL updates can act as destructive interference for models already residing near a task-specific optimum.

Figures (4)

• Figure 1: Low-Rank Adaptation (LoRA) mechanism. By optimizing only the low-rank matrices $A$ and $B$, we significantly reduce memory usage while retaining the ability to learn task-specific features.
• Figure 2: Evolution of Mean Reward. High-rank adapters ($r=256$, blue lines) drive consistent learning for generalist models (Top Row), whereas models that underwent less conventional training (Bottom Row) struggle to optimize the reward signal.
• Figure 3: Validation Accuracy on MATH500. Successful models (Top Row) show correlation between training reward and validation score. Qwen-Math (Bottom Center) exhibits "specialist collapse" at high ranks.
• Figure 4: Evolution of Response Length. Plastic models (Top Row) dynamically increased their context usage ("thinking") to maximize reward. Rigid models (Bottom Row) failed to adapt or suffered length collapse.