Evaluating Parameter Efficient Methods for RLVR

TL;DR

This paperAddresses the question of which Parameter-Efficient Fine-Tuning (PEFT) method best supports Reinforcement Learning with Verifiable Rewards (RLVR) for mathematical reasoning. It conducts a large-scale, cross-method evaluation across 12 PEFT variants using the DeepSeek-R1-Distill math benchmark suite, revealing that structural variants such as DoRA, AdaLoRA, and MiSS consistently outperform standard LoRA and can rival or exceed full fine-tuning. The study also uncovers a spectral collapse when using SVD-based initializations (PiSSA, MiLoRA) due to misalignment with RLVR’s off-principal updates, and shows that extreme parameter reduction (VeRA, Rank-1) bottlenecks the model’s reasoning capacity. Across ablations and scaling experiments to a 7B model, the results emphasize maintaining sufficient expressivity and adopting geometry-aware adapters to achieve robust RLVR performance, offering practical guidance for researchers and practitioners.

Abstract

We systematically evaluate Parameter-Efficient Fine-Tuning (PEFT) methods under the paradigm of Reinforcement Learning with Verifiable Rewards (RLVR). RLVR incentivizes language models to enhance their reasoning capabilities through verifiable feedback; however, while methods like LoRA are commonly used, the optimal PEFT architecture for RLVR remains unidentified. In this work, we conduct the first comprehensive evaluation of over 12 PEFT methodologies across the DeepSeek-R1-Distill families on mathematical reasoning benchmarks. Our empirical results challenge the default adoption of standard LoRA with three main findings. First, we demonstrate that structural variants, such as DoRA, AdaLoRA, and MiSS, consistently outperform LoRA. Second, we uncover a spectral collapse phenomenon in SVD-informed initialization strategies (\textit{e.g.,} PiSSA, MiLoRA), attributing their failure to a fundamental misalignment between principal-component updates and RL optimization. Furthermore, our ablations reveal that extreme parameter reduction (\textit{e.g.,} VeRA, Rank-1) severely bottlenecks reasoning capacity. We further conduct ablation studies and scaling experiments to validate our findings. This work provides a definitive guide for advocating for more exploration for parameter-efficient RL methods.

Figures (5)

• Figure 1: Left: Comparison of average accuracy vs. percentage of trainable parameters (log scale) for various parameter efficient methods under our RLVR evaluations. The shaded area represents the performance frontier. Right: Training dynamics showing accuracy reward over training steps for different methods.
• Figure 2: Overview of our evaluation. Left: We systematically evaluate a wide range of parameter-efficient methods categorized. Center: The experimental setup spans diverse base models and is validated on mathematical reasoning datasets. Right: Comprehensive ablation studies are conducted across RLVR loss types, learning rates, LoRA ranks, and batch sizes to ensure the robustness of our findings.
• Figure 3: Left: Normalized magnitude of updates across singular value indices. Center: Cumulative proportion of energy explained by the top-$k$ components. Right: Accuracy reward curves during training, illustrating the performance collapse of SVD-based initializations in the RLVR setting compared to standard baselines.
• Figure 4: Hyperparameters for RLVR training across model scales.
• Figure 5: Accuracy reward of batch size 128 and 32.