Planning vs Reasoning: Ablations to Test Capabilities of LoRA layers
Neel Redkar
TL;DR
This work investigates whether Low-Rank Adaptation (LoRA) layers can enhance reasoning and planning in language models. It introduces HashHop and HashChain Reasoning as deterministic benchmarks to differentiate planning from reasoning capabilities and to measure the impact of LoRA on each. The results indicate reasoning tasks benefit from low-rank representations and can be substantially improved with LoRA, while planning remains more challenging and shows limited gains, leading to the proposal of ELoRA (Entropy LoRA) which improves convergence and performance. Together, these findings suggest a separation of planning and reasoning in evaluation and point toward reasoning-focused, low-rank adaptations as a scalable path for extending latent capabilities, with future work spanning broader benchmarks and entropy-based priors.
Abstract
Low-Rank Adaptation (LoRA) layers have emerged as a promising approach for efficient model fine-tuning, but their capabilities and limitations have not been fully explored. This paper: 1) Investigates the fundamental question of whether LoRA layers are effective at increasing reasoning + planning abilities 2) We introduce HashChain Reasoning, a novel evaluation dataset that deterministically tests reasoning capabilities. Through systematic ablation studies on GPT-2, we demonstrate that reasoning capabilities appear to exist primarily in low-rank spaces and can be effectively enhanced using LoRA layers. The effective rank analysis of trained LoRA matrices reveals a 2-3x lower rank requirement for reasoning tasks compared to planning tasks, giving context on where LoRA layers would be effective. This also provides evidence for reasoning fundamentally preferring low-parameter spaces for generalization.