How Does Prefix Matter in Reasoning Model Tuning?

TL;DR

The paper revisits the standard practice of removing boilerplate prefixes during supervised fine-tuning, proposing that short, semantically loaded prefixes can serve as lightweight alignment signals to improve safety and reasoning in large reasoning models. Through a controlled ablation across reasoning, safety, and factuality tasks using three distilled R1 models with LoRA-based PEFT, the authors find that prefix conditioning yields consistent Safe@1 gains on adversarial benchmarks and improved GSM8K reasoning, while effects on coding and factuality are task-dependent. A token-level loss analysis shows prefix tokens carry higher losses and thus larger gradient updates, supporting the view that prefixes can anchor alignment during training. The findings suggest prefix-based conditioning as a scalable, interpretable complement to RLHF and other alignment methods, prompting a rethink of dataset cleaning strategies and their role in model safety and capability development.

Abstract

Recent alignment studies commonly remove introductory boilerplate phrases from supervised fine-tuning (SFT) datasets. This work challenges that assumption. We hypothesize that safety- and reasoning-oriented prefix sentences serve as lightweight alignment signals that can guide model decoding toward safer and more coherent responses. To examine this, we fine-tune three R1 series models across three core model capabilities: reasoning (mathematics, coding), safety, and factuality, systematically varying prefix inclusion from 0% to 100%. Results show that prefix-conditioned SFT improves both safety and reasoning performance, yielding up to +6% higher Safe@1 accuracy on adversarial benchmarks (WildJailbreak, StrongReject) and +7% improvement on GSM8K reasoning. However, factuality and coding tasks show marginal or negative effects, indicating that prefix-induced narrowing of the search space benefits structured reasoning. Token-level loss analysis further reveals that prefix tokens such as "revised" and "logically" incur higher gradient magnitudes, acting as alignment anchors that stabilize reasoning trajectories. Our findings suggest that prefix conditioning offers a scalable and interpretable mechanism for improving reasoning safety, serving as an implicit form of alignment that complements traditional reward-based methods.

Figures (1)

• Figure 1: Impact of prefix inclusion on reasoning performance. GSM8K accuracy (%) for the R1-7B model trained with increasing proportions of prefix-included samples with "revised" token added to the prefix.