Disentangling Reasoning Tokens and Boilerplate Tokens For Language Model Fine-tuning

TL;DR

The paper addresses the problem that agent-tuning data contain tokens with distinct roles, notably reasoning versus boilerplate tokens, which are learned at different rates. It introduces SHAD, a shuffle-aware discriminator, which classifies tokens by comparing token-level losses after shuffling input-output pairs, using $LD(y_k)=l_s(y_k)-l_o(y_k)$ to label boilerplate ($LD(y_k)\le0$) or reasoning tokens. Building on SHAD, it proposes Reasoning-highlighted Fine-Tuning (RFT), which applies adaptive weights with $\mathcal{L}_{RFT}= \omega_b \mathcal{L}_b + \omega_r \mathcal{L}_r$, where $\omega_b$ and $\omega_r$ are softmax-normalized by a temperature $\tau$, to prioritize learning of reasoning tokens. Empirical results on ToolBench, APIGen, and ShareGPT-based data show SHAD+RFT improves agent capabilities across held-in and held-out benchmarks, illustrating the value of token-level discrimination and adaptive weighting for fine-tuning large language models. The work highlights a practical approach to mitigate overfitting to boilerplate patterns and enhance multi-step reasoning and tool use in real-world tasks.

Abstract

When using agent-task datasets to enhance agent capabilities for Large Language Models (LLMs), current methodologies often treat all tokens within a sample equally. However, we argue that tokens serving different roles - specifically, reasoning tokens versus boilerplate tokens (e.g., those governing output format) - differ significantly in importance and learning complexity, necessitating their disentanglement and distinct treatment. To address this, we propose a novel Shuffle-Aware Discriminator (SHAD) for adaptive token discrimination. SHAD classifies tokens by exploiting predictability differences observed after shuffling input-output combinations across samples: boilerplate tokens, due to their repetitive nature among samples, maintain predictability, whereas reasoning tokens do not. Using SHAD, we propose the Reasoning-highlighted Fine-Tuning (RFT) method, which adaptively emphasizes reasoning tokens during fine-tuning, yielding notable performance gains over common Supervised Fine-Tuning (SFT).

Figures (10)

• Figure 1: Examples of reasoning tokens (green) and boilerplate tokens (yellow and blue). Boilerplate tokens can be further categorized into format tokens (yellow) and template-connecting tokens (blue).
• Figure 2: Loss changes for different types of tokens in the sampled test data that the model fails to answer for the regular SFT training.
• Figure 3: Illustration of the SHAD method, which classifies tokens through three steps. In step 1, a small subset of the data is sampled, and the output of the sampled data is shuffled. In step 2, the LLM is tuned using the shuffled data. In step 3, tokens are classified by comparing the prediction losses between the tuned and original models.
• Figure 4: Example of shuffled data. After shuffling, the assistant's responses no longer correspond to the original queries. However, some tokens (boilerplate tokens, red) remain semantically similar to the original response and are therefore predictable. In contrast, reasoning tokens (green) no longer align with the query, resulting in noise. Note that 'Action' and 'Action Input' are directly copied from 'Thought' and could be considered as non-reasoning.
• Figure 5: Case study of tokens classified by SHAD. The blue regions represent reasoning tokens, identified by an increase in loss on the model tuned with shuffled data compared to the original model. In contrast, the brown regions indicate boilerplate tokens, characterized by a decrease in loss on the tuned model.