Revealing Behavioral Plasticity in Large Language Models: A Token-Conditional Perspective

TL;DR

This work proposes Token-Conditioned Reinforcement Learning (ToCoRL), a principled framework that leverages RL to internalize this chameleon-like plasticity, transforming transient inference-time adaptations into stable and learnable behavioral patterns.

Abstract

In this work, we reveal that Large Language Models (LLMs) possess intrinsic behavioral plasticity-akin to chameleons adapting their coloration to environmental cues-that can be exposed through token-conditional generation and stabilized via reinforcement learning. Specifically, by conditioning generation on carefully selected token prefixes sampled from responses exhibiting desired behaviors, LLMs seamlessly adapt their behavioral modes at inference time (e.g., switching from step-by-step reasoning to direct answering) without retraining. Based on this insight, we propose Token-Conditioned Reinforcement Learning (ToCoRL), a principled framework that leverages RL to internalize this chameleon-like plasticity, transforming transient inference-time adaptations into stable and learnable behavioral patterns. ToCoRL guides exploration with token-conditional generation and keep enhancing exploitation, enabling emergence of appropriate behaviors. Extensive experiments show that ToCoRL enables precise behavioral control without capability degradation. Notably, we show that large reasoning models, while performing strongly on complex mathematics, can be effectively adapted to excel at factual question answering, which was a capability previously hindered by their step-by-step reasoning patterns.

Figures (6)

• Figure 1: Like chameleons changing their coloration in response to external stimuli (top), language models can adapt their behavior according to the token prefix (bottom). $\oplus$ stands for concatenation.
• Figure 2: Top: Conditioning response generation on a direct-answer prefix enables LRM to switch from step-by-step reasoning to direct knowledge retrieval, revealing a new perspective on language model behavioral plasticity. Middle: ToCoRL leverages token-conditional generation to guide exploration during RL training, inducing the emergence of new factual answering behaviors. Bottom: After an initial direct answer, the ToCoRL-trained model adopts an emergent recalibrative reasoning behavior for factual problems (see \ref{['subsection: Underlying Mechanisms of Performance and Behavioral Emergence']}), while its complex math problem–solving behavior remains unchanged. We provide concrete examples for further illustration in \ref{['fig: detailed demonstration (with example) for token conditional generation and influence of ToCoRL']}.
• Figure 3: For clearer visualization, we plot the curves of these metrics over the first training epoch. Compared to ToCoRL, GRPO extends reasoning length while keeping the original reasoning style. Since unnecessary associations and unverified information persist, longer reasoning yields minor improvements, and response lengths eventually return to their initial levels. Adaptive-Thinking penalizes all reasoning, so evaluation scores rise quickly at first as ineffective context is removed. However, without additional reasoning capacity, the LRM’s potential is underutilized, limiting further gains. Instruct-tuning RL initially shortens reasoning, but the prompt instruction gradually loses control over the behavior.
• Figure 4: Detailed demonstration of the behavior change brought by token-conditional generation and ToCoRL. Concrete queries and responses are provided.
• Figure 5: Since most experiments are conducted with large reasoning models, we only illustrate the procedure in this setting. Moreover, we only show the case where the length of token prefix $k=3$ as an example. That is, besides the <think> token, two initial tokens from direct answer (in orange) are used for constructing token prefix.

Theorems & Definitions (7)

• Theorem 4.1
• Theorem 4.2
• Definition 4.3
• Theorem 4.4
• proof
• proof
• proof