Precise Attribute Intensity Control in Large Language Models via Targeted Representation Editing

TL;DR

This work tackles the problem of precise, continuous control of attribute intensity in LLM outputs, addressing gaps where existing alignment methods only steer outputs directionally. The authors introduce Pre-Control, a three-part framework that reframes alignment as target-reaching, trains a lightweight TD(lambda) value function to predict final attribute scores from partial generations, and performs gradient-based interventions in the model's hidden space at test time to reach exact targets. They demonstrate the approach on LLaMA-3.2-3b and Phi-4-mini, achieving high accuracy in matching user-specified attribute intensities and enabling efficient Pareto frontier exploration and controllable distillation. The method yields significant gains in both multi-objective control quality and computational efficiency, suggesting practical impact for customizable and scalable LLM alignment. The work further contributes a Pareto frontier interpolation strategy and provides open-source code to facilitate adoption and extension.

Abstract

Precise attribute intensity control--generating Large Language Model (LLM) outputs with specific, user-defined attribute intensities--is crucial for AI systems adaptable to diverse user expectations. Current LLM alignment methods, however, typically provide only directional or open-ended guidance, failing to reliably achieve exact attribute intensities. We address this limitation with three key designs: (1) reformulating precise attribute intensity control as a target-reaching problem, rather than simple maximization; (2) training a lightweight value function via temporal-difference learning to predict final attribute intensity scores from partial generations, thereby steering LLM outputs; and (3) employing gradient-based interventions on hidden representations to navigate the model precisely towards specific attribute intensity targets. Our method enables fine-grained, continuous control over attribute intensities, moving beyond simple directional alignment. Experiments on LLaMA-3.2-3b and Phi-4-mini confirm our method's ability to steer text generation to user-specified attribute intensities with high accuracy. Finally, we demonstrate efficiency enhancements across three downstream tasks: preference data synthesis, Pareto frontier approximation and optimization, and distillation of aligned behaviors for intervention-free inference. Our code is available on https://github.com/Pre-Control/pre-control

Figures (11)

• Figure 1: Overview of Pre-Control. For precise attribute intensity control, we formalize it as a target-reaching problem. We train a value function on the hidden space of an LLM to predict the attribute-wise reward. During test-time, we leverage this value function to guide the LLM generating text towards the specified attribute scores through targeted representation editing.
• Figure 2: Iterative intervention results.
• Figure 3: Pareto frontier comparison.
• Figure 4: Attribute-wise reward distribution.
• Figure 5: LLaMA-3.2-3b attribute intensity distributions. Base is the before-intervene distribution, ours is the after-intervene distribution.