ControlLM: Crafting Diverse Personalities for Language Models

TL;DR

ControlLM presents a framework for inference-time, fine-grained control of LLM personalities by learning and applying activation-space directions corresponding to human-like traits. The method uses an Extraction Phase to build trait vectors and a Control Phase to modulate activations with a tunable factor $ abla$, without retraining. The AutoControlActivate toolkit automates direction discovery, enabling rapid expansion of controllable traits. Empirical results show improved reasoning and language modeling performance, and targeted reductions in sycophancy, across multiple models and benchmarks, highlighting practical potential and ethical considerations for deploying diverse, value-aligned AI personas.

Abstract

As language models continue to scale in size and capability, they display an array of emerging behaviors, both beneficial and concerning. This heightens the need to control model behaviors. We hope to be able to control the personality traits of language models at the inference-time so as to have various character features, on top of which the requirements of different types of tasks can be met. Personality is a higher-level and more abstract behavioral representation for language models. We introduce ControlLM, which leverages differential activation patterns, derived from contrasting behavioral prompts in the model's latent space, to influence the model's personality traits at inference. This approach allows for the precise, real-time adjustment of model behavior. First, we demonstrate ControlLM's capacity to elicit diverse persona behaviors without any training, while precision control allows personality traits to closely match average human values. Subsequently, we showcase improved reasoning and question answering through selective amplification of beneficial attributes like conscientiousness and friendliness. We hope that this work will inspire research on controlling human-like behaviors of language models and provide insights for future research. Our code is publicly available at: https://github.com/wengsyx/ControlLM.

Figures (7)

• Figure 1: ControlLM adjusts LM's outputs by modulating key personality traits, enabling the real-time craftion of diverse personas with fine-grained behavioral control.
• Figure 2: The ControlLM framework. The (a) and the (b) illustrate the Extraction Phase and Control Phase of ControlLM, respectively. The (c) shows an example of adding Conscientiousness and Openness personalities and reducing Neuroticism personality.
• Figure 3: ControlLM performs fine-grained personality manipulation by adjusting $\gamma$ values when oriented towards different controlled personality targets.
• Figure 4: Controlling the Language model's (Llama-2-Chat 70B) Warmth Personality by adjusting $\gamma$ values in Language Modeling Tasks. The results show that appropriately reducing the Warmth personality of the LM can improve the language modeling ability of the language model. To some extent, ControlLM reduces the alignment tax phenomenon caused by the RLHF process.
• Figure 5: Controlling the Warmth Personality of Language Models in Alpaca-Eval Tasks. It enhance its general question answering capability.