Funny or Persuasive, but Not Both: Evaluating Fine-Grained Multi-Concept Control in LLMs

TL;DR

This work presents a formal, model-agnostic framework to evaluate fine-grained multi-concept control in LLMs, focusing on single- and dual-concept prompts across humor, persuasiveness, clarity, politeness, assertiveness, and formality. It uses judge-based pairwise comparisons to derive rank correlations between intended and realized levels, aggregating results with Spearman’s $\rho$ and Fisher’s $z$, and applying statistical tests to quantify cross-concept interference. Across three mid-sized models and three tasks, the study finds strong single-concept control but notable degradation when a second concept is introduced, indicating challenges in compositionality for prompting-based control. The framework and findings establish a benchmark for developing robust, multi-dimensional style control methods with clearer interpretability and reliability in real-world applications.

Abstract

Large Language Models (LLMs) offer strong generative capabilities, but many applications require explicit and \textit{fine-grained} control over specific textual concepts, such as humor, persuasiveness, or formality. Prior approaches in prompting and representation engineering can provide coarse or single-attribute control, but systematic evaluation of multi-attribute settings remains limited. We introduce an evaluation framework for fine-grained controllability for both single- and dual-concept scenarios, focusing on linguistically distinct concept pairs (e.g., persuasiveness vs.~humor). Surprisingly, across multiple LLMs and generative tasks, we find that performance often drops in the dual-concept setting, even though the chosen concepts should in principle be separable. This reveals a fundamental limitation of naive prompting-based control: models struggle with compositionality even when concepts are intuitively independent. Our framework provides systematic evidence of this gap and offers a principled approach for measuring the ability of future methods for multi-concept control.

Figures (13)

• Figure 1: Illustrative example: an LLM can perform single-concept control, but the explicit presence of a second concept at the input can compromise the ability of the model to control the former concept in its response.
• Figure 2: Model-generated response rank of the target concept versus the desired level. Point size and density indicate the number of samples at each coordinate. Results shown for Llama-11B with the secondary concept level randomly sampled. For example, "Humor $|$ Persuasiveness" denotes responses generated independently for each humor level (target concept) while persuasiveness is randomly set for each inference.
• Figure 3: ARGUMENT generation - score tie proportions across six concepts for each model.
• Figure 4: STORY generation - score tie proportions across six concepts for each model.
• Figure 5: STRUCTURED generation - score tie proportions across six concepts for each model.