PromptCD: Test-Time Behavior Enhancement via Polarity-Prompt Contrastive Decoding

TL;DR

This work introduces Polarity-Prompt Contrastive Decoding (PromptCD), a test-time behavior control method that generalizes contrastive decoding to broader enhancement settings and is applicable to both LLMs and Vision-Language Models without additional training.

Abstract

Reliable AI systems require large language models (LLMs) to exhibit behaviors aligned with human preferences and values. However, most existing alignment approaches operate at training time and rely on additional high-quality data, incurring significant computational and annotation costs. While recent work has shown that contrastive decoding can leverage a model's internal distributions to improve specific capabilities, its applicability remains limited to narrow behavioral scopes and scenarios. In this work, we introduce Polarity-Prompt Contrastive Decoding (PromptCD), a test-time behavior control method that generalizes contrastive decoding to broader enhancement settings. PromptCD constructs paired positive and negative guiding prompts for a target behavior and contrasts model responses-specifically token-level probability distributions in LLMs and visual attention patterns in VLMs-to reinforce desirable outcomes. This formulation extends contrastive decoding to a wide range of enhancement objectives and is applicable to both LLMs and Vision-Language Models (VLMs) without additional training. For LLMs, experiments on the "3H" alignment objectives (helpfulness, honesty, and harmlessness) demonstrate consistent and substantial improvements, indicating that post-trained models can achieve meaningful self-enhancement purely at test time. For VLMs, we further analyze contrastive effects on visual attention, showing that PromptCD significantly improves VQA performance by reinforcing behavior-consistent visual grounding. Collectively, these results highlight PromptCD as a simple, general, and cost-efficient strategy for reliable behavior control across modalities.

Figures (9)

• Figure 1: Overview of PromptCD. A unified framework for test-time behavior enhancement. By contrasting the logits (for text) or visual attention maps (for images) induced by positive and negative prompts, PromptCD effectively steers models toward desired goals without retraining. We demonstrate this capability through faithfulness enhancement in LLMs and visual attentiveness improvement in VLMs.
• Figure 2: Comparison of contextual and parametric knowledge shifts under positive and negative prompts. The first discriminative tokens are identified to represent the corresponding knowledge, and their logit and rank distributions are tracked to illustrate the underlying distributional dynamics.
• Figure 3: Selected cases showing changes in the first token for both parametric and contextual knowledge. The cases are obtained by introducing positive prompts carrying counterfactual information into the LLaMA2-7B-chat model. '$\rightarrow$' denotes the knowledge shift after incorporating positive prompts. 'Logits' and 'rank' refer to the first discriminative token in the knowledge answer, indicating the model's knowledge confidence.
• Figure 4: Illustration of PromptCD enhancing context-faithfulness. By contrasting the probability distributions induced by positive and negative prompts, PromptCD amplifies contextual signals, guiding the LLM to generate faithful responses to the input question.
• Figure 5: The PromptCD framework for cross-modal attention refinement. By contrasting the task-specific attention distribution $A^{(P)}$ (derived from positive prompts) against the general attention pattern $A^{(N)}$ (derived from negative prompts), the method computes a refined attention map $\hat{A}$. This contrastive mechanism effectively isolates and highlights request-relevant visual regions, thereby enhancing visual grounding.