Less Diverse, Less Safe: The Indirect But Pervasive Risk of Test-Time Scaling in Large Language Models

TL;DR

This work identifies a previously unrecognized vulnerability in test-time scaling (TTS) for large language models: constraining candidate diversity can cause TTS to generate unsafe outputs. It introduces RefDiv, a reference-guided diversity stress test that uses a population-based genetic algorithm to progressively reduce diversity while steering toward harmful completions. Across multiple open-source and closed-source LLMs and two common TTS strategies (Best-of-N and Monte Carlo Tree Search), RefDiv yields higher attack success rates than state-of-the-art jailbreaks and transfers effectively across strategies, models, and guardrails. The results emphasize the need for diversity-aware, robust TTS designs and provide RefDiv as a practical stress-test framework for assessing and improving the safety of TTS-enabled LLM systems.

Abstract

Test-Time Scaling (TTS) improves LLM reasoning by exploring multiple candidate responses and then operating over this set to find the best output. A tacit premise behind TTS is that sufficiently diverse candidate pools enhance reliability. In this work, we show that this assumption in TTS introduces a previously unrecognized failure mode. When candidate diversity is curtailed, even by a modest amount, TTS becomes much more likely to produce unsafe outputs. We present a reference-guided diversity reduction protocol (RefDiv) that serves as a diagnostic attack to stress test TTS pipelines. Through extensive experiments across four open-source models (Qwen3, Mistral, Llama3.1, Gemma3) and two widely used TTS strategies (Monte Carlo Tree Search and Best-of-N), constraining diversity consistently signifies the rate at which TTS produces unsafe results. The effect is often stronger than that produced by prompts directly with high adversarial intent scores. This observed phenomenon also transfers across TTS strategies and to closed-source models (e.g. OpenAI o3 and Gemini-2.5-Pro), thus indicating that this is a general and extant property of TTS rather than a model-specific artifact. Additionally, we find that numerous widely used safety guardrail classifiers (e.g. Llama-Guard and OpenAI Moderation API), are unable to flag the adversarial input prompts generated by RefDiv, demonstrating that existing defenses offer limited protection against this diversity-driven failure mode. Through this work, we hope to motivate future research on designing robust TTS strategies that are both effective and secure against diversity-targeted stress tests as illustrated by RefDiv.

Figures (14)

• Figure 1: In initial iterations of RefDiv ($\alpha_t$ is small for small $t$), the stress test steers candidates (which are comparatively more diverse) towards affirmative reference tokens. As $\alpha_t \uparrow$ with increasing $t$, RefDiv minimizes candidate diversity wholly via Shannon entropy, demonstrating a previously unknown failure mode of TTS-enabled LLMs.
• Figure 2: ASR trends across iterations for AutoDAN, GCG, and RefDiv with Best-of-$N$ TTS.
• Figure 3: ASR trends across iterations for AutoDAN, GCG, and RefDiv with MCTS TTS.
• Figure 4: Analyzing the Shannon Entropy trend across iterations for RefDiv and AutoDAN.
• Figure 5: Transferability of RefDiv prompts for Best-of-$N$$\rightarrow$ MCTS and MCTS $\rightarrow$ Best-of-$N$ across LLMs.