Breaking the Pre-Sampling Barrier: Activation-Informed Difficulty-Aware Self-Consistency

TL;DR

ACTSC tackles the high cost of Self-Consistency by replacing pre-sampling difficulty estimation with an activation-informed probe that predicts problem difficulty from internal FFN signals. It trains a lightweight Difficulty Probe using Difficulty-Sensitive Neurons, enabling a single forward pass to decide between single-sample inference and dynamic window Self-Consistency. The method achieves substantial reductions in sampling and token costs (up to 87.1% fewer samples vs SC and significant gains over DSC) while maintaining or even improving accuracy across mathematical and non-mathematical benchmarks. This activation-based, dataset-adaptive approach offers a practical, dataset-agnostic mechanism for efficient, reliable test-time scaling in large language models.

Abstract

Self-Consistency (SC) is an effective decoding strategy that improves the reasoning performance of Large Language Models (LLMs) by generating multiple chain-of-thought reasoning paths and selecting the final answer via majority voting. However, it suffers from substantial inference costs because it requires a large number of samples. To mitigate this issue, Difficulty-Adaptive Self-Consistency (DSC) was proposed to reduce unnecessary token usage for easy problems by adjusting the number of samples according to problem difficulty. However, DSC requires additional model calls and pre-sampling to estimate difficulty, and this process is repeated when applying to each dataset, leading to significant computational overhead. In this work, we propose Activation-Informed Difficulty-Aware Self-Consistency (ACTSC) to address these limitations. ACTSC leverages internal difficulty signals reflected in the feed-forward network neuron activations to construct a lightweight difficulty estimation probe, without any additional token generation or model calls. The probe dynamically adjusts the number of samples for SC and can be applied to new datasets without requiring pre-sampling for difficulty estimation. To validate its effectiveness, we conduct experiments on five benchmarks. Experimental results show that ACTSC effectively reduces inference costs while maintaining accuracy relative to existing methods.

Figures (5)

• Figure 1: Comparison of existing methods and ACTSC. Existing methods need pre-sampling and extra model calls to estimate difficulty. ACTSC uses internal model signals from the input, eliminating additional token generation and model call costs.
• Figure 2: Overview of the ACTSC framework. ACTSC consists of two sequential steps. (Left) In Step 1, a lightweight difficulty probe is trained based on internal FFN activation patterns while keeping the LLM frozen. (Right) In Step 2, the trained probe estimates problem difficulty from activation signals observed during a forward pass and adaptively controls the number of inference samples by selecting either single-sample reasoning or dynamic window reasoning.
• Figure 3: Total token cost vs. average number of samples for Gemma3-4B on (a) MATH-500 and (b) AIME average. The dashed line indicates a linear scaling baseline. Markers represent adaptive consistency methods.
• Figure 4: Distribution of difficulty prediction logits produced by the DSN probe across different actual difficulty levels. Violin plots show the distribution of probe logits for MATH-500 Level 1–5 and AIME 2024/2025. Blue and red dashed lines represent the mean and median, respectively.
• Figure 5: Distribution of easy and hard problems across MATH difficulty levels using model-specific probes trained on internal activations of Qwen2.5-7B model. Percentages indicate the proportion of problems classified as easy or hard at each difficulty level.