Scaling Unverifiable Rewards: A Case Study on Visual Insights

TL;DR

This paper tackles the problem of scaling unverifiable rewards in complex data-science tasks by distributing inference across a staged, multi-agent pipeline and pruning low-quality branches with stage-local evaluators. It introduces Selective Test-Time Scaling (Selective TTS), which reallocates compute across Data Profiling, Visualization, and Insight Generation stages, guided by human-aligned judges and a controlled budget. Through two experimental regimes on tabular datasets (including the VIS Publication dataset), the authors show that Selective TTS increases mean insight quality and reduces variance under a fixed compute budget, with an optimal pruning ratio around 0.6 and over-pruning leading to degraded performance. The work advances practical methods for scaling open-ended reasoning tasks like data insight generation while mitigating judge drift and providing a modular, auditable framework for future exploration in unverifiable AI applications.

Abstract

Large Language Model (LLM) agents can increasingly automate complex reasoning through Test-Time Scaling (TTS), iterative refinement guided by reward signals. However, many real-world tasks involve multi-stage pipeline whose final outcomes lack verifiable rewards or sufficient data to train robust reward models, making judge-based refinement prone to accumulate error over stages. We propose Selective TTS, a process-based refinement framework that scales inference across different stages in multi-agent pipeline, instead of repeated refinement over time by prior work. By distributing compute across stages and pruning low-quality branches early using process-specific judges, Selective TTS mitigates the judge drift and stabilizes refinement. Grounded in the data science pipeline, we build an end-to-end multi-agent pipeline for generating visually insightful charts and report of given dataset, and design a reliable LLM-based judge model, aligned with human experts (Kendall's τ=0.55). Our proposed selective TTS then improves insight quality under a fixed compute budget, increasing mean scores from 61.64 to 65.86 while reducing variance. We hope our findings serve as the first step toward to scaling complex, open-ended tasks with unverifiable rewards, such as scientific discovery and story generation.

Figures (25)

• Figure 1: Comparison between traditional TTS through refinement over time (left) novikov2025alphaevolvecodingagentscientific, and our proposed process refinement in multi-agent pipeline (right). They show different styles of scaling behaviors.
• Figure 2: Our multi-agent data analysis pipeline aims to answer two key questions: (RQ1) whether judge-guided scaling can align with human experts, and (RQ2) how much performance can be further scaled within the same compute budget using the proposed selective TTS.
• Figure 3: Overview of Selective Test-Time Scaling under a fixed compute budget (e.g., 30). Stage-specific evaluators perform pruning across the multi-agent pipeline while controlling the branching factor at each stage. Higher pruning ratios ($\rho$) promote more rounds of exploration and diversity across runs, while reducing within-round breadth. Details on budget accounting and branching control appear in §\ref{['sec:budget']}.
• Figure 4: Sorted overall score curves under three judgers (easy, moderate, harsh) on VIS Publication dataset. See Appendix §\ref{['app:medical']} for the Medical Insurance dataset.
• Figure 5: Effects of pruning ratio $\rho$ on compute allocation and quality. (a) Higher $\rho$ shifts compute from within-run breadth (fewer reports per run) to cross-run exploration (more runs). (b) The mean score increases with stronger pruning and peaks at $\rho = 0.6$, while the standard deviation generally decreases, indicating improved stability in report quality. (c) Increasing $\rho$ removes the low-quality tail and shifts the score distribution upward.