BoRP: Bootstrapped Regression Probing for Scalable and Human-Aligned LLM Evaluation

TL;DR

BoRP tackles the metric gap in open-ended LLM evaluation by reframing user satisfaction as a regression task in latent space. It combines a geometry-aware probing approach with polarization-index bootstrapping and a Partial Least Squares mapper to convert latent differences into continuous satisfaction scores, enabling fast, human-aligned assessment without generative decoding. On industrial data, BoRP outperforms larger generative baselines in alignment with human judgments while dramatically reducing inference costs, and it enables sensitive A/B testing via CUPED for scalable product iteration. The approach offers a practical pathway to scalable, reliable, and cost-efficient evaluation for open-ended conversational AI systems.

Abstract

Accurate evaluation of user satisfaction is critical for iterative development of conversational AI. However, for open-ended assistants, traditional A/B testing lacks reliable metrics: explicit feedback is sparse, while implicit metrics are ambiguous. To bridge this gap, we introduce BoRP (Bootstrapped Regression Probing), a scalable framework for high-fidelity satisfaction evaluation. Unlike generative approaches, BoRP leverages the geometric properties of LLM latent space. It employs a polarization-index-based bootstrapping mechanism to automate rubric generation and utilizes Partial Least Squares (PLS) to map hidden states to continuous scores. Experiments on industrial datasets show that BoRP (Qwen3-8B/14B) significantly outperforms generative baselines (even Qwen3-Max) in alignment with human judgments. Furthermore, BoRP reduces inference costs by orders of magnitude, enabling full-scale monitoring and highly sensitive A/B testing via CUPED.

Figures (13)

• Figure 1: The BoRP Framework.(Left) Probe-Guided Bootstrapping: Illustration of the Polarization Mining process, extracting extreme samples for rubric generation. (Right) Geometry-Aware Probing Engine: BoRP extracts hidden states via contrastive suffix prompting. A PLS regressor projects the difference vector to a continuous score. Notably, we aggregate signals from both intermediate and final layers to quantify model uncertainty via the layer-wise score gap(see Sec \ref{['sec:analysis']}). This architecture leverages KV-cache reuse, achieving high-throughput evaluation.
• Figure 3: Uncertainty Analysis. (a) Performance stabilizes after Layer 25. (b) Inconsistency correlates with RMSE.
• Figure 4: The generic prompt used for unsupervised mining (Phase 1) and blind probing.
• Figure 5: The detailed prompt for reverse-engineering rubrics from extreme cases.
• Figure 6: The prompt for fusing multiple rubric drafts into a final standard.