On the Optimality of Tracking Fisher Information in Adaptive Testing with Stochastic Binary Responses

TL;DR

The paper addresses estimating a continuous ability parameter from sequential binary responses in an adaptive testing setting. It introduces FIT-Q, a Fisher-information tracking algorithm paired with a method-of-moments estimator and a novel stopping statistic, and proves its asymptotic optimality in both fixed-budget and fixed-confidence regimes. The work provides universal information-theoretic lower bounds and matching upper bounds, aided by large-deviation tools and Ville's inequality to handle estimate-query endogeneity. Numerical experiments with logistic and bimodal Fisher-information models corroborate the theoretical results, showing superior sample efficiency over baselines and practical feasibility.

Abstract

We study the problem of estimating a continuous ability parameter from sequential binary responses by actively asking questions with varying difficulties, a setting that arises naturally in adaptive testing and online preference learning. Our goal is to certify that the estimate lies within a desired margin of error, using as few queries as possible. We propose a simple algorithm that adaptively selects questions to maximize Fisher information and updates the estimate using a method-of-moments approach, paired with a novel test statistic to decide when the estimate is accurate enough. We prove that this Fisher-tracking strategy achieves optimal performance in both fixed-confidence and fixed-budget regimes, which are commonly invested in the best-arm identification literature. Our analysis overcomes a key technical challenge in the fixed-budget setting -- handling the dependence between the evolving estimate and the query distribution -- by exploiting a structural symmetry in the model and combining large deviation tools with Ville's inequality. Our results provide rigorous theoretical support for simple and efficient adaptive testing procedures.

Figures (5)

• Figure 1: Illustration of proposed $\phi$ and $\lambda_*$: (a) the proposed function $\phi(\lambda, p)$ (solid curve) and the log-moment generating functions $\psi(\lambda, p)$ and $\psi(-\lambda, p)$ (dashed curves), plotted over $p \in [0,1]$ for a fixed $\lambda > 0$; (b) the objective functions $\lambda \mapsto \lambda \cdot |f(z) - f(z_*)| - \phi(\lambda, f(z))$ for $z = z_* \pm \epsilon$ (dashed curves), used in the selection of $\lambda_*$.
• Figure 2: Fisher information function $h(z)$ for logistic and algebraic models.
• Figure 3: Negative log failure probability under the fixed-budget setting for (a) logistic and (b) algebraic-4 models. The background diagonals correspond to the optimal slope $\frac{1}{2} I(x_*; \theta_*) \epsilon^2$ predicted by Theorem \ref{['thm:FB-bound']}.
• Figure 4: Expected stopping times under the fixed-confidence setting, plotted against $\log(1/\delta)$ for (a) logistic and (b) algebraic-4 models. The background diagonals correspond to the theoretical slope predicted by Theorem \ref{['thm:FC-bound']}.
• Figure 5: Comparison of average stopping times for FIT-Q, and the A1 algorithm suggested in bassamboo2023learning adapted to our setting via discretization. The plot measures the mean stopping time as a function of the true parameter $\theta_*$. The x-axis covers the range $[-0.4, 0.4)$, which is one of the subintervals created by partitioning the full parameter space $\Theta = [-2,2]$.

Theorems & Definitions (31)

• Definition 1: FB-consistent algorithms
• Theorem 1: Universal performance limit in the FB setting
• Theorem 2: Performance of FIT-Q in FB setting
• proof : Proof sketch of Theorem \ref{['thm:FIT-Q-FB']}
• Definition 2: FC-consistent algorithms
• Theorem 3: Universal performance limit in FC setting
• Theorem 4: FC-consistency of FIT-Q
• proof : Proof sketch of Theorem \ref{['thm:FIT-Q-FC-consistency']}
• Theorem 5: Performance of FIT-Q in FC setting
• proof : Proof sketch of Theorem \ref{['thm:FIT-Q-FC-optimality']}