Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Reduced Basis Decomposition Approach to Efficient Data Collection in Pairwise Comparison Studies

Jiahua Jiang, Joseph Marsh, Rowland G Seymour

Abstract

Comparative judgement studies elicit quality assessments through pairwise comparisons, typically analysed using the Bradley-Terry model. A challenge in these studies is experimental design, specifically, determining the optimal pairs to compare to maximize statistical efficiency. Constructing static experimental designs for these studies requires spectral decomposition of a covariance matrix over pairs of pairs, which becomes computationally infeasible for studies with more than approximately 150 objects. We propose a scalable method based on reduced basis decomposition that bypasses explicit construction of this matrix, achieving computational savings of two to three orders of magnitude. We establish eigenvalue bounds guaranteeing approximation quality and characterise the rank structure of the design matrix. Simulations demonstrate speedup factors exceeding 100 for studies with 64 or more objects, with negligible approximation error. We apply the method to construct designs for a 452-region spatial study in under 7 minutes and enable real-time design updates for classroom peer assessment, reducing computation time from 15 minutes to 15 seconds.

A Reduced Basis Decomposition Approach to Efficient Data Collection in Pairwise Comparison Studies

Abstract

Comparative judgement studies elicit quality assessments through pairwise comparisons, typically analysed using the Bradley-Terry model. A challenge in these studies is experimental design, specifically, determining the optimal pairs to compare to maximize statistical efficiency. Constructing static experimental designs for these studies requires spectral decomposition of a covariance matrix over pairs of pairs, which becomes computationally infeasible for studies with more than approximately 150 objects. We propose a scalable method based on reduced basis decomposition that bypasses explicit construction of this matrix, achieving computational savings of two to three orders of magnitude. We establish eigenvalue bounds guaranteeing approximation quality and characterise the rank structure of the design matrix. Simulations demonstrate speedup factors exceeding 100 for studies with 64 or more objects, with negligible approximation error. We apply the method to construct designs for a 452-region spatial study in under 7 minutes and enable real-time design updates for classroom peer assessment, reducing computation time from 15 minutes to 15 seconds.
Paper Structure (18 sections, 4 theorems, 27 equations, 3 figures, 2 tables, 4 algorithms)

This paper contains 18 sections, 4 theorems, 27 equations, 3 figures, 2 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. The Bradley--Terry model
  3. Likelihood function and inference
  4. Experimental design
  5. Reduced basis decomposition for the Bradley--Terry model
  6. Efficient construction of $\Delta$
  7. Reduced-basis spectral decomposition of $\boldsymbol{E}$
  8. Eigenpairs approximation of $\Delta$
  9. Theoretical analysis
  10. Implementing the method
  11. Empirical results
  12. Scalability in size of study
  13. Sensitivity to the sparsity of the prior covariance matrix
  14. Sensitivity to the tolerance parameter
  15. Applications to real data
  16. ...and 3 more sections

Key Result

Theorem 1

Let $\{\alpha_i\}_{i=1}^{\frac{N(N-1)}{2}}$ be the eigenvalues of $\Delta$. Let $\{\sigma_i\}_{i=1}^d$ be the largest $d$ eigenvalues of $\widetilde{\Delta}$, which approximate the largest $d$ eigenvalues of $\Delta$. Both of them are sorted in non-increasing order. Then for $i = 1, \dots,d.$

Figures (3)

  • Figure 1: Comparison of speed up of the RBD method compared to the standard method for studies with different numbers of objects (N) in. The subfigures show (a) Laplacian, (b) Toeplitz, and (c) Inverse Wishart. Note the y-axes are on the log scale.
  • Figure 2: Comparison of KL diveregnce for studies with different numbers of objects (N). The subfigures show (a) Laplacian, (b) Toeplitz, and (c) Inverse Wishart. Note the y-axes are on the log scale.
  • Figure 3: Comparisons of the KL divergence for different values of the tolerance parameter $\varepsilon_R$.

Theorems & Definitions (9)

  • Definition 1
  • Theorem 1
  • proof
  • Lemma 3.1
  • proof
  • Theorem 2
  • proof
  • Corollary 3
  • Remark 1