Eliciting Thinking Hierarchy without a Prior
Yuqing Kong, Yunqi Li, Yubo Zhang, Zhihuan Huang, Jinzhao Wu
TL;DR
This work addresses eliciting thinking hierarchy without priors to improve crowd-based problem solving. It introduces a generative model where each thinking type t has an answer distribution w_t, predictions guided by p_{t→t'}, and a joint distribution M = W^\top Λ W with an upper-triangular Λ, enabling hierarchy recovery via Non-negative Congruence Triangularization. Two practical ranking algorithms, AR and AR^{+}, identify the latent hierarchy from data, and a proxy Answer-Prediction matrix A allows empirical estimation of M from open-response data where respondents provide an answer and a predicted answer of others. Four empirical studies across math, Go, general knowledge, and Chinese pronunciation demonstrate that the top-ranked answers produced by the method outperform plurality voting, with a tighter goodness-of-fit when the top answer is correct. The findings offer a priors-free, empirically validated framework for eliciting a rich thinking hierarchy, with implications for better decision-making in new domains and policy contexts, while suggesting avenues for incentive design and natural-language extensions.
Abstract
When we use the wisdom of the crowds, we usually rank the answers according to their popularity, especially when we cannot verify the answers. However, this can be very dangerous when the majority make systematic mistakes. A fundamental question arises: can we build a hierarchy among the answers \textit{without any prior} where the higher-ranking answers, which may not be supported by the majority, are from more sophisticated people? To address the question, we propose 1) a novel model to describe people's thinking hierarchy; 2) two algorithms to learn the thinking hierarchy without any prior; 3) a novel open-response based crowdsourcing approach based on the above theoretic framework. In addition to theoretic justifications, we conduct four empirical crowdsourcing studies and show that a) the accuracy of the top-ranking answers learned by our approach is much higher than that of plurality voting (In one question, the plurality answer is supported by 74 respondents but the correct answer is only supported by 3 respondents. Our approach ranks the correct answer the highest without any prior); b) our model has a high goodness-of-fit, especially for the questions where our top-ranking answer is correct. To the best of our knowledge, we are the first to propose a thinking hierarchy model with empirical validations in the general problem-solving scenarios; and the first to propose a practical open-response based crowdsourcing approach that beats plurality voting without any prior.