Do great minds think alike? Investigating Human-AI Complementarity in Question Answering with CAIMIRA

TL;DR

CAIMIRA reframes QA evaluation as a multidimensional, content-aware IRT problem, enabling quantitative comparison of humans and AI across latent skills and question traits. By mapping questions to a learnable relevance vector $\mathbf{r_j}$ and a zero-centered difficulty vector $\mathbf{d_j}$, and by representing agents with a learnable skill matrix $\mathbf{E}^a$, the model predicts correctness via $p(U_{i, j}) = \sigma\left((\mathbf{s_i}-\mathbf{d_j})^{\top}\mathbf{r_j}\right)$. In extensive Protobowl-based experiments with ~155 humans and ~70 AI systems, CAIMIRA reveals that humans outperform AI in abductive recall and conceptual reasoning, while LLMs excel at targeted information retrieval and fact-based reasoning when information is accessible, with retrievers and RAG setups showing strong performance on many factual tasks. The work highlights complementary strengths between humans and AI and proposes latent-factor benchmarks to guide future QA tasks toward more nuanced linguistic interpretation and cross-contextual knowledge application. Collectively, CAIMIRA provides a principled, interpretable framework for benchmarking and guiding the development of AI systems that better complement human problem-solving in real-world QA settings.

Abstract

Recent advancements of large language models (LLMs) have led to claims of AI surpassing humans in natural language processing (NLP) tasks such as textual understanding and reasoning. This work investigates these assertions by introducing CAIMIRA, a novel framework rooted in item response theory (IRT) that enables quantitative assessment and comparison of problem-solving abilities of question-answering (QA) agents: humans and AI systems. Through analysis of over 300,000 responses from ~70 AI systems and 155 humans across thousands of quiz questions, CAIMIRA uncovers distinct proficiency patterns in knowledge domains and reasoning skills. Humans outperform AI systems in knowledge-grounded abductive and conceptual reasoning, while state-of-the-art LLMs like GPT-4 and LLaMA show superior performance on targeted information retrieval and fact-based reasoning, particularly when information gaps are well-defined and addressable through pattern matching or data retrieval. These findings highlight the need for future QA tasks to focus on questions that challenge not only higher-order reasoning and scientific thinking, but also demand nuanced linguistic interpretation and cross-contextual knowledge application, helping advance AI developments that better emulate or complement human cognitive abilities in real-world problem-solving.

Figures (31)

• Figure 1: Response Correctness prediction using Agent skills and Question difficulty over relevant latent factors. We list the five latent factors that caimira discovers, and highlight the relevant ones (green), which contribute to estimating whether an agent will respond to the example question correctly. The agent skills over these relevant factors are highlighted in red boxes.
• Figure 2: Distribution of question categories and subcategories over our dataset of 3042 questions.
• Figure 3: The caimira workflow. It predicts the probability of agent-$i$ correctly answering question-$j$ using a model in Eq. \ref{['equ:kira']}. Here, the question's raw relevance $\mathbf{r'_j}$ and raw difficulty $\mathbf{d;_j}$ are multidimensional and computed by learnt linear transformations over the question embedding $\mathbf{E}^q_j$ (§ \ref{['subsec:equations']}), and the agent skill $\mathbf{s_i}$ is extracted from a learnable agent embedding matrix $\mathbf{E}^a$. $\mathbf{r_j}$ is a probability distribution computed from the raw reference $\mathbf{r'_j}$ and improves the interpretability of the multidimensional model (§ \ref{['sec:relevance']}); $\mathbf{d_j}$ is achieved by zero centering of the raw difficulty $\mathbf{d'_j}$, which addresses the non-identifiability issue of $\mathbf{s_i}$ and $\mathbf{d_j}$ in $(\mathbf{s_i}-\mathbf{d_j})$ (§ \ref{['subsec:zero-mean']}).
• Figure 4: Ablation study showing caimira performance with varying latent dimensions $m$, indicating sufficiency at $m=5$, beyond which gains are marginal.
• Figure 5: Interpretation of the five latent dimensions in caimira. We use Logistic Regression to predict the binary relevance label, $\mathbf{r_{jk}} > 0.6$, for each dimension $k$. For question features, we use topical categories and linguistic properties. We report the classification accuracy and the statistically significant features. Coefficients are positive if the features positively affect classification, negative otherwise. This demonstrates the efficacy of predicting the relevance from a question's sbert embedding.