Characterising Topic Familiarity and Query Specificity Using Eye-Tracking Data
Jiaman He, Zikang Leng, Dana McKay, Johanne R. Trippas, Damiano Spina
TL;DR
The paper investigates whether eye-tracking signals, specifically pupil dilation and gaze velocity, can reveal a user's topic familiarity and query specificity without contextual information. Using a lab study with 18 participants and novel NL-based annotations for specificity, it trains multiple classifiers to predict these cognitive states, achieving Macro F1 scores up to 71.25% for topic familiarity and 60.54% for query specificity under cross-validation, with LOSO analyses showing reduced performance due to inter-user variability. The work demonstrates the feasibility of real-time cognitive state estimation from eye-tracking data and introduces a new annotation guideline for query specificity, suggesting potential for adaptive information retrieval systems. The findings highlight pupillometry as a stronger indicator of memory-related processes than gaze dynamics and point to the need for larger datasets and advanced models to improve generalization and robustness.
Abstract
Eye-tracking data has been shown to correlate with a user's knowledge level and query formulation behaviour. While previous work has focused primarily on eye gaze fixations for attention analysis, often requiring additional contextual information, our study investigates the memory-related cognitive dimension by relying solely on pupil dilation and gaze velocity to infer users' topic familiarity and query specificity without needing any contextual information. Using eye-tracking data collected via a lab user study (N=18), we achieved a Macro F1 score of 71.25% for predicting topic familiarity with a Gradient Boosting classifier, and a Macro F1 score of 60.54% with a k-nearest neighbours (KNN) classifier for query specificity. Furthermore, we developed a novel annotation guideline -- specifically tailored for question answering -- to manually classify queries as Specific or Non-specific. This study demonstrates the feasibility of eye-tracking to better understand topic familiarity and query specificity in search.