Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gaze patterns predict preference and confidence in pairwise AI image evaluation

Nikolas Papadopoulos, Shreenithi Navaneethan, Sheng Bai, Ankur Samanta, Paul Sajda

Abstract

Preference learning methods, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), rely on pairwise human judgments, yet little is known about the cognitive processes underlying these judgments. We investigate whether eye-tracking can reveal preference formation during pairwise AI-generated image evaluation. Thirty participants completed 1,800 trials while their gaze was recorded. We replicated the gaze cascade effect, with gaze shifting toward chosen images approximately one second before the decision. Cascade dynamics were consistent across confidence levels. Gaze features predicted binary choice (68% accuracy), with chosen images receiving more dwell time, fixations, and revisits. Gaze transitions distinguished high-confidence from uncertain decisions (66% accuracy), with low-confidence trials showing more image switches per second. These results show that gaze patterns predict both choice and confidence in pairwise image evaluations, suggesting that eye-tracking provides implicit signals relevant to the quality of preference annotations.

Gaze patterns predict preference and confidence in pairwise AI image evaluation

Abstract

Preference learning methods, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), rely on pairwise human judgments, yet little is known about the cognitive processes underlying these judgments. We investigate whether eye-tracking can reveal preference formation during pairwise AI-generated image evaluation. Thirty participants completed 1,800 trials while their gaze was recorded. We replicated the gaze cascade effect, with gaze shifting toward chosen images approximately one second before the decision. Cascade dynamics were consistent across confidence levels. Gaze features predicted binary choice (68% accuracy), with chosen images receiving more dwell time, fixations, and revisits. Gaze transitions distinguished high-confidence from uncertain decisions (66% accuracy), with low-confidence trials showing more image switches per second. These results show that gaze patterns predict both choice and confidence in pairwise image evaluations, suggesting that eye-tracking provides implicit signals relevant to the quality of preference annotations.

Paper Structure

This paper contains 19 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Experimental Setup
  4. Task Overview.
  5. Procedure.
  6. Stimulus Assignment.
  7. Participants.
  8. Apparatus.
  9. Eye-Tracking Data Processing.
  10. Image Stimuli
  11. Open-Source Datasets
  12. Custom-Generated Pairs
  13. Results
  14. Gaze Cascade Analysis
  15. Predicting Preference from Gaze Features
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Example trial stimulus.
  • Figure 2: Gaze cascade effect in AI image evaluation. Likelihood of fixating on the eventually chosen image as a function of time before decision (key press). The blue line shows grand average across participants (N=30), with individual participants shown in gray lines.
  • Figure 3: Gaze cascade dynamics by confidence level.
  • Figure 4: Permutation importance for preference prediction (top 5 features). Error bars show SD across 5 folds.