Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UniAR: A Unified model for predicting human Attention and Responses on visual content

Peizhao Li, Junfeng He, Gang Li, Rachit Bhargava, Shaolei Shen, Nachiappan Valliappan, Youwei Liang, Hongxiang Gu, Venky Ramachandran, Golnaz Farhadi, Yang Li, Kai J Kohlhoff, Vidhya Navalpakkam

TL;DR

UniAR is proposed -- a unified model of human attention and preference behavior across diverse visual content that leverages a multimodal transformer to predict subjective feedback, along with the underlying human attention or interaction heatmaps and viewing order.

Abstract

Progress in human behavior modeling involves understanding both implicit, early-stage perceptual behavior, such as human attention, and explicit, later-stage behavior, such as subjective preferences or likes. Yet most prior research has focused on modeling implicit and explicit human behavior in isolation; and often limited to a specific type of visual content. We propose UniAR -- a unified model of human attention and preference behavior across diverse visual content. UniAR leverages a multimodal transformer to predict subjective feedback, such as satisfaction or aesthetic quality, along with the underlying human attention or interaction heatmaps and viewing order. We train UniAR on diverse public datasets spanning natural images, webpages, and graphic designs, and achieve SOTA performance on multiple benchmarks across various image domains and behavior modeling tasks. Potential applications include providing instant feedback on the effectiveness of UIs/visual content, and enabling designers and content-creation models to optimize their creation for human-centric improvements.

UniAR: A Unified model for predicting human Attention and Responses on visual content

TL;DR

UniAR is proposed -- a unified model of human attention and preference behavior across diverse visual content that leverages a multimodal transformer to predict subjective feedback, along with the underlying human attention or interaction heatmaps and viewing order.

Abstract

Progress in human behavior modeling involves understanding both implicit, early-stage perceptual behavior, such as human attention, and explicit, later-stage behavior, such as subjective preferences or likes. Yet most prior research has focused on modeling implicit and explicit human behavior in isolation; and often limited to a specific type of visual content. We propose UniAR -- a unified model of human attention and preference behavior across diverse visual content. UniAR leverages a multimodal transformer to predict subjective feedback, such as satisfaction or aesthetic quality, along with the underlying human attention or interaction heatmaps and viewing order. We train UniAR on diverse public datasets spanning natural images, webpages, and graphic designs, and achieve SOTA performance on multiple benchmarks across various image domains and behavior modeling tasks. Potential applications include providing instant feedback on the effectiveness of UIs/visual content, and enabling designers and content-creation models to optimize their creation for human-centric improvements.
Paper Structure (45 sections, 3 equations, 3 figures, 7 tables)

This paper contains 45 sections, 3 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. UniAR.
  3. Related Work
  4. Saliency prediction.
  5. Scanpath prediction.
  6. Subjective rating prediction.
  7. Limitations of prior work.
  8. Multi-tasking unified model for language & vision.
  9. Unifying Human Attention and Responses
  10. Model Architecture
  11. Heatmap Predictor
  12. Scanpath (Sequence) Predictor
  13. Decoding during inference.
  14. Rating Predictor
  15. Text Prompt
  16. ...and 30 more sections

Figures (3)

  • Figure 1: Overview of our UniAR model. UniAR is a multimodal model that takes an image (could be a natural image, screenshot of a webpage, graphic design, or UI) along with a text prompt as input, and outputs heatmaps of human attention/interaction, scanpath or sequence of viewing/interaction, and subjective preference/likes. Example inputs and corresponding outputs for saliency, scanpath, and rating are shown on the left side, and the detailed model architecture is shown on the right side.
  • Figure 2: Examples of UniAR's predictions across different tasks/domains. Images in green border are ground-truth, while images in orange border are UniAR's predictions. First row: attention/saliency heatmap prediction on natural images (Salicon) and webpages (WS-Saliency). Second row: importance heatmap on graphic designs (Imp1k), and saliency heatmap on Mobile UI. Third row: scanpath-sequence during free-viewing of webpages (WS-Scanpath) and object-searching within images (COCO-Search18). Fourth row: preference/rating prediction for natural images (Koniq-10k) and webpages (Web Aesthetics).
  • Figure 3: Another set of visualizations on UniAR's predictions. Images in green border are ground-truth, while images in orange border are UniAR's predictions. First row: saliency heatmap on Salicon and WS-Saliency. Second row: importance heatmap on Imp1k, and saliency heatmap on Mobile UI. Third row: free-viewing scanpath on WS-Scanpath and object-searching scanpath on COCO-Search18. Fourth row: rating prediction on Koniq-10k and Web Aesthetics datasets.