Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Aligning Human and Machine Attention for Enhanced Supervised Learning

Avihay Chriqui, Inbal Yahav, Dov Teeni, Ahmed Abbasi

TL;DR

HuMAL addresses the challenge of aligning machine attention with human attention to improve supervised learning, especially when labeled data are scarce or imbalanced. It introduces three strategies (HuMAL-AL, HuMAL-AN, HuMAL-AP) that integrate human attention into transformer-based classifiers, and evaluates them on Yelp sentiment and myPersonality tasks. The results show that HuMAL-AL consistently outperforms baselines and other HuMAL variants, with notable gains under data scarcity and across different base models (BERT, GPT-2, XLNet). This work suggests that incorporating task-specific human attention can enhance performance and interpretability in real-world NLP applications, particularly for nuanced or low-resource tasks.

Abstract

Attention, or prioritization of certain information items over others, is a critical element of any learning process, for both humans and machines. Given that humans continue to outperform machines in certain learning tasks, it seems plausible that machine performance could be enriched by aligning machine attention with human attention mechanisms -- yet research on this topic is sparse and has achieved only limited success. This paper proposes a new approach to address this gap, called Human-Machine Attention Learning (HuMAL). This approach involves reliance on data annotated by humans to reflect their self-perceived attention during specific tasks. We evaluate several alternative strategies for integrating such human attention data into machine learning (ML) algorithms, using a sentiment analysis task (review data from Yelp) and a personality-type classification task (data from myPersonality). The best-performing HuMAL strategy significantly enhances the task performance of fine-tuned transformer models (BERT, as well as GPT-2 and XLNET), and the benefit is particularly pronounced under challenging conditions of imbalanced or sparse labeled data. This research contributes to a deeper understanding of strategies for integrating human attention into ML models and highlights the potential of leveraging human cognition to augment ML in real-world applications.

Aligning Human and Machine Attention for Enhanced Supervised Learning

TL;DR

HuMAL addresses the challenge of aligning machine attention with human attention to improve supervised learning, especially when labeled data are scarce or imbalanced. It introduces three strategies (HuMAL-AL, HuMAL-AN, HuMAL-AP) that integrate human attention into transformer-based classifiers, and evaluates them on Yelp sentiment and myPersonality tasks. The results show that HuMAL-AL consistently outperforms baselines and other HuMAL variants, with notable gains under data scarcity and across different base models (BERT, GPT-2, XLNet). This work suggests that incorporating task-specific human attention can enhance performance and interpretability in real-world NLP applications, particularly for nuanced or low-resource tasks.

Abstract

Attention, or prioritization of certain information items over others, is a critical element of any learning process, for both humans and machines. Given that humans continue to outperform machines in certain learning tasks, it seems plausible that machine performance could be enriched by aligning machine attention with human attention mechanisms -- yet research on this topic is sparse and has achieved only limited success. This paper proposes a new approach to address this gap, called Human-Machine Attention Learning (HuMAL). This approach involves reliance on data annotated by humans to reflect their self-perceived attention during specific tasks. We evaluate several alternative strategies for integrating such human attention data into machine learning (ML) algorithms, using a sentiment analysis task (review data from Yelp) and a personality-type classification task (data from myPersonality). The best-performing HuMAL strategy significantly enhances the task performance of fine-tuned transformer models (BERT, as well as GPT-2 and XLNET), and the benefit is particularly pronounced under challenging conditions of imbalanced or sparse labeled data. This research contributes to a deeper understanding of strategies for integrating human attention into ML models and highlights the potential of leveraging human cognition to augment ML in real-world applications.

Paper Structure

This paper contains 18 sections, 5 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Mechanisms of Human vs. Machine Attention in Learning Processes
  3. HuMAL Strategies
  4. Attention as Loss (HuMAL-AL)
  5. Attention as Normalizer (HuMAL-AN)
  6. Attention as Prior (HuMAL-AP)
  7. Experiment
  8. Sentiment Classification Task
  9. Personality Type Classification Task
  10. Experiment Mechanism
  11. Results
  12. Main Results: Comparing the Performance of the Three HuMAL Strategies
  13. Data Requirements of HuMAL-AL vs. BERT
  14. Impact on the Classification Loss
  15. Impact of Text Length on HuMAL Performance
  16. ...and 3 more sections

Figures (7)

  • Figure 1: Examples of human attention (blue in top two texts) compared with machine attention (red in bottom text) in a Yelp review. Adapted from "Human Attention Maps for Text Classification: Do Humans and Neural Networks Focus on the Same Words?" by sen2020human. Copyright 1963–2023 by ACL.
  • Figure 2: HuMAL illustration: Alternative strategies for leveraging human attention into machine attention. Example using BERT as a base model
  • Figure 3: Comparison between the performance of HuMAL-AL and BERT, as a function of data size and imbalance ratio, in the sentiment classification task.
  • Figure 4: Performance of HuMAL-AL compared to BERT across different dataset sizes in the sentiment classification task.
  • Figure 5: Loss Convergence Analysis for Sentiment Classification
  • ...and 2 more figures