Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MObyGaze: a film dataset of multimodal objectification densely annotated by experts

Julie Tores, Elisa Ancarani, Lucile Sassatelli, Hui-Yin Wu, Clement Bergman, Lea Andolfi, Victor Ecrement, Remy Sun, Frederic Precioso, Thierry Devars, Magali Guaresi, Virginie Julliard, Sarah Lecossais

TL;DR

MObyGaze introduces a dense, expert-annotated multimodal film dataset to study objectification, a high-level construct rooted in film studies and psychology. It defines a thesaurus of 5 sub-constructs and 11 concepts across visual, textual, and audio modalities, and annotates 20 feature-length films to yield 6072 segments over 43 hours with detailed unitization and categorization. The paper formulates and benchmarks multiple learning tasks (classification and localization) under label diversity, using vision, text, and audio models, and proposes strategies to aggregate or separate labels across annotators. Results demonstrate feasibility of detecting objectification from vision and text signals, reveal the limited standalone value of audio, and highlight the value of label-diversity-aware training, with rich opportunities for explainability, fairness studies, and media studies applications.

Abstract

Characterizing and quantifying gender representation disparities in audiovisual storytelling contents is necessary to grasp how stereotypes may perpetuate on screen. In this article, we consider the high-level construct of objectification and introduce a new AI task to the ML community: characterize and quantify complex multimodal (visual, speech, audio) temporal patterns producing objectification in films. Building on film studies and psychology, we define the construct of objectification in a structured thesaurus involving 5 sub-constructs manifesting through 11 concepts spanning 3 modalities. We introduce the Multimodal Objectifying Gaze (MObyGaze) dataset, made of 20 movies annotated densely by experts for objectification levels and concepts over freely delimited segments: it amounts to 6072 segments over 43 hours of video with fine-grained localization and categorization. We formulate different learning tasks, propose and investigate best ways to learn from the diversity of labels among a low number of annotators, and benchmark recent vision, text and audio models, showing the feasibility of the task. We make our code and our dataset available to the community and described in the Croissant format: https://anonymous.4open.science/r/MObyGaze-F600/.

MObyGaze: a film dataset of multimodal objectification densely annotated by experts

TL;DR

MObyGaze introduces a dense, expert-annotated multimodal film dataset to study objectification, a high-level construct rooted in film studies and psychology. It defines a thesaurus of 5 sub-constructs and 11 concepts across visual, textual, and audio modalities, and annotates 20 feature-length films to yield 6072 segments over 43 hours with detailed unitization and categorization. The paper formulates and benchmarks multiple learning tasks (classification and localization) under label diversity, using vision, text, and audio models, and proposes strategies to aggregate or separate labels across annotators. Results demonstrate feasibility of detecting objectification from vision and text signals, reveal the limited standalone value of audio, and highlight the value of label-diversity-aware training, with rich opportunities for explainability, fairness studies, and media studies applications.

Abstract

Characterizing and quantifying gender representation disparities in audiovisual storytelling contents is necessary to grasp how stereotypes may perpetuate on screen. In this article, we consider the high-level construct of objectification and introduce a new AI task to the ML community: characterize and quantify complex multimodal (visual, speech, audio) temporal patterns producing objectification in films. Building on film studies and psychology, we define the construct of objectification in a structured thesaurus involving 5 sub-constructs manifesting through 11 concepts spanning 3 modalities. We introduce the Multimodal Objectifying Gaze (MObyGaze) dataset, made of 20 movies annotated densely by experts for objectification levels and concepts over freely delimited segments: it amounts to 6072 segments over 43 hours of video with fine-grained localization and categorization. We formulate different learning tasks, propose and investigate best ways to learn from the diversity of labels among a low number of annotators, and benchmark recent vision, text and audio models, showing the feasibility of the task. We make our code and our dataset available to the community and described in the Croissant format: https://anonymous.4open.science/r/MObyGaze-F600/.

Paper Structure

This paper contains 46 sections, 4 equations, 10 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Bias analysis in film datasets
  4. Annotation of audiovisual and multimodal content
  5. Dataset creation for interpretive tasks
  6. Dataset
  7. Thesaurus of multimodal objectification
  8. Data selection
  9. Annotation
  10. Data format
  11. Validation of the data
  12. Analysis of the data
  13. New ML tasks: definitions, label diversity, models and experiments
  14. Categorizing and localizing objectification: possible task formulations
  15. Learning under label diversity
  16. ...and 31 more sections

Figures (10)

  • Figure 1: Examples of segments tagged with a Sure level of objectification. Top left: vision modality only. Top right: text modality only. Bottom: multimodal concepts producing objectification.
  • Figure 2: Thesaurus for the construct of objectification: 5 sub-constructs (left table) manifested through 11 concepts spanning 3 modalities (right table).
  • Figure 3: Descriptive analysis of the MObyGaze data. Left: distribution of label frequencies and corresponding durations. Right: number of concept occurrences for each objectification level.
  • Figure 4: Example of two dataset instances.
  • Figure 5: The annotation interface.
  • ...and 5 more figures