Auditing ImageNet: Towards a Model-driven Framework for Annotating Demographic Attributes of Large-Scale Image Datasets

TL;DR

This paper introduces a model-driven framework to annotate apparent age and gender in ImageNet to enable a demographic audit of the ILSVRC 2012 subset and the 'person' subset. It combines FaceBoxes-based face detection with DEX-based age and gender inference, evaluates biases in annotation models using APPA-REAL and PPB, and reports preliminary demographic imbalances (e.g., 41.62% female in ILSVRC, 1.71% over 60). The authors acknowledge annotation biases and propose future work toward fairer models and broader diversity metrics, aiming to assess downstream effects on pretrained CNN representations. Overall, it establishes a starting point for systematic demographic auditing of large-scale image datasets and their influence on transfer learning.

Abstract

The ImageNet dataset ushered in a flood of academic and industry interest in deep learning for computer vision applications. Despite its significant impact, there has not been a comprehensive investigation into the demographic attributes of images contained within the dataset. Such a study could lead to new insights on inherent biases within ImageNet, particularly important given it is frequently used to pretrain models for a wide variety of computer vision tasks. In this work, we introduce a model-driven framework for the automatic annotation of apparent age and gender attributes in large-scale image datasets. Using this framework, we conduct the first demographic audit of the 2012 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) subset of ImageNet and the "person" hierarchical category of ImageNet. We find that 41.62% of faces in ILSVRC appear as female, 1.71% appear as individuals above the age of 60, and males aged 15 to 29 account for the largest subgroup with 27.11%. We note that the presented model-driven framework is not fair for all intersectional groups, so annotation are subject to bias. We present this work as the starting point for future development of unbiased annotation models and for the study of downstream effects of imbalances in the demographics of ImageNet. Code and annotations are available at: http://bit.ly/ImageNetDemoAudit