Visual Semantic Role Labeling

TL;DR

This work defines Visual Semantic Role Labeling (V-SRL), a task to detect agents, actions, and objects across semantic roles in images, moving beyond coarse action labels. It introduces V-COCO, a COCO-derived dataset annotating 16K people in 10K images with 26 verbs and role-bound objects, plus baseline CNN-based methods for agent and role localization. The study analyzes agent and role detection performance, reveals predominant error modes (e.g., incorrect action labels and mislocalization for small objects), and provides a foundation for grounded, fine-grained action understanding. Overall, the paper establishes a new benchmark for joint grounding of actions and object roles, enabling more context-aware visual reasoning and future improvements in scene understanding.

Abstract

In this paper we introduce the problem of Visual Semantic Role Labeling: given an image we want to detect people doing actions and localize the objects of interaction. Classical approaches to action recognition either study the task of action classification at the image or video clip level or at best produce a bounding box around the person doing the action. We believe such an output is inadequate and a complete understanding can only come when we are able to associate objects in the scene to the different semantic roles of the action. To enable progress towards this goal, we annotate a dataset of 16K people instances in 10K images with actions they are doing and associate objects in the scene with different semantic roles for each action. Finally, we provide a set of baseline algorithms for this task and analyze error modes providing directions for future work.

Figures (7)

• Figure 1: Visual Semantic Role Labeling: We want to go beyond classifying the action occurring in the image to being able to localize the agent, and the objects in various semantic roles associated with the action.
• Figure 2: Visualizations of the images in the dataset: We show examples of annotations in the dataset. We show the agent in the blue box and objects in various semantic roles in red. People can be doing multiple actions at the same time. First row shows: person skateboarding, person sitting and riding on horse, person sitting on chair and riding on elephant, person drinking from a glass and sitting on a chair, person sitting on a chair eating a doughnut.
• Figure 3: Interface for collecting annotations: The top row shows the interface for annotating the action the person is doing, and the bottom row shows the interface for associating the roles.
• Figure 4: Statistics on V-COCO: The bar plot on left shows the distribution of the number of annotated people per image. The bar plot on right shows the distribution of the number of actions a person is doing. Note that X-axis is on $\log$ scale.
• Figure 5: Human Agreement