Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Multiple Object States from Actions via Large Language Models

Masatoshi Tateno, Takuma Yagi, Ryosuke Furuta, Yoichi Sato

TL;DR

Experimental results show that the model trained on LLM-generated pseudo-labels significantly outperforms strong vision-language models, demonstrating the effectiveness of the pseudo-labeling framework that considers past context via LLMs.

Abstract

Recognizing the states of objects in a video is crucial in understanding the scene beyond actions and objects. For instance, an egg can be raw, cracked, and whisked while cooking an omelet, and these states can coexist simultaneously (an egg can be both raw and whisked). However, most existing research assumes a single object state change (e.g., uncracked -> cracked), overlooking the coexisting nature of multiple object states and the influence of past states on the current state. We formulate object state recognition as a multi-label classification task that explicitly handles multiple states. We then propose to learn multiple object states from narrated videos by leveraging large language models (LLMs) to generate pseudo-labels from the transcribed narrations, capturing the influence of past states. The challenge is that narrations mostly describe human actions in the video but rarely explain object states. Therefore, we use the LLMs knowledge of the relationship between actions and states to derive the missing object states. We further accumulate the derived object states to consider past state contexts to infer current object state pseudo-labels. We newly collect a dataset called the Multiple Object States Transition (MOST) dataset, which includes manual multi-label annotation for evaluation purposes, covering 60 object states across six object categories. Experimental results show that our model trained on LLM-generated pseudo-labels significantly outperforms strong vision-language models, demonstrating the effectiveness of our pseudo-labeling framework that considers past context via LLMs.

Learning Multiple Object States from Actions via Large Language Models

TL;DR

Experimental results show that the model trained on LLM-generated pseudo-labels significantly outperforms strong vision-language models, demonstrating the effectiveness of the pseudo-labeling framework that considers past context via LLMs.

Abstract

Recognizing the states of objects in a video is crucial in understanding the scene beyond actions and objects. For instance, an egg can be raw, cracked, and whisked while cooking an omelet, and these states can coexist simultaneously (an egg can be both raw and whisked). However, most existing research assumes a single object state change (e.g., uncracked -> cracked), overlooking the coexisting nature of multiple object states and the influence of past states on the current state. We formulate object state recognition as a multi-label classification task that explicitly handles multiple states. We then propose to learn multiple object states from narrated videos by leveraging large language models (LLMs) to generate pseudo-labels from the transcribed narrations, capturing the influence of past states. The challenge is that narrations mostly describe human actions in the video but rarely explain object states. Therefore, we use the LLMs knowledge of the relationship between actions and states to derive the missing object states. We further accumulate the derived object states to consider past state contexts to infer current object state pseudo-labels. We newly collect a dataset called the Multiple Object States Transition (MOST) dataset, which includes manual multi-label annotation for evaluation purposes, covering 60 object states across six object categories. Experimental results show that our model trained on LLM-generated pseudo-labels significantly outperforms strong vision-language models, demonstrating the effectiveness of our pseudo-labeling framework that considers past context via LLMs.
Paper Structure (36 sections, 1 equation, 19 figures, 22 tables)

This paper contains 36 sections, 1 equation, 19 figures, 22 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Learning Object States from Videos
  4. Learning from Video Narrations
  5. LLMs for Visual Learning
  6. Problem Setting
  7. Method
  8. Object State Label Extraction from Narrations via LLMs
  9. Interval Alignment
  10. Self-Training from Pseudo-Labels
  11. Experiments
  12. Multiple Object States Transition Dataset (MOST)
  13. Implementation Details
  14. Evaluation on MOST Dataset
  15. Ablation Study
  16. ...and 21 more sections

Figures (19)

  • Figure 1: We formulate object state recognition as a multi-label frame-wise classification task (bottom). Compared to formulation focusing on state change in previous works (top), our formulation explicitly considers coexisting multiple object states.
  • Figure 1: Data distribution of MOST dataset. Y-axis denotes number of videos that each state presents.
  • Figure 2: Three-stage framework of inferring presence of object state labels from video narrations. (a) Manipulation action extraction from narrations. (b) Object state description generation from actions. (c) Context-aware object state label inference.
  • Figure 2: Temporal alignment between video frames and action/state description. (a) Given video frame, we select most likely actions from candidates using VLMs. (b) Interval is further filtered by matching frame and state description.
  • Figure 3: Annotation example of MOST dataset. Red bands show presence of object states by time (seconds). All labels are marked negative if target objects are not visible.
  • ...and 14 more figures