Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Lifelong Scene Graph Generation with Knowledge-ware In-context Prompt Learning

Tao He, Tongtong Wu, Dongyang Zhang, Guiduo Duan, Ke Qin, Yuan-Fang Li

TL;DR

This work tackles lifelong scene graph generation by treating predicate prediction as a streaming continual-learning problem and addresses forgetting with knowledge-aware in-context prompts. It maps visual features to symbolic textual representations and uses a memory-augmented prompt-retrieval strategy to assemble contextual demonstrations for a pretrained language model. The approach, ICSGG, demonstrates strong performance gains over state-of-the-art SGG models on Visual Genome and Open-Image under lifelong settings, while also improving standard SGG after conventional training. The results highlight the practical potential of prompting-based continual learning for structured visual reasoning, with interpretable knowledge prompts and exemplars guiding relation prediction. Limitations include not training the object detector end-to-end, suggesting future work toward an end-to-end LSGG framework.

Abstract

Scene graph generation (SGG) endeavors to predict visual relationships between pairs of objects within an image. Prevailing SGG methods traditionally assume a one-off learning process for SGG. This conventional paradigm may necessitate repetitive training on all previously observed samples whenever new relationships emerge, mitigating the risk of forgetting previously acquired knowledge. This work seeks to address this pitfall inherent in a suite of prior relationship predictions. Motivated by the achievements of in-context learning in pretrained language models, our approach imbues the model with the capability to predict relationships and continuously acquire novel knowledge without succumbing to catastrophic forgetting. To achieve this goal, we introduce a novel and pragmatic framework for scene graph generation, namely Lifelong Scene Graph Generation (LSGG), where tasks, such as predicates, unfold in a streaming fashion. In this framework, the model is constrained to exclusive training on the present task, devoid of access to previously encountered training data, except for a limited number of exemplars, but the model is tasked with inferring all predicates it has encountered thus far. Rigorous experiments demonstrate the superiority of our proposed method over state-of-the-art SGG models in the context of LSGG across a diverse array of metrics. Besides, extensive experiments on the two mainstream benchmark datasets, VG and Open-Image(v6), show the superiority of our proposed model to a number of competitive SGG models in terms of continuous learning and conventional settings. Moreover, comprehensive ablation experiments demonstrate the effectiveness of each component in our model.

Towards Lifelong Scene Graph Generation with Knowledge-ware In-context Prompt Learning

TL;DR

This work tackles lifelong scene graph generation by treating predicate prediction as a streaming continual-learning problem and addresses forgetting with knowledge-aware in-context prompts. It maps visual features to symbolic textual representations and uses a memory-augmented prompt-retrieval strategy to assemble contextual demonstrations for a pretrained language model. The approach, ICSGG, demonstrates strong performance gains over state-of-the-art SGG models on Visual Genome and Open-Image under lifelong settings, while also improving standard SGG after conventional training. The results highlight the practical potential of prompting-based continual learning for structured visual reasoning, with interpretable knowledge prompts and exemplars guiding relation prediction. Limitations include not training the object detector end-to-end, suggesting future work toward an end-to-end LSGG framework.

Abstract

Scene graph generation (SGG) endeavors to predict visual relationships between pairs of objects within an image. Prevailing SGG methods traditionally assume a one-off learning process for SGG. This conventional paradigm may necessitate repetitive training on all previously observed samples whenever new relationships emerge, mitigating the risk of forgetting previously acquired knowledge. This work seeks to address this pitfall inherent in a suite of prior relationship predictions. Motivated by the achievements of in-context learning in pretrained language models, our approach imbues the model with the capability to predict relationships and continuously acquire novel knowledge without succumbing to catastrophic forgetting. To achieve this goal, we introduce a novel and pragmatic framework for scene graph generation, namely Lifelong Scene Graph Generation (LSGG), where tasks, such as predicates, unfold in a streaming fashion. In this framework, the model is constrained to exclusive training on the present task, devoid of access to previously encountered training data, except for a limited number of exemplars, but the model is tasked with inferring all predicates it has encountered thus far. Rigorous experiments demonstrate the superiority of our proposed method over state-of-the-art SGG models in the context of LSGG across a diverse array of metrics. Besides, extensive experiments on the two mainstream benchmark datasets, VG and Open-Image(v6), show the superiority of our proposed model to a number of competitive SGG models in terms of continuous learning and conventional settings. Moreover, comprehensive ablation experiments demonstrate the effectiveness of each component in our model.
Paper Structure (39 sections, 11 equations, 3 figures, 6 tables)

This paper contains 39 sections, 11 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Scene Graph Generation
  4. Prompt-based Learning
  5. Lifelong learning
  6. Preliminaries
  7. Problem Statement
  8. Prompt-based learning
  9. In-context learning
  10. Method
  11. Visual features to symbolic representations
  12. Visual feature extraction
  13. Visual-language to symbolic representations
  14. In-context prompts for LSGG
  15. Knowledge-aware prompts learning
  16. ...and 24 more sections

Figures (3)

  • Figure 1: The overall framework of our proposed in-context-based prompt method for life long scene graph generation. There are three main components in our method: (1) visual to textual embedding transformation; (2) knowledge-aware prompts learning; and (3) in-context exemplars selection for the prompt.
  • Figure 2: Results of four tasks at different training stages on the PredCls of mR@$100$ on VG.
  • Figure 3: Visualization results of four images from the VG. We show our transformed textual representations for the context, relationship and the comparison results on the task of PredCls with VS$^3$zhang2023learning.