Salient Temporal Encoding for Dynamic Scene Graph Generation

TL;DR

The paper tackles the problem of modeling dynamic scenes by constructing sparse, explicit spatial-temporal scene graphs. It introduces the Salient Temporal Relation Encoder (STRE), which first builds a spatial scene graph per frame, then selectively connects temporally relevant object pairs through Saliency Attention, and finally encodes these temporal relations as explicit features to refine the graph. Key contributions include a sparse, explicit temporal encoding scheme, a joint refinement framework that improves standard scene graph generation by up to $4.4\%$ in SGDet and yields downstream gains in action recognition (up to $0.6\%$ mAP). The approach also provides a reusable temporal feature bank (ST-SGFB) for action recognition and demonstrates improved efficiency and interpretability over dense temporal connections, enabling better real-world dynamic scene understanding.

Abstract

Representing a dynamic scene using a structured spatial-temporal scene graph is a novel and particularly challenging task. To tackle this task, it is crucial to learn the temporal interactions between objects in addition to their spatial relations. Due to the lack of explicitly annotated temporal relations in current benchmark datasets, most of the existing spatial-temporal scene graph generation methods build dense and abstract temporal connections among all objects across frames. However, not all temporal connections are encoding meaningful temporal dynamics. We propose a novel spatial-temporal scene graph generation method that selectively builds temporal connections only between temporal-relevant objects pairs and represents the temporal relations as explicit edges in the scene graph. The resulting sparse and explicit temporal representation allows us to improve upon strong scene graph generation baselines by up to $4.4\%$ in Scene Graph Detection. In addition, we show that our approach can be leveraged to improve downstream vision tasks. Particularly, applying our approach to action recognition, shows 0.6\% gain in mAP in comparison to the state-of-the-art

Figures (5)

• Figure 1: Top row shows a person pouring water into a cup and then drinking the water from the cup. Second and third row compare state-of-the-art dense temporal connections and our sparse and explicit temporal connections for dynamic scene graph generation. Circular nodes (blue, green and orange) represent the detected objects, and rectangular nodes (brown and red) represent our encoded temporal relations.
• Figure 2: Overview of Salient Temporal Relation Encoder (STRE). From left to right, frame $t$ is input into PISP herzig2018mapping to obtain an initial spatial scene graph proposal. To aggregate spatial context for each node, the spatial scene graph prediction from the previous frame $t-1$ are input into a Graph Convolutional Network. The updated node features are passed into the Saliency Attention module to calculate temporal relevance score. The pairs with the highest temporal relevance scores are selected for encoding the temporal relation via the Temporal Relation Encoder. The spatial context and encoded temporal relations are summed up and passed into the spatial-temporal scene graph refinement module to obtain the final graph.
• Figure 3: Qualitative comparison of scene graphs generated by: (b) PISPherzig2018mapping, (c) STTrancong2021spatial and (d) our STRE model. Green box in the bottom-left image indicates a detected floor which does not exist in the ground-truth. For better visualization, only the top 12 relations from no constraint results are plotted, excluding the relations with the floor. The orange-colored relations are correct predictions, the red-colored relations are incorrect predictions that do not exist in the ground-truth, and the gray-colored relations are missing predictions.
• Figure 4: Two examples showing temporal connections between different objects across frames predicted by STRE. Green arrows represent temporal connections.
• Figure 5: Action Recognition qualitative results from two models: (1) S-SGFB which uses spatial scene graph as feature bank, (2) ST-SGFB which uses spatial-temporal scene graph as feature bank. Red means wrong predictions.