Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Frames to Sequences: Temporally Consistent Human-Centric Dense Prediction

Xingyu Miao, Junting Dong, Qin Zhao, Yuhang Yang, Junhao Chen, Yang Long

TL;DR

This work tackles temporally consistent, multi-task dense prediction for humans in videos by pairing a scalable synthetic data pipeline with a ViT-based predictor that injects explicit human geometry priors and uses adaptive channel weighting. A two-stage training regime combines static image pretraining with dynamic sequence supervision via flow-based temporal stabilization, yielding robust depth, normals, and segmentation across motion. The approach achieves state-of-the-art results on THuman2.1 and Hi4D and generalizes well to in-the-wild video, highlighting the value of large-scale synthetic supervision and human priors for temporal human-centric perception. Overall, the combination of photorealistic synthetic data, temporal supervision, and geometry-aware modeling provides a practical path toward temporally coherent, multi-task dense predictions for human-centric video analysis.

Abstract

In this work, we focus on the challenge of temporally consistent human-centric dense prediction across video sequences. Existing models achieve strong per-frame accuracy but often flicker under motion, occlusion, and lighting changes, and they rarely have paired human video supervision for multiple dense tasks. We address this gap with a scalable synthetic data pipeline that generates photorealistic human frames and motion-aligned sequences with pixel-accurate depth, normals, and masks. Unlike prior static data synthetic pipelines, our pipeline provides both frame-level labels for spatial learning and sequence-level supervision for temporal learning. Building on this, we train a unified ViT-based dense predictor that (i) injects an explicit human geometric prior via CSE embeddings and (ii) improves geometry-feature reliability with a lightweight channel reweighting module after feature fusion. Our two-stage training strategy, combining static pretraining with dynamic sequence supervision, enables the model first to acquire robust spatial representations and then refine temporal consistency across motion-aligned sequences. Extensive experiments show that we achieve state-of-the-art performance on THuman2.1 and Hi4D and generalize effectively to in-the-wild videos.

From Frames to Sequences: Temporally Consistent Human-Centric Dense Prediction

TL;DR

This work tackles temporally consistent, multi-task dense prediction for humans in videos by pairing a scalable synthetic data pipeline with a ViT-based predictor that injects explicit human geometry priors and uses adaptive channel weighting. A two-stage training regime combines static image pretraining with dynamic sequence supervision via flow-based temporal stabilization, yielding robust depth, normals, and segmentation across motion. The approach achieves state-of-the-art results on THuman2.1 and Hi4D and generalizes well to in-the-wild video, highlighting the value of large-scale synthetic supervision and human priors for temporal human-centric perception. Overall, the combination of photorealistic synthetic data, temporal supervision, and geometry-aware modeling provides a practical path toward temporally coherent, multi-task dense predictions for human-centric video analysis.

Abstract

In this work, we focus on the challenge of temporally consistent human-centric dense prediction across video sequences. Existing models achieve strong per-frame accuracy but often flicker under motion, occlusion, and lighting changes, and they rarely have paired human video supervision for multiple dense tasks. We address this gap with a scalable synthetic data pipeline that generates photorealistic human frames and motion-aligned sequences with pixel-accurate depth, normals, and masks. Unlike prior static data synthetic pipelines, our pipeline provides both frame-level labels for spatial learning and sequence-level supervision for temporal learning. Building on this, we train a unified ViT-based dense predictor that (i) injects an explicit human geometric prior via CSE embeddings and (ii) improves geometry-feature reliability with a lightweight channel reweighting module after feature fusion. Our two-stage training strategy, combining static pretraining with dynamic sequence supervision, enables the model first to acquire robust spatial representations and then refine temporal consistency across motion-aligned sequences. Extensive experiments show that we achieve state-of-the-art performance on THuman2.1 and Hi4D and generalize effectively to in-the-wild videos.
Paper Structure (35 sections, 18 equations, 9 figures, 19 tables)

This paper contains 35 sections, 18 equations, 9 figures, 19 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Human vision data
  4. Human vision Task
  5. Dense Prediction Architectures
  6. Methodology
  7. Human-centric Synthetic Data Pipeline
  8. Composition stage.
  9. Rendering stage.
  10. Model Architecture
  11. Encoder and Decoder.
  12. Local Geometry Enhancement.
  13. Channel Weight Adaptation (CWA).
  14. Human Geometric Prior.
  15. Training Pipeline
  16. ...and 20 more sections

Figures (9)

  • Figure 1: Synthesis Data sample. Ground-truth synthetic annotations of depth, surface normals, and masks.
  • Figure 2: Pipeline overview. Given a sequence of RGB frames, our model extracts DINO features, global image features, and human geometric priors. These features are fused and re-weighted to generate enhanced representations for predicting temporally consistent depth, surface normals, and segmentation masks.
  • Figure 3: Qualitative comparison on challenging images in the wild.
  • Figure 4: Qualitative comparison on video depth estimation. For better visualization, we also show the time slice on the red lines of each video on their right side.
  • Figure 5: Qualitative comparison on video surface normal estimation. For better visualization, we also show the time slice on the red lines of each video on their right side.
  • ...and 4 more figures