ShowMak3r: Compositional TV Show Reconstruction

TL;DR

ShowMak3r tackles reconstructing dynamic radiance fields from TV-show videos with shot changes by introducing a compositional 3D Gaussian representation for the stage and actors. The pipeline integrates 3DLocator for SMPL-to-stage alignment using depth priors, ShotMatcher for cross-shot actor association, and a face-fitting network to refine expressions, enabling novel-view rendering and per-actor editing. Quantitative and qualitative experiments on Sitcoms3D demonstrate improved stage reconstruction and accurate actor placement, while applications include actor relocation, insertion, deletion, and pose manipulation. This approach provides a practical, editable 4D reconstruction framework for TV shows, with potential impact on production analytics, post-production, and content editing.

Abstract

Reconstructing dynamic radiance fields from video clips is challenging, especially when entertainment videos like TV shows are given. Many challenges make the reconstruction difficult due to (1) actors occluding with each other and having diverse facial expressions, (2) cluttered stages, and (3) small baseline views or sudden shot changes. To address these issues, we present ShowMak3r, a comprehensive reconstruction pipeline that allows the editing of scenes like how video clips are made in a production control room. In ShowMak3r, a 3DLocator module locates recovered actors on the stage using depth prior and estimates unseen human poses via interpolation. The proposed ShotMatcher module then tracks the actors under shot changes. Furthermore, ShowMak3r introduces a face-fitting network that dynamically recovers the actors' expressions. Experiments on Sitcoms3D dataset show that our pipeline can reassemble TV show scenes with new cameras at different timestamps. We also demonstrate that ShowMak3r enables interesting applications such as synthetic shot-making, actor relocation, insertion, deletion, and pose manipulation. Project page : https://nstar1125.github.io/showmak3r

Figures (13)

• Figure 1: We introduce ShowMak3r, a comprehensive pipeline that reconstructs dynamic radiance fields from TV shows. Given a video clip that shows limited viewpoints and abrupt shot changes, our pipeline recovers the background stage and dynamic actors that are trackable across the shot changes. The scene is compositional and editable, so we can render it from novel viewpoints while selectively editing individual actors. Our method also recovers detailed human appearances, including facial expressions.
• Figure 2: Overview of our ShowMak3r pipeline. Given a TV show video clip, we perform dense reconstruction of the stage (Sec. \ref{['sec:background']}), locate SMPL models to the stage (Sec. \ref{['sec:3DLocator']}), associate SMPL models across shots to track actors (Sec. \ref{['sec:ShotMatcher']}), and recover the detailed appearance of the actors (Sec. \ref{['sec:human']}). 3D Gaussians of the stage and the actors are rendered to produce novel frames.
• Figure 3: An example of transient object removal.
• Figure 4: Results of actor association. ShotMatcher can associate actors even when some individuals do not appear in a shot. If the distance of the matched actors is above the matching threshold, ShotMatcher identifies them as different.
• Figure 5: An effect of using the proposed foreground masking.