RealityEffects: Augmenting 3D Volumetric Videos with Object-Centric Annotation and Dynamic Visual Effects

TL;DR

RealityEffects addresses the lack of authoring tools for object-centric augmentation of 3D volumetric videos by introducing a desktop interface that binds dynamic visual effects to tracked physical objects. It presents a taxonomy of object-centric augmentation techniques (text labels, highlights, embedded visuals, connected links, and motion effects) grounded in an analysis of 120 edited videos and demonstrates a working system with streaming, processing, and augmenting modules. A lab usability study with 19 participants indicates that direct manipulation lowers the barrier to editing volumetric captures while enabling expressive, interactive viewing experiences. The work contributes a taxonomy, a practical authoring tool, and initial evidence of the approach's potential across domains such as sports analysis, education, and product visualization, while outlining avenues for automation, multi-camera capture, and immersive MR extensions.

Abstract

This paper introduces RealityEffects, a desktop authoring interface designed for editing and augmenting 3D volumetric videos with object-centric annotations and visual effects. RealityEffects enhances volumetric capture by introducing a novel method for augmenting captured physical motion with embedded, responsive visual effects, referred to as object-centric augmentation. In RealityEffects, users can interactively attach various visual effects to physical objects within the captured 3D scene, enabling these effects to dynamically move and animate in sync with the corresponding physical motion and body movements. The primary contribution of this paper is the development of a taxonomy for such object-centric augmentations, which includes annotated labels, highlighted objects, ghost effects, and trajectory visualization. This taxonomy is informed by an analysis of 120 edited videos featuring object-centric visual effects. The findings from our user study confirm that our direct manipulation techniques lower the barriers to editing and annotating volumetric captures, thereby enhancing interactive and engaging viewing experiences of 3D volumetric videos.

Figures (9)

• Figure 1: Design Space Analysis: We collected a set of 120 existing edited videos and images and observed the five most common design techniques, namely, Text Annotation, Object Highlight, Embedded Visual, Connected Link, and Motion Effect. The screenshots are copyrighted by each video creator. We listed the link for each video in the Appendix.
• Figure 2: RealityEffects consists of three modules -- (1) Streaming: Azure Kinect SDK provides the depth camera data feed, (2) Processing: Mediapipe and OpenCV for body and color tracking (3) Augmenting: Three.js for visual rendering.
• Figure 3: Authoring Workflow: Collection of examples from RealityEffects's workflow to demonstrate features such as dynamic parameters, highlighting and annotations.
• Figure 4: Product Showcase: Use case scenario demonstrating a sales pitch for a handheld camera, where labels, visualizations, and highlights are used to enhance the product's appeal and provide information.
• Figure 5: Chemistry Lab Training: Use case scenario highlighting safe practices, necessary precautions, and potential lab dangers related to cross-contamination.