SketchPlay: Intuitive Creation of Physically Realistic VR Content with Gesture-Driven Sketching

TL;DR

SketchPlay addresses the barrier to creating physically plausible VR content by enabling users to sketch in air for structure and gesture dynamics for motion, then automatically synthesizing executable Blender physics scripts into coherent 4D scenes. It introduces a three-stage pipeline—Sketch Drawing and Gesture Intent Recognition, Physically Realistic Motion Simulation, and Scenario Synthesis—leveraging MediaPipe, Grounded-SAM, SketchDream, a Vision-Language Model, GPT-4o, and 4D Gaussian Splatting to ensure physical plausibility and visual fidelity. The work's key contributions include a novel two-input creation paradigm, a GPT-4o-driven material/mass inference mechanism to drive motion, and a multi-stage synthesis workflow that yields photorealistic, temporally coherent 4D content. Results show SketchPlay surpasses text-driven and baselines in expressiveness, realism, and user experience, highlighting strong potential for education, art, and immersive storytelling.

Abstract

Creating physically realistic content in VR often requires complex modeling tools or predefined 3D models, textures, and animations, which present significant barriers for non-expert users. In this paper, we propose SketchPlay, a novel VR interaction framework that transforms humans' air-drawn sketches and gestures into dynamic, physically realistic scenes, making content creation intuitive and playful like drawing. Specifically, sketches capture the structure and spatial arrangement of objects and scenes, while gestures convey physical cues such as velocity, direction, and force that define movement and behavior. By combining these complementary forms of input, SketchPlay captures both the structure and dynamics of user-created content, enabling the generation of a wide range of complex physical phenomena, such as rigid body motion, elastic deformation, and cloth dynamics. Experimental results demonstrate that, compared to traditional text-driven methods, SketchPlay offers significant advantages in expressiveness, and user experience. By providing an intuitive and engaging creation process, SketchPlay lowers the entry barrier for non-expert users and shows strong potential for applications in education, art, and immersive storytelling.

Figures (8)

• Figure 1: SketchPlay Pipeline. Our pipeline consists of three core stages: (1) Sketch Drawing and Gesture Intent Recognition, where dynamic physical information like velocity and direction is extracted from the user's gesture; (2) Physically Realistic Motion Simulation, where a physics engine uses this information to simulate realistic behaviors; and (3) Scenario Synthesis, which uses physical priors like edge and depth maps from the simulation to render a photorealistic video.
• Figure 2: Overview of Motion Simulation. Inputs (Prompt, Gesture, Sketch) are fused by a VLM to (1) parse intent and layout, (2) infer materials and mass $m$, compute $v_{\text{obj}}$ from the gesture with $\alpha_{\text{material}}$, and set collision/friction, and (3) generate a Blender Python script to run the simulation, yielding physically consistent dynamics aligned with the sketch.
• Figure 3: Illustration of the gesture capture process in our 'Air Drawing' stage. The system uses a hand-tracking algorithm (e.g., MediaPipezhang2020mediapipe) to capture the keypoints of a user's hand motion in 3D space. This raw trajectory is then refined into a clean sketch, which forms the basis for the subsequent physics simulation.
• Figure 4: Qualitative comparison of Blender simulations for the 'domino fall' task. The frames are rendered from scripts generated by four different methods. SketchPlay produces a physically plausible and coherent chain reaction, whereas scripts from baseline models result in chaotic collapses (BlenderGPT), unnatural scattering (SceneCraft), or implausible physics (GPT4Motion).
• Figure 5: Qualitative evaluation of SketchPlay against sketch-driven baselines (SparseCtrl, SketchVideo, VidSketch). Across a variety of challenging physical scenarios shown here, SketchPlay consistently generates videos with higher physical realism and fewer visual artifacts, demonstrating the effectiveness of our physics-based approach.