Sketch2Anim: Towards Transferring Sketch Storyboards into 3D Animation

TL;DR

This work tackles the challenge of converting sketch-based storyboards into 3D animations by introducing a two-module system: a neural mapper that aligns 2D storyboard cues with 3D keyposes and trajectories in a shared embedding space, and a multi-conditional motion generator that conditions on 3D keyposes, 3D trajectories, and action words to synthesize per-frame 3D motions via diffusion models. A trajectory ControlNet and a trajectory-aware keypose adapter enable simultaneous, balanced control over dynamic and static motion aspects, while a 2D-3D alignment module allows direct 2D inputs to drive generation during inference. Extensive experiments on synthetic and real sketches, including ablations and a user perceptual study, demonstrate that Sketch2Anim achieves superior motion realism, precise control of keyposes and trajectories, and strong text-motion alignment compared with baselines. The approach supports 3D motion editing and real-world storyboard workflows, offering a practical path toward automated storyboard-to-animation pipelines. The work also discusses limitations (e.g., lack of object interactions and physical constraints) and outlines future directions such as speed control and scene recovery to further enhance applicability.

Abstract

Storyboarding is widely used for creating 3D animations. Animators use the 2D sketches in storyboards as references to craft the desired 3D animations through a trial-and-error process. The traditional approach requires exceptional expertise and is both labor-intensive and time-consuming. Consequently, there is a high demand for automated methods that can directly translate 2D storyboard sketches into 3D animations. This task is under-explored to date and inspired by the significant advancements of motion diffusion models, we propose to address it from the perspective of conditional motion synthesis. We thus present Sketch2Anim, composed of two key modules for sketch constraint understanding and motion generation. Specifically, due to the large domain gap between the 2D sketch and 3D motion, instead of directly conditioning on 2D inputs, we design a 3D conditional motion generator that simultaneously leverages 3D keyposes, joint trajectories, and action words, to achieve precise and fine-grained motion control. Then, we invent a neural mapper dedicated to aligning user-provided 2D sketches with their corresponding 3D keyposes and trajectories in a shared embedding space, enabling, for the first time, direct 2D control of motion generation. Our approach successfully transfers storyboards into high-quality 3D motions and inherently supports direct 3D animation editing, thanks to the flexibility of our multi-conditional motion generator. Comprehensive experiments and evaluations, and a user perceptual study demonstrate the effectiveness of our approach.

Figures (16)

• Figure 1: Traditional 3D animation workflow. Given one frame of a sketch storyboard (top-left), animators first imagine the desired keypose sequence guided by the trajectory and the action word. Two such representative keyposes are shown in blue. These keyposes (including the first keypose) are imported into 3D software (e.g., Blender) as keypose and motion references. The animators then manually place the pre-defined 3D joints to match the keypose, while crafting the desired motion to explain the trajectory and action word. After a long trial-and-error process, a high-quality animation is produced. See the supplemental video for the whole manual process.
• Figure 2: Overview of Sketch2Anim. Our pipeline consists of two core modules - the multi-conditional motion generator (Sec. \ref{['sec:generator']}) and the 2D-3D neural mapper (Sec. \ref{['sec:mapper']}). Instead of directly lifting the 2D keypose and trajectory into their 3D counterparts, we train a neural mapper dedicated to aligning the two domains in the embedding space. Because of this shared embedding, it enables the employment of more informative and precise 3D keyposes and trajectories as the motion conditions in the motion generator, while exploiting the 2D keypose and trajectory detected from the sketch storyboard at inference time. The legend indicates the data flow at training and inference of both modules . See the following sections for detailed technical designs.
• Figure 3: User interface and joint detection. (a) In the interface, the white canvas is the main drawing area. Users can sketch or type the action word. We use different pens for character strokes and trajectories. (b) We use Sketch2Pose to detect the 2D joint points (red points), which serve as our input, instead of the raw sketch in (a); and we draw the line segments between joint points only for intuitive visualization. The uniformly re-sampled trajectory points are drawn in cyan.
• Figure 4: The structure of our motion generator. A trajectory ControlNet is exploited to incorporate the trajectory control, while a keypose adapter is placed between the ControlNet and the diffusion model to inject the keypose condition. The same color and style of the common elements and data flow as in Fig. \ref{['fig:pipeline']} are used. Refer to Sec. \ref{['sec:generator']} for a detailed explanation.
• Figure 5: Result Gallery. Three more storyboards are successfully transferred into their high-quality animations. Only the motion clips before blending are shown for clear visualization of keyposes, trajectories, and motion. The complete animation after blending can be seen in the supplemental video. For each clip, we highlight the keypose in red, while the trajectories are spatial curves shown in green. The motions of "Kneel", "Cartwheel", and "Sit" are usually hard to generate, and our results are of high quality conforming with the sketches.