SketchDynamics: Exploring Free-Form Sketches for Dynamic Intent Expression in Animation Generation

TL;DR

SketchDynamics demonstrates how free-form sketches can communicate dynamic animation intent to AI without rigid command vocabularies. By coupling a vision–language model with adaptive clarification and iterative refinement cues, the approach converts sketch storyboards into vector animations, validated through a three-stage user study that reveals both expressiveness and the need for user-in-the-loop guidance. The results show improved alignment between user intent and output as stages progress, supporting a lazy-specification, co-creative workflow. The work also outlines generalizable extensions to video generation and 3D scenes, suggesting a scalable framework for sketch-driven, human–AI collaborative content creation.

Abstract

Sketching provides an intuitive way to convey dynamic intent in animation authoring (i.e., how elements change over time and space), making it a natural medium for automatic content creation. Yet existing approaches often constrain sketches to fixed command tokens or predefined visual forms, overlooking their freeform nature and the central role of humans in shaping intention. To address this, we introduce an interaction paradigm where users convey dynamic intent to a vision-language model via free-form sketching, instantiated here in a sketch storyboard to motion graphics workflow. We implement an interface and improve it through a three-stage study with 24 participants. The study shows how sketches convey motion with minimal input, how their inherent ambiguity requires users to be involved for clarification, and how sketches can visually guide video refinement. Our findings reveal the potential of sketch and AI interaction to bridge the gap between intention and outcome, and demonstrate its applicability to 3D animation and video generation.

Figures (11)

• Figure 1: Prior systems davis2008ksketchkazi2016motionkazi2014dracokazi2014kitty treat sketches as symbols to be mapped to animation commands (Top). In contrast, we treat sketches as semantic expressions of motion intent (Bottom), leveraging VLM-based commonsense reasoning and human-AI clarification to interpret why and how motion happens.
• Figure 2: Study illustration showing the three stages of our investigation: (1) sketch input and direct generation, (2) clarification through disambiguation cues, and (3) iterative refinement with contextual editing. Each stage involved eight new participants in the study.
• Figure 3: Interface design for the first Stage. (A) The user sketches a frame-level storyboard to explain the law of reflection, drawing incident and reflected light rays on a mirror surface. (B) adds brief notes at the bottom to specify the cues, and (C) generates a preview video from the storyboard.
• Figure 4: Selected sketches with scripts and animation excerpts from participants’ storyboards in Stage 1. Above the sketch is the script(N/A means no script); the gray text below describes the animation. These examples illustrate common ways sketches were used to convey animation intentions (e.g., translation, scaling, rotation, appearance). The categories shown here are not meant as a strict taxonomy but as illustrative examples; in practice, participants’ free-form drawings were far more varied and often mixed multiple notations within a single sketch. Some animations shown are refined versions to better reflect participants’ intended outcomes.
• Figure 5: Illustration of ambiguity clarification. From left to right, the examples show sketch ambiguities ranging from low to high. Top: different types of ambiguities in storyboards, Bottom: the corresponding types of information needed to resolve them, mapped to our clarification strategies.