Generative AI for 2D Character Animation

TL;DR

The paper addresses the bottleneck of producing educational 2D animation by developing AI-driven workflows that empower artists to realize character animation more efficiently. It leverages a shared SDXL fine-tuning backbone with DreamBooth, supplemented by Conditioning via ControlNet, SegmentAnything, AnimateDiff, and SDEdit across four pipelines to generate ink, paint, and motion. Key findings include the need for depth conditioning and background matting to stabilize outputs, and the trade-off between automation and manual cleanup, with about 50 shots completed by a small team in eight weeks. The work demonstrates a practical, artist-centric pipeline for 2D educational animation and highlights avenues for improvement through larger, more domain-specific data and refined conditioning strategies.

Abstract

In this pilot project, we teamed up with artists to develop new workflows for 2D animation while producing a short educational cartoon. We identified several workflows to streamline the animation process, bringing the artists' vision to the screen more effectively.

Figures (9)

• Figure 1: Schematic diagram showing three of our generative AI animation workflows
• Figure 2: A sample frame of input and output for the character LUTHER from our AI ink and paint workflow.
• Figure A.1: Hand-drawn illustrations used to fine-tune SDXL for the character VERNA. Character design by Kelly McNutt.
• Figure A.2: Sample text-to-image outputs for the VERNA DreamBooth model with no additional conditioning. Note the proportions of her head to body and eyes to face are inconsistent across different random seeds. However, other characteristics like the curl of her hair or the shape of her dress and hands are well-preserved across generations.
• Figure A.3: Part of a ComfyUI workflow for FRED MID-EVIL using depth and edge ControlNets.