FilmAgent: A Multi-Agent Framework for End-to-End Film Automation in Virtual 3D Spaces

TL;DR

Human evaluation shows that FilmAgent outperforms all baselines across all aspects and scores 3.98 out of 5 on average, showing the feasibility of multi-agent collaboration in filmmaking, and reveals that FilmAgent, despite using the less advanced GPT-4o model, surpasses the single-agent o1, showing the advantage of a well-coordinated multi-agent system.

Abstract

Virtual film production requires intricate decision-making processes, including scriptwriting, virtual cinematography, and precise actor positioning and actions. Motivated by recent advances in automated decision-making with language agent-based societies, this paper introduces FilmAgent, a novel LLM-based multi-agent collaborative framework for end-to-end film automation in our constructed 3D virtual spaces. FilmAgent simulates various crew roles, including directors, screenwriters, actors, and cinematographers, and covers key stages of a film production workflow: (1) idea development transforms brainstormed ideas into structured story outlines; (2) scriptwriting elaborates on dialogue and character actions for each scene; (3) cinematography determines the camera setups for each shot. A team of agents collaborates through iterative feedback and revisions, thereby verifying intermediate scripts and reducing hallucinations. We evaluate the generated videos on 15 ideas and 4 key aspects. Human evaluation shows that FilmAgent outperforms all baselines across all aspects and scores 3.98 out of 5 on average, showing the feasibility of multi-agent collaboration in filmmaking. Further analysis reveals that FilmAgent, despite using the less advanced GPT-4o model, surpasses the single-agent o1, showing the advantage of a well-coordinated multi-agent system. Lastly, we discuss the complementary strengths and weaknesses of OpenAI's text-to-video model Sora and our FilmAgent in filmmaking.

Figures (24)

• Figure 1: We introduce FilmAgent, a multi-agent collaborative framework for end-to-end film automation powered by large language models (LLMs). A team of LLM-based agents takes on film crew roles, and simulates the human workflow in 3D virtual spaces by sequentially engaging in idea development, scriptwriting, and cinematography, finally completing the filmmaking process.
• Figure 2: A vertical view of one of the 3D spaces (the living room) in FilmAgent built with Unity. The environment is pre-configured with designated positions for actors and various camera setups for cinematography. These include static shots from multiple distances and dynamic shots that either follow or orbit around characters. Full camera setup of this space is provided in Figure \ref{['fig:shots']}.
• Figure 3: Workflow of FilmAgent. Given a story idea and 3D virtual spaces, the director creates character profiles and a scene outline. Actors, the screenwriter, and the director then collaborate on dialogue and movements. Cinematographers annotate camera setups for each line. Finally, the film is shot within the 3D spaces. LLM-based agents take on various film crew roles, collaborating through Critique-Correct-Verify and Debate-Judge strategies.
• Figure 4: The responsibilities of a screenwriter extend beyond writing dialogues; they also involve annotating the corresponding action for each line.
• Figure 5: Compared with the original version, the win, tie, and lose rates of the updated script and camera choices after multi-agent collaboration.