Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Versatile Multimodal Agent for Multimedia Content Generation

Daoan Zhang, Wenlin Yao, Xiaoyang Wang, Yebowen Hu, Jiebo Luo, Dong Yu

TL;DR

This work addresses the challenge of end-to-end multimodal content generation by introducing MultiMedia-Agent, a tool- and modality-centric system guided by Skill Acquisition Theory. It constructs a Multimedia Content Playground with 18 real-world tasks, a structured tool library, and a two-stage plan-curation process that combines self-correction and model-preference optimization. Training proceeds in three stages (cognitive, associative, autonomous), leveraging a data-generation pipeline and preference-based feedback to align outputs with human aesthetics and needs. Empirical results show progressive improvements in plan quality and multimodal coherence, demonstrating the approach's potential to outperform existing models in complex multimedia generation tasks and to better reflect user preferences in generated content.

Abstract

With the advancement of AIGC (AI-generated content) technologies, an increasing number of generative models are revolutionizing fields such as video editing, music generation, and even film production. However, due to the limitations of current AIGC models, most models can only serve as individual components within specific application scenarios and are not capable of completing tasks end-to-end in real-world applications. In real-world applications, editing experts often work with a wide variety of images and video inputs, producing multimodal outputs -- a video typically includes audio, text, and other elements. This level of integration across multiple modalities is something current models are unable to achieve effectively. However, the rise of agent-based systems has made it possible to use AI tools to tackle complex content generation tasks. To deal with the complex scenarios, in this paper, we propose a MultiMedia-Agent designed to automate complex content creation. Our agent system includes a data generation pipeline, a tool library for content creation, and a set of metrics for evaluating preference alignment. Notably, we introduce the skill acquisition theory to model the training data curation and agent training. We designed a two-stage correlation strategy for plan optimization, including self-correlation and model preference correlation. Additionally, we utilized the generated plans to train the MultiMedia-Agent via a three stage approach including base/success plan finetune and preference optimization. The comparison results demonstrate that the our approaches are effective and the MultiMedia-Agent can generate better multimedia content compared to novel models.

A Versatile Multimodal Agent for Multimedia Content Generation

TL;DR

This work addresses the challenge of end-to-end multimodal content generation by introducing MultiMedia-Agent, a tool- and modality-centric system guided by Skill Acquisition Theory. It constructs a Multimedia Content Playground with 18 real-world tasks, a structured tool library, and a two-stage plan-curation process that combines self-correction and model-preference optimization. Training proceeds in three stages (cognitive, associative, autonomous), leveraging a data-generation pipeline and preference-based feedback to align outputs with human aesthetics and needs. Empirical results show progressive improvements in plan quality and multimodal coherence, demonstrating the approach's potential to outperform existing models in complex multimedia generation tasks and to better reflect user preferences in generated content.

Abstract

With the advancement of AIGC (AI-generated content) technologies, an increasing number of generative models are revolutionizing fields such as video editing, music generation, and even film production. However, due to the limitations of current AIGC models, most models can only serve as individual components within specific application scenarios and are not capable of completing tasks end-to-end in real-world applications. In real-world applications, editing experts often work with a wide variety of images and video inputs, producing multimodal outputs -- a video typically includes audio, text, and other elements. This level of integration across multiple modalities is something current models are unable to achieve effectively. However, the rise of agent-based systems has made it possible to use AI tools to tackle complex content generation tasks. To deal with the complex scenarios, in this paper, we propose a MultiMedia-Agent designed to automate complex content creation. Our agent system includes a data generation pipeline, a tool library for content creation, and a set of metrics for evaluating preference alignment. Notably, we introduce the skill acquisition theory to model the training data curation and agent training. We designed a two-stage correlation strategy for plan optimization, including self-correlation and model preference correlation. Additionally, we utilized the generated plans to train the MultiMedia-Agent via a three stage approach including base/success plan finetune and preference optimization. The comparison results demonstrate that the our approaches are effective and the MultiMedia-Agent can generate better multimedia content compared to novel models.
Paper Structure (17 sections, 4 figures, 5 tables)

This paper contains 17 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Tool Agent
  4. Any-to-any generation
  5. Data Curation
  6. Multi-media Tasks
  7. Tool Library
  8. Hierarchical Plan Curation
  9. Two-stage correlation of plan curation
  10. Data Statistics
  11. MultiMedia-Agent
  12. Agent Skill Acquisition
  13. Experiment Settings
  14. Skill Acquisition Theory Can Benefit Tool Agent Training
  15. Visualization Results
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Formats for tool library and generated plan.
  • Figure 2: Two-stage correlation of plan curation for content creation.
  • Figure 3: The detailed structure of MultiMedia-Agent.
  • Figure 4: Visualization of the multimedia content created from the plan generated by MultiMedia-Agent. The user query is: use the images and the corresponding video to create a satisfying video.