Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

M2M-Gen: A Multimodal Framework for Automated Background Music Generation in Japanese Manga Using Large Language Models

Megha Sharma, Muhammad Taimoor Haseeb, Gus Xia, Yoshimasa Tsuruoka

TL;DR

The effectiveness of M2M Gen is confirmed through extensive subjective evaluations, showcasing its capability to generate higher quality, more relevant and consistent music that complements specific scenes when compared to the authors' baselines.

Abstract

This paper introduces M2M Gen, a multi modal framework for generating background music tailored to Japanese manga. The key challenges in this task are the lack of an available dataset or a baseline. To address these challenges, we propose an automated music generation pipeline that produces background music for an input manga book. Initially, we use the dialogues in a manga to detect scene boundaries and perform emotion classification using the characters faces within a scene. Then, we use GPT4o to translate this low level scene information into a high level music directive. Conditioned on the scene information and the music directive, another instance of GPT 4o generates page level music captions to guide a text to music model. This produces music that is aligned with the mangas evolving narrative. The effectiveness of M2M Gen is confirmed through extensive subjective evaluations, showcasing its capability to generate higher quality, more relevant and consistent music that complements specific scenes when compared to our baselines.

M2M-Gen: A Multimodal Framework for Automated Background Music Generation in Japanese Manga Using Large Language Models

TL;DR

The effectiveness of M2M Gen is confirmed through extensive subjective evaluations, showcasing its capability to generate higher quality, more relevant and consistent music that complements specific scenes when compared to the authors' baselines.

Abstract

This paper introduces M2M Gen, a multi modal framework for generating background music tailored to Japanese manga. The key challenges in this task are the lack of an available dataset or a baseline. To address these challenges, we propose an automated music generation pipeline that produces background music for an input manga book. Initially, we use the dialogues in a manga to detect scene boundaries and perform emotion classification using the characters faces within a scene. Then, we use GPT4o to translate this low level scene information into a high level music directive. Conditioned on the scene information and the music directive, another instance of GPT 4o generates page level music captions to guide a text to music model. This produces music that is aligned with the mangas evolving narrative. The effectiveness of M2M Gen is confirmed through extensive subjective evaluations, showcasing its capability to generate higher quality, more relevant and consistent music that complements specific scenes when compared to our baselines.

Paper Structure

This paper contains 7 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Evaluation
  5. Subjective Evaluation
  6. Objective Evaluation
  7. Conclusion

Figures (4)

  • Figure 1: Pipeline for M2M-Gen. It takes as input the images and dialogue transcript of a manga book and outputs an audio file containing tailored background music. Manga image taken from Manga109 mtap_matsui_2017multimedia_aizawa_2020 courtesy of Yoshi Masako.
  • Figure 2: Baseline pipeline. It takes as input a manga page and dialogue transcripts and outputs an audio file containing tailored background music for the page. Manga image taken from Manga109 mtap_matsui_2017multimedia_aizawa_2020 courtesy of Yoshi Masako.
  • Figure 3: Final mean scores for each of the metrics using within-subject ANOVA on the within-scene study
  • Figure 4: Final mean scores for each of the metric using within-subject ANOVA on the between-scenes study