Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The NES Video-Music Database: A Dataset of Symbolic Video Game Music Paired with Gameplay Videos

Igor Cardoso, Rubens O. Moraes, Lucas N. Ferreira

TL;DR

The paper introduces NES-VMDB, a large dataset pairing 98,940 NES gameplay clips with their original MIDI soundtracks to enable symbolic music generation from video data. It assembles the dataset by collecting long-play videos, segmenting into 15-second clips, and using a Shazam-like audio fingerprinting approach to map clips to synthesized MIDI pieces from the NES-MDB. A Controllable Music Transformer (CMT) baseline is trained to generate NES music conditioned on gameplay rhythm features, and its outputs are evaluated against an unconditional CMT and human pieces using objective music-structure metrics. A genre classifier tests whether the conditioned music captures genre signals, showing partial success in aligning generated pieces with game genres. Overall, the work demonstrates the feasibility of video-conditioned symbolic music generation for retro games and outlines clear avenues for improving modeling fidelity and genre understanding.

Abstract

Neural models are one of the most popular approaches for music generation, yet there aren't standard large datasets tailored for learning music directly from game data. To address this research gap, we introduce a novel dataset named NES-VMDB, containing 98,940 gameplay videos from 389 NES games, each paired with its original soundtrack in symbolic format (MIDI). NES-VMDB is built upon the Nintendo Entertainment System Music Database (NES-MDB), encompassing 5,278 music pieces from 397 NES games. Our approach involves collecting long-play videos for 389 games of the original dataset, slicing them into 15-second-long clips, and extracting the audio from each clip. Subsequently, we apply an audio fingerprinting algorithm (similar to Shazam) to automatically identify the corresponding piece in the NES-MDB dataset. Additionally, we introduce a baseline method based on the Controllable Music Transformer to generate NES music conditioned on gameplay clips. We evaluated this approach with objective metrics, and the results showed that the conditional CMT improves musical structural quality when compared to its unconditional counterpart. Moreover, we used a neural classifier to predict the game genre of the generated pieces. Results showed that the CMT generator can learn correlations between gameplay videos and game genres, but further research has to be conducted to achieve human-level performance.

The NES Video-Music Database: A Dataset of Symbolic Video Game Music Paired with Gameplay Videos

TL;DR

The paper introduces NES-VMDB, a large dataset pairing 98,940 NES gameplay clips with their original MIDI soundtracks to enable symbolic music generation from video data. It assembles the dataset by collecting long-play videos, segmenting into 15-second clips, and using a Shazam-like audio fingerprinting approach to map clips to synthesized MIDI pieces from the NES-MDB. A Controllable Music Transformer (CMT) baseline is trained to generate NES music conditioned on gameplay rhythm features, and its outputs are evaluated against an unconditional CMT and human pieces using objective music-structure metrics. A genre classifier tests whether the conditioned music captures genre signals, showing partial success in aligning generated pieces with game genres. Overall, the work demonstrates the feasibility of video-conditioned symbolic music generation for retro games and outlines clear avenues for improving modeling fidelity and genre understanding.

Abstract

Neural models are one of the most popular approaches for music generation, yet there aren't standard large datasets tailored for learning music directly from game data. To address this research gap, we introduce a novel dataset named NES-VMDB, containing 98,940 gameplay videos from 389 NES games, each paired with its original soundtrack in symbolic format (MIDI). NES-VMDB is built upon the Nintendo Entertainment System Music Database (NES-MDB), encompassing 5,278 music pieces from 397 NES games. Our approach involves collecting long-play videos for 389 games of the original dataset, slicing them into 15-second-long clips, and extracting the audio from each clip. Subsequently, we apply an audio fingerprinting algorithm (similar to Shazam) to automatically identify the corresponding piece in the NES-MDB dataset. Additionally, we introduce a baseline method based on the Controllable Music Transformer to generate NES music conditioned on gameplay clips. We evaluated this approach with objective metrics, and the results showed that the conditional CMT improves musical structural quality when compared to its unconditional counterpart. Moreover, we used a neural classifier to predict the game genre of the generated pieces. Results showed that the CMT generator can learn correlations between gameplay videos and game genres, but further research has to be conducted to achieve human-level performance.
Paper Structure (10 sections, 4 figures, 3 tables)

This paper contains 10 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The NES-MDB dataset
  4. Symbolic Music Generation from Videos
  5. Symbolic Music Generation for Video Games
  6. The NES-VMDB Database
  7. Experiments
  8. Music Structure Metrics
  9. Game Genre Classification
  10. Conclusions and Future Work

Figures (4)

  • Figure 1: The piano roll representation of a snippet of a piece from the NES-MDB dataset. The horizontal axis represents time and the vertical axis represents pitch. The color of the notes represents their channel donahue2018nes.
  • Figure 2: One of the first frames of the Contra long-play from the World of Longplay channel.
  • Figure 3: CMT approach to condition music generation from input videos di2021video.
  • Figure 4: Distribution of genres in the NES-VMDB database.