Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GAN-based Content-Conditioned Generation of Handwritten Musical Symbols

Gerard Asbert, Pau Torras, Lei Kang, Alicia Fornés, Josep Lladós

TL;DR

This work tackles Optical Music Recognition under data scarcity by proposing a content-conditioned GAN that generates realistic handwritten music symbols at the symbol level and then engraves them into full scores using Smashcima. The approach combines a GANWriting-inspired architecture with a CNN-based symbol classifier, along with an engraving pipeline, to produce diverse, controllable handwriting styles while maintaining symbol fidelity. Quantitative and qualitative evaluations show that the generated symbols exhibit strong realism and that the integrated line-level pipeline yields scores whose structure approximates real manuscripts, indicating the potential of synthetic data to augment OMR training. The study also outlines practical avenues for improvement, including exploring diffusion or transformer-based models and expanding the symbol set to further bridge the gap between synthetic and real handwritten music notation.

Abstract

The field of Optical Music Recognition (OMR) is currently hindered by the scarcity of real annotated data, particularly when dealing with handwritten historical musical scores. In similar fields, such as Handwritten Text Recognition, it was proven that synthetic examples produced with image generation techniques could help to train better-performing recognition architectures. This study explores the generation of realistic, handwritten-looking scores by implementing a music symbol-level Generative Adversarial Network (GAN) and assembling its output into a full score using the Smashcima engraving software. We have systematically evaluated the visual fidelity of these generated samples, concluding that the generated symbols exhibit a high degree of realism, marking significant progress in synthetic score generation.

GAN-based Content-Conditioned Generation of Handwritten Musical Symbols

TL;DR

This work tackles Optical Music Recognition under data scarcity by proposing a content-conditioned GAN that generates realistic handwritten music symbols at the symbol level and then engraves them into full scores using Smashcima. The approach combines a GANWriting-inspired architecture with a CNN-based symbol classifier, along with an engraving pipeline, to produce diverse, controllable handwriting styles while maintaining symbol fidelity. Quantitative and qualitative evaluations show that the generated symbols exhibit strong realism and that the integrated line-level pipeline yields scores whose structure approximates real manuscripts, indicating the potential of synthetic data to augment OMR training. The study also outlines practical avenues for improvement, including exploring diffusion or transformer-based models and expanding the symbol set to further bridge the gap between synthetic and real handwritten music notation.

Abstract

The field of Optical Music Recognition (OMR) is currently hindered by the scarcity of real annotated data, particularly when dealing with handwritten historical musical scores. In similar fields, such as Handwritten Text Recognition, it was proven that synthetic examples produced with image generation techniques could help to train better-performing recognition architectures. This study explores the generation of realistic, handwritten-looking scores by implementing a music symbol-level Generative Adversarial Network (GAN) and assembling its output into a full score using the Smashcima engraving software. We have systematically evaluated the visual fidelity of these generated samples, concluding that the generated symbols exhibit a high degree of realism, marking significant progress in synthetic score generation.
Paper Structure (19 sections, 3 equations, 5 figures, 1 table)

This paper contains 19 sections, 3 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Previous Work
  3. Methodology
  4. Preliminaries
  5. Baseline Model
  6. Adaptation to Music Symbol Generation
  7. Datasets
  8. Experimental Setup
  9. Hyperparameter Tuning
  10. Additional Modifications
  11. Line-level Generation
  12. Considered Metrics
  13. Evaluation
  14. Qualitative Results
  15. Quantitative Results
  16. ...and 4 more sections

Figures (5)

  • Figure 1: Architecture of the proposed handwritten musical symbol generation model. The input to the generator is a style example and the symbol class encoded as a one-hot encoded vector. The architecture is trained combining a standard GAN discriminator loss and a symbol classification loss, each produced by two sub-networks.
  • Figure 2: (a) An example of an average generation. (b) An example of a faulty generation. Top row: Input symbols | Bottom row: Generated symbols.
  • Figure 3: Mix of real and synthetic symbols. Check footnote $^3$ to know which ones are generated.
  • Figure 4: On the left, reference images of an accidentalSharp and a gClef from the input dataset. On the right, faulty images generated by the GAN of each respective class.
  • Figure 5: Musical sheets made with the symbols generated by the GAN, then cut into a measure and cropped vertically.