Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MelodyVis: Visual Analytics for Melodic Patterns in Sheet Music

Matthias Miller, Daniel Fürst, Maximilian T. Fischer, Hanna Hauptmann, Daniel Keim, Mennatallah El-Assady

TL;DR

This work tackles the challenge that manual melody detection is labor-intensive and fully automated methods often lack expressive power. It introduces MelodyVis, a visual analytics tool with five interconnected views, including a Melody Operator Graph and a Voicing Timeline, and eight atomic melodic operators to capture transformations like transposition and inversion. A user study with 25 participants demonstrates that enabling operators doubles the number of patterns identified while maintaining manageable cognitive load, validating a mixed-initiative approach. The framework preserves analyst control, supports scalable exploration of melodic patterns in sheet music, and suggests future extensions to broader data formats, audio playback, and richer operator configurations to broaden applicability across musicology research and education.

Abstract

Manual melody detection is a tedious task requiring high expertise level, while automatic detection is often not expressive or powerful enough. Thus, we present MelodyVis, a visual application designed in collaboration with musicology experts to explore melodic patterns in digital sheet music. MelodyVis features five connected views, including a Melody Operator Graph and a Voicing Timeline. The system utilizes eight atomic operators, such as transposition and mirroring, to capture melody repetitions and variations. Users can start their analysis by manually selecting patterns in the sheet view, and then identifying other patterns based on the selected samples through an interactive exploration process. We conducted a user study to investigate the effectiveness and usefulness of our approach and its integrated melodic operators, including usability and mental load questions. We compared the analysis executed by 25 participants with and without the operators. The study results indicate that the participants could identify at least twice as many patterns with activated operators. MelodyVis allows analysts to steer the analysis process and interpret results. Our study also confirms the usefulness of MelodyVis in supporting common analytical tasks in melodic analysis, with participants reporting improved pattern identification and interpretation. Thus, MelodyVis addresses the limitations of fully-automated approaches, enabling music analysts to step into the analysis process and uncover and understand intricate melodic patterns and transformations in sheet music.

MelodyVis: Visual Analytics for Melodic Patterns in Sheet Music

TL;DR

This work tackles the challenge that manual melody detection is labor-intensive and fully automated methods often lack expressive power. It introduces MelodyVis, a visual analytics tool with five interconnected views, including a Melody Operator Graph and a Voicing Timeline, and eight atomic melodic operators to capture transformations like transposition and inversion. A user study with 25 participants demonstrates that enabling operators doubles the number of patterns identified while maintaining manageable cognitive load, validating a mixed-initiative approach. The framework preserves analyst control, supports scalable exploration of melodic patterns in sheet music, and suggests future extensions to broader data formats, audio playback, and richer operator configurations to broaden applicability across musicology research and education.

Abstract

Manual melody detection is a tedious task requiring high expertise level, while automatic detection is often not expressive or powerful enough. Thus, we present MelodyVis, a visual application designed in collaboration with musicology experts to explore melodic patterns in digital sheet music. MelodyVis features five connected views, including a Melody Operator Graph and a Voicing Timeline. The system utilizes eight atomic operators, such as transposition and mirroring, to capture melody repetitions and variations. Users can start their analysis by manually selecting patterns in the sheet view, and then identifying other patterns based on the selected samples through an interactive exploration process. We conducted a user study to investigate the effectiveness and usefulness of our approach and its integrated melodic operators, including usability and mental load questions. We compared the analysis executed by 25 participants with and without the operators. The study results indicate that the participants could identify at least twice as many patterns with activated operators. MelodyVis allows analysts to steer the analysis process and interpret results. Our study also confirms the usefulness of MelodyVis in supporting common analytical tasks in melodic analysis, with participants reporting improved pattern identification and interpretation. Thus, MelodyVis addresses the limitations of fully-automated approaches, enabling music analysts to step into the analysis process and uncover and understand intricate melodic patterns and transformations in sheet music.
Paper Structure (25 sections, 8 figures)

This paper contains 25 sections, 8 figures.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Sheet Music Retrieval based on Melodic Sequences
  4. Melodic Pattern Mining and Extraction
  5. Applying Methodology Transfer
  6. Structure of Melodic Patterns
  7. Atomic Melodic Operators
  8. Complex Chaining of Melodic Operators
  9. Analysis Tasks and Requirements
  10. MelodyVis -- Visual Interface for Melody Analysis
  11. Data Preparation
  12. Interaction Components
  13. Visual Interaction Pipeline
  14. Use Cases
  15. The Four Seasons: Spring -- by Antonio Vivaldi
  16. ...and 10 more sections

Figures (8)

  • Figure 1: This manual fugue analysis example shows how a music analyst would perform a melodic pattern analysis using color and annotations. We took this example as a solution from an exam published by the School of Music from Western Michigan University wmich2019fuguereview.
  • Figure 2: http://visual-musicology.com/melodyvis/ consists of five core interaction components interconnected through linking and brushing keim2002infovisanddatamining: (1) Sheet Music View, (2) Melodic Operator Selection, (3) Voice Separation View, (4) Melodic Transformation Graph, and (5) Melodic Pattern Configuration. Together, they facilitate a comprehensive and interactive analysis workflow for the exploration of melodic patterns in digital sheet music.
  • Figure 3: Visual analysis of Vivaldi's https://musescore.com/musikmann/scores/135805vivaldi1680spring. (Top) VSV with color-coded highlights indicating corresponding melodic patterns across the voices. (Bottom Left) The SV displays the selected patterns with overlay annotations. (Bottom Right) The MTG illustrates the relationships between the identified patterns and their transformations within the composition showing dashed edges with pattern sets that have similar patterns.
  • Figure 4: Analysis of J.S. Bach's https://musescore.com/user/101554/scores/117279, focusing on the arpeggio patterns in the first voice, where the most interesting melodic development occurs, while illustrating the simple repetitions occurring in the second and third voice.
  • Figure 5: Tetris theme analysis: On the left, the highlighted Sheet View displays the recurring motifs and contrasting melodic sequences. The Melodic Transformation Graph on the right reveals the simple relationships yet memorable patterns used in this iconic melody.
  • ...and 3 more figures