Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bridging Paintings and Music -- Exploring Emotion based Music Generation through Paintings

Tanisha Hisariya, Huan Zhang, Jinhua Liang

TL;DR

This research develops a model capable of generating music that resonates with the emotions depicted in visual arts, integrating emotion labeling, image captioning, and language models to transform visual inputs into musical compositions.

Abstract

Rapid advancements in artificial intelligence have significantly enhanced generative tasks involving music and images, employing both unimodal and multimodal approaches. This research develops a model capable of generating music that resonates with the emotions depicted in visual arts, integrating emotion labeling, image captioning, and language models to transform visual inputs into musical compositions. Addressing the scarcity of aligned art and music data, we curated the Emotion Painting Music Dataset, pairing paintings with corresponding music for effective training and evaluation. Our dual-stage framework converts images to text descriptions of emotional content and then transforms these descriptions into music, facilitating efficient learning with minimal data. Performance is evaluated using metrics such as Fréchet Audio Distance (FAD), Total Harmonic Distortion (THD), Inception Score (IS), and KL divergence, with audio-emotion text similarity confirmed by the pre-trained CLAP model to demonstrate high alignment between generated music and text. This synthesis tool bridges visual art and music, enhancing accessibility for the visually impaired and opening avenues in educational and therapeutic applications by providing enriched multi-sensory experiences.

Bridging Paintings and Music -- Exploring Emotion based Music Generation through Paintings

TL;DR

This research develops a model capable of generating music that resonates with the emotions depicted in visual arts, integrating emotion labeling, image captioning, and language models to transform visual inputs into musical compositions.

Abstract

Rapid advancements in artificial intelligence have significantly enhanced generative tasks involving music and images, employing both unimodal and multimodal approaches. This research develops a model capable of generating music that resonates with the emotions depicted in visual arts, integrating emotion labeling, image captioning, and language models to transform visual inputs into musical compositions. Addressing the scarcity of aligned art and music data, we curated the Emotion Painting Music Dataset, pairing paintings with corresponding music for effective training and evaluation. Our dual-stage framework converts images to text descriptions of emotional content and then transforms these descriptions into music, facilitating efficient learning with minimal data. Performance is evaluated using metrics such as Fréchet Audio Distance (FAD), Total Harmonic Distortion (THD), Inception Score (IS), and KL divergence, with audio-emotion text similarity confirmed by the pre-trained CLAP model to demonstrate high alignment between generated music and text. This synthesis tool bridges visual art and music, enhancing accessibility for the visually impaired and opening avenues in educational and therapeutic applications by providing enriched multi-sensory experiences.
Paper Structure (16 sections, 3 figures, 2 tables)

This paper contains 16 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Image feature extraction and conversion
  4. Deep Learning methods for Music Generation
  5. Multi-Modality Audio and Music generation
  6. Method
  7. Image Emotion Labelling Model
  8. Image Description Model
  9. LLM model
  10. Music Generation
  11. Experiments
  12. Datasets
  13. Evaluation metrics
  14. Training
  15. Results
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Architecture of Emotion Labelling model with pretrained ResNet50 and additional two bidirectional GRU and one Attention layer proceeding with dropout along fully connected layer.
  • Figure 2: Overview architecture of our model , which encompasses working flow of all the four models represented as different colour giving the output as MS-G$_{E}$ (single label text from ResNet50+ MusicGen), MS-G$_{N}$ (image descritive text from BLIP+MusicGen), MS-G$_{L}$ (Enhanced description from Falcon+ MusicGen), MS-G$_{O}$ (Enhanced description from Falcon+enhanced finetuning method of MusicGen).
  • Figure 3: CLAP analysis of generated song across model with their emotions. It is being meausred with providing “emotion song” as text during CLAP calculation.