Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DreamLLM-3D: Affective Dream Reliving using Large Language Model and 3D Generative AI

Pinyao Liu, Keon Ju Lee, Alexander Steinmaurer, Claudia Picard-Deland, Michelle Carr, Alexandra Kitson

TL;DR

DreamLLM-3D presents a multimodal AI system that relives dreams by analyzing whispered dream reports with a local LLM and visualizing identified dream entities as 3D point clouds, with affective color and soundscapes shaping the experience. The pipeline uses a local LLM (Mistral 7B) with $T = 0$, Nomic-Embed-Text embeddings, Chroma retrieval, and a Point-E–based text-to-3D diffusion model to generate 3D dream objects in about 17 seconds on an $A100$ GPU and render them in Unity3D. It encodes emotional and social information in the dream via HVDC categories, mapping to colors and context-aware audio layers to create an immersive, affective re-experiencing. The authors also propose an experiential AI-Dreamworker Hybrid paradigm to combine AI-driven insights with human interpretation, and discuss ethical considerations and potential longitudinal applications.

Abstract

We present DreamLLM-3D, a composite multimodal AI system behind an immersive art installation for dream re-experiencing. It enables automated dream content analysis for immersive dream-reliving, by integrating a Large Language Model (LLM) with text-to-3D Generative AI. The LLM processes voiced dream reports to identify key dream entities (characters and objects), social interaction, and dream sentiment. The extracted entities are visualized as dynamic 3D point clouds, with emotional data influencing the color and soundscapes of the virtual dream environment. Additionally, we propose an experiential AI-Dreamworker Hybrid paradigm. Our system and paradigm could potentially facilitate a more emotionally engaging dream-reliving experience, enhancing personal insights and creativity.

DreamLLM-3D: Affective Dream Reliving using Large Language Model and 3D Generative AI

TL;DR

DreamLLM-3D presents a multimodal AI system that relives dreams by analyzing whispered dream reports with a local LLM and visualizing identified dream entities as 3D point clouds, with affective color and soundscapes shaping the experience. The pipeline uses a local LLM (Mistral 7B) with , Nomic-Embed-Text embeddings, Chroma retrieval, and a Point-E–based text-to-3D diffusion model to generate 3D dream objects in about 17 seconds on an GPU and render them in Unity3D. It encodes emotional and social information in the dream via HVDC categories, mapping to colors and context-aware audio layers to create an immersive, affective re-experiencing. The authors also propose an experiential AI-Dreamworker Hybrid paradigm to combine AI-driven insights with human interpretation, and discuss ethical considerations and potential longitudinal applications.

Abstract

We present DreamLLM-3D, a composite multimodal AI system behind an immersive art installation for dream re-experiencing. It enables automated dream content analysis for immersive dream-reliving, by integrating a Large Language Model (LLM) with text-to-3D Generative AI. The LLM processes voiced dream reports to identify key dream entities (characters and objects), social interaction, and dream sentiment. The extracted entities are visualized as dynamic 3D point clouds, with emotional data influencing the color and soundscapes of the virtual dream environment. Additionally, we propose an experiential AI-Dreamworker Hybrid paradigm. Our system and paradigm could potentially facilitate a more emotionally engaging dream-reliving experience, enhancing personal insights and creativity.

Paper Structure

This paper contains 9 sections, 3 figures.

Table of Contents

  1. Introduction
  2. AI-assisted Affective Dream Reliving Overview
  3. LLM System
  4. Affective Color Mapping
  5. Context-Aware Soundscape System
  6. Discussion and Conclusion
  7. A Composite Multimodal AI System for Affective Dream Re-experiencing.
  8. Towards an Experiential AI-Dreamworker Hybrid Paradigm
  9. Ethical implication:

Figures (3)

  • Figure 1: During the dream-reliving experience, the immersant whispers their dream into the system. The spoken dream characters and objects are then represented as 3D point clouds in the shared immersive space.
  • Figure 2: A composite multimodal AI system performing real-time dream content analysis for immersive dream-reliving, using LLM and 3D generative AI.
  • Figure 3: Example of a graphic score for soundscape composition illustrating the integration of sound layers for the emotion classes and social interaction classes (X-axis = time, and Y-axis = sound layers).