Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Making Your Dreams A Reality: Decoding the Dreams into a Coherent Video Story from fMRI Signals

Yanwei Fu, Jianxiong Gao, Baofeng Yang, Jianfeng Feng

TL;DR

This work tackles decoding visual content from sleep-time fMRI to produce coherent dream video narratives. It introduces a three-stage pipeline: Visual Stimulus Perception Reconstruction (wakeful image reconstruction from fMRI using cross-subject NSD pretraining and diffusion-based generation), Dream Visual Imagery Decoding (zero-shot decoding of dream content via shared real-dream brain patterns), and Dream Narrative Integration (LLM-driven synthesis of decoded frames into a cohesive dream story). The framework leverages the Natural Scenes Dataset for cross-subject generalization and evaluates both visual reconstructions and narrative quality using CLIP-based similarity and qualitative LLMS-generated scripts. The results demonstrate the feasibility of translating subjective dream experiences into interpretable visual narratives, with implications for dream research, neuroscience, and creative visualization.

Abstract

This paper studies the brave new idea for Multimedia community, and proposes a novel framework to convert dreams into coherent video narratives using fMRI data. Essentially, dreams have intrigued humanity for centuries, offering glimpses into our subconscious minds. Recent advancements in brain imaging, particularly functional magnetic resonance imaging (fMRI), have provided new ways to explore the neural basis of dreaming. By combining subjective dream experiences with objective neurophysiological data, we aim to understand the visual aspects of dreams and create complete video narratives. Our process involves three main steps: reconstructing visual perception, decoding dream imagery, and integrating dream stories. Using innovative techniques in fMRI analysis and language modeling, we seek to push the boundaries of dream research and gain deeper insights into visual experiences during sleep. This technical report introduces a novel approach to visually decoding dreams using fMRI signals and weaving dream visuals into narratives using language models. We gather a dataset of dreams along with descriptions to assess the effectiveness of our framework.

Making Your Dreams A Reality: Decoding the Dreams into a Coherent Video Story from fMRI Signals

TL;DR

This work tackles decoding visual content from sleep-time fMRI to produce coherent dream video narratives. It introduces a three-stage pipeline: Visual Stimulus Perception Reconstruction (wakeful image reconstruction from fMRI using cross-subject NSD pretraining and diffusion-based generation), Dream Visual Imagery Decoding (zero-shot decoding of dream content via shared real-dream brain patterns), and Dream Narrative Integration (LLM-driven synthesis of decoded frames into a cohesive dream story). The framework leverages the Natural Scenes Dataset for cross-subject generalization and evaluates both visual reconstructions and narrative quality using CLIP-based similarity and qualitative LLMS-generated scripts. The results demonstrate the feasibility of translating subjective dream experiences into interpretable visual narratives, with implications for dream research, neuroscience, and creative visualization.

Abstract

This paper studies the brave new idea for Multimedia community, and proposes a novel framework to convert dreams into coherent video narratives using fMRI data. Essentially, dreams have intrigued humanity for centuries, offering glimpses into our subconscious minds. Recent advancements in brain imaging, particularly functional magnetic resonance imaging (fMRI), have provided new ways to explore the neural basis of dreaming. By combining subjective dream experiences with objective neurophysiological data, we aim to understand the visual aspects of dreams and create complete video narratives. Our process involves three main steps: reconstructing visual perception, decoding dream imagery, and integrating dream stories. Using innovative techniques in fMRI analysis and language modeling, we seek to push the boundaries of dream research and gain deeper insights into visual experiences during sleep. This technical report introduces a novel approach to visually decoding dreams using fMRI signals and weaving dream visuals into narratives using language models. We gather a dataset of dreams along with descriptions to assess the effectiveness of our framework.
Paper Structure (19 sections, 3 equations, 7 figures, 2 tables)

This paper contains 19 sections, 3 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dream Visual Decoding
  4. Visual Stimulus Decoding
  5. Problem setups
  6. Data Collection and preprocessing
  7. Method
  8. Visual Stimulus Perception Reconstruction
  9. Dream Visual Imagery Decoding
  10. Dream Narrative Integration
  11. Experiments
  12. Dataset
  13. Natural Scenes Dataset.
  14. fMRI-Dream Dataset.
  15. Implementation Details
  16. ...and 4 more sections

Figures (7)

  • Figure 1: We use a 3 Tesla (3T) MRI scanner for data collection. Participants are positioned within the MRI scanner.
  • Figure 2: The fMRI dream decoding process is divided into three stages: i) Visual Stimulus Perception Reconstruction: Decoding brain activity patterns associated with real visual stimuli perception. ii) Dream Visual Imagery Decoding: Shared brain activity patterns between real visual stimuli and dream-induced visual experiences aid in decoding snapshots of dream content from fMRI data. iii) Dream Narrative Integration: By leveraging LLM, we synthesize fragmented dream visualizations into cohesive narratives, providing a complete interpretation of the dream experience.
  • Figure 3: Pipeline of Dream Narrative Integration. This process includes three steps: Single-Shot Dream Description, Dream Story Composition, and Video Integration.
  • Figure 4: Visualization of Dream Narrative Video "some cat".
  • Figure 5: Visualization of Dream Narrative Video "skiing with a snowboard".
  • ...and 2 more figures