Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Immersed in my Ideas: Using Virtual Reality and Multimodal Interactions to Visualize Users' Ideas and Thoughts

Yunhao Xing, Jerrick Ban, Timothy D. Hubbard, Michael Villano, Diego Gomez-Zara

TL;DR

VIVRA (Voice Interactive Virtual Reality Annotation), a VR application combining multimodal interaction with large language models (LLMs) to transform users' ideas into interactive 3D visualizations to create powerful ideation and reflection tools is introduced.

Abstract

This paper introduces VIVRA (Voice Interactive Virtual Reality Annotation), a VR application combining multimodal interaction with large language models (LLMs) to transform users' ideas into interactive 3D visualizations. VIVRA converts verbalized thoughts into "idea balloons" that summarize and expand on detected topics by an LLM. VIVRA allows users to verbalize their thoughts in real time or record their ideas to display the topics later. We evaluated the effectiveness of VIVRA in an exploratory study with 29 participants and a user study with 10 participants. Our results show that VIVRA enhanced users' ability to reflect on and develop ideas, achieving high levels of satisfaction, usability, and engagement. Participants valued VIVRA as a reflective tool for exploring personal thoughts and ideas. We discuss the potential advantages and uses of this application, highlighting the potential of combining immersive technologies with LLMs to create powerful ideation and reflection tools.

Immersed in my Ideas: Using Virtual Reality and Multimodal Interactions to Visualize Users' Ideas and Thoughts

TL;DR

VIVRA (Voice Interactive Virtual Reality Annotation), a VR application combining multimodal interaction with large language models (LLMs) to transform users' ideas into interactive 3D visualizations to create powerful ideation and reflection tools is introduced.

Abstract

This paper introduces VIVRA (Voice Interactive Virtual Reality Annotation), a VR application combining multimodal interaction with large language models (LLMs) to transform users' ideas into interactive 3D visualizations. VIVRA converts verbalized thoughts into "idea balloons" that summarize and expand on detected topics by an LLM. VIVRA allows users to verbalize their thoughts in real time or record their ideas to display the topics later. We evaluated the effectiveness of VIVRA in an exploratory study with 29 participants and a user study with 10 participants. Our results show that VIVRA enhanced users' ability to reflect on and develop ideas, achieving high levels of satisfaction, usability, and engagement. Participants valued VIVRA as a reflective tool for exploring personal thoughts and ideas. We discuss the potential advantages and uses of this application, highlighting the potential of combining immersive technologies with LLMs to create powerful ideation and reflection tools.
Paper Structure (44 sections, 5 figures, 1 table, 1 algorithm)

This paper contains 44 sections, 5 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. User Text Input in VR/AR Systems
  4. Topic Visualization
  5. Supporting reflection through VR and LLMs
  6. The VIVRA System
  7. Design Goals
  8. DG1: Integrating multimodal techniques to leverage user interaction with ideas and thoughts.
  9. DG2: Providing a rich narrative that seamlessly integrates both linear temporal and interactive real-time storytelling elements.
  10. DG3: Displaying users' ideas and thoughts in VR to enhance reflection and creativity.
  11. DG4: Enabling users to record, summarize, visualize, and expand relevant topics from their ideas in real-time.
  12. Example Scenario using VIVRA
  13. System Workflow
  14. Key Features
  15. Multimodal Interaction with the balloons
  16. ...and 29 more sections

Figures (5)

  • Figure 1: The workflow of VIVRA
  • Figure 2: The one-click organization function using gaze: (a) before organization, (b) after organization.
  • Figure 3: The functions and animation of the interactive balloon. (a) The functions of the interactive balloon. (b) The animation after a user clicks the Start balloon. The balloon will float to the corner and turn into a timer. (c) The animation of creating a balloon. A portal animation will emerge from the ground, and the balloon will ascend from the portal to the middle of the room.
  • Figure 4: The three conditions of our evaluation: (a) Transcript, (b) Word Cloud, (c) VIVRA.
  • Figure 5: The post-treatment questionnaire results