Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From 2D Document Interactions into Immersive Information Experience: An Example-Based Design by Augmenting Content, Spatializing Placement, Enriching Long-Term Interactions, and Simplifying Content Creations

Chen Chen

TL;DR

This research demonstrates how to transform such of today's 2D document interactions into an immersive information experience by augmenting content with virtual reality, spatializing document placements with mixed reality, enriching long-term and continuous interactions with voice assistants, and simplify document creation workflow with generative AI.

Abstract

Documents serve as a crucial and indispensable medium for everyday workplace tasks. However, understanding, interacting and creating such documents on today's planar interfaces without any intelligent support are challenging due to our natural cognitive constraints on remembering, processing, understanding and interacting with these information. My doctorate research investigates how to bring 2D document interactions into immersive information experience using multiple of today's emergent technologies. With the examples of four specific types of documents -- medical scans, instruction document, self-report diary survey, and reference images for visual artists -- my research demonstrates how to transform such of today's 2D document interactions into an immersive information experience, by augmenting content with virtual reality, spatializing document placements with mixed reality, enriching long-term and continuous interactions with voice assistants, and simplify document creation workflow with generative AI.

From 2D Document Interactions into Immersive Information Experience: An Example-Based Design by Augmenting Content, Spatializing Placement, Enriching Long-Term Interactions, and Simplifying Content Creations

TL;DR

This research demonstrates how to transform such of today's 2D document interactions into an immersive information experience by augmenting content with virtual reality, spatializing document placements with mixed reality, enriching long-term and continuous interactions with voice assistants, and simplify document creation workflow with generative AI.

Abstract

Documents serve as a crucial and indispensable medium for everyday workplace tasks. However, understanding, interacting and creating such documents on today's planar interfaces without any intelligent support are challenging due to our natural cognitive constraints on remembering, processing, understanding and interacting with these information. My doctorate research investigates how to bring 2D document interactions into immersive information experience using multiple of today's emergent technologies. With the examples of four specific types of documents -- medical scans, instruction document, self-report diary survey, and reference images for visual artists -- my research demonstrates how to transform such of today's 2D document interactions into an immersive information experience, by augmenting content with virtual reality, spatializing document placements with mixed reality, enriching long-term and continuous interactions with voice assistants, and simplify document creation workflow with generative AI.

Paper Structure

This paper contains 5 sections, 3 figures.

Table of Contents

  1. Introduction
  2. VRContour: Boosting Contouring Experience by Augmenting Medical Scans with VR
  3. Paper-To-Place: Improving Instruction Consumption Experience by Spatializing Procedural Steps into Workspace with MR
  4. VOLI: Enriching Long-Term Interactions with Diary Survey among Aging Populations with VAs
  5. Conclusions and Future Work: Toward GenAI-Powered Reference Images Creations for Supporting Visual Artists' Creative Works

Figures (3)

  • Figure 1: VRContour Chen2022VRContour. (a - b) A VR stylus and a tracked tablet were used for supporting contour delineation workflow. (c - e) The virtual tablet rendered inside VR scene could be zoomed for supporting inspecting detailed structure. (f - l) The contour could be delineated and refined on 2D and 3D interfaces. (m) The design taxonomy of VRContour. (n - v) The final implementations of the VRContour.
  • Figure 2: PaperToPlace Chen2023PaperToPlace. We assume a spatial profile has been pre-created (h). The author uses the authoring pipeline to extract the document profile for the MR experience (a - g). In the consuming pipeline (i), the instruction steps are displayed such that the consumers could easily refer to the instruction step while not being occluded by the virtual graphics.
  • Figure 3: We integrated the diary survey into Echo Dot and Echo Show, which were deployed into the residences of 16 older adults participants. The additional built-in touchscreen on Echo Show enables older adults to see the survey prompt (c) and input responses by touch (e - f).