A Review of Prototyping in XR: Linking Extended Reality to Digital Fabrication

TL;DR

The paper addresses how XR prototyping can accelerate design and fabrication by bridging immersive virtual prototyping with physical production. It adopts a systematic review of 54 manuscripts from 2008–2023 to map six building blocks (display, control, transform, model construction/rendering, non-visual feedback, link to fabrication) and three prototyping workflows (physical-based, preset-block assembly, blank-canvas) while detailing display and input modalities and how prototypes transfer to fabrication. The study highlights trends in display devices (screens, CAVE, and especially HMDs), multiple control methods (controller, pen, hand gestures, touchscreen, physical artefacts), and two pathways to fabrication (manual and machine) along with benefits and challenges, including cross-platform integration and fidelity. It also discusses future directions, notably AI-generated content and more seamless integration with fabrication pipelines, with implications for researchers and industry adoption.

Abstract

Extended Reality (XR) has expanded the horizons of entertainment and social life and shows great potential in the manufacturing industry. Prototyping in XR can help designers make initial proposals and iterations at low cost before manufacturers and investors decide whether to invest in research, development or even production. According to the literature (54 manuscripts in the last 15 years) prototyping in XR in XR is easier to use than three-dimensional (3D) modeling with a personal computer and more capable of displaying 3D structures than paper drawing. In this comprehensive review, we systematically surveyed the literature on prototyping in XR and discussed the possibility of transferring created virtual prototypes from XR to commonly used 3D modeling software and reality. We proposed five research questions regarding prototyping in XR. They are: what the constituent elements and workflow of prototyping are; which display devices can deliver satisfying immersive and interactive experiences; how user control input is obtained and what methods are available for users to interact with virtual elements and create XR prototypes; what approaches can facilitate the connection with fabrication to ensure a smooth transition from the virtual to the physical world; and what the challenges are and what the future holds for this research domain. Based on these questions, we summarized the components and workflows of prototyping in XR. Moreover, we present an overview of the latest trends in display device evolution, control technologies, digital model construction, and manufacturing processes. In view of these latest developments and gaps, we speculated on the challenges and opportunities in the field of prototyping in XR, especially in linking extended reality to digital fabrication, with the aim of guiding researchers towards new research directions.

Figures (6)

• Figure 1: Various XR display device, (a) Holodesk hilliges2012holodesk, (b) smartphone for AR poke, (c) CAVE cave, (d) Hybrid Virtual Environment 3D (Hyve-3D) is designed by Hybridlab to facilitate the initial stage of 3D content creation in virtual environments, (e) HMD for AR bibAR, and (f) HMD for VR varjo.
• Figure 2: Illustration of the six research topics with typical application examples, which include (a) Control: (i)controller, (ii)hand gesture and (iii)touch screen gesture with stylus pen); (b)Transform; (c) Model Construction and Rendering; (d) Non-Visual Feedback: (i)audio and (ii)haptic; (e)Display: (i)VR HMD and VR view and (ii) hologram; and (f) Link to Fabrication: (i) printable files for auto fabrication by 3D printer and (ii) instructions for manual fabrication).
• Figure 3: Examples of four of the six building blocks (a) Control: (i) Feng et al. employed a handwriting pad and pressure-sensitive pen as input devices for carving and relief creation in VR feng2022pressure, (ii) Both the surface drawing on phone and the phone's posture and position serve as creative inputs for prototyping kwan2019mobi3dsketch. (b) Display: Indicator bubbles utilized to address depth perception limitations in VR displays. wacker2020heatmaps. (c) Optional port: (i) RoMA that allows for almost simultaneous prototyping and fabrication peng2018roma, (ii) Convert the carving steps calculated from the digital model into projections to provide visual guidance hattab2019rough. And (d) Render: (i) The branch shaped brush specifically designed for creating tree prototypes yuan2021immersive, (ii) Liftoff for rendering complex and exquisite surfaces with a imported 2D sketched and 3D pen sweeping jackson2016lift.
• Figure 4: Scatter chart depicting publication counts by year and research topic using bubbles and pie chart depicting percentage in each topic.
• Figure 5: Workflow of prototyping within a metaverse environment and its interconnected fabrication process