Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards AI-Architecture Liberty: A Comprehensive Survey on Design and Generation of Virtual Architecture by Deep Learning

Anqi Wang, Jiahua Dong, Lik-Hang Lee, Jiachuan Shen, Pan Hui

TL;DR

This survey analyzes how deep learning enables the design and generation of virtual architecture within immersive environments, focusing on three pillars: 3D shape generation techniques, 3D-aware image synthesis, and virtual-environment design principles. It identifies four critical production challenges—datasets, multimodality, design intuition, and generative frameworks—and analyzes current DL approaches (GANs, VAEs, diffusion models) and 3D representations (voxel, mesh, point cloud, neural fields). The authors propose four research agendas—agency between designers and machines, communication and explainability, user diversity and collective participation, and adaptive multimodal tooling—along with four enablers for ubiquitous interaction. The work aims to foster designer–AI collaboration, broadening access to virtual-building content creation and governance in metaverse-like environments. Overall, it advocates interdisciplinary collaboration to advance equitable, efficient, and ethically grounded AI-generated architecture in virtual spaces.

Abstract

3D shape generation techniques leveraging deep learning have garnered significant interest from both the computer vision and architectural design communities, promising to enrich the content in the virtual environment. However, research on virtual architectural design remains limited, particularly regarding designer-AI collaboration and deep learning-assisted design. In our survey, we reviewed 149 related articles (81.2% of articles published between 2019 and 2023) covering architectural design, 3D shape techniques, and virtual environments. Through scrutinizing the literature, we first identify the principles of virtual architecture and illuminate its current production challenges, including datasets, multimodality, design intuition, and generative frameworks. We then introduce the latest approaches to designing and generating virtual buildings leveraging 3D shape generation and summarize four characteristics of various approaches to virtual architecture. Based on our analysis, we expound on four research agendas, including agency, communication, user consideration, and integrating tools. Additionally, we highlight four important enablers of ubiquitous interaction with immersive systems in deep learning-assisted architectural generation. Our work contributes to fostering understanding between designers and deep learning techniques, broadening access to designer-AI collaboration. We advocate for interdisciplinary efforts to address this timely research topic, facilitating content designing and generation in the virtual environment.

Towards AI-Architecture Liberty: A Comprehensive Survey on Design and Generation of Virtual Architecture by Deep Learning

TL;DR

This survey analyzes how deep learning enables the design and generation of virtual architecture within immersive environments, focusing on three pillars: 3D shape generation techniques, 3D-aware image synthesis, and virtual-environment design principles. It identifies four critical production challenges—datasets, multimodality, design intuition, and generative frameworks—and analyzes current DL approaches (GANs, VAEs, diffusion models) and 3D representations (voxel, mesh, point cloud, neural fields). The authors propose four research agendas—agency between designers and machines, communication and explainability, user diversity and collective participation, and adaptive multimodal tooling—along with four enablers for ubiquitous interaction. The work aims to foster designer–AI collaboration, broadening access to virtual-building content creation and governance in metaverse-like environments. Overall, it advocates interdisciplinary collaboration to advance equitable, efficient, and ethically grounded AI-generated architecture in virtual spaces.

Abstract

3D shape generation techniques leveraging deep learning have garnered significant interest from both the computer vision and architectural design communities, promising to enrich the content in the virtual environment. However, research on virtual architectural design remains limited, particularly regarding designer-AI collaboration and deep learning-assisted design. In our survey, we reviewed 149 related articles (81.2% of articles published between 2019 and 2023) covering architectural design, 3D shape techniques, and virtual environments. Through scrutinizing the literature, we first identify the principles of virtual architecture and illuminate its current production challenges, including datasets, multimodality, design intuition, and generative frameworks. We then introduce the latest approaches to designing and generating virtual buildings leveraging 3D shape generation and summarize four characteristics of various approaches to virtual architecture. Based on our analysis, we expound on four research agendas, including agency, communication, user consideration, and integrating tools. Additionally, we highlight four important enablers of ubiquitous interaction with immersive systems in deep learning-assisted architectural generation. Our work contributes to fostering understanding between designers and deep learning techniques, broadening access to designer-AI collaboration. We advocate for interdisciplinary efforts to address this timely research topic, facilitating content designing and generation in the virtual environment.
Paper Structure (50 sections, 9 figures, 5 tables)

This paper contains 50 sections, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Preamble: 3D Virtual Architecture Generated by Deep Learning
  3. Methodology and Related Articles
  4. Scope and Structure
  5. Overview of Generated Virtual Architecture
  6. Virtual Architecture and Design Discipline
  7. Virtual Architecture: Mission and Scope
  8. Design Discipline
  9. Deep Generative Models and 3D Representations
  10. 3D Shape Generation
  11. 3D-Aware Image Synthesis
  12. 3D Representations
  13. Generated Virtual Architecture with Deep Learning
  14. Related Works
  15. Dataset
  16. ...and 35 more sections

Figures (9)

  • Figure 1: Virtual architecture provides rich use affordances, benefiting various aspects, including users, communities, and organizations. (a) Hub Rooms in VRChat; (b) Simulated city scene with various buildings; (c) Interior of individual house in a virtual world game, Second Life; (d) Saint Motel Fan Meet on Mozzila hub; (e) Sci-fi nightclub; (f) Center building in pride month event held on Dencentraland; (g) Realism Sotheby’s virtual gallery; (h) The interactive digital experience in a virtual building for the brand Jose Cuervo designed by Rojkind Arquitectos; (i) The office building of Vice Media Group was designed in metaverse by BIG Architect; (j) "NFTism" is a virtual NFT gallery developed collaboratively by Zaha Hadid Architects (ZHA) and Journee. The architectural design focuses on user experience, social interaction, and "dramaturgical" compositions, supporting MMO (massively multiplayer online) technology and integrating audio-video interaction; (k) The virtual city designed and developed by ZHA, filled with fluid style virtual architecture; (l) Each piece of land and each item in the virtual land as a non-fungible token could trade in the virtual world of Decentraland; (m) A mapping showing ownership of non-fungible token land in Sandbox; (n) A user-friendly tool for editing and creating buildings and land in social VR, Mozzila hub.
  • Figure 2: Applications of deep learning impact our lives in all aspects. (a) Self-drive cars; (b) AlfaGO; (c) Segmentation in the city recognition with computer vision; (d) Mirror World NFT, which shows AI dialogue character with personality and development from learning; (e) OpenCV recognizing the object types in the camera view; (f) Apple watch paired with deep learning detect atrial fibrillation with 97 % accuracy; (g) ChatGPT developed by OpenAI; (h) Recommendation system in the Tiktok; (i) Smart agriculture implemented by deep learning with drones; (j) DALLE-2, one powerful painting tool empowered by machine learning; (k) D.O.U.G, a collaborative robotic arms interacted with human, learning human behaviors and gestures, performance and created by artist Soug Wen; (l) digital human body generation by 3D reconstruction technique; (m) An AI art movie created by GANs (Casey Reas); (n) BCI (Brain-computer interface).
  • Figure 3: (a) This survey investigates the intersection of various areas. (b) A profile of the number of cited works categorized by different years and topics. The survey's scope and profile of related articles: a -- architectural studies on DGMs; c -- computer vision studies; v -- works on rules in VWs.
  • Figure 4: Virtual architecture projects exhibit various forms: (a) Joris Putteneers' "Synesthesia" (2016) creates a surreal and complex architectural construction through algorithmic simulation in Houdini; (b) "E-motion" (2020) by Fei Chen et al. features a digital interface allowing real-time data interaction for rethinking co-living among species; (c) "ISOS" by Viviane Toraci Fiorella, Taza Celilia, and Prandini Alvaro Campo in the "Volumetric Cinema" workshop by Current.CAM (2022) explores the interaction between virtual avatars and dramatic environments; (d) Tane Moleta and Mizuho Nishioka's collaborative project (2021), "Populating Virtual Worlds Together," leads to an autonomous virtual space; (e) Utilizing BCI to capture affective-driven dynamic noise, Barsan et al. (2020) demonstrate 3D volumetric architecture in virtual environments barsan2020affective; (f) Current.CAM's VR gallery (2021) features continuous partitioned spaces with fluid blue, reinforcing the shaping of digital interfaces on the human senses.
  • Figure 5: The frameworks of GANs, VAEs, and Diffusion Models.
  • ...and 4 more figures