DTBIA: An Immersive Visual Analytics System for Brain-Inspired Research

TL;DR

DTBIA addresses the challenge of visualizing high-dimensional, temporally dynamic, and spatially complex Digital Twin Brain data by introducing an immersive visual analytics system with a hierarchical Region–Voxel–Slice exploration, Real-Scale functional visualization, and Large-Scale structural navigation. It integrates BOLD and DTI data and employs 3D edge bundling to reduce clutter, validated through two case studies with brain researchers that demonstrated improved interpretation of neural patterns and connectivity. The work advances cross-disciplinary brain-inspired research by enabling embodied exploration, cross-species comparisons, and interactive model validation in VR, with potential to enhance collaboration among neuroscience, biology, and AI communities. Practical impact includes enhanced insight into spatiotemporal brain activity, structural networks, and the DMN, and a framework adaptable to larger datasets and additional species.

Abstract

The Digital Twin Brain (DTB) is an advanced artificial intelligence framework that integrates spiking neurons to simulate complex cognitive functions and collaborative behaviors. For domain experts, visualizing the DTB's simulation outcomes is essential to understanding complex cognitive activities. However, this task poses significant challenges due to DTB data's inherent characteristics, including its high-dimensionality, temporal dynamics, and spatial complexity. To address these challenges, we developed DTBIA, an Immersive Visual Analytics System for Brain-Inspired Research. In collaboration with domain experts, we identified key requirements for effectively visualizing spatiotemporal and topological patterns at multiple levels of detail. DTBIA incorporates a hierarchical workflow - ranging from brain regions to voxels and slice sections - along with immersive navigation and a 3D edge bundling algorithm to enhance clarity and provide deeper insights into both functional (BOLD) and structural (DTI) brain data. The utility and effectiveness of DTBIA are validated through two case studies involving with brain research experts. The results underscore the system's role in enhancing the comprehension of complex neural behaviors and interactions.

Figures (6)

• Figure 1: The figure illustrates the workflow of DTBIA, combining hierarchical exploration from region to voxel and slice levels with immersive navigation from real-scale to large-scale brain models. The green exploration paths begin with DTB data, comparing it to biological brain data to identify regions of interest (RoIs). Users can follow the deep green path to navigate the large-scale brain for relevant RoIs or take the light green path to move to the voxel level, where encoding animations and FDEB refine voxel selection. The red exploration path starts with biological brain data (human or macaque), leading to slice-level analysis. Through directional slicing, users examine voxel activity, then transition to large-scale brain models for further analysis using FDEB. Functional BOLD data is represented in brown, and structural DTI data in blue.
• Figure 2: This figure illustrates three exploration paths of DTBIA. The green paths begin by comparing biological brain data with DTB data through encoding animations to assess model validity. The deep green path identifies peak brain activities with a time slider, selects RoIs, and navigates to the large-scale brain for further exploration, highlighting regions and connections. Users then analyze static 3D line charts and apply thresholding to the FDEB results, refining connections to show the most relevant links. The light green path diverges by focusing on Voxel-Level encoding animations, allowing for the identification of high-activity voxels and further refinement through FDEB and threshold selection. The red path, starting from biological brain data, involves Slice-Section-Level exploration, utilizing sagittal, horizontal, and coronal planes to find voxels within the slices, followed by encoding animations, FDEB, and threshold refinement.
• Figure 3: This figure demonstrates the system's capability to integrate macaque brain data, showing (A) functional data by region and (B) DTI connections among voxels within a region.
• Figure 4: This figure showcases key findings from the first case study: (A) Time slider use reveals peak brain activity at time point 119. (B) Region 16 (Front_Inf_Orb_R) shows high BOLD signals, leading to a teleportation to it. (C) Explores extensive connections in this region. (D) Reveals DTI connections among voxels. (E) Line charts near the brain's center highlight above-average activity.
• Figure 5: We use different colors to display each of these 7 regions in a graphical representation of the DMN to make it easier to distinguish and analyze them. Results can be viewed from different angles: (A) Front view (B) Back view (C) Top view