Multimodal Data Fusion to Capture Dynamic Interactions between Built Environment and Vulnerable Older Adults

TL;DR

This study tackles the difficulty of capturing micro-scale interactions between the built environment and mobility for vulnerable older adults by proposing a multimodal data fusion framework that integrates eye-tracking, gait, physiological signals, GPS/SLAM, and video in real urban settings. Using a Singaporean field protocol, the approach reconstructs accurate trajectories in GPS-denied spaces, extracts BE features (surface materials, width, signage), and links them to perceptual and physiological responses at high temporal resolution. Key contributions include automated walkway material classification via Grounding DINO and SAM, gait and arousal analyses aligned to synchronized trajectories, and a prototype dashboard enabling time–space visualization for planners. The work advances inclusive urban design by providing dynamic, evidence-based indicators of accessibility and comfort, and lays the foundation for integrating such multimodal outputs into city-scale digital twins for policy guidance.

Abstract

Ensuring safe and inclusive mobility for vulnerable older adults is an emerging priority in urban planning. However, existing data sources such as surveys or GIS-based audits provide limited insight into how micro-scale built environment (BE) features influence real-world behavior and perception. This study presents a novel multimodal data-fusion approach that integrates wearable and environmental sensing to dynamically represent human-environment interactions and quantify the BE impacts on mobility among vulnerable older adults, specifically those with knee osteoarthritis or a history of falls. Data collected during naturalistic walking sessions in Singapore, are used to demonstrate this framework of synchronized streams from eye tracking, kinematic sensors, physiological monitors, GPS, and video recordings. Preliminary results show how AI-driven data fusion can uncover behaviorally and perceptually significant urban segments, providing a basis for actionable insights in inclusive design. This human-centered analytical approach advances the representation of urban environments from the perspective of vulnerable pedestrians, establishing a foundation for evidence-based, age-friendly city planning.

Figures (4)

• Figure 1: Overview of the multimodal data collection setup used to represent built-environment impacts on older adults’ mobility. Participants were equipped with eye-tracking glasses, wearable physiological sensors, and body-mounted IMUs while walking along urban routes in Singapore. A trailing GoPro camera provided third-person video. The integrated configuration captures synchronized gaze, gait, physiological, and environmental data for dynamic and fine-grained analysis of human–environment interactions.
• Figure 2: Activities in GPS denied environment while SLAM trajectory with GPS data points.
• Figure 3: Reconstructed walking trajectories annotated with walkway attributes. (Left) Trajectory color-coded by estimated walkway width, showing transitions between narrow and wide path segments. (Right) Trajectory labeled with recognized walkway surface materials, including brushed concrete, asphalt, granite, and exposed aggregate concrete. Sample third-person video frames (bottom) illustrate representative walking contexts corresponding to points A–E along the trajectory.
• Figure 4: Prototype interactive dashboard for synchronized visualization of multimodal data streams. The interface displays third-person video, first-person video with fixation overlays, and physiological signals along the walking trajectory.