Incorporating Eye-Tracking Signals Into Multimodal Deep Visual Models For Predicting User Aesthetic Experience In Residential Interiors

TL;DR

This work tackles predicting interior aesthetic experiences from dynamic video by introducing a dual-branch CNN–LSTM that fuses visual content with eye-tracking signals. Eye-tracking data are used as privileged information during training to enable video-only deployment at inference, demonstrating robust performance on both objective and subjective aesthetic dimensions. The authors contribute a multimodal dataset of 224 interior-design videos with synchronized gaze from 28 participants, a spatio-temporal fusion model with a cross-modal transfer mechanism, and a training strategy that preserves performance without gaze. Results show 72.2% accuracy on objective tasks and 66.8% on subjective tasks, with pupil dynamics mainly aiding objective judgments and gaze-augmented cues boosting subjective evaluations, accompanied by interpretable Grad-CAM visualizations. This approach supports practical, scalable tools for assessing architectural aesthetics in interior design using readily available video inputs and gaze data during training.

Abstract

Understanding how people perceive and evaluate interior spaces is essential for designing environments that promote well-being. However, predicting aesthetic experiences remains difficult due to the subjective nature of perception and the complexity of visual responses. This study introduces a dual-branch CNN-LSTM framework that fuses visual features with eye-tracking signals to predict aesthetic evaluations of residential interiors. We collected a dataset of 224 interior design videos paired with synchronized gaze data from 28 participants who rated 15 aesthetic dimensions. The proposed model attains 72.2% accuracy on objective dimensions (e.g., light) and 66.8% on subjective dimensions (e.g., relaxation), outperforming state-of-the-art video baselines and showing clear gains on subjective evaluation tasks. Notably, models trained with eye-tracking retain comparable performance when deployed with visual input alone. Ablation experiments further reveal that pupil responses contribute most to objective assessments, while the combination of gaze and visual cues enhances subjective evaluations. These findings highlight the value of incorporating eye-tracking as privileged information during training, enabling more practical tools for aesthetic assessment in interior design.

Figures (4)

• Figure 1: Architecture of the proposed dual-branch CNN–LSTM model.
• Figure 2: Grad-CAM visualization of task-specific attention.
• Figure 3: Temporal attention for 'Leave-Enter' dimension.
• Figure 4: Progressive attention for organizational assessment.