Capsule Network-Based Multimodal Fusion for Mortgage Risk Assessment from Unstructured Data Sources

TL;DR

This work tackles mortgage risk prediction using unstructured, publicly available data by introducing FusionCapsNet, a capsule-inspired multimodal fusion framework that simultaneously processes text, LiDAR-derived imagery, and sentiment signals. The method operates in two phases: first identifying the best unimodal encoders (BERT for text, VGG for images, and MLP for sentiment features), then fusing their outputs through a capsule-based architecture with dynamic routing and adaptive modality weighting to preserve spatial, contextual, and modality-specific information. Compared with traditional fusion strategies, FusionCapsNet achieves higher predictive performance (AUC, $pAUC$, and F1) and offers improved interpretability via GradCAM and token-level attributions, demonstrating the value of unstructured data in mortgage risk modeling. The approach provides a cost-free, scalable early-warning layer for lenders and policymakers, with potential applicability to other financial risk domains and markets, given open data availability and further extension with transfer learning and expanded data sources.

Abstract

Mortgage risk assessment traditionally relies on structured financial data, which is often proprietary, confidential, and costly. In this study, we propose a novel multimodal deep learning framework that uses cost-free, publicly available, unstructured data sources, including textual information, images, and sentiment scores, to generate credit scores that approximate commercial scorecards. Our framework adopts a two-phase approach. In the unimodal phase, we identify the best-performing models for each modality, i.e. BERT for text, VGG for image data, and a multilayer perceptron for sentiment-based features. In the fusion phase, we introduce the capsule-based fusion network (FusionCapsNet), a novel fusion strategy inspired by capsule networks, but fundamentally redesigned for multimodal integration. Unlike standard capsule networks, our method adapts a specific mechanism in capsule networks to each modality and restructures the fusion process to preserve spatial, contextual, and modality-specific information. It also enables adaptive weighting so that stronger modalities dominate without ignoring complementary signals. Our framework incorporates sentiment analysis across distinct news categories to capture borrower and market dynamics and employs GradCAM-based visualizations as an interpretability tool. These components are designed features of the framework, while our results later demonstrate that they effectively enrich contextual understanding and highlight the influential factors driving mortgage risk predictions. Our results show that our multimodal FusionCapsNet framework not only exceeds individual unimodal models but also outperforms benchmark fusion strategies such as addition, concatenation, and cross attention in terms of AUC, partial AUC, and F1 score, demonstrating clear gains in both predictive accuracy and interpretability for mortgage risk assessment.

Figures (10)

• Figure 1: Overview of the framework consisting of two main phases. In Phase 1 (Unified Analysis), news articles undergo sentiment analysis and category selection, followed by independent model selection for text, numeric sentiment features, and LiDAR images. In Phase 2 (Multimodal Analysis), the best-performing models serve as modality-specific encoders whose outputs are fused using the proposed FusionCapsNet architecture to predict mortgage default risk.
• Figure 2: Multimodal capsule-based fusion architecture. Encoded outputs from text, image, and numeric modalities are projected into primary capsules and subsequently routed to digit capsules. Modality-specific confidence metrics ($C_{\mathrm{Txt}}$, $C_{\mathrm{Img}}$, $C_{\mathrm{Num}}$) are computed and scaled by learnable weights ($W_{1}$, $W_{2}$, $W_{3}$), then passed through an adaptive weighted gate to produce the final risk prediction.
• Figure 3: Class distribution of the mortgage dataset. The high-risk (default) group constitutes 14% of the observations, while the low-risk (non-default) group accounts for 86%, indicating a class imbalance that must be addressed in the modeling process.
• Figure 4: Selected keyword categories for text data extraction, representing economic, housing, and policy factors potentially influencing mortgage risk.
• Figure 5: Sample LiDAR maps illustrating elevation and spatial features, including green space, building density, water bodies, and transport infrastructure.