A Multimodal Approach to Heritage Preservation in the Context of Climate Change

TL;DR

We address automatic degradation assessment of cultural heritage under climate stress using a lightweight multimodal architecture that fuses environmental sensor readings with visual imagery. The method adapts PerceiverIO with simplified 64-dimensional encoders and introduces Adaptive Barlow Twins loss to promote modality complementarity rather than redundancy, enabling effective learning from small data. On Strasbourg Cathedral data, the approach achieves 76.9% accuracy, beating strong baselines by up to 43%, with ablations showing sensor data provides stronger signal than imagery alone and a modal complementarity gain. The optimal partial correlation target is $\tau = 0.3$, and a 12M-parameter model with data augmentation and an ensemble over seeds yields robust results. This work demonstrates that architectural simplicity plus contrastive regularization enables practical AI-driven conservation support when large-scale pre-training is unavailable, with potential for cross-site transfer and interpretability enhancements in future work.

Abstract

Cultural heritage sites face accelerating degradation due to climate change, yet tradi- tional monitoring relies on unimodal analysis (visual inspection or environmental sen- sors alone) that fails to capture the complex interplay between environmental stres- sors and material deterioration. We propose a lightweight multimodal architecture that fuses sensor data (temperature, humidity) with visual imagery to predict degradation severity at heritage sites. Our approach adapts PerceiverIO with two key innovations: (1) simplified encoders (64D latent space) that prevent overfitting on small datasets (n=37 training samples), and (2) Adaptive Barlow Twins loss that encourages modality complementarity rather than redundancy. On data from Strasbourg Cathedral, our model achieves 76.9% accu- racy, a 43% improvement over standard multimodal architectures (VisualBERT, Trans- former) and 25% over vanilla PerceiverIO. Ablation studies reveal that sensor-only achieves 61.5% while image-only reaches 46.2%, confirming successful multimodal synergy. A systematic hyperparameter study identifies an optimal moderate correlation target (τ =0.3) that balances align- ment and complementarity, achieving 69.2% accuracy compared to other τ values (τ =0.1/0.5/0.7: 53.8%, τ =0.9: 61.5%). This work demonstrates that architectural sim- plicity combined with contrastive regularization enables effective multimodal learning in data-scarce heritage monitoring contexts, providing a foundation for AI-driven con- servation decision support systems.

Figures (8)

• Figure 1: Schema of the dataset
• Figure 2: Example of sensor data between two dates
• Figure 3: Image modality example
• Figure 4: Architecture overview.
• Figure 5: Model ranking by accuracy (left) and F1-score (right). Our approach achieves 76.9% accuracy, substantially outperforming all baselines.