Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FusionNet: Physics-Aware Representation Learning for Multi-Spectral and Thermal Data via Trainable Signal-Processing Priors

Georgios Voulgaris

TL;DR

This work addresses the fragility of deep learning models in cross-spectral remote sensing by aligning representations with signal-formation physics. It introduces FusionNet, a physics-aware intermediate fusion framework that combines Thermal Infrared and Short Wave Infrared cues using trainable signal-processing priors, including a Dilated Gabor Mix Pool CNN backbone. The study shows that a SWIR spectral ratio Band 7:6 captures persistent environmental signatures better than direct thermal signals, with DGCNN achieving 88.7% on this ratio and FusionNet reaching 90.6% across five spectral configurations; transfer learning from ImageNet degrades TIR performance, underscoring the need for modality-aware training. Together, these results demonstrate that physics-informed spectral priors and wider receptive fields yield robust, generalizable cross-spectral representations for cement-plant detection in real-world data.

Abstract

Modern deep learning models operating on multi-modal visual signals often rely on inductive biases that are poorly aligned with the physical processes governing signal formation, leading to brittle performance under cross-spectral and real-world conditions. In particular, approaches that prioritise direct thermal cues struggle to capture indirect yet persistent environmental alterations induced by sustained heat emissions. This work introduces a physics-aware representation learning framework that leverages multi-spectral information to model stable signatures of long-term physical processes. Specifically, a geological Short Wave Infrared (SWIR) ratio sensitive to soil property changes is integrated with Thermal Infrared (TIR) data through an intermediate fusion architecture, instantiated as FusionNet. The proposed backbone embeds trainable differential signal-processing priors within convolutional layers, combines mixed pooling strategies, and employs wider receptive fields to enhance robustness across spectral modalities. Systematic ablations show that each architectural component contributes to performance gains, with DGCNN achieving 88.7% accuracy on the SWIR ratio and FusionNet reaching 90.6%, outperforming state-of-the-art baselines across five spectral configurations. Transfer learning experiments further show that ImageNet pretraining degrades TIR performance, highlighting the importance of modality-aware training for cross-spectral learning. Evaluated on real-world data, the results demonstrate that combining physics-aware feature selection with principled deep learning architectures yields robust and generalisable representations, illustrating how first-principles signal modelling can improve multi-spectral learning under challenging conditions.

FusionNet: Physics-Aware Representation Learning for Multi-Spectral and Thermal Data via Trainable Signal-Processing Priors

TL;DR

This work addresses the fragility of deep learning models in cross-spectral remote sensing by aligning representations with signal-formation physics. It introduces FusionNet, a physics-aware intermediate fusion framework that combines Thermal Infrared and Short Wave Infrared cues using trainable signal-processing priors, including a Dilated Gabor Mix Pool CNN backbone. The study shows that a SWIR spectral ratio Band 7:6 captures persistent environmental signatures better than direct thermal signals, with DGCNN achieving 88.7% on this ratio and FusionNet reaching 90.6% across five spectral configurations; transfer learning from ImageNet degrades TIR performance, underscoring the need for modality-aware training. Together, these results demonstrate that physics-informed spectral priors and wider receptive fields yield robust, generalizable cross-spectral representations for cement-plant detection in real-world data.

Abstract

Modern deep learning models operating on multi-modal visual signals often rely on inductive biases that are poorly aligned with the physical processes governing signal formation, leading to brittle performance under cross-spectral and real-world conditions. In particular, approaches that prioritise direct thermal cues struggle to capture indirect yet persistent environmental alterations induced by sustained heat emissions. This work introduces a physics-aware representation learning framework that leverages multi-spectral information to model stable signatures of long-term physical processes. Specifically, a geological Short Wave Infrared (SWIR) ratio sensitive to soil property changes is integrated with Thermal Infrared (TIR) data through an intermediate fusion architecture, instantiated as FusionNet. The proposed backbone embeds trainable differential signal-processing priors within convolutional layers, combines mixed pooling strategies, and employs wider receptive fields to enhance robustness across spectral modalities. Systematic ablations show that each architectural component contributes to performance gains, with DGCNN achieving 88.7% accuracy on the SWIR ratio and FusionNet reaching 90.6%, outperforming state-of-the-art baselines across five spectral configurations. Transfer learning experiments further show that ImageNet pretraining degrades TIR performance, highlighting the importance of modality-aware training for cross-spectral learning. Evaluated on real-world data, the results demonstrate that combining physics-aware feature selection with principled deep learning architectures yields robust and generalisable representations, illustrating how first-principles signal modelling can improve multi-spectral learning under challenging conditions.
Paper Structure (17 sections, 6 equations, 9 figures, 3 tables)

This paper contains 17 sections, 6 equations, 9 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Remotely Sensed Data
  4. Models
  5. CNN5
  6. Dilated Gabor Mix Pool CNN (DGCNN)
  7. Channel Attention Mechanism
  8. Model Training
  9. Results and Discussion
  10. Ablation Studies
  11. Backbone Comparison: DGCNN vs CNN5
  12. t-Distributed Stochastic Neighbour Embedding (t-SNE)
  13. Confusion Matrices
  14. Feature Extraction Analysis
  15. State Of The Art (SOTA) Comparison
  16. ...and 2 more sections

Figures (9)

  • Figure 1: a) Cement Chip and Surrounding Landcover, b) Each Image is Comprised of 3-Channels, Each Channel Represents a Specific Date: January 2018, April 2017, January 2017.
  • Figure 2: a) FusionNet: TIR (Bands 11-10) and SWIR (Bands 7-6 and Ratio 7:6), b) Backbone Networks: $f_1-f_5$: 5 Convolutional Layer Network (DGCNN) with an Initial Parameterised Gabor Convolutional Layer, a Mixture of Average and Maximum Pooling, and a Dilated Convolutional Layer; f: 5 Convolutional Layer Network (CNN5), c) Channel Attention Mechanism.
  • Figure 3: CNN5 vs DGCNN Average Accuracy ($\%$) Scores Across TIR (Bands 11–10) and SWIR (Bands 7–6 and Ratio 7:6) Inputs, Evaluated per Cement and Landcover Classes. The Proposed DGCNN Outperforms the Conventional Convolutional Backbone in Both Thermal and SWIR Datasets, with the Proposed Ratio Yielding the Highest Overall Accuracy.
  • Figure 4: t-Distributed Stochastic Neighbour Embedding (t-SNE) a) CNN5, b) DGCNN per Cement and Landcover Classes. The Proposed DGCNN Model, Yields a Clearer Separation Between Classes Compared to Traditional CNNs.
  • Figure 5: Confusion Matrix Results a) CNN5, b) DGCNN per Cement and Landcover Classes. The Proposed DGCNN Model Outperforms the Conventional Convolutional Architecture.
  • ...and 4 more figures