Universal computational thermal imaging overcoming the ghosting effect

Abstract

Thermal imaging is crucial for night vision but fundamentally hampered by the ghosting effect, a loss of detailed texture in cluttered photon streams. While conventional ghosting mitigation has relied on data post-processing, the recent breakthrough in heat-assisted detection and ranging (HADAR) opens a promising frontier for hyperspectral computational thermal imaging that produces night vision with day-like visibility. However, universal anti-ghosting imaging remains elusive, as state-of-the-art HADAR applies only to limited scenes with uniform materials, whereas material non-uniformity is ubiquitous in the real world. Here, we propose a universal computational thermal imaging framework, TAG (thermal anti-ghosting), to address material non-uniformity and overcome ghosting for high-fidelity night vision. TAG takes hyperspectral photon streams for nonparametric texture recovery, enabling our experimental demonstration of unprecedented expression recovery in thus-far-elusive ghostly human faces -- the archetypal, long-recognized ghosting phenomenon. Strikingly, TAG not only universally outperforms HADAR across various scenes, but also reveals the influence of material non-uniformity, shedding light on HADAR's effectiveness boundary. We extensively test facial texture and expression recovery across day and night, and demonstrate, for the first time, thermal 3D topological alignment and mood detection. This work establishes a universal foundation for high-fidelity computational night vision, with potential applications in autonomous navigation, reconnaissance, healthcare, and wildlife monitoring.

Figures (6)

• Figure 1: Universal thermal anti-ghosting (TAG) for high-fidelity night vision with day-like visibility. a, The TAG framework. TAG takes hyperspectral thermal imagery for nonparametric TeX decomposition and produces high-fidelity textures, along with non-uniform temperature and emissivity. b, The SLOT principle. To break TeX degeneracy, SLOT uses a B-spline basis expansion for continuous emissivity and imposes a smoothness constraint, universally applicable to non-uniform materials. In contrast, HADAR relies on rigid material categorization and suffers from classification errors when material non-uniformity is present. c, Traditional thermal imaging cannot separate ambient scattering from direct emission, yielding textureless silhouettes that resemble ghosts. d, TAG vividly recovers otherwise hidden geometric textures and facial expressions that are crucial for various perception tasks. From left to right: neutral (sad), frown with eyes open (angry), grin with eyes and mouth open (happy), and pout.
• Figure 2: The challenge of TeX degeneracy for texture recovery. a, Illustration of TeX degeneracy, where multiple TeX triplets produce an identical thermal radiation spectrum (taken from the red cross in b). b, Texture comparison. Left: poor texture recovery by panchromatic thermal imaging and CLAHE failing to break the TeX degeneracy. Right: vivid texture recovery by TAG, which successfully breaks the TeX degeneracy.
• Figure 3: Shedding light on the effectiveness boundary of HADAR. a, Comparable texture recovery between HADAR and TAG when intra-class material non-uniformity is smaller (negligible) than the inter-class emissivity contrast. b, Failure of HADAR TeX vision when intra-class material non-uniformity is larger (significant) than the inter-class emissivity contrast. c, HADAR is robust to emissivity/library inaccuracy when material non-uniformity is negligible. The leftmost panel uses the baseline candidate emissivity estimated by the HADAR approach. The middle and right panels use candidate curves from TAG controlled by the regularization parameter ($\lambda = 10^{-3}, 10^6$).
• Figure 4: Zero-shot cross-modal machine perception enabled by TAG. a, Input data comparison showing ghosting traditional thermal imaging and TAG with vivid textures. b, Task I: AI colorization. With textures, TAG perfectly accommodates color mapping and preserves spatial patterns. Without textures, traditional thermal images produce blurry color artifacts. c, Task II: 3D topological alignment. TAG recovers textures to flawlessly anchor a 468-point facial mesh, whereas alignment completely fails or drifts on traditional thermal images. d, Task III: Semantic perception and recognition. Standard RGB-trained vision engines successfully localize faces and predict nuanced human emotions (i.e., Sad, Happy) on TAG images, but suffer catastrophic breakdowns on thermal images.
• Figure 5: Passive recovery of facial texture and expression with high-fidelity night vision in pitch darkness. a, Ghosting thermal imaging at night. b, Nighttime facial expression recovery by TAG. Left: neutral expression with mouth and eyes closed. Right: Smiling with mouth and eyes open.