End-to-end metasurface design for temperature imaging via broadband Planck-radiation regression

TL;DR

This paper tackles temperature imaging from broadband LWIR radiation using a compact end-to-end optics and computation framework. It introduces Planck regression, a nonlinear reconstruction that enforces Planck's blackbody law to recover a temperature map $\mathbf{T}(x,y)$ from a grayscale sensor image, and jointly optimizes a single-layer metasurface by end-to-end design to minimize reconstruction error. The end-to-end system achieves robust, high-quality reconstructions of arbitrary temperature maps (including random patterns) with about $2-4\%$ RMSE under realistic sensor noise, and reduces error roughly fourfold relative to non-end-to-end baselines. Planck regression outperforms CNN-based reconstructions in generalization, and the metasurface design approach paves the way for ultra-compact thermal-imaging devices that exploit physics-informed priors rather than purely data-driven models.

Abstract

We present a theoretical framework for temperature imaging from long-wavelength infrared thermal radiation (e.g. 8-12 $μ$m) through the end-to-end design of a metasurface-optics frontend and a computational-reconstruction backend. We introduce a new nonlinear reconstruction algorithm, ``Planck regression," that reconstructs the temperature map from a grayscale sensor image, even in the presence of severe chromatic aberration, by exploiting blackbody and optical physics particular to thermal imaging. We combine this algorithm with an end-to-end approach that optimizes a manufacturable, single-layer metasurface to yield the most accurate reconstruction. Our designs demonstrate high-quality, noise-robust reconstructions of arbitrary temperature maps (including completely random images) in simulations of an ultra-compact thermal-imaging device. We also show that Planck regression is much more generalizable to arbitrary images than a straightforward neural-network reconstruction, which requires a large training set of domain-specific images.

Figures (5)

• Figure 1: The end-to-end optimization pipeline. The differentiable forward pipeline includes a physical model of the monochrome sensor image formed from the blackbody emitting object with temperature map $\vb{T}$, together with a computational reconstruction of the object’s temperature map $\vb{T}_{\text{est}}$. The reconstruction error $\mathcal{L}(\vb{T}, \vb{T}_{\text{est}})$ is formed at the end of the forward pipeline. The optimization minimizes this error with respect to the metasurface parameters $\vb{p}$ and the reconstruction hyper-parameter $\alpha$ by back-propagating the gradients through the image formation and reconstruction algorithm of the forward pipeline.
• Figure 2: (a) The metasurface unit cell, consisting of a square silicon pillar with a variable width of [range-phrase=–, range-units=single]1.82.7 on a silicon substrate. (b) The phase $\angle t$ and the amplitude squared $T = |t|^2$ of the unit cell complex transmission coefficient $t$ as a function of pillar width, calculated at the center frequency $(\lambda = 9.6µm)$ and at the bandwidth limits $(\lambda = 8µm, \lambda = 12µm)$.
• Figure 3: An end-to-end optimized single-layer metasurface design used to reconstruct temperature maps of $32^2$ pixels from $128^2$ monochrome sensor images of LWIR thermal radiation ([range-phrase=–, range-units=single]812), with a sensor pixel size of $12µm$. The metasurface consists of $2048^2$ unit cells with a $4µm$ period (Figure \ref{['results_geometry_figure']}), and the NA is $0.20$. In terms of the wavelength corresponding to the center frequency, $\lambda = 9.6µm$, the metasurface diameter is $853.33\lambda$, and the distance from the metasurface to the sensor is $2083.33 \lambda$. In the end-to-end optimization, the metasurface is initialized as an inverse-designed monochromatic lens at the center frequency, and converges to the structure depicted in (a), which plots the width of each pillar over the metasurface area. (b--d): The end-to-end design accurately reconstructs temperature maps of arbitrary objects such as the MIT logo, with a relative RMSE (root mean square error) of $2.09 \%$ for $4 \%$ sensor noise. (e): The evolution of the objective function [Equation \ref{['endtoend_objective']}] (reconstruction of random temperature maps, blue) and the reconstruction error of a test MIT logo (red) throughout the end-to-end optimization, and (f) the evolution of the reconstruction hyper-parameter $\alpha$. (g): Using the end-to-end optimized design, the reconstruction error vs. sensor noise amplitude for a random test temperature map (light blue) and the test MIT logo (red). (h): The PSFs of the metasurface at the 21 Chebyshev points used for the spectral integration scheme of the image formation [Equation \ref{['imageformation']}].
• Figure 4: (a): End-to-end optimized single-layer metasurface designs for varying sensor sizes: $64^2$, $128^2$, and $256^2$ sensor pixels, and varying metasurface initalizations: uniform pillar widths, and three inverse-designed metalenses---a monochromatic lens optimized to maximize focal intensity pestourieInverseDesignLargearea2018 at the center frequency $\lambda = 9.6µm$, an "achromatic" lens optimized to maximize the worst-case focal intensity pestourieInverseDesignLargearea2018liInverseDesignEnables2022 over the whole LWIR bandwidth, and a polychromatic lens optimized to focus five different frequencies to five different spots on the sensor. The parameters of the metasurfaces are the same as in Figure \ref{['results1_figure']}. For the three sensor sizes, we apply $2 \%, 4 \%$ and $8 \%$ noise (relative to the image mean) since we observe that the image mean roughly decreases by half when we double the sensor width. (b and c): The relative RMSE of the reconstructions of a test MIT logo and a test random object for the end-to-end designs shown in (a). For a fixed sensor size, the end-to-end designs resulting from different metasurface initializations all have similar performance and improve upon the inverse-designed baselines (initial inverse-designed metalenses combined with the Planck regression). The reconstruction error of the end-to-end designs decreases as the sensor size increases.
• Figure 5: Reconstruction of various test temperature maps using both the Planck-based reconstruction algorithm and a baseline CNN reconstruction algorithm with a U-Net architecture. The sensor images are generated using the end-to-end designed metasurface from Figure \ref{['results1_figure']} with $4 \%$ sensor noise. The CNN is trained on a set of 52,000 images of circles with randomly sampled radii and temperature between $T_{\text{low}} = 263.15K$ to $T_{\text{high}} = 623.15K$, over a constant background $T_{\text{bg}} = 443.15K$. (a) and (b): Reconstructions of "test” circles that are similar to the CNN training set, for which the U-Net reconstruction is extremely accurate and outperforms the Planck-based algorithm. (c) and (d): Reconstructions of test temperature maps that are different from the CNN training set: a square and the MIT logo against constant backgrounds of $T_{\text{bg}}$. The U-Net fails to generalize to these cases and produces unrecognizable reconstructions, in contrast to the Planck-based algorithm which demonstrates accurate and recognizable reconstructions.