Information-driven design of imaging systems

TL;DR

This work reframes imaging system design around the mutual information between noiseless encodings and noisy measurements, enabling direct, decoder-independent assessment of information preserved by the encoder. By decomposing $I(\mathbf{X};\mathbf{Y}) = H(\mathbf{Y}) - H(\mathbf{Y}|\mathbf{X})$ and leveraging three probabilistic models to upper-bound $H(\mathbf{Y})$, the authors provide practical estimators that work across diverse imaging modalities. They validate that information estimates predict downstream decoder performance in color photography, radio astronomy (black hole imaging), lensless imaging, and microscopy, and introduce IDEAL to optimize encoders via gradient ascent on information content—achieving comparable results to end-to-end optimization with reduced complexity. The framework offers a unified, scalable approach to evaluating and designing imaging systems under real-world noise, with potential extensions to stochastic encoders and task-specific information objectives. This can accelerate principled design in previously intractable imaging domains and unify cross-disciplinary performance criteria.

Abstract

Imaging systems have traditionally been designed to mimic the human eye and produce visually interpretable measurements. Modern imaging systems, however, process raw measurements computationally before or instead of human viewing. As a result, the information content of raw measurements matters more than their visual interpretability. Despite the importance of measurement information content, current approaches for evaluating imaging system performance do not quantify it: they instead either use alternative metrics that assess specific aspects of measurement quality or assess measurements indirectly with performance on secondary tasks. We developed the theoretical foundations and a practical method to directly quantify mutual information between noisy measurements and unknown objects. By fitting probabilistic models to measurements and their noise characteristics, our method estimates information by upper bounding its true value. By applying gradient-based optimization to these estimates, we also developed a technique for designing imaging systems called Information-Driven Encoder Analysis Learning (IDEAL). Our information estimates accurately captured system performance differences across four imaging domains (color photography, radio astronomy, lensless imaging, and microscopy). Systems designed with IDEAL matched the performance of those designed with end-to-end optimization, the prevailing approach that jointly optimizes hardware and image processing algorithms. These results establish mutual information as a universal performance metric for imaging systems that enables both computationally efficient design optimization and evaluation in real-world conditions. A video summarizing this work can be found at: https://waller-lab.github.io/EncodingInformationWebsite/

Figures (26)

• Figure 1: Information estimation for imaging systems. An encoder (e.g., an optical system) maps unknown objects to noiseless images $\mathbf{X}$. Noise corrupts these images to produce measurements $\mathbf{Y}$. The information estimator uses these measurements and a noise model to quantify how well measurements distinguish noiseless images (and thus objects under deterministic encoding). The estimate $\hat{I}(\mathbf{X}; \mathbf{Y})$ enables both encoder evaluation and design optimization.
• Figure 2: Information estimates predict decoder performance across four imaging applications. Each row shows representative objects, three encoder designs, example measurements, and the relationship between information estimates and a representative decoder performance metric (see Fig. \ref{['fig:decoder_performance_all_metrics']} for all metrics). a) Color photography: Bayer, learned, and random filter arrays with neural network demosaicing chakrabartiLearningSensorMultiplexing. b) Radio astronomy: Three telescope array configurations with inverse problem reconstruction. c) Lensless imaging: Lens, microlens array, and diffuser with Wiener deconvolution. d) LED array microscopy: Brightfield, differential phase contrast, and single-LED illumination with neural network protein expression prediction.
• Figure 3: Information-Driven Encoder Analysis Learning (IDEAL) designs encoders through gradient ascent on information estimates. a) IDEAL framework applied to color filter design: gradient feedback from information estimates updates parameters of a differentiable filter model to maximize information capture. b) Estimated information increases monotonically during optimization. c) The final IDEAL-optimized encoder matches the performance of jointly optimizing encoders and decoders end-to-end chakrabartiLearningSensorMultiplexing in terms of both downstream reconstruction error and measurement mutual information, while avoiding decoder complexity during training. We further study IDEAL in comparison to end-to-end in subsequent work markleyComputationallyEfficientInformationDriven2025.
• Figure S1: A minimal probabilistic graphical model of an imaging system. Three key variables describe the imaging process: (1) the object, which has unknown physical properties we want to measure, (2) the noiseless image, created when the imaging system encodes object properties through a deterministic process, and (3) the noisy measurement, produced when detection adds random noise to the noiseless image. The arrows show how information flows: the measurement can only reveal information about the object if that information was preserved in the noiseless image.
• Figure S2: Probabilistic modeling and visualization of imaging systems.a) Probabilistic model of microscopy, in which a distribution of objects (cells) are encoded to a distribution of images by a microscope and detected as noisy measurements. b) Two complementary ways to visualize measurement distributions: spatial coordinates show all pixels for a few samples, while energy coordinates show the full probability distribution for selected pixels. c) Visualization of how measurement noise affects distributions: as noise increases, measurement probability spreads further from the noiseless value. d) Demonstration of how encoder design affects information preservation: more distinct noiseless images (right) create measurements that remain separable even with noise, while similar images (left) produce overlapping distributions that make object discrimination harder.

Theorems & Definitions (1)

• proof