Real-time Noise Source Estimation of a Camera System from an Image and Metadata

TL;DR

This work presents a real-time, memory-efficient noise-source estimator for camera systems that fuses image data with camera metadata to decompose total image noise into photon shot noise, dark current shot noise, readout noise, and an unexpected-noise term. A four-branch DNN (PN, DCSN, RN, and $\xi_{ ext{M/I}}$) extends a DRNE-based baseline, with metadata configuration into w/o-Meta, Min-Meta, and Full-Meta variants; training relies on simulated physics-based noise on a large, mostly noise-free dataset. Across six datasets and two real camera systems, Full-Meta consistently yields the most accurate and robust per-noise-source estimates and total noise, even handling unexpected noise via $oldsymbol{\xi}_{ ext{M/I}}$ detection. The approach demonstrates real-time performance (approximately 1–2 ms per patch) and can improve downstream tasks such as denoising, while providing a foundation for automatic countermeasures in autonomous perception systems. Overall, this work advances noise characterization from symptom suppression to root-cause identification by leveraging physics-informed metadata fusion in a compact neural architecture.

Abstract

Autonomous machines must self-maintain proper functionality to ensure the safety of humans and themselves. This pertains particularly to its cameras as predominant sensors to perceive the environment and support actions. A fundamental camera problem addressed in this study is noise. Solutions often focus on denoising images a posteriori, that is, fighting symptoms rather than root causes. However, tackling root causes requires identifying the noise sources, considering the limitations of mobile platforms. This work investigates a real-time, memory-efficient and reliable noise source estimator that combines data- and physically-based models. To this end, a DNN that examines an image with camera metadata for major camera noise sources is built and trained. In addition, it quantifies unexpected factors that impact image noise or metadata. This study investigates seven different estimators on six datasets that include synthetic noise, real-world noise from two camera systems, and real field campaigns. For these, only the model with most metadata is capable to accurately and robustly quantify all individual noise contributions. This method outperforms total image noise estimators and can be plug-and-play deployed. It also serves as a basis to include more advanced noise sources, or as part of an automatic countermeasure feedback-loop to approach fully reliable machines.

Figures (16)

• Figure 1: Proposed Camera Noise Source Estimation. Different noise sources affect the image formation process of a scene. Our noise source estimator quantifies major noise source contributions $\hat{\sigma}_{i \in \{ \text{PN, DCSN, RN}\}}$, unexpected noise $\hat{\xi}_{\text{M/I}}$, and the total image noise $\hat{\sigma}_{\text{Total}}$ using an image and camera metadata.
• Figure 2: Noise level estimator vs. proposed noise source and level estimator. Left: Customized baseline estimator $\text{DRNE}_\text{cust.}$, which predicts the noise level of the input image's total noise. Right: Proposed noise source estimator that additionally employs camera metadata and predicts the noise levels of four different noise types. Architectural changes from the baseline are highlighted.
• Figure 3: Datasets. Exemplary image snippets from Sim ($896 \times 768 \,\px$), TAMPERE17 ($512 \times 512 \,\px$), Udacity ($1920 \times 1200 \,\px$), KITTI ($1242 \times 375 \,\px$), Cellar and Parking Lot (ICX285: $1360 \times 1024 \,\px$, EV76C661: $1280 \times 1024 \,\px$).
• Figure 4: Noise estimation of uncorrupted KITTI and Udacity datasets. The reference methods estimate significant noise in KITTI images ($\hat{\sigma} \leq 5$) and low noise in Udacity data ($\hat{\sigma} \leq 1.25$).
• Figure 5: Camera systems.ICX285 is attached on an autonomous robotic platform and EV76C661 on an inspection helmet.