Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TGTM: TinyML-based Global Tone Mapping for HDR Sensors

Peter Todorov, Julian Hartig, Jan Meyer-Siemon, Martin Fiedler, Gregor Schewior

TL;DR

TGTM introduces a lightweight histogram-based global tone-mapping pipeline for HDR sensors in automotive imaging. It uses two histograms derived from a 26-bit HDR input to predict four parameters of a modified Reinhard tone-curve, producing a 12-bit LDR output with temporal stability and low compute (about 9k FLOPS per image). The network is trained via data simulation that inverts random tone curves on 8-bit sRGB sources, enabling supervision without ground-truth tone curves. On real HDR data, TGTM achieves up to 5.85 dB PSNR improvements over state-of-the-art methods while consuming orders of magnitude less computation, making it well-suited for embedding in ISP pipelines and real-time ADAS applications.

Abstract

Advanced driver assistance systems (ADAS) relying on multiple cameras are increasingly prevalent in vehicle technology. Yet, conventional imaging sensors struggle to capture clear images in conditions with intense illumination contrast, such as tunnel exits, due to their limited dynamic range. Introducing high dynamic range (HDR) sensors addresses this issue. However, the process of converting HDR content to a displayable range via tone mapping often leads to inefficient computations, when performed directly on pixel data. In this paper, we focus on HDR image tone mapping using a lightweight neural network applied on image histogram data. Our proposed TinyML-based global tone mapping method, termed as TGTM, operates at 9,000 FLOPS per RGB image of any resolution. Additionally, TGTM offers a generic approach that can be incorporated to any classical tone mapping method. Experimental results demonstrate that TGTM outperforms state-of-the-art methods on real HDR camera images by up to 5.85 dB higher PSNR with orders of magnitude less computations.

TGTM: TinyML-based Global Tone Mapping for HDR Sensors

TL;DR

TGTM introduces a lightweight histogram-based global tone-mapping pipeline for HDR sensors in automotive imaging. It uses two histograms derived from a 26-bit HDR input to predict four parameters of a modified Reinhard tone-curve, producing a 12-bit LDR output with temporal stability and low compute (about 9k FLOPS per image). The network is trained via data simulation that inverts random tone curves on 8-bit sRGB sources, enabling supervision without ground-truth tone curves. On real HDR data, TGTM achieves up to 5.85 dB PSNR improvements over state-of-the-art methods while consuming orders of magnitude less computation, making it well-suited for embedding in ISP pipelines and real-time ADAS applications.

Abstract

Advanced driver assistance systems (ADAS) relying on multiple cameras are increasingly prevalent in vehicle technology. Yet, conventional imaging sensors struggle to capture clear images in conditions with intense illumination contrast, such as tunnel exits, due to their limited dynamic range. Introducing high dynamic range (HDR) sensors addresses this issue. However, the process of converting HDR content to a displayable range via tone mapping often leads to inefficient computations, when performed directly on pixel data. In this paper, we focus on HDR image tone mapping using a lightweight neural network applied on image histogram data. Our proposed TinyML-based global tone mapping method, termed as TGTM, operates at 9,000 FLOPS per RGB image of any resolution. Additionally, TGTM offers a generic approach that can be incorporated to any classical tone mapping method. Experimental results demonstrate that TGTM outperforms state-of-the-art methods on real HDR camera images by up to 5.85 dB higher PSNR with orders of magnitude less computations.
Paper Structure (14 sections, 2 equations, 7 figures, 2 tables)

This paper contains 14 sections, 2 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Solution
  4. Inference Pipeline
  5. Data Simulation
  6. Neural Network Architecture
  7. Computational Complexity
  8. Training Details
  9. Solution Constraints
  10. Experiments
  11. Histogram Evaluation
  12. Results with Simulated Images
  13. Results with Real Images
  14. Conclusion

Figures (7)

  • Figure 1: Breakdown of computation in different phases on a 2 Mpx Full HD image.
  • Figure 2: Inference pipeline. The CNN uses two image histograms to output four parameters for tone curve creation.
  • Figure 3: Data simulation pipeline. An unmapped image is created by applying an inverted tone curve (orange) to a ground truth image.
  • Figure 4: Visual effect of applying an inverted tone curve on the right to the source image at the top left. The higher the curve gain is, the stronger the "L-shape" of the curve is. Whenever the gain is above 25, an image is visually completely black and a linear histogram is ineffective to represent the image information.
  • Figure 5: TGTM model architecture with a breakdown of the number of parameters and FLOPS.
  • ...and 2 more figures