Adaptive Illumination Control for Robot Perception

TL;DR

The paper addresses the challenge of robust robot perception under difficult lighting by enabling active onboard illumination that is tuned to perception goals. It introduces Lightning, a three-stage pipeline comprising a CLID relighting network for data expansion, an offline DP-based Optimal Intensity Schedule (OIS) to balance image utility and power, and a real-time Illumination Control Policy (ILC) learned via imitation learning. The CLID network enables synthesis of scene appearances across light intensities, the Oracle computes globally optimal light sequences, and the ILC deploys these policies on hardware to improve SLAM robustness while saving power. This approach demonstrates tangible gains in SLAM reliability across sequences and provides a practical, deployable method for task-aware active illumination in mobile robotics.

Abstract

Robot perception under low light or high dynamic range is usually improved downstream - via more robust feature extraction, image enhancement, or closed-loop exposure control. However, all of these approaches are limited by the image captured these conditions. An alternate approach is to utilize a programmable onboard light that adds to ambient illumination and improves captured images. However, it is not straightforward to predict its impact on image formation. Illumination interacts nonlinearly with depth, surface reflectance, and scene geometry. It can both reveal structure and induce failure modes such as specular highlights and saturation. We introduce Lightning, a closed-loop illumination-control framework for visual SLAM that combines relighting, offline optimization, and imitation learning. This is performed in three stages. First, we train a Co-Located Illumination Decomposition (CLID) relighting model that decomposes a robot observation into an ambient component and a light-contribution field. CLID enables physically consistent synthesis of the same scene under alternative light intensities and thereby creates dense multi-intensity training data without requiring us to repeatedly re-run trajectories. Second, using these synthesized candidates, we formulate an offline Optimal Intensity Schedule (OIS) problem that selects illumination levels over a sequence trading off SLAM-relevant image utility against power consumption and temporal smoothness. Third, we distill this ideal solution into a real-time controller through behavior cloning, producing an Illumination Control Policy (ILC) that generalizes beyond the initial training distribution and runs online on a mobile robot to command discrete light-intensity levels. Across our evaluation, Lightning substantially improves SLAM trajectory robustness while reducing unnecessary illumination power.

Figures (5)

• Figure 1: (a) Oracle intensity schedule over time (frames vs. discrete levels $K$); orange curve denotes $k^t$ and red boxes mark representative frames. Intensity is reduced near a reflective whiteboard to prevent specularities/saturation, then increased afterward. (b) Predicted intensity traces comparing the oracle OIS and learned controller (ILC). (c) CLID relighting: predicted ambient component $A$ and light-contribution map $S_F$ are used to synthesize $I^k$ for candidate intensities $k\in[0,1]$. (d) Examples of light sensitivity (specular reflections and blown highlights) motivating adaptive control.
• Figure 2: CLID relighting on robot sequences. A random frame from sequence lab2113 shown after relighting. Raw images are captured by setting light intensity to 50% $(k=0.5)$, these are then relit using the CLID network to generate images at light-intensities 0-100% at 10% increments (left to right)
• Figure 3: Lightning Framework: Our pipeline consists of three stages: (1) Data Expansion (Relighting): A Co-Located Illumination Decomposition (CLID) network takes a real observation at some light-intensity (50% for our experiments) and decomposes it into an ambient component $A$, a scalar light contribution map $S_F$, and a color vector $C_F$. This allows for the synthesis of a dense virtual dataset spanning the full range of candidate light intensities $\mathcal{K}$. (2) Expert Generation (Oracle): As illustrated in the trellis graph (Stage 2), we formulate active illumination as a global optimization problem. The optimization identifies the Optimal Intensity Schedule (OIS) by traversing a trellis structure where nodes represent discrete light levels and edges represent transitions. The path is determined by minimizing a sequence-level energy function comprising unary costs (balancing power consumption and image utility) and pairwise costs (balancing feature matching quality and temporal smoothness). (3) Imitation Policy (ILC): The offline Oracle is distilled into a real-time policy via behavior cloning. The resulting controller uses a visual encoder from CLID to process the current image $I_t$ and previous light state $k_{t-1}$ to predict the next optimal illumination intensity $k_t$ for online deployment.
• Figure 4: Our robot platform for data-collection and evaluation. Left, an illustration of our hardware including the stereo camera setup and custom-built co-located light and right shows our robot operating in a low-light environment
• Figure 5: ILC's intensity schedule versus fixed baselines. The imitation policy, deployed on a robot, outputs a per-frame light intensity (blue), and is compared against 0% (green-dashed) and 100% (red-dashed) fixed-intensity baselines. Insets show frames at the corresponding timestamps: (a) ILC increases illumination when entering a low-light region to preserve image utility; (b) ILC reduces illumination near a reflective whiteboard to mitigate specular saturation; (c) ILC chooses an intermediate illumination level to balance competing effects of low light and specular reflection.