Optimal OnTheFly Feedback Control of Event Sensors

TL;DR

This work tackles video reconstruction from event-based vision sensors by introducing On-The-Fly control of per-column activation thresholds $\Delta$, enabling data-dependent, dynamic sensing that balances reconstruction quality and event rate. A sensor control network and a reconstruction network are trained end-to-end using a physically accurate EVS simulator, employing a probabilistic relaxation with the Gumbel-Softmax to backpropagate through stochastic threshold masks and a fused multi-threshold event representation. Empirical results show 6–12% LPIPS improvements and up to 49% event-rate reductions compared with fixed or random threshold regimes, with interpretable threshold patterns that adapt to scene dynamics. The approach paves the way for smarter EVS acquisition and potential silicon hardware implementations, while highlighting future work to extend the control to additional EVS parameters and joint spatial bias control.

Abstract

Event-based vision sensors produce an asynchronous stream of events which are triggered when the pixel intensity variation exceeds a predefined threshold. Such sensors offer significant advantages, including reduced data redundancy, micro-second temporal resolution, and low power consumption, making them valuable for applications in robotics and computer vision. In this work, we consider the problem of video reconstruction from events, and propose an approach for dynamic feedback control of activation thresholds, in which a controller network analyzes the past emitted events and predicts the optimal distribution of activation thresholds for the following time segment. Additionally, we allow a user-defined target peak-event-rate for which the control network is conditioned and optimized to predict per-column activation thresholds that would eventually produce the best possible video reconstruction. The proposed OnTheFly control scheme is data-driven and trained in an end-to-end fashion using probabilistic relaxation of the discrete event representation. We demonstrate that our approach outperforms both fixed and randomly-varying threshold schemes by 6-12% in terms of LPIPS perceptual image dissimilarity metric, and by 49% in terms of event rate, achieving superior reconstruction quality while enabling a fine-tuned balance between performance accuracy and the event rate. Additionally, we show that sampling strategies provided by our OnTheFly control are interpretable and reflect the characteristics of the scene. Our results, derived from a physically-accurate simulator, underline the promise of the proposed methodology in enhancing the utility of event cameras for image reconstruction and other downstream tasks, paving the way for hardware implementation of dynamic feedback EVS control in silicon.

Figures (5)

• Figure 1: (a) Illustration of image reconstruction from events with spatiotemporally-variable parameters providing complimentary incoherence in the data, allowing for an optimal combination. (b) Data-driven acquisition optimization, illustrated for EVS. A ground-truth image $\mathbf{I}$ is passed to the simulator "Simul", yielding discretized events $\mathbf{D}$, which are then mapped back by a reconstruction network $\mathcal{V}_\theta$ to match the initial image. The control network $\mathcal{C}_\psi$ adjusts the sensor acquisition parameters to achieve the best possible image reconstruction accuracy. Reconstruction and control network weights are optimized jointly at the training stage to minimize the target image dissimilarity metric $\mathcal{L}$. During inference, the simulator can be swapped with a real-camera setup.
• Figure 2: Exemplary ground truth image $\mathbf{L}$ and corresponding discretized events $\mathbf{D}^\Delta$ for different thresholds $\Delta$. Reconstructions are given for the globally fixed threshold value $\Delta$. A histogram of effective threshold values are given for 5 target threshold values: 1.15, 1.25, 1.4, 1.7, 2.2. Mean event rate decreases with target activation threshold increase.
• Figure 3: Illustration of the simulation-reconstruction-control feedback loop during training. In this example $N_c$=3. Recurrent states are indicated with yellow arrows.
• Figure 4: Reconstruction accuracy vs. event rate trade-off evaluation for the OnTheFly control. a) LPIPS vs. event rate trade-off for the OnTheFly, Random and fixed globally constant threshold control regimes. Random control is achieved by drawing the distribution of thresholds from the Dirichlet distribution and then sampling threshold configurations from it. b) Mean image reconstruction quality improvement by OnTheFly control compared to the constant threshold configuration as a function of the event rate averaged over validation data. To compare different sequences, we assume the smallest value of $\Delta$ to achieve 100% event rate. The corresponding LPIPS values are then interpolated via cubic Hermite polynomials. c) Exemplary reconstructed images together with corresponding discretized events for the constant activation threshold ($\Delta$=1.7) configuration and OnTheFly control ($\lambda$=1.8) with matching average event rates.
• Figure 5: Fixed target event rate control. a) Mean event rate for the control strategies, fixed global activation thresholds $\Delta$ and the target event rate. Note that the fixed event rate control (red) is controlled not to exceed the target event rate. b) Corresponding $\lambda$ values for fixed event rate and fixed $\lambda$ control. c,d) Activation threshold $\Delta$ distribution per column over time for two OnTheFly control strategies. e,f) discretized event frames and corresponding reconstructions for the fixed event rate control at time points indicated with dashed cyan lines. Green dashed line indicates the middle column of the imaging sensor in $X$-Time and $X$-$Y$ panels.