TaCOS: Task-Specific Camera Optimization with Simulation

TL;DR

This work presents a novel end-to-end optimization approach that co-designs cameras with spe-cific vision tasks and believes this approach can advance the fully automated de-sign of cameras.

Abstract

The performance of perception tasks is heavily influenced by imaging systems. However, designing cameras with high task performance is costly, requiring extensive camera knowledge and experimentation with physical hardware. Additionally, cameras and perception tasks are mostly designed in isolation, whereas recent methods that jointly design cameras and tasks have shown improved performance. Therefore, we present a novel end-to-end optimization approach that co-designs cameras with specific vision tasks. This method combines derivative-free and gradient-based optimizers to support both continuous and discrete camera parameters within manufacturing constraints. We leverage recent computer graphics techniques and physical camera characteristics to simulate the cameras in virtual environments, making the design process cost-effective. We validate our simulations against physical cameras and provide a procedurally generated virtual environment. Our experiments demonstrate that our method designs cameras that outperform common off-the-shelf options, and more efficiently compared to the state-of-the-art approach, requiring only 2 minutes to design a camera on an example experiment compared with 67 minutes for the competing method. Designed to support the development of cameras under manufacturing constraints, multiple cameras, and unconventional cameras, we believe this approach can advance the fully automated design of cameras.

Figures (16)

• Figure 1: Our method combines a derivative-free optimizer and gradient-based optimizer to co-design the camera with perception tasks in simulation, which utilizes ray tracing and a physics-based noise model. Our approach supports optimising discrete and continuous camera parameters for manufacture constraints and the generalization to other camera design problems.
• Figure 2: We establish a virtual environment and capture scene renders using a ray-traced scene capture camera. We then add physics-based, sensor-specific noise to the renders and input them into perception tasks for evaluation. In our optimization process, we jointly optimize the camera parameters $\Phi_{camera}$ using a fitness function $F$ with a derivative-free optimizer (blue arrow), as well as the parameters of perception tasks $\Phi_{model}$ (if trainable) on their corresponding loss function $l_{perception}$ with gradient-based optimizers (red arrow).
• Figure 3: Comparison of captured and synthetic images in terms of (a) variance in pixel intensities and (b) perception task performance. In (a), despite differences in color intensities due to manufacturing variations of the test target, the variances of pixel values in synthetic images match those in captured images, validating the accuracy of our noise model. In (b), the ranking of camera performance in our simulation aligns with physical cameras, and the differences in their performance between captured and synthetic images are consistent.
• Figure 4: Training curves comparing the design of cameras using our method with and without joint optimization, and curve of DISeR klinghoffer2023diser, are plotted. Zoomed-in windows from 0 to 50 and 950 to 1000 timesteps are provided for visualization. The curves demonstrate that our method with joint optimization achieves superior task performance with fewer steps and smoother behaviour.
• Figure 5: The vertical (a) and horizontal (b) FOVs of cameras optimized by our method with joint optimization, alongside those designed by humans under the daytime scenario, and example evaluation (c). Our method designs a camera with the largest FOV, enabling the capture of all obstacles and objects, extracting more features while maintaining sufficient resolution for effective object and feature detection.