Discovering Closed-Loop Failures of Vision-Based Controllers via Reachability Analysis

TL;DR

The paper tackles safety verification for vision-based controllers in robotics by blending simulation-based sampling with Hamilton-Jacobi reachability to compute an approximate Backward Reachable Tube (BRT). This hybrid approach enables tractable identification of unsafe visual inputs without requiring an explicit analytic sensor model, addressing high-dimensional perception challenges. It contributes a general framework and demonstrates its effectiveness on two case studies—TaxiNet (3D) and indoor navigation (5D)—revealing concrete failure modes such as runway-marking confusion and obstacle occlusions. The work advances systematic closed-loop safety analysis for perception-driven controllers and suggests practical pathways to improve robustness via targeted data collection and training.

Abstract

Machine learning driven image-based controllers allow robotic systems to take intelligent actions based on the visual feedback from their environment. Understanding when these controllers might lead to system safety violations is important for their integration in safety-critical applications and engineering corrective safety measures for the system. Existing methods leverage simulation-based testing (or falsification) to find the failures of vision-based controllers, i.e., the visual inputs that lead to closed-loop safety violations. However, these techniques do not scale well to the scenarios involving high-dimensional and complex visual inputs, such as RGB images. In this work, we cast the problem of finding closed-loop vision failures as a Hamilton-Jacobi (HJ) reachability problem. Our approach blends simulation-based analysis with HJ reachability methods to compute an approximation of the backward reachable tube (BRT) of the system, i.e., the set of unsafe states for the system under vision-based controllers. Utilizing the BRT, we can tractably and systematically find the system states and corresponding visual inputs that lead to closed-loop failures. These visual inputs can be subsequently analyzed to find the input characteristics that might have caused the failure. Besides its scalability to high-dimensional visual inputs, an explicit computation of BRT allows the proposed approach to capture non-trivial system failures that are difficult to expose via random simulations. We demonstrate our framework on two case studies involving an RGB image-based neural network controller for (a) autonomous indoor navigation, and (b) autonomous aircraft taxiing.

Figures (11)

• Figure 1: Closed-loop BRTs of the ideal system (the grey area) and the actual system (the striped red area) during the morning when the aircraft starts at $p_y$(a) 110m, (b) 160m, and (c) 210m. The vision-based controller leads to particularly unsafe behaviors near the boundary of the runway.
• Figure 2: (a) The autonomous aircraft taxiing example. $p_x$, $p_y$, $\theta$ denote the state of the aircraft; the FoV of the camera is shown with dashed-white lines. (b) Heatmap showing the error in prediction over $p_x\text{ and }\theta$ for $p_y=160m$. Asymmetric images (c), (d) seen by the CNN at symmetric locations about the centerline.
• Figure 3: (a) Top-view of the runway in the morning. The trajectory followed by the aircraft under the CNN policy (red line) takes it off the runway. The successful trajectory (in green) takes the aircraft from "A" to "C", on adding the patch over the runway marking during ablation. The trajectory (in cyan) from "A" to "D" is followed at night. (b) The actual image that the CNN sees at "A" (yellow star in Fig. 3(a)). The CNN confuses the runway marking as the centreline. (c) Modified image with an artificial patch over the runway marking.
• Figure 4: (a) The morning (red shaded), and the night (blue shaded) BRTs overlaid for a $p_y$ of 110m. The state of interest, shown with a yellow star, is only contained in the morning BRT and not in the night BRT. (b) The absolute difference in $\hat{p_x}$ and the ground truth $p_x$ vs. the different blends of the runway marking at the state of interest (a lower value is better). (c) The images (at the yellow star in (a)) corresponding to different blends. The right-most image has a blend of 1, which is the unmodified image that the aircraft observes at night. The left-most image with a blend of 0 is obtained by manually cropping the runway marking and replacing it by the patch from the image observed in the morning. Any intermediate image is an interpolation between these two according to the blend value.
• Figure 5: (a) The morning (red shaded) and night (blue shaded) BRTs overlaid for $p_y$ = 190m. The state, shown with a yellow star, is only included in the night BRT. (b) Top view of the runway. In the morning, the CNN policy accomplishes the taxiing task by taking the red trajectory from "A" (yellow star in (a)) to "C." At night, the policy takes the aircraft outside the runway along the blue trajectory from "A" to "B". (c) The centreline in the image cannot be vividly seen by the CNN at location "A" at night due to poor illumination, whereas it can be seen clearly in the morning (d).