Beyond the Field-of-View: Enhancing Scene Visibility and Perception with Clip-Recurrent Transformer

TL;DR

This paper proposes the concept of online video inpainting for autonomous vehicles to expand the field of view, thereby enhancing scene visibility, perception, and system safety and introduces the FlowLens architecture, which explicitly employs optical flow and implicitly incorporates a novel clip-recurrent transformer for feature propagation.

Abstract

Vision sensors are widely applied in vehicles, robots, and roadside infrastructure. However, due to limitations in hardware cost and system size, camera Field-of-View (FoV) is often restricted and may not provide sufficient coverage. Nevertheless, from a spatiotemporal perspective, it is possible to obtain information beyond the camera's physical FoV from past video streams. In this paper, we propose the concept of online video inpainting for autonomous vehicles to expand the field of view, thereby enhancing scene visibility, perception, and system safety. To achieve this, we introduce the FlowLens architecture, which explicitly employs optical flow and implicitly incorporates a novel clip-recurrent transformer for feature propagation. FlowLens offers two key features: 1) FlowLens includes a newly designed Clip-Recurrent Hub with 3D-Decoupled Cross Attention (DDCA) to progressively process global information accumulated over time. 2) It integrates a multi-branch Mix Fusion Feed Forward Network (MixF3N) to enhance the precise spatial flow of local features. To facilitate training and evaluation, we derive the KITTI360 dataset with various FoV mask, which covers both outer- and inner FoV expansion scenarios. We also conduct both quantitative assessments and qualitative comparisons of beyond-FoV semantics and beyond-FoV object detection across different models. We illustrate that employing FlowLens to reconstruct unseen scenes even enhances perception within the field of view by providing reliable semantic context. Extensive experiments and user studies involving offline and online video inpainting, as well as beyond-FoV perception tasks, demonstrate that FlowLens achieves state-of-the-art performance. The source code and dataset are made publicly available at https://github.com/MasterHow/FlowLens.

Figures (12)

• Figure 1: High-level logic comparison between FlowLens' Clip-Recurrent Transformer and the recent Video Inpainting Transformer (VI-Trans) li2022towardsliu2021fuseformer. FlowLens provides immediate online output using only past streams and current clips. In contrast, VI-Trans generates delayed outputs dependent on future iterations. FlowLens' instant output capability makes it a more suitable choice for potential real-time applications, such as intelligent vehicle systems.
• Figure 2: Illustrations of our proposed FlowLens. (a) An overview. From left to right, it consists of 1) a convolution stem to extract shallow features, 2) an explicit flow-guided feature propagation module, 3) a clip-recurrent transformer to implicitly propagate features and fuse past information stream, 4) output convolution layers to restore the completed frames. (b)-(c) Our proposed Mix Focal Transformer Block and Mix Fusion Feed Forward Network (MixF3N).
• Figure 3: The histogram of the movement for the KITTI360. The distribution of pixel displacement exhibits a long-tailed feature and contains large motion, especially for the pinhole camera.
• Figure 4: 3D-Decoupled Cross Attention. By decoupling the dimensions of time, width, and height, we are able to efficiently query the most correlated features from the past. With an additional non-local strip pooling window, the information flows flexibly in intersecting directions during spatial query.
• Figure 5: Qualitative comparison on KITTI360 outwards beyond-FoV scene estimation with LaMa suvorov2022resolution, FuseFormer liu2021fuseformer, and E2FGVI li2022towards.