Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Opening the Black Box of 3D Reconstruction Error Analysis with VECTOR

Racquel Fygenson, Kazi Jawad, Isabel Li, Francois Ayoub, Robert G. Deen, Scott Davidoff, Dominik Moritz, Mauricio Hess-Flores

TL;DR

The paper addresses the lack of interpretable error insight in stereo reconstruction BA by introducing VECTOR, a visual analytics tool that reveals pre- and post-BA relationships among feature tracks, camera pose, and 3D points. VECTOR combines four coordinated panels (Scene, Image Grid, Image, Track) with a Filtering Panel to enable error-focused debugging and iterative refinement, demonstrated on NASA JPL’s Mars Perseverance and Ingenuity data. The authors outline a user-centered workflow, key data formats, and concrete use cases (e.g., Flight 26), showing that analysts can rapidly identify and remove high-error tracks and improve BA convergence. The approach has practical impact for high-precision terrain reconstruction in science and space exploration, with open-source availability and plans to scale and generalize to other stereo-vision domains.

Abstract

Reconstruction of 3D scenes from 2D images is a technical challenge that impacts domains from Earth and planetary sciences and space exploration to augmented and virtual reality. Typically, reconstruction algorithms first identify common features across images and then minimize reconstruction errors after estimating the shape of the terrain. This bundle adjustment (BA) step optimizes around a single, simplifying scalar value that obfuscates many possible causes of reconstruction errors (e.g., initial estimate of the position and orientation of the camera, lighting conditions, ease of feature detection in the terrain). Reconstruction errors can lead to inaccurate scientific inferences or endanger a spacecraft exploring a remote environment. To address this challenge, we present VECTOR, a visual analysis tool that improves error inspection for stereo reconstruction BA. VECTOR provides analysts with previously unavailable visibility into feature locations, camera pose, and computed 3D points. VECTOR was developed in partnership with the Perseverance Mars Rover and Ingenuity Mars Helicopter terrain reconstruction team at the NASA Jet Propulsion Laboratory. We report on how this tool was used to debug and improve terrain reconstruction for the Mars 2020 mission.

Opening the Black Box of 3D Reconstruction Error Analysis with VECTOR

TL;DR

The paper addresses the lack of interpretable error insight in stereo reconstruction BA by introducing VECTOR, a visual analytics tool that reveals pre- and post-BA relationships among feature tracks, camera pose, and 3D points. VECTOR combines four coordinated panels (Scene, Image Grid, Image, Track) with a Filtering Panel to enable error-focused debugging and iterative refinement, demonstrated on NASA JPL’s Mars Perseverance and Ingenuity data. The authors outline a user-centered workflow, key data formats, and concrete use cases (e.g., Flight 26), showing that analysts can rapidly identify and remove high-error tracks and improve BA convergence. The approach has practical impact for high-precision terrain reconstruction in science and space exploration, with open-source availability and plans to scale and generalize to other stereo-vision domains.

Abstract

Reconstruction of 3D scenes from 2D images is a technical challenge that impacts domains from Earth and planetary sciences and space exploration to augmented and virtual reality. Typically, reconstruction algorithms first identify common features across images and then minimize reconstruction errors after estimating the shape of the terrain. This bundle adjustment (BA) step optimizes around a single, simplifying scalar value that obfuscates many possible causes of reconstruction errors (e.g., initial estimate of the position and orientation of the camera, lighting conditions, ease of feature detection in the terrain). Reconstruction errors can lead to inaccurate scientific inferences or endanger a spacecraft exploring a remote environment. To address this challenge, we present VECTOR, a visual analysis tool that improves error inspection for stereo reconstruction BA. VECTOR provides analysts with previously unavailable visibility into feature locations, camera pose, and computed 3D points. VECTOR was developed in partnership with the Perseverance Mars Rover and Ingenuity Mars Helicopter terrain reconstruction team at the NASA Jet Propulsion Laboratory. We report on how this tool was used to debug and improve terrain reconstruction for the Mars 2020 mission.
Paper Structure (14 sections, 3 figures)

This paper contains 14 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Understanding Machine Learning Output
  4. Error Analysis for Stereo Reconstruction Algorithms
  5. How Users Find Errors with VECTOR
  6. User Goals
  7. Data
  8. VECTOR Panels
  9. Scene View
  10. Image Grid View
  11. Image View
  12. Track View
  13. Filtering Panel
  14. Conclusions and next steps

Figures (3)

  • Figure 1: The stereo scene reconstruction process. Starting with a set of 2D images for stereo reconstruction (step 1), the scene reconstruction algorithm computes tiepoints and tracks, triangulates their 3D position and back-projects them into the original images creating residuals (step 2). Then, the BA algorithm optimizes camera poses and 3D structure to minimize residuals (step 3), outputting optimized residuals in each image along with descriptive statistics in XML (step 4). Analysts then manually remove erroneous tracks (e.g., top-left of step 5) and camera poses (e.g., bottom-right of step 5), and re-run the process.
  • Figure 2: Top: VECTOR panels which are used in tandem to detect and eliminate erroneous feature tracks and camera poses that adversely affect BA in stereo reconstruction. Bottom: example visualizations populating the shown panels.
  • Figure 3: Annotated example visualizations from Flight 26. (a) An example camera (circled) does not align with others. It could be removed to improve BA accuracy. (b) Ideally, residuals are short and uniformly distributed in angle. (c) Ideally, residuals decrease when a new BA is applied. (d) Inaccurate computed terrain points which should lie on the plane.