Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

nvTorchCam: An Open-source Library for Camera-Agnostic Differentiable Geometric Vision

Daniel Lichy, Hang Su, Abhishek Badki, Jan Kautz, Orazio Gallo

TL;DR

NvTorchCam abstracts critical camera operations such as projection and unprojection, allowing developers to implement algorithms once and apply them across diverse camera models--including pinhole, fisheye, and 360 equirectangular panoramas, which are commonly used in automotive and real estate capture applications.

Abstract

We introduce nvTorchCam, an open-source library under the Apache 2.0 license, designed to make deep learning algorithms camera model-independent. nvTorchCam abstracts critical camera operations such as projection and unprojection, allowing developers to implement algorithms once and apply them across diverse camera models--including pinhole, fisheye, and 360 equirectangular panoramas, which are commonly used in automotive and real estate capture applications. Built on PyTorch, nvTorchCam is fully differentiable and supports GPU acceleration and batching for efficient computation. Furthermore, deep learning models trained for one camera type can be directly transferred to other camera types without requiring additional modification. In this paper, we provide an overview of nvTorchCam, its functionality, and present various code examples and diagrams to demonstrate its usage. Source code and installation instructions can be found on the nvTorchCam GitHub page at https://github.com/NVlabs/nvTorchCam.

nvTorchCam: An Open-source Library for Camera-Agnostic Differentiable Geometric Vision

TL;DR

NvTorchCam abstracts critical camera operations such as projection and unprojection, allowing developers to implement algorithms once and apply them across diverse camera models--including pinhole, fisheye, and 360 equirectangular panoramas, which are commonly used in automotive and real estate capture applications.

Abstract

We introduce nvTorchCam, an open-source library under the Apache 2.0 license, designed to make deep learning algorithms camera model-independent. nvTorchCam abstracts critical camera operations such as projection and unprojection, allowing developers to implement algorithms once and apply them across diverse camera models--including pinhole, fisheye, and 360 equirectangular panoramas, which are commonly used in automotive and real estate capture applications. Built on PyTorch, nvTorchCam is fully differentiable and supports GPU acceleration and batching for efficient computation. Furthermore, deep learning models trained for one camera type can be directly transferred to other camera types without requiring additional modification. In this paper, we provide an overview of nvTorchCam, its functionality, and present various code examples and diagrams to demonstrate its usage. Source code and installation instructions can be found on the nvTorchCam GitHub page at https://github.com/NVlabs/nvTorchCam.

Paper Structure

This paper contains 40 sections, 50 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Prior Work
  3. Library Structure
  4. Preliminary Notation
  5. Doc-string Formatting and Inferred Batch Size
  6. Mathematical Notations
  7. Cameras
  8. Mathematical Model of Cameras
  9. Camera Models and Their Usage
  10. Supported Models
  11. Creating Cameras
  12. Essential Camera Functions
  13. Function: project_to_pixel
  14. Function: pixel_to_ray
  15. Tensor-like Properties
  16. ...and 25 more sections

Figures (6)

  • Figure 1: Shows an equirectangular panorama (ERP) being resampled to a cubemap, two pinhole images with different extrinsic rotations, and a fisheye projection.
  • Figure 2: Image coordinate systems
  • Figure 3: Example of the result of warping the source image at different distances from the camera that captured the reference image.
  • Figure 4: Illustration for Subsection \ref{['subsec:consistency_checking']}
  • Figure 5: Mutual visibility masks calculated by backward-forward warping. (c) is white at locations in (a) that are visible in (b). (d) is white at locations in (b) that are visible in (a)
  • ...and 1 more figures