Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Under-Canopy Terrain Reconstruction in Dense Forests Using RGB Imaging and Neural 3D Reconstruction

Refael Sheffer, Chen Pinchover, Haim Zisman, Dror Ozeri, Roee Litman

TL;DR

The paper tackles the challenge of reconstructing terrain beneath dense forest canopies using only RGB imagery. It introduces a Neural Radiance Field (NeRF) pipeline trained on RGB data, augmented with a low-light RAW loss and two canopy-removal strategies to produce ground-only renderings, enabling downstream tasks such as SAR-based person detection and tree-counting. The authors formalize image-capture guidelines, implement the pipeline with COLMAP and Instant-NGP, and demonstrate that RGB-only NeRF can achieve competitive ground reconstruction and useful analytics compared to specialized sensors like LiDAR and thermal AOS. While promising and cost-effective, the approach is currently sensitive to lighting and SfM accuracy, suggesting future work in multi-spectral fusion, online reconstruction, and improved acquisition planning.

Abstract

Mapping the terrain and understory hidden beneath dense forest canopies is of great interest for numerous applications such as search and rescue, trail mapping, forest inventory tasks, and more. Existing solutions rely on specialized sensors: either heavy, costly airborne LiDAR, or Airborne Optical Sectioning (AOS), which uses thermal synthetic aperture photography and is tailored for person detection. We introduce a novel approach for the reconstruction of canopy-free, photorealistic ground views using only conventional RGB images. Our solution is based on the celebrated Neural Radiance Fields (NeRF), a recent 3D reconstruction method. Additionally, we include specific image capture considerations, which dictate the needed illumination to successfully expose the scene beneath the canopy. To better cope with the poorly lit understory, we employ a low light loss. Finally, we propose two complementary approaches to remove occluding canopy elements by controlling per-ray integration procedure. To validate the value of our approach, we present two possible downstream tasks. For the task of search and rescue (SAR), we demonstrate that our method enables person detection which achieves promising results compared to thermal AOS (using only RGB images). Additionally, we show the potential of our approach for forest inventory tasks like tree counting. These results position our approach as a cost-effective, high-resolution alternative to specialized sensors for SAR, trail mapping, and forest-inventory tasks.

Under-Canopy Terrain Reconstruction in Dense Forests Using RGB Imaging and Neural 3D Reconstruction

TL;DR

The paper tackles the challenge of reconstructing terrain beneath dense forest canopies using only RGB imagery. It introduces a Neural Radiance Field (NeRF) pipeline trained on RGB data, augmented with a low-light RAW loss and two canopy-removal strategies to produce ground-only renderings, enabling downstream tasks such as SAR-based person detection and tree-counting. The authors formalize image-capture guidelines, implement the pipeline with COLMAP and Instant-NGP, and demonstrate that RGB-only NeRF can achieve competitive ground reconstruction and useful analytics compared to specialized sensors like LiDAR and thermal AOS. While promising and cost-effective, the approach is currently sensitive to lighting and SfM accuracy, suggesting future work in multi-spectral fusion, online reconstruction, and improved acquisition planning.

Abstract

Mapping the terrain and understory hidden beneath dense forest canopies is of great interest for numerous applications such as search and rescue, trail mapping, forest inventory tasks, and more. Existing solutions rely on specialized sensors: either heavy, costly airborne LiDAR, or Airborne Optical Sectioning (AOS), which uses thermal synthetic aperture photography and is tailored for person detection. We introduce a novel approach for the reconstruction of canopy-free, photorealistic ground views using only conventional RGB images. Our solution is based on the celebrated Neural Radiance Fields (NeRF), a recent 3D reconstruction method. Additionally, we include specific image capture considerations, which dictate the needed illumination to successfully expose the scene beneath the canopy. To better cope with the poorly lit understory, we employ a low light loss. Finally, we propose two complementary approaches to remove occluding canopy elements by controlling per-ray integration procedure. To validate the value of our approach, we present two possible downstream tasks. For the task of search and rescue (SAR), we demonstrate that our method enables person detection which achieves promising results compared to thermal AOS (using only RGB images). Additionally, we show the potential of our approach for forest inventory tasks like tree counting. These results position our approach as a cost-effective, high-resolution alternative to specialized sensors for SAR, trail mapping, and forest-inventory tasks.
Paper Structure (38 sections, 5 equations, 7 figures, 3 tables)

This paper contains 38 sections, 5 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Our contributions
  3. Related Work
  4. LiDAR based mapping.
  5. Thermal under canopy imaging.
  6. Aerial photogrammetry and MVS‐based reconstruction.
  7. Obstruction free photography.
  8. Differentiable volumetric rendering.
  9. Semantic image segmentation.
  10. Method
  11. Image Capture Protocol
  12. Viewing angle.
  13. Sampling density.
  14. Ground sample distance (GSD).
  15. Lighting conditions.
  16. ...and 23 more sections

Figures (7)

  • Figure 1: An example of the several under-canopy imaging methods discussed in this paper, on the 'F6' scene from the drone imagery dataset schedl2021airborne. Images are approximately aligned for easy comparison. See Section \ref{['sec:quality']} for more details.
  • Figure 2: Proposed method pipeline overview: (a) High overlap image capture; (b) bundle adjustment; (c) NeRF reconstruction; (d) ground only rendering revealing occluded terrain.
  • Figure 3: Effect of lighting conditions on the result on two sample views of the same location in different times, from schedl2021airborne. Direct sunlight (F11, top left) creates cast shadows, which might cause over exposed canopy and/or underexposed understory, as opposed to indirect lighting (F6, top right). This is also visible in the histogram (bottom) where even after equalization, a big portion of the dynamic range in direct lighting is spent on the lit area, and make the shadows more sensitive to quantization. See Figure \ref{['fig:loss']} for more results.
  • Figure 4: A comparison of two NeRF loss functions, L1 \ref{['eq:l1loss']} and RAW \ref{['eq:rawloss']},on two scenes from schedl2021airborne. The two images top are from the F11 scene, which includes direct lighting and high dynamic range, the L1 loss struggles with the cast shadows and shade boundaries. The two bottom images are from the F5 scene, where dynamic range is more balanced, and RAW loss better captures the most occluded person on the right part of the image, which can be partially seen in the L1 image. Additionally, other persons are slightly more blurred in the L1 image. See the anc/supp.pdf for results on more loss functions.
  • Figure 5: A Comparison of canopy removal methods. The two methods described in Section \ref{['sec:peal']}, are applied on scene F5 from schedl2021airborne. The full scene (before canopy removal) is in the top left, 3D crop bottom left, and 3D segmentation bottom right. In addition, the 3D color-based segmentation mask is rendered as grayscale image, where e.g. the trunks in the right side are excluded. We recommend the reader to view the anc/supp_Figure5.mp4 of this segmentation result. See Section \ref{['sec:peal']} for more details.
  • ...and 2 more figures