Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VizFlyt: Perception-centric Pedagogical Framework For Autonomous Aerial Robots

Kushagra Srivastava, Rutwik Kulkarni, Manoj Velmurugan, Nitin J. Sanket

TL;DR

VizFlyt tackles the challenge of teaching perception-based autonomous aerial robotics by offering a hardware-agnostic, hardware-in-the-loop HITL framework that hallucinates photorealistic RGBD sensors from real-time pose using 3D Gaussian Splatting. The approach decouples coursework from fragile hardware, enabling repeatable testing at high update rates and supporting an open-source curriculum with practical experiments in racing windows, unknown gaps, dynamic windows, and high-speed obstacle avoidance. The paper details the digital-twin pipeline, HITL architecture, and a six-project course design, validated through real-world lab and field experiments and supported by autograding. The work promises reduced hardware costs, safer testing, and broader adoption of vision-based aerial autonomy education and research, with future extensions to adverse weather and additional sensors.

Abstract

Autonomous aerial robots are becoming commonplace in our lives. Hands-on aerial robotics courses are pivotal in training the next-generation workforce to meet the growing market demands. Such an efficient and compelling course depends on a reliable testbed. In this paper, we present VizFlyt, an open-source perception-centric Hardware-In-The-Loop (HITL) photorealistic testing framework for aerial robotics courses. We utilize pose from an external localization system to hallucinate real-time and photorealistic visual sensors using 3D Gaussian Splatting. This enables stress-free testing of autonomy algorithms on aerial robots without the risk of crashing into obstacles. We achieve over 100Hz of system update rate. Lastly, we build upon our past experiences of offering hands-on aerial robotics courses and propose a new open-source and open-hardware curriculum based on VizFlyt for the future. We test our framework on various course projects in real-world HITL experiments and present the results showing the efficacy of such a system and its large potential use cases. Code, datasets, hardware guides and demo videos are available at https://pear.wpi.edu/research/vizflyt.html

VizFlyt: Perception-centric Pedagogical Framework For Autonomous Aerial Robots

TL;DR

VizFlyt tackles the challenge of teaching perception-based autonomous aerial robotics by offering a hardware-agnostic, hardware-in-the-loop HITL framework that hallucinates photorealistic RGBD sensors from real-time pose using 3D Gaussian Splatting. The approach decouples coursework from fragile hardware, enabling repeatable testing at high update rates and supporting an open-source curriculum with practical experiments in racing windows, unknown gaps, dynamic windows, and high-speed obstacle avoidance. The paper details the digital-twin pipeline, HITL architecture, and a six-project course design, validated through real-world lab and field experiments and supported by autograding. The work promises reduced hardware costs, safer testing, and broader adoption of vision-based aerial autonomy education and research, with future extensions to adverse weather and additional sensors.

Abstract

Autonomous aerial robots are becoming commonplace in our lives. Hands-on aerial robotics courses are pivotal in training the next-generation workforce to meet the growing market demands. Such an efficient and compelling course depends on a reliable testbed. In this paper, we present VizFlyt, an open-source perception-centric Hardware-In-The-Loop (HITL) photorealistic testing framework for aerial robotics courses. We utilize pose from an external localization system to hallucinate real-time and photorealistic visual sensors using 3D Gaussian Splatting. This enables stress-free testing of autonomy algorithms on aerial robots without the risk of crashing into obstacles. We achieve over 100Hz of system update rate. Lastly, we build upon our past experiences of offering hands-on aerial robotics courses and propose a new open-source and open-hardware curriculum based on VizFlyt for the future. We test our framework on various course projects in real-world HITL experiments and present the results showing the efficacy of such a system and its large potential use cases. Code, datasets, hardware guides and demo videos are available at https://pear.wpi.edu/research/vizflyt.html

Paper Structure

This paper contains 26 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Learnings From Current Curricula
  3. Others' Aerial Robotics Coursework
  4. Our Latest Past Course Offering
  5. VizFlyt Framework
  6. Digital Twin Scene Generation
  7. Hardware Setup
  8. Collision Detection
  9. Framework Architecture
  10. Proposed Course Curriculum
  11. Learning Objectives
  12. Course Setup
  13. Course Curriculum
  14. Grading and Evaluation
  15. Logistics and Cost of Operation
  16. ...and 11 more sections

Figures (4)

  • Figure 2: Termination of attempt during a crash (yellow highlight). (a) Real crash in the old version of the course leading to damages and manual grading, (b) automatic crash reporting from the hallucinated camera with no robot damages.
  • Figure 3: (a) Artistic impression of the previous iteration of the course with experiments run with real obstacles and sensors, (b) Artistic impression of the proposed course with experiments run using the VizFlyt framework to hallucinate sensors for HITL testing, and (c) VizFlyt framework Architecture.
  • Figure 4: (a) Flight test for proposed project 3: flight through racing windows, (b) Flight test for proposed project 4: flight through unknown gap, (c) Proposed project 5's dynamic window and (d) High-speed flight experiment. The red highlights denoted as VizFlyt view show hallucinated cameras seen by the robot in (a) to (c). In (d), the entire scene is hallucinated with the real-robot overlaid.
  • Figure 5: Comparison of various visual odometry methods on (a) square, (b) spiral and (c) lemniscate trajectories.