SketchPlan: Diffusion Based Drone Planning From Human Sketches

TL;DR

SketchPlan introduces a diffusion-based planner that converts hand-drawn sketches over depth images into 3D drone trajectories, enabling intuitive, real-time navigation with zero-shot sim-to-real transfer. The architecture decouples sketch interpretation (SketchAdapter) from 3D path inference (DiffPath) and trains on a large auto-labeled 3DGS dataset with a small set of human sketches, achieving robust performance in unseen environments. A collision-filtering step ensures safety, and hardware experiments show successful navigation across easy and medium clutter, with partial success in hard clutter due to challenging openings. Overall, the method demonstrates that combining sketch-conditioned diffusion with partial labeling and synthetic environments can close the sim-to-real gap for complex UAV planning tasks.

Abstract

We propose SketchPlan, a diffusion-based planner that interprets 2D hand-drawn sketches over depth images to generate 3D flight paths for drone navigation. SketchPlan comprises two components: a SketchAdapter that learns to map the human sketches to projected 2D paths, and DiffPath, a diffusion model that infers 3D trajectories from 2D projections and a first person view depth image. Our model achieves zero-shot sim-to-real transfer, generating accurate and safe flight paths in previously unseen real-world environments. To train the model, we build a synthetic dataset of 32k flight paths using a diverse set of photorealistic 3D Gaussian Splatting scenes. We automatically label the data by computing 2D projections of the 3D flight paths onto the camera plane, and use this to train the DiffPath diffusion model. However, since real human 2D sketches differ significantly from ideal 2D projections, we additionally label 872 of the 3D flight paths with real human sketches and use this to train the SketchAdapter to infer the 2D projection from the human sketch. We demonstrate SketchPlan's effectiveness in both simulated and real-world experiments, and show through ablations that training on a mix of human labeled and auto-labeled data together with a modular design significantly boosts its capabilities to correctly interpret human intent and infer 3D paths. In real-world drone tests, SketchPlan achieved 100\% success in low/medium clutter and 40\% in unseen high-clutter environments, outperforming key ablations by 20-60\% in task completion.

Figures (6)

• Figure 1: SketchPlan Architecture. SketchAdapter takes a 2D human sketch and predicts the intended 2D path projection, trained with a small set of human-labeled data. DiffPath infers a 3D path from its 2D projection and an FPV depth image, trained with a large volume of auto-labeled 3D paths from diverse 3DGS scenes. Dashed arrows indicate loss computation for training.
• Figure 2: Comparison of human sketch and projection (left) for a 3D path (right). Note that the camera projection of the 3D path differs significantly from the human sketch. The projection maps the straight part of the path to a tight pixel cluster in the middle of the image while the human instead sketches a straight vertical line.
• Figure 3: DiffPath Encoder. Our encoder combines the depth image and 2D projection into a compact embedding $z^{(j)}$ by sampling patches along the projection on a granular feature map. Convolutional kernels denoted as $K\times K \times C /S$, where $K$ is the kernel size, $C$ the channels and $S$ the stride.
• Figure 4: Left: Sample Sketches -- Right: Resulting 3D Paths. Note that more curvature in the sketch results in greater side translation in the generated 3D path.
• Figure 5: Rollout of the same 2D sketch in two different poses. Note how the curvature of the 3D path (bottom) changes depending on the background scene (top). Arrows in bottom panels denote path direction.