TBD Pedestrian Data Collection: Towards Rich, Portable, and Large-Scale Natural Pedestrian Data

TL;DR

This work describes a portable data collection system, coupled with a semi-autonomous labeling pipeline that enables large-scale data collection in diverse environments and fast trajectory label production, and designs a label correction web application that facilitates human verification of automated pedestrian tracking outcomes.

Abstract

Social navigation and pedestrian behavior research has shifted towards machine learning-based methods and converged on the topic of modeling inter-pedestrian interactions and pedestrian-robot interactions. For this, large-scale datasets that contain rich information are needed. We describe a portable data collection system, coupled with a semi-autonomous labeling pipeline. As part of the pipeline, we designed a label correction web app that facilitates human verification of automated pedestrian tracking outcomes. Our system enables large-scale data collection in diverse environments and fast trajectory label production. Compared with existing pedestrian data collection methods, our system contains three components: a combination of top-down and ego-centric views, natural human behavior in the presence of a socially appropriate "robot", and human-verified labels grounded in the metric space. To the best of our knowledge, no prior data collection system has a combination of all three components. We further introduce our ever-expanding dataset from the ongoing data collection effort -- the TBD Pedestrian Dataset and show that our collected data is larger in scale, contains richer information when compared to prior datasets with human-verified labels, and supports new research opportunities.

Figures (5)

• Figure 1: This set of images represent the same moment recorded from multiple sensors: a) Top-down view image taken by a static camera with grounded pedestrian trajectory labels shown. b) Ego-centric point cloud from a 3D LiDAR with the projected trajectories from (a). c) Ego-centric RBG and depth images from the mounted stereo camera. Green vertical bars represent the projected labels. Note that two pedestrians at the back are partially and completely occluded from the stereo camera.
• Figure 2: Sensor setup used to collect the TBD Pedestrian Dataset. (left) One of the nodes used to capture top-down RGB views. (middle) The cart used to capture ego-centric sensor views during data collection for Set 1. (right) The suitcase robot used to capture ego-centric sensor views during data collection for Set 2.
• Figure 3: Hardware setup for the TBD Pedestrian Dataset. Blue circles indicate positions of RGB cameras. Green box shows our suitcase robot pushed through the scene. The white area is where trajectory labels are collected.
• Figure 4: Application interface for the human verification process. It contains a media player and various options to fix tracking errors automatically and manually.
• Figure 5: Examples from the TBD Set 1. a) a dynamic group. b) a static conversational group. c) a tour group with 14 pedestrians. d) a pedestrian affecting other pedestrians by asking them to come to the table. e) pedestrians stop and look at their phones. f) two pedestrians change their navigation goals and turn toward the table. g) a group of pedestrians change their navigation goals multiple times. h) a crowded scene where pedestrians are heading in different directions.