Human-In-The-Loop Machine Learning for Safe and Ethical Autonomous Vehicles: Principles, Challenges, and Opportunities

TL;DR

Towards safe and ethical autonomy, a review of HITL-ML for AVs is presented, focusing on Curriculum Learning (CL), Human-In-The-Loop Reinforcement Learning (HITL-RL), Active Learning (AL), and ethical principles.

Abstract

Rapid advances in Machine Learning (ML) have triggered new trends in Autonomous Vehicles (AVs). ML algorithms play a crucial role in interpreting sensor data, predicting potential hazards, and optimizing navigation strategies. However, achieving full autonomy in cluttered and complex situations, such as intricate intersections, diverse sceneries, varied trajectories, and complex missions, is still challenging, and the cost of data labeling remains a significant bottleneck. The adaptability and robustness of humans in complex scenarios motivate the inclusion of humans in the ML process, leveraging their creativity, ethical power, and emotional intelligence to improve ML effectiveness. The scientific community knows this approach as Human-In-The-Loop Machine Learning (HITL-ML). Towards safe and ethical autonomy, we present a review of HITL-ML for AVs, focusing on Curriculum Learning (CL), Human-In-The-Loop Reinforcement Learning (HITL-RL), Active Learning (AL), and ethical principles. In CL, human experts systematically train ML models by starting with simple tasks and gradually progressing to more difficult ones. HITL-RL significantly enhances the RL process by incorporating human input through techniques like reward shaping, action injection, and interactive learning. AL streamlines the annotation process by targeting specific instances that need to be labeled with human oversight, reducing the overall time and cost associated with training. Ethical principles must be embedded in AVs to align their behavior with societal values and norms. In addition, we provide insights and specify future research directions.

Figures (4)

• Figure 1: The workflow of HITL-ML for AVs includes several key steps: 1) developing an ML model for decision-making and predictive modeling using PPO, 2) applying human validation and annotation through HITL-RL and AL to improve safety and foster public trust, 3) conducting model training where data is organized and refined with CL to enhance learning efficiency and performance, and 4) real-world deployment while considering ethical implications.
• Figure 2: With CL for AVs tasks are divided into smaller, manageable parts, progressing from easy to more difficult challenges. Data associated with each part is sequentially fed to the model to enhance learning efficiency and improve overall performance.
• Figure 3: The actor-critic setup of HITL-RL for AVs allows human experts to inject their actions and guide the agent in critical situations, ensuring that the AV navigates safely and effectively despite unexpected challenges. Additionally, the environmental reward is supplemented with human-defined rewards to guide the learning process. Both the environmental reward and the human reward are fed to the critic.
• Figure 4: AVs face unknown and unexpected scenarios and utilize AL to annotate them. The bottom right shows the AL loop comprising the steps of training, object selection, annotating, and appending. In the annotation step, human experts annotate the selected examples. AL impacts on unseen situations and training are shown in the top right.