Looking for a better fit? An Incremental Learning Multimodal Object Referencing Framework adapting to Individual Drivers

TL;DR

This paper tackles the problem of reliably referencing outside-the-vehicle objects under diverse driving conditions and driver behaviors. It introduces IcRegress, an incremental-learning regression framework that fuses multimodal cues—pointing, gaze, head pose, and speech—and adapts continuously to individual drivers and sensor availability. The authors propose an exemplar-based forgetting-aware strategy and validate it against baselines, introducing new metrics (SegObj, MRDE, MinDT) for outside-vehicle referencing in a driving simulation, while providing an open-source implementation. Overall, IcRegress demonstrates improved personalization and generalization for multimodal object referencing, with implications for safer and more natural human–vehicle interaction in real-world vehicles.

Abstract

The rapid advancement of the automotive industry towards automated and semi-automated vehicles has rendered traditional methods of vehicle interaction, such as touch-based and voice command systems, inadequate for a widening range of non-driving related tasks, such as referencing objects outside of the vehicle. Consequently, research has shifted toward gestural input (e.g., hand, gaze, and head pose gestures) as a more suitable mode of interaction during driving. However, due to the dynamic nature of driving and individual variation, there are significant differences in drivers' gestural input performance. While, in theory, this inherent variability could be moderated by substantial data-driven machine learning models, prevalent methodologies lean towards constrained, single-instance trained models for object referencing. These models show a limited capacity to continuously adapt to the divergent behaviors of individual drivers and the variety of driving scenarios. To address this, we propose \textit{IcRegress}, a novel regression-based incremental learning approach that adapts to changing behavior and the unique characteristics of drivers engaged in the dual task of driving and referencing objects. We suggest a more personalized and adaptable solution for multimodal gestural interfaces, employing continuous lifelong learning to enhance driver experience, safety, and convenience. Our approach was evaluated using an outside-the-vehicle object referencing use case, highlighting the superiority of the incremental learning models adapted over a single trained model across various driver traits such as handedness, driving experience, and numerous driving conditions. Finally, to facilitate reproducibility, ease deployment, and promote further research, we offer our approach as an open-source framework at \url{https://github.com/amrgomaaelhady/IcRegress}.

Figures (8)

• Figure 1: Setup overview showing our driving simulation with three 55-inch screens, steering wheels, and pedals. We simulate the car's left door with a plastic barrier beside the driving seat. Sensor cameras are attached to a top beam to simulate their location on the roof of modern vehicles.
• Figure 2: Buildings' clusters with the three possible lateral building offsets. The Left is the top view, and the right is the driver's view.
• Figure 3: A top view of 8-building and 16-building clusters.
• Figure 4: Cluster with target building visualized using ray casting in OpenDS simulator (top view and lateral view).
• Figure 5: Accuracy and error results comparing single and multiple modalities performance during the multimodal object referencing task.