Sage Deer: A Super-Aligned Driving Generalist Is Your Copilot

TL;DR

Sage Deer tackles the need for personalized, safe, and capable driving copilots by integrating multi-view and multi-modal inputs with a super-aligned, generalist LLM framework. It combines a learnable retrieval-augmented generation (RAG) module to tailor responses to individual user preferences and a Continuous Latent Chain Eliciting (CLCE) mechanism to activate latent reasoning without explicit chain-of-thought labels. The approach is validated on driving-focused, multi-task datasets (AIDE and DMD), where Sage Deer outperforms strong baselines in generalist tasks and demonstrates robust super-alignment through user-specific retrievals. The work also introduces a data-curation and benchmarking pipeline for driving copilots, highlighting practical implications for personalized, context-aware vehicle cockpits.

Abstract

The intelligent driving cockpit, an important part of intelligent driving, needs to match different users' comfort, interaction, and safety needs. This paper aims to build a Super-Aligned and GEneralist DRiving agent, SAGE DeeR. Sage Deer achieves three highlights: (1) Super alignment: It achieves different reactions according to different people's preferences and biases. (2) Generalist: It can understand the multi-view and multi-mode inputs to reason the user's physiological indicators, facial emotions, hand movements, body movements, driving scenarios, and behavioral decisions. (3) Self-Eliciting: It can elicit implicit thought chains in the language space to further increase generalist and super-aligned abilities. Besides, we collected multiple data sets and built a large-scale benchmark. This benchmark measures the deer's perceptual decision-making ability and the super alignment's accuracy.

Figures (8)

• Figure 1: The Capability of Sage Deer. (a) Sage Deer can perform corresponding actions according to the preferences of each user. Users can save their preferences in a document and update them anytime. (b) Sage Deer can integrate multi-view and multi-model inputs for driver and scene understanding. Notably, Sage Deer actually has both super-aligned and generalist capabilities at the same time.
• Figure 2: Data construction process of Sage Deer. (1) We use existent GPT-4o tools to generate captions for the videos likes li2023videochatdamonlpsg2023videollama. Then, we set a reasonable prompt to merge the information from different videos. (2) We took advantage of the existing labels (including physiological indicators, emotional indicators, action indicators, behavioral indicators, scene understanding, and reasoning decision-making) to correct and supplement the captain. (3) Next, we use GPT4 achiam2023gpt as an assistant to build question-answering pairs for different tasks (including physiological indicators, emotion, behavior, and so on.). (4) We design multiple user preferences, and GPT4 responds to the current scenario based on user preferences. All of the above processes are double-checked manually.
• Figure 3: Sage Deer uses pre-trained video encoders (visual tokenizers) to tokenize the different modes and views of the video, especially the physiological encoder used to extract the physiological signals of the face video. Using tokenized visual embedding, large language models can provide general responses for physiological indicators, emotional states, gestures, human behavior, and scene understanding. It is worth emphasizing that the individual needs of the user can be edited in a single document. We then query this information with learnable retrieval augmentation generation, generating results super aligned with user preferences.
• Figure 4: A comparison of our continuous latent chain eliciting with the existent chain-of-thought method. (a) The traditional CoT model generates the reasoning process with the supervision of COT tokens. (b) Coconut uses the LLM to reason in an unrestricted latent space instead of a language space hao2024training. But he still distills knowledge from the COT label. (c) Our method tries to activate the implicit language space so that the model learns the implicit COT from itself.
• Figure 5: The trivial solutions of the CL-COT embedding $E_{cot}$. (a) Compared with normal embedding, COT embedding tends to be self-similar. (b) Compared with normal embedding, COT embedding tends to have a smaller range of values. These two phenomena indicate that the COT embedding tends to be invalid.