INSIGHT: INference-time Sequence Introspection for Generating Help Triggers in Vision-Language-Action Models

TL;DR

INSIGHT introduces an introspective framework for Vision-Language-Action models that leverages token-level uncertainty signals to decide when a robot should request human help. By extracting entropy, negative log-probability, and Dirichlet-based aleatoric and epistemic uncertainties from token distributions produced by a π0-FAST autoregressive policy, and training a compact transformer to predict help triggers, INSIGHT demonstrates that temporal modeling of uncertainty outperforms static sequence-level scores. The work systematically compares strong (step-level) and weak (episode-level) supervision across in-distribution and out-of-distribution settings, showing a clear trade-off between labeling effort and predictive fidelity, with strong supervision providing the most reliable performance and weak supervision offering scalability. Across multiple scenarios, INSIGHT achieves meaningful improvements over conformal-prediction baselines, enabling timely, uncertainty-guided human intervention and opening avenues for active learning and lifelong improvement in embodied AI systems.

Abstract

Recent Vision-Language-Action (VLA) models show strong generalization capabilities, yet they lack introspective mechanisms for anticipating failures and requesting help from a human supervisor. We present \textbf{INSIGHT}, a learning framework for leveraging token-level uncertainty signals to predict when a VLA should request help. Using $π_0$-FAST as the underlying model, we extract per-token \emph{entropy}, \emph{log-probability}, and Dirichlet-based estimates of \emph{aleatoric and epistemic uncertainty}, and train compact transformer classifiers to map these sequences to help triggers. We explore supervision regimes for strong or weak supervision, and extensively compare them across in-distribution and out-of-distribution tasks. Our results show a trade-off: strong labels enable models to capture fine-grained uncertainty dynamics for reliable help detection, while weak labels, though noisier, still support competitive introspection when training and evaluation are aligned, offering a scalable path when dense annotation is impractical. Crucially, we find that modeling the temporal evolution of token-level uncertainty signals with transformers provides far greater predictive power than static sequence-level scores. This study provides the first systematic evaluation of uncertainty-based introspection in VLAs, opening future avenues for active learning and for real-time error mitigation through selective human intervention.

Figures (4)

• Figure 1: We use the $\pi_{0}$-FAST model as an underlying policy, translating inputs into autoregressive action tokens $T^1_t, \dots, T^n_t$. Our method, INSIGHT, uses the probability distribution that each token is sampled from, and extracts tokenwise uncertainty features $u_t^{1:n}$. We train a lightweight transformer to classify these features and predict if help is needed at that step.
• Figure 2: Hierarchy of episodes, steps, actions, and tokens. Each step involves one round of observation, inference, and action execution, shown in Fig. \ref{['fig:teaser']}.
• Figure 3: Results for the transformer (INSIGHT) and Conformal Prediction based on entropy (CP-E) and perplexity (CP-P). Each box plot indicates mean (dashed horizontal lines) and median (solid horizontal lines) performance across folds. Error bars indicate 1 standard deviation. Significance by paired Wilcoxon (two-sided) across folds: * $p<0.05$, ** $p<0.01$.
• Figure 4: Simulation-based OOD evaluation. We compare transformer variants under different supervision regimes: regular (trained on the real-world, formerly in-distribution, dataset), jumbo (trained on the combined in-distribution + distribution-shift data), and sim-only (weakly-supervised). Significance by paired Wilcoxon (two-sided) across folds: * $p<0.05$, ** $p<0.01$.