Retrospective Learning from Interactions

TL;DR

ReSpect proposes a fully annotation-free, retrospective learning framework that decodes implicit feedback from deployment interactions to iteratively improve a multimodal LLM policy. By grounding learning in a new multi-turn reference task (MultiRef) and employing three objective families—Filtered fine-tuning, RL, and Kahneman-Tversky-style optimization—the approach demonstrates large, sustained gains in task completion (from 31% to 82%) without external labels. Key findings show positive-only feedback is more informative than including negative signals, that a simple FFT strategy outperforms more complex RL and KTO in live settings, and that the feedback decoder remains robust across rounds with high precision. The work highlights a scalable path for continuous self-improvement of LLMs in real deployments, while acknowledging limitations in credit assignment, generalization beyond MultiRef, and potential data-poisoning risks in unsupervised feedback loops.

Abstract

Multi-turn interactions between large language models (LLMs) and users naturally include implicit feedback signals. If an LLM responds in an unexpected way to an instruction, the user is likely to signal it by rephrasing the request, expressing frustration, or pivoting to an alternative task. Such signals are task-independent and occupy a relatively constrained subspace of language, allowing the LLM to identify them even if it fails on the actual task. We introduce ReSpect, a method to learn from such signals in past interactions via retrospection without additional annotations. We deploy ReSpect in a new multimodal interaction scenario, where humans instruct a multimodal LLM to solve an abstract reasoning task with a combinatorial solution space. Through thousands of interactions with humans, we show how ReSpect gradually improves task completion rate from 31% to 82%, all without any external annotation.

Figures (23)

• Figure 1: The ReSpect process. We deploy an MLLM policy $\pi_{\theta_\rho}(a \vert x)$ in rounds $\rho$, to interact with users in multi-turn interactions. Following each round, the MLLM retrospectively analyzes each of its actions (highlighted in blue) to decode feedback given the interaction context and follow-up utterances. The decoded feedback can be positive (thumbs up as illustrated), negative or neutral. After each round, the model is retrained using all data aggregated so far $D_{\leq\rho}$. The MLLM improves over time without any external annotations. The plot on the right shows the performance curve in our experiments -- task success rate improves from 31% to 82% over six rounds.
• Figure 2: The interaction scenario we use in our experiments. MultiRef is a multi-turn reference game. A speaker and a listener both observe a shared set of tangram shapes in different orders. The speaker describes targets for the listener to select, often gradually over multiple turns. As an interaction progresses, the speaker naturally produces implicit feedback signals that validate or reject the listener's actions.
• Figure 3: The text-only prompt used to decode feedback from past interactions. This figure combines the prompts for both binary and ternary feedback decoding. Green: binary case only. Orange: ternary case only. The verbal feedback generated by the model is in bold. Additional comments for readability are in blue italics.
• Figure 4: Task performance and efficiency improve as the policy learns from more past interactions. We present deployment results across three rounds for six concurrent systems, and three more rounds for the best system b-fft, together with human-human references (hh) and a redeployment of the initial policy $\pi_{\theta_0}$ (control). Left: interaction-level success rate ($\uparrow$, higher is better). Right: interaction-level efficiency by # turns per interactions ($\downarrow$). Shades are 95% confidence intervals by bootstrapping with 10,000 resamples.
• Figure 5: Turn-level performance of b-fft evaluated by post-hoc human annotations. Left: % turns where the policy's action $\hat{a}$ matches exactly the human listener's action $a^*$ ($\uparrow$). Center: similarity between the policy's action and the human listener's action ($\uparrow$). Even actions that receive negative feedback in deployment (neg fb) are increasingly similar to human actions. Right: % policy actions annotated to have received positive implicit feedback from human listeners ($\uparrow$). Shades are 95% confidence intervals by bootstrapping with 10,000 resamples.