Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tap-to-Adapt: Learning User-Aligned Response Timing for Speech Agents

Zihong He, Hai-Ning Liang, Chen Liang

Abstract

Response timing judgment is a critical component of interactive speech agents. Although there exists substantial prior work on turn modeling and voice wake-up, there is a lack of research on response timing judgments continuously aligned with user intent. To address this, we propose the Tap-to-Adapt framework, which enables users to naturally activate or interrupt the agent via tap interactions to construct online learning labels for response timing models. Under this framework, Dilated TCN and a sequential replay strategy play significant roles, as demonstrated through data-driven experiments and user studies. Additionally, we develop an evaluation and continuous data mining system tailored for the Tap-to-Adapt framework, through which we have collected approximately 20,000 samples from the user studies involving 20 participants.

Tap-to-Adapt: Learning User-Aligned Response Timing for Speech Agents

Abstract

Response timing judgment is a critical component of interactive speech agents. Although there exists substantial prior work on turn modeling and voice wake-up, there is a lack of research on response timing judgments continuously aligned with user intent. To address this, we propose the Tap-to-Adapt framework, which enables users to naturally activate or interrupt the agent via tap interactions to construct online learning labels for response timing models. Under this framework, Dilated TCN and a sequential replay strategy play significant roles, as demonstrated through data-driven experiments and user studies. Additionally, we develop an evaluation and continuous data mining system tailored for the Tap-to-Adapt framework, through which we have collected approximately 20,000 samples from the user studies involving 20 participants.
Paper Structure (25 sections, 2 equations, 6 figures, 1 table)

This paper contains 25 sections, 2 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Speech Turn-Initialization Modeling
  4. Speech Wake-Up Systems
  5. User-Feedback-Based Online Speech Learning
  6. Method
  7. Framework Overview
  8. Model and Learning Strategy
  9. Dilated TCN-based Model
  10. Sequential Replay Online Learning Strategy
  11. Sample Mining and Evaluation
  12. Simulated Sample Construction
  13. Real Sample Mining
  14. Experiment
  15. Validation Experiment
  16. ...and 10 more sections

Figures (6)

  • Figure 1: In human–speech agent interaction scenarios, the Tap-to-Adapt Framework drives the agent to learn to respond according to the human’s expectations by constructing online learning samples through manual activations and interruptions performed by the human.
  • Figure 2: The figure illustrates the architecture of the Tap-to-Adapt Framework.
  • Figure 3: The figure illustrates the architecture of an efficient Dilated TCN-based model within the Tap-to-Adapt framework. The model first extracts Mel-spectrogram features from the input audio, then passes them through multiple TCN blocks with a dilation mechanism to enlarge the temporal receptive field. The resulting features are subsequently fed into a main model, which by default adopts Transformer blocks vaswani2017attention, consisting of an Input Projection Layer, Positional Encoding, and a Transformer Encoder.
  • Figure 4: Results of the validation experiment using 1,000 simulated samples for Tap-to-Adapt Framework. The related model and online learning strategy are described in Sec. \ref{['model_and_learning']}. Here, without Transformer Blocks indicates that the main model is implemented as a simple MLP.
  • Figure 5: The temporal trend of average activation and interruption frequencies in continuous participant interactions under consistent environmental conditions over time.
  • ...and 1 more figures