Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

F-Actor: Controllable Conversational Behaviour in Full-Duplex Models

Maike Züfle, Ondrej Klejch, Nicholas Sanders, Jan Niehues, Alexandra Birch, Tsz Kin Lam

TL;DR

The paper tackles controllable, instruction-following full-duplex speech by introducing F-Actor, an open model that can be trained efficiently by freezing the audio encoder and fine-tuning only the LLM. It demonstrates data-efficient training on ~2{,}000 hours and shows controllable behavior over voice, topic, and conversational dynamics, including backchannels and interruptions. Through a single-stage pipeline and comprehensive evaluations, it provides evidence of strong instruction-following performance and competitive turn-taking behavior, while releasing code to support reproducibility. The work advances practical, customizable conversational AI with real-time, proactive dialogue capabilities, offering a path toward more natural and controllable spoken interactions in constrained academic settings.

Abstract

Spoken conversational systems require more than accurate speech generation to have human-like conversations: to feel natural and engaging, they must produce conversational behaviour that adapts dynamically to the context. Current spoken conversational systems, however, rarely allow such customization, limiting their naturalness and usability. In this work, we present the first open, instruction-following full-duplex conversational speech model that can be trained efficiently under typical academic resource constraints. By keeping the audio encoder frozen and finetuning only the language model, our model requires just 2,000 hours of data, without relying on large-scale pretraining or multi-stage optimization. The model can follow explicit instructions to control speaker voice, conversation topic, conversational behaviour (e.g., backchanneling and interruptions), and dialogue initiation. We propose a single-stage training protocol and systematically analyze design choices. Both the model and training code will be released to enable reproducible research on controllable full-duplex speech systems.

F-Actor: Controllable Conversational Behaviour in Full-Duplex Models

TL;DR

The paper tackles controllable, instruction-following full-duplex speech by introducing F-Actor, an open model that can be trained efficiently by freezing the audio encoder and fine-tuning only the LLM. It demonstrates data-efficient training on ~2{,}000 hours and shows controllable behavior over voice, topic, and conversational dynamics, including backchannels and interruptions. Through a single-stage pipeline and comprehensive evaluations, it provides evidence of strong instruction-following performance and competitive turn-taking behavior, while releasing code to support reproducibility. The work advances practical, customizable conversational AI with real-time, proactive dialogue capabilities, offering a path toward more natural and controllable spoken interactions in constrained academic settings.

Abstract

Spoken conversational systems require more than accurate speech generation to have human-like conversations: to feel natural and engaging, they must produce conversational behaviour that adapts dynamically to the context. Current spoken conversational systems, however, rarely allow such customization, limiting their naturalness and usability. In this work, we present the first open, instruction-following full-duplex conversational speech model that can be trained efficiently under typical academic resource constraints. By keeping the audio encoder frozen and finetuning only the language model, our model requires just 2,000 hours of data, without relying on large-scale pretraining or multi-stage optimization. The model can follow explicit instructions to control speaker voice, conversation topic, conversational behaviour (e.g., backchanneling and interruptions), and dialogue initiation. We propose a single-stage training protocol and systematically analyze design choices. Both the model and training code will be released to enable reproducible research on controllable full-duplex speech systems.
Paper Structure (34 sections, 1 equation, 4 figures, 7 tables)

This paper contains 34 sections, 1 equation, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Related Work
  4. Model
  5. Evaluation
  6. Dialogue Generation for Evaluation.
  7. General System Capabilities Eval.
  8. Instruction-Following Capabilities Eval.
  9. Experimental Setup
  10. Data
  11. Training Details
  12. Experiments
  13. Baselines
  14. Results
  15. General Modeling Abilities
  16. ...and 19 more sections

Figures (4)

  • Figure 1: Overview of our controllable full-duplex model, which can be prompted to control (i) speaker voice, (ii) conversation topic, (iii) conversational behaviour (e.g., backchanneling and interruptions), and (iv) dialogue initiation. Only the LLM and audio heads are trained in a single-stage training, other components remain frozen.
  • Figure 2: Example prompts from the train set.
  • Figure 3: Prompt to rewrite the narrative from Behavior-SD.
  • Figure 4: Prompt for the LLM Judge to judge the instruction following capabilities of our full-duplex model.