An LLM Is All You Need: Data-Efficient Adaptation to Full-Duplex, Word-Level Spoken Dialogue
Abstract
Full-duplex spoken dialogue, in which a system listens while it speaks and can backchannel, yield the floor, and be interrupted, is usually obtained by training on hundreds of thousands to millions of hours of speech. This is costly, rewrites the language model that supplies the system's knowledge, and still leaves turn-taking either implicit or decided coarsely rather than word by word. We show that a frozen, general-purpose multimodal LLM can instead be adapted into a word-level full-duplex system with little data. Our system is a thinker-talker cascade on an 80 ms frame clock. The thinker keeps the LLM frozen and trains only lightweight adapters and a few new token embeddings (97M parameters); at every frame it emits one token of agent text, with pauses, yields, and onsets represented as tokens of the same vocabulary and kept legal by a finite-state machine. A four-stage curriculum, each stage anchored to the model's own behavior under the previous format, performs the conversion in about 30 GPU-hours. The talker reads only the thinker's discrete tokens, so the two halves are trained independently. On Full-Duplex-Bench and InteractionBench, the system is competitive with open-source full-duplex systems built through large-scale speech training: among them, it is the least likely to take the floor when the user merely pauses, backchannels most often, and leads in spoken-task accuracy, groundedness, and coherence over long conversations.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a data-efficient framework for adapting frozen multimodal LLMs into full-duplex spoken dialogue systems using a thinker-talker cascade and finite-state turn-taking, achieving competitive performance with large-scale models at a fraction of the training cost.
Methodology
The paper proposes a highly efficient adaptation strategy for converting a frozen, general-purpose multimodal LLM (Qwen3-Omni) into a full-duplex spoken dialogue system. The core methodological contribution is the "thinker-talker" cascade operating on an 80ms frame clock. The "thinker" retains the LLM backbone frozen and trains only lightweight adapters (LoRA) and new token embeddings for timing controls (pauses, yields, onsets). A key innovation is the finite-state machine (FSM) that masks illegal tokens during sampling, ensuring that turn-taking decisions (represented as tokens in the same vocabulary as words) remain structurally valid. The "talker" is a separate streaming speech synthesis model that consumes only the discrete text tokens from the thinker, allowing independent training. The training curriculum is a four-stage process, each stage anchored to the model's previous behavior via self-distillation or KL divergence, requiring only ~30 GPU-hours. This approach effectively decouples the high-level dialogue logic (which remains in the frozen LLM) from the low-level acoustic generation and timing, avoiding the catastrophic forgetting and high data costs associated with full fine-tuning of speech models.
Results & evaluation
The system is evaluated on Full-Duplex-Bench (v1.0 and v1.5) and InteractionBench. The results show that the proposed system is competitive with large-scale open-source systems like Moshi and PersonaPlex, which require millions of hours of speech training. Specifically, the system excels in pause handling (least likely to take the floor when the user pauses) and backchannel frequency. It also leads in spoken-task accuracy, groundedness, and coherence on InteractionBench. However, it lags behind some baselines in responsiveness (latency) and whisper/volume control. The ablation study on noise robustness is particularly strong, demonstrating that training the channel-fusion layer and audio adapter is critical for handling real-world room tone, providing a clear mechanistic understanding of the model's robustness.
Reproducibility
The paper provides extensive implementation details, including specific hyperparameters, learning rates, batch sizes, and architectural dimensions for both the thinker and talker. The training data sources (synthetic dialogues, specific hours of audio) and the four-stage curriculum are described in detail. The use of standard components like Mimi codec and LoRA enhances reproducibility. However, the reliance on a specific proprietary multimodal LLM (Qwen3-Omni) and the specific synthetic data generation pipeline may limit immediate reproducibility for groups without access to these specific assets or the compute resources to generate similar synthetic data.
Limitations
The primary limitation is the dependence on a pre-trained, general-purpose multimodal LLM that already understands speech; the paper does not claim to learn full-duplex behavior from scratch with small data. The talker is currently trained only on synthesized speech and does not utilize the user's audio channel for voice conditioning (it is zeroed out), limiting its ability to adapt to user-specific acoustic characteristics or control volume/whispering effectively. The evaluation benchmarks, while standard, have relatively small sample sizes for some behaviors, leading to high variance in some metrics. The backchannel training data is small (701 unique dialogues), raising questions about generalization.
Broader impact
This work significantly lowers the barrier to entry for developing full-duplex spoken dialogue systems. By demonstrating that a frozen LLM can be adapted with minimal data and compute, it makes advanced conversational AI more accessible to smaller research groups and companies. The discrete token interface between thinker and talker offers a modular design that could be extended to other modalities or tasks. The explicit modeling of turn-taking via FSM provides a transparent and controllable mechanism for dialogue management, which is valuable for safety and user experience in real-world applications. The paper presents a data-efficient framework for adapting frozen multimodal LLMs into full-duplex spoken dialogue systems using a thinker-talker cascade and finite-state turn-taking, achieving competitive performance with large-scale models at a fraction of the training cost.
Author of this paper? Send a correction.