Duo: A Reciprocal World Model for Full-Duplex Interaction
Abstract
In this report, we introduce the Reciprocal World Model (RWM) and present Duo v1, a full-duplex world model that continuously perceives audiovisual inputs while responding to users in real time. The RWM treats intelligence and physical reality as equally fundamental to what constitutes a world. Existing world models largely focus on simulating the latter, leaving reasoning and decision-making as external controls rather than intrinsic parts of the world itself. In RWM, we identify two equally fundamental capabilities: Intent, which provides the cognition underlying world interactions, and Realization, which generates coherent audio-visual behavior. To realize this formulation, Duo v1 introduces three key technical designs: (1) Leveraging autoregressive modeling for continuous interaction reasoning and participation decisions, rather than restricting it to instruction understanding for diffusion models. (2) Constructing chunk-specific conditions and synchronize tokens across Intent and Realization on a unified timeline. Module-specific attention masks enforce causal dependencies both within and across chunks, preventing access to future inputs. (3) Mitigating distribution shift in both modules through dual rollout distillation, training on autoregressively generated Intent tokens and synthesized audiovisual histories from Realization. Both modules continuously perceive real-time audio, video, and text, enabling full-duplex interaction with spontaneous conversations and interruptions. Beyond conversation, users can share real objects, reshape virtual scenes, guide assistants, or play audiovisual games without mouse or keyboard controls. The release of Duo v1 makes our first step, with the next version coming soon. We look forward to broader exploration of this emerging field.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a novel Reciprocal World Model architecture that tightly couples autoregressive intent reasoning with causal audiovisual generation to enable full-duplex, real-time interaction. The technical contribution is significant, particularly the dual rollout distillation and unified timeline synchronization, which address critical challenges in streaming multimodal generation. While the experimental section is truncated, the architectural innovation and open-source release suggest a high-impact contribution to the field of interactive world models.
Methodology
The paper proposes the Reciprocal World Model (RWM), a framework that integrates an autoregressive "Intent" module (based on a multimodal LLM) with a causal diffusion "Realization" module (based on a DiT). The core methodological contribution is the tight coupling of these two modules on a unified timeline using chunk-specific conditions and synchronize tokens. The Intent module handles high-level reasoning and participation decisions (speak/wait), while the Realization module generates synchronized audio and video. A key technical innovation is the "dual rollout distillation" strategy, which mitigates distribution shift by training the Realization module on autoregressively generated Intent tokens and synthesized audiovisual histories, ensuring that the inference-time conditioning matches the training distribution. The use of module-specific attention masks to enforce strict causality within and across chunks is a robust design choice for real-time streaming.
Results & evaluation
The provided text is truncated before the detailed experimental results section. However, the abstract and introduction claim that the system enables full-duplex interaction, spontaneous conversations, and interruptions. The paper mentions the release of "Duo v1" and provides links to a homepage and Hugging Face collection, suggesting that qualitative results and potentially some quantitative metrics are available externally. Without the full experimental section, it is difficult to rigorously assess the quantitative performance (e.g., latency, FID, CLAP scores, human preference studies) compared to baselines like Moshi or Qwen-Omni. The reliance on external links for primary results is a minor weakness for a standalone paper evaluation.
Reproducibility
The paper provides links to the GitHub repository and Hugging Face model weights, which significantly enhances reproducibility. The architectural details, such as the VAE compression ratios (40 latent vectors/sec for audio, 17 frames to 5 latent frames for video) and the specific attention mask strategies, are described in sufficient detail for implementation. The training data construction pipelines are mentioned but not fully detailed in the truncated text.
Limitations
The primary limitation is the lack of visible quantitative evaluation in the provided text. The system relies on a modular approach (LLM + Diffusion), which may introduce latency or synchronization challenges compared to unified end-to-end models, although the paper claims to address this with the unified timeline. The "dual rollout distillation" is computationally expensive as it requires generating rollouts during training. The paper is a "report" rather than a full conference paper, which may mean some theoretical justifications or ablation studies are omitted.
Broader impact
This work has significant potential impact on the field of human-computer interaction and embodied AI. By treating intelligence and physical reality as equally fundamental, it moves beyond passive video generation to active, reciprocal interaction. This could lead to more natural and engaging AI companions, virtual assistants, and interactive media. The full-duplex capability is a major step towards human-like conversational agents that can listen and speak simultaneously. The paper introduces a novel Reciprocal World Model architecture that tightly couples autoregressive intent reasoning with causal audiovisual generation to enable full-duplex, real-time interaction. The technical contribution is significant, particularly the dual rollout distillation and unified timeline synchronization, which address critical challenges in streaming multimodal generation. While the experimental section is truncated, the architectural innovation and open-source release suggest a high-impact contribution to the field of interactive world models.
Author of this paper? Send a correction.