This week’s shortlist
Recent advances in large audio-language models (LALMs) have broadened audio understanding across speech, sound, and music, with tool-based agents further supporting these tasks by invoking predefined audio tools. However, a persistent gap remains between perceiving acoustic…
Abstract
Recent advances in large audio-language models (LALMs) have broadened audio understanding across speech, sound, and music, with tool-based agents further supporting these tasks by invoking predefined audio tools. However, a persistent gap remains between perceiving acoustic content and using it as evidence for reasoning: recognizing what is audible does not ensure that a model can identify which cues resolve an uncertain decision. To bridge this gap, we propose Think2Listen, an evidence-centric learning framework for agentic audio understanding. Guided by evidence before commitment, Think2Listen couples iterative reasoning with a targeted LALM interaction to actively seek acoustic cues that distinguish competing hypotheses and revise candidate judgments based on the observations returned. It learns what to ask, how to use evidence, and when to stop, using observation-induced changes in candidate-state to inform training credit. Extensive evaluations on two benchmarks spanning seven audio categories demonstrate that Think2Listen achieves (I) enhanced multi-step audio reasoning, with gains in overall accuracy of 5.08%--17.67% over competing audio agents; and (II) improved audio understanding and reasoning across speech, sound, and music, boosting overall accuracy by 2.29%--13.10% over the corresponding baselines. The source code is available at https://github.com/JinWang79/Think2Listen.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces an evidence-centric learning framework for agentic audio understanding that couples iterative reasoning with targeted LALM interactions, using observation-induced candidate-state changes to inform policy training. This approach effectively bridges the gap between acoustic perception and discriminative reasoning, demonstrating significant performance gains over existing audio agents by learning what to ask, how to use evidence, and when to stop.
Methodology
The paper proposes "Think2Listen," a framework that integrates an "Adaptive Audio Evidence Loop" with a novel training signal called "Transition-as-Credit." The methodology addresses the gap between acoustic perception and evidence-based reasoning by allowing an agent to iteratively query a Large Audio-Language Model (LALM) for specific acoustic cues before committing to an answer. The core technical contribution is the "Transition-as-Credit" mechanism, which uses a semantic extractor to reconstruct how observations change the agent's candidate state (eliminating or confirming hypotheses). This process-level feedback is used to calibrate rewards in a Group Relative Policy Optimization (GRPO) style update, distinguishing between useful evidence-seeking and misleading or redundant queries. The approach is theoretically sound, leveraging intermediate state changes to provide denser supervision signals than terminal accuracy alone.
Results & evaluation
The authors evaluate the framework on two benchmarks spanning seven audio categories (speech, sound, music). The results show significant improvements over competing audio agents (5.08%–17.67% accuracy gain) and baselines (2.29%–13.10% gain). The inclusion of ablation studies on the "Transition-as-Credit" component and the analysis of untrained interaction patterns (Figure 2) provides strong empirical support for the necessity of the proposed learning strategy. The experiments effectively demonstrate that learning *what* to ask and *when* to stop is crucial for agentic audio understanding.
Reproducibility
The paper provides a GitHub repository link, which is a positive indicator for reproducibility. The methodology is described with sufficient detail regarding the state formulation, the semantic extractor's role, and the reward calibration formulas. However, specific hyperparameters for the LALM and the actor policy are likely in the appendix (truncated in the provided text), which is standard. The reliance on a specific LALM for the "listener" role may limit immediate reproducibility if that model is not open-source or easily accessible, but the framework itself is clearly defined.
Limitations
The framework relies on the quality of the underlying LALM for generating observations; if the LALM hallucinates acoustic features, the agent's reasoning will be flawed, although the "misleading trajectory" flag attempts to mitigate this. The computational cost of iterative querying and the semantic extraction step for every turn may be high compared to single-shot inference. Additionally, the method is tested on specific benchmarks; generalization to open-ended audio tasks without predefined candidate sets $C$ is not explicitly addressed.
Broader impact
This work contributes to the emerging field of agentic multimodal AI, specifically in the audio domain. By formalizing "evidence-centric" learning, it offers a template for other modalities where perception and reasoning are decoupled. The technique of using state-transition changes for credit assignment is applicable to other tool-using agents. It advances the capability of AI systems to handle complex, multi-step audio reasoning tasks that require active information gathering rather than passive listening. The paper introduces an evidence-centric learning framework for agentic audio understanding that couples iterative reasoning with targeted LALM interactions, using observation-induced candidate-state changes to inform policy training. This approach effectively bridges the gap between acoustic perception and discriminative reasoning, demonstrating significant performance gains over existing audio agents by learning what to ask, how to use evidence, and when to stop.
Zero-shot voice cloning typically emphasizes preserving the timbre and prosodic characteristics of a reference speaker. However, in real-world use, speech should adapt its delivery to the target text rather than remain tied to a fixed speaking style. In this work, we formulate…
Abstract
Zero-shot voice cloning typically emphasizes preserving the timbre and prosodic characteristics of a reference speaker. However, in real-world use, speech should adapt its delivery to the target text rather than remain tied to a fixed speaking style. In this work, we formulate this problem as text-adaptive prosody synthesis for zero-shot voice cloning and study how post-training can endow a cloning model with this capability. We develop a three-stage framework. First, Direct Preference Optimization (DPO) provides a cold start by aligning the model with text-appropriate expressive speech. Second, Group Sequence Policy Optimization (GSPO) jointly optimizes prosodic expressiveness, acoustic quality, intelligibility, and speaker similarity. Finally, context-conditioned On-Policy Self-Distillation (OPSD) enables self-improvement by transferring expressive behavior from strong-reference to weak-reference conditions. On Qwen3-TTS, our framework improves Emotion from 18.5 to 81.2 and Paralinguistics from 38.4 to 73.9. Extensive ablations validate the role of each stage.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces a three-stage post-training framework (DPO, GSPO, OPSD) to achieve text-adaptive prosody in zero-shot TTS, significantly improving expressiveness while maintaining speaker identity. The technical contribution is strong, with a well-designed methodology that leverages modern RLHF techniques and LLM-based data pipelines, supported by extensive experiments and ablations that validate the effectiveness of each stage.
Methodology
The paper proposes a three-stage post-training framework (DPO, GSPO, OPSD) to enable text-adaptive prosody in zero-shot TTS. The methodology is sound, leveraging established RLHF techniques (DPO, GSPO) and a novel application of self-distillation (OPSD) to decouple speaker identity from reference prosody. The use of LLMs (Gemini) for automated preference data construction and reward modeling is a strong, modern approach that addresses the scarcity of high-quality expressive speech data. The formulation of the problem as "text-adaptive prosody synthesis" rather than just "voice cloning" is a meaningful conceptual shift.
Results & evaluation
Experiments are extensive, covering multiple benchmarks (Emergent TTS-Eval, Seed-TTS-Eval) and subjective evaluations (MOS, ABX). The ablation studies are thorough, isolating the contributions of each stage and hyperparameter choices (e.g., speaker diversity, reward weights). The results show significant improvements in emotion and paralinguistics scores, though there is a noted trade-off with speaker similarity, which is honestly reported. The use of human ABX tests to validate reduced reliance on reference expressiveness is a strong experimental design choice.
Reproducibility
Reproducibility is moderate. While the framework is well-described, reliance on proprietary models (Gemini 3.1, Qwen3-TTS) and specific datasets (Emilia, SeedVC) limits open reproduction. The automated data pipeline is detailed, but the exact prompts and filtering thresholds for Gemini-based verification are not fully specified. The code is not explicitly linked, though a demo page is provided.
Limitations
The primary limitation is the trade-off between expressiveness and speaker similarity, which the authors acknowledge but do not fully resolve. The method relies heavily on the quality of the LLM-based reward model and data generation, introducing potential biases. The evaluation is primarily on English and Chinese, with limited cross-lingual generalization analysis. The computational cost of the multi-stage training and LLM-in-the-loop data generation is not discussed.
Broader impact
This work has significant implications for the development of more natural and expressive TTS systems, moving beyond static voice cloning towards dynamic, context-aware speech generation. The framework could be adapted to other speech tasks and languages. The use of LLMs for data augmentation and reward modeling sets a precedent for future work in speech AI. The paper introduces a three-stage post-training framework (DPO, GSPO, OPSD) to achieve text-adaptive prosody in zero-shot TTS, significantly improving expressiveness while maintaining speaker identity. The technical contribution is strong, with a well-designed methodology that leverages modern RLHF techniques and LLM-based data pipelines, supported by extensive experiments and ablations that validate the effectiveness of each stage.
Audio phase relations encode source locations, yet existing large audio-language models (LALM) reduce audio to single-channel content to stabilise general audio understanding, leaving spatial information for direction, distance, and source-relation questions unused. We present…
Abstract
Audio phase relations encode source locations, yet existing large audio-language models (LALM) reduce audio to single-channel content to stabilise general audio understanding, leaving spatial information for direction, distance, and source-relation questions unused. We present SpheraAudio, which accepts recordings with any channel count -- mono, stereo, First-Order Ambisonics (FOA), or higher -- and maps them into a single token stream. A spatial encoder turns complex multichannel spectra into spatial tokens interleaved with audio tokens at corresponding temporal positions. Spatial Multimodal RoPE (SMRoPE) makes each token's modality identity and temporal slot visible to rotary attention without altering the language model's original RoPE. Training runs in three stages -- spatial-encoder pretraining, projector alignment, and joint supervised-and-reinforcement learning -- progressing from geometric supervision to language-based spatial decisions, with answers trained as explicit spatial chain-of-thought: step-by-step geometric reasoning over the heard scene. Current benchmarks tie questions to a fixed channel-count format and score answers without spatial reasoning evidence. We introduce SpheraQA, which spans mono, stereo, and FOA recordings and organises questions hierarchically from single-attribute perception through cross-source relations to multi-step reasoning, scored programmatically against simulated scene geometry and real-recording annotations with step-by-step solution references. Across SpheraQA and public spatial-audio benchmarks, SpheraAudio sets a new performance standard for spatial audio understanding. It supports high-precision localisation, tracking, and reasoning over static and moving sound sources while preserving strong general-audio capabilities. We will release code, SpheraQA, and evaluation metrics to support reproducibility.
AI-generated analysis. Check claims against the paper.
Summary
The paper presents SpheraAudio, a novel LALM framework that integrates spatial audio understanding via a dedicated spatial encoder and SMRoPE, achieving state-of-the-art performance on spatial reasoning tasks while preserving general audio capabilities. The contribution is significant due to its comprehensive handling of multichannel inputs, the introduction of the SpheraQA benchmark, and the rigorous training pipeline combining geometric pretraining, CoT supervision, and reinforcement learning, effectively bridging the gap between signal processing and language-based spatial reasoning.
Methodology
The paper proposes SpheraAudio, a framework integrating spatial audio understanding into Large Audio-Language Models (LALMs). The core methodological contributions are the "Spatial AuT" encoder, which processes complex multichannel spectra (FOA, stereo, mono) into spatial tokens, and "SMRoPE" (Spatial Multimodal RoPE), a novel rotary position embedding scheme that encodes modality identity and temporal alignment without altering the base LLM's RoPE. The training pipeline is rigorous, involving geometric pretraining of the spatial encoder, projector alignment, and joint SFT with explicit spatial Chain-of-Thought (CoT) supervision, followed by GRPO reinforcement learning. The approach effectively bridges the gap between raw multichannel signal processing and high-level language reasoning by explicitly pairing audio and spatial tokens in time.
Results & evaluation
The authors introduce SpheraQA, a substantial benchmark (20,492 questions) covering perception, relations, and reasoning across mono, stereo, and FOA formats. Experimental results show SpheraAudio outperforming strong baselines like Spatial-Omni-7B and Gemini 3.1 Pro on spatial tasks while retaining or improving general audio capabilities. Ablation studies convincingly demonstrate the necessity of token interleaving (vs. addition) and the specific SMRoPE design for temporal correspondence. The inclusion of stereo-specific experiments and general audio retention tests provides a comprehensive evaluation of the model's versatility.
Reproducibility
The paper provides detailed implementation specifications, including architecture dimensions, loss functions, hyperparameters for SFT and RL, and precise evaluation protocols for SpheraQA. The commitment to release code, the benchmark, and evaluation metrics enhances reproducibility. The use of GPT-4o for answer extraction in free-form responses is a practical but potentially biased evaluation choice, though mitigated by programmatic scoring for structured answers.
Limitations
The model relies on a 4B parameter backbone (Qwen3-Omni), which may limit its performance on extremely complex reasoning compared to larger LLMs. The spatial encoder is pretrained on simulated FOA data; while real recordings are included in evaluation, the generalization to uncalibrated, noisy real-world microphone arrays beyond the tested LOCATA/STARSS23 datasets is not fully explored. The "Motion direction" task shows a negative delta compared to the baseline, indicating a weakness in specific dynamic spatial reasoning.
Broader impact
This work significantly advances the field of spatial audio AI by providing a unified framework for handling arbitrary channel counts. It enables embodied agents to understand sound scenes with precision, supporting applications in navigation, robotics, and immersive media. The introduction of SpheraQA provides a much-needed standardized benchmark for evaluating spatial reasoning in LALMs, moving beyond simple localization to complex geometric reasoning. The paper presents SpheraAudio, a novel LALM framework that integrates spatial audio understanding via a dedicated spatial encoder and SMRoPE, achieving state-of-the-art performance on spatial reasoning tasks while preserving general audio capabilities. The contribution is significant due to its comprehensive handling of multichannel inputs, the introduction of the SpheraQA benchmark, and the rigorous training pipeline combining geometric pretraining, CoT supervision, and reinforcement learning, effectively bridging the gap between signal processing and language-based spatial reasoning.
Rapid progress in text-to-audio (TTA) generation demands reliable and explainable automatic evaluation. However, existing metrics fail to simultaneously deliver high correlation with human judgments and meaningful, audio-grounded explanations, forcing reliance on labor-intensive…
Abstract
Rapid progress in text-to-audio (TTA) generation demands reliable and explainable automatic evaluation. However, existing metrics fail to simultaneously deliver high correlation with human judgments and meaningful, audio-grounded explanations, forcing reliance on labor-intensive human evaluation that bottlenecks model development iteration. To address this, we propose CoTTAGE, a reasoning-based evaluator for text relevance (REL) and audio quality (OVL). Following a reason-then-predict paradigm, CoTTAGE first generates an audio-grounded rationale to diagnose errors before predicting scores. Supported by over 10K newly collected human OVL annotations, CoTTAGE is first trained via supervised fine-tuning and then refined with reinforcement learning using a Thurstone-based preference objective that converts sampled score distributions into pairwise preference probabilities and rewards agreement with rankings derived from human judgments. Across six benchmarks spanning REL and OVL, CoTTAGE achieves state-of-the-art correlation with human judgments while producing human-preferred explanations to accelerate TTA development.
AI-generated analysis. Check claims against the paper.
Summary
CoTTAGE introduces a reasoning-based evaluator for text-to-audio generation that achieves state-of-the-art correlation with human judgments by combining supervised fine-tuning with a novel Thurstone-based reinforcement learning objective, while also providing human-preferred, audio-grounded explanations for its scores.
Methodology
The paper proposes CoTTAGE, a reasoning-based evaluator for text-to-audio (TTA) generation that assesses both text relevance (REL) and overall audio quality (OVL). The core methodological contribution is the "reason-then-predict" paradigm, where the model generates an audio-grounded rationale before predicting a score. This is implemented via a two-stage training pipeline: (1) Supervised Fine-Tuning (SFT) on reasoning traces generated by a proprietary LLM (Gemini-3.1 Pro) conditioned on ground-truth MOS scores, and (2) Reinforcement Learning (RL) using a Thurstone-based pairwise preference objective. The RL stage is particularly interesting as it leverages the variance of sampled scores to create uncertainty-aware preference probabilities, rewarding the model for aligning with human rankings rather than just pointwise scores. The use of a Bhattacharyya coefficient for the reward signal is a solid technical choice for handling probabilistic preferences.
Results & evaluation
The experimental setup is robust, utilizing six benchmarks (RELATE, XACLE, PAM, Clotho) to test both in-domain and out-of-domain generalization. The collection of 10K+ new human OVL annotations is a significant contribution, as OVL data was previously scarce compared to REL data. The results show CoTTAGE outperforming strong baselines, including embedding-based metrics (CLAPScore, ELSA) and general-purpose LALMs (Gemini-3.1 Pro, Qwen2.5-Omni). The ablation study clearly demonstrates the benefit of the RL stage over SFT alone. The human preference evaluation of the rationales further validates the explainability claim, showing that CoTTAGE's explanations are preferred over those of the base model and even the proprietary teacher model in some cases.
Reproducibility
The paper provides high reproducibility. The code and data are publicly available on GitHub. Detailed hyperparameters, prompts, and annotation protocols are included in the appendices. The use of standard models (Qwen2.5-Omni-7B) and LoRA fine-tuning makes the method accessible to the community. The detailed description of the MTurk annotation process, including quality control measures, adds to the transparency of the dataset creation.
Limitations
The primary limitation is the reliance on a proprietary model (Gemini-3.1 Pro) for generating the reasoning traces during SFT. This introduces a dependency on a closed-source system and potential biases from that specific model. Additionally, the evaluation is limited to short audio clips (<30s), and the model does not provide temporal grounding (timestamps) for errors, which limits its utility for fine-grained debugging of long-form audio. The domain coverage is also limited to environmental sounds, with no evaluation on music or speech-centric generation.
Broader impact
This work addresses a critical bottleneck in TTA development: the lack of scalable, explainable automatic evaluation. By providing a model that not only scores audio but also explains *why* it gave that score, CoTTAGE enables more efficient iteration cycles for TTA model developers. The release of the 10K OVL annotations is a valuable resource for the community. The approach of using RL with preference objectives for audio evaluation is a promising direction that could be extended to other audio tasks. CoTTAGE introduces a reasoning-based evaluator for text-to-audio generation that achieves state-of-the-art correlation with human judgments by combining supervised fine-tuning with a novel Thurstone-based reinforcement learning objective, while also providing human-preferred, audio-grounded explanations for its scores.
Speech tokenizers convert audio into discrete units, enabling language models to process and generate speech. The goal is to learn compact, text-like representations while preserving enough detail to reconstruct a speaker's voice and delivery. We propose Facto: a factorized…
Abstract
Speech tokenizers convert audio into discrete units, enabling language models to process and generate speech. The goal is to learn compact, text-like representations while preserving enough detail to reconstruct a speaker's voice and delivery. We propose Facto: a factorized tokenizer that disentangles time-varying information (content and prosody) from static components (speaker identity and recording conditions). We build on a recent linear factorization method for self-supervised features, extending it to handle unseen speakers. Then, we cluster the factorized space and train a lightweight decoder to reconstruct audio from the resulting tokens. By removing the static components before clustering, our tokens are more robust to small noise perturbations. We validate this by testing token stability under various noise conditions. Next, we theoretically analyze Facto's convergence and scaling properties, quantifying how the factorization improves with the number of training speakers. Finally, we evaluate Facto on a range of downstream tasks. We show that greater token stability translates into improved text-to-speech performance and competitive results in voice conversion and spoken language modeling. Code and checkpoints at https://github.com/ubisoft/ubisoft-laforge-facto/ and demo page at https://ubisoft-laforge.github.io/speech/facto/.
AI-generated analysis. Check claims against the paper.
Summary
Facto proposes a factorized speech tokenizer that disentangles content from speaker identity using linear factorization of self-supervised features, providing theoretical guarantees on convergence and stability. The paper makes a significant contribution by extending linear factorization to unseen speakers and rigorously analyzing the scaling properties with the number of training speakers, resulting in more robust tokens for downstream tasks like TTS and voice conversion.
Methodology
The paper proposes "Facto," a factorized speech tokenizer that disentangles time-varying content/prosody from static speaker/channel information. The core method builds on a linear factorization of self-supervised features (WavLM), extending it to handle unseen speakers via a linear content estimator. The methodology is theoretically grounded, providing convergence analysis for the factorization and a bias-variance trade-off analysis for the content estimator. The approach uses PCA and k-means clustering on the factorized content space, followed by a lightweight HiFiGAN-based decoder with FiLM conditioning for speaker identity reintroduction. The theoretical contributions (Theorem 1 and 2, Corollary 1) are a strong differentiator, offering rigorous bounds on error scaling with the number of training speakers.
Results & evaluation
The experiments validate the theoretical predictions, showing that the factorization stabilizes as the number of speakers increases (O(1/K) decay). The paper demonstrates improved token stability under noise perturbations compared to baselines. Downstream tasks include text-to-speech, voice conversion, and spoken language modeling. The results show competitive performance, with clear improvements in TTS due to token stability. The use of LibriSpeech and standard metrics provides a solid empirical foundation.
Reproducibility
The paper provides code and checkpoints on GitHub and a demo page. The methodology is described in sufficient detail, including hyperparameters (e.g., rank R=12, WavLM layer 6) and training setups. The theoretical derivations are provided in the main text or appendices, enhancing reproducibility.
Limitations
The method relies on the quality of the self-supervised features (WavLM) and the linear factorization assumption, which may not capture all non-linear interactions. The content estimator is linear, which might limit its ability to handle highly complex speaker variations. The paper focuses on English speech (LibriSpeech), and generalization to other languages or noisy real-world environments beyond small perturbations is not extensively tested.
Broader impact
The work contributes to the field of speech tokenization by providing a principled, theoretically sound approach to disentangling content and speaker identity. This can benefit downstream tasks like TTS, voice conversion, and speech language modeling by providing more robust and compact representations. The emphasis on stability and theoretical guarantees sets a high standard for future work in speech representation learning. Facto proposes a factorized speech tokenizer that disentangles content from speaker identity using linear factorization of self-supervised features, providing theoretical guarantees on convergence and stability. The paper makes a significant contribution by extending linear factorization to unseen speakers and rigorously analyzing the scaling properties with the number of training speakers, resulting in more robust tokens for downstream tasks like TTS and voice conversion.
In this report, we introduce the Reciprocal World Model (RWM) and present Duo v1, a full-duplex world model that continuously perceives audiovisual inputs while responding to users in real time. The RWM treats intelligence and physical reality as equally fundamental to what…
Abstract
In this report, we introduce the Reciprocal World Model (RWM) and present Duo v1, a full-duplex world model that continuously perceives audiovisual inputs while responding to users in real time. The RWM treats intelligence and physical reality as equally fundamental to what constitutes a world. Existing world models largely focus on simulating the latter, leaving reasoning and decision-making as external controls rather than intrinsic parts of the world itself. In RWM, we identify two equally fundamental capabilities: Intent, which provides the cognition underlying world interactions, and Realization, which generates coherent audio-visual behavior. To realize this formulation, Duo v1 introduces three key technical designs: (1) Leveraging autoregressive modeling for continuous interaction reasoning and participation decisions, rather than restricting it to instruction understanding for diffusion models. (2) Constructing chunk-specific conditions and synchronize tokens across Intent and Realization on a unified timeline. Module-specific attention masks enforce causal dependencies both within and across chunks, preventing access to future inputs. (3) Mitigating distribution shift in both modules through dual rollout distillation, training on autoregressively generated Intent tokens and synthesized audiovisual histories from Realization. Both modules continuously perceive real-time audio, video, and text, enabling full-duplex interaction with spontaneous conversations and interruptions. Beyond conversation, users can share real objects, reshape virtual scenes, guide assistants, or play audiovisual games without mouse or keyboard controls. The release of Duo v1 makes our first step, with the next version coming soon. We look forward to broader exploration of this emerging field.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces a novel Reciprocal World Model architecture that tightly couples autoregressive intent reasoning with causal audiovisual generation to enable full-duplex, real-time interaction. The technical contribution is significant, particularly the dual rollout distillation and unified timeline synchronization, which address critical challenges in streaming multimodal generation. While the experimental section is truncated, the architectural innovation and open-source release suggest a high-impact contribution to the field of interactive world models.
Methodology
The paper proposes the Reciprocal World Model (RWM), a framework that integrates an autoregressive "Intent" module (based on a multimodal LLM) with a causal diffusion "Realization" module (based on a DiT). The core methodological contribution is the tight coupling of these two modules on a unified timeline using chunk-specific conditions and synchronize tokens. The Intent module handles high-level reasoning and participation decisions (speak/wait), while the Realization module generates synchronized audio and video. A key technical innovation is the "dual rollout distillation" strategy, which mitigates distribution shift by training the Realization module on autoregressively generated Intent tokens and synthesized audiovisual histories, ensuring that the inference-time conditioning matches the training distribution. The use of module-specific attention masks to enforce strict causality within and across chunks is a robust design choice for real-time streaming.
Results & evaluation
The provided text is truncated before the detailed experimental results section. However, the abstract and introduction claim that the system enables full-duplex interaction, spontaneous conversations, and interruptions. The paper mentions the release of "Duo v1" and provides links to a homepage and Hugging Face collection, suggesting that qualitative results and potentially some quantitative metrics are available externally. Without the full experimental section, it is difficult to rigorously assess the quantitative performance (e.g., latency, FID, CLAP scores, human preference studies) compared to baselines like Moshi or Qwen-Omni. The reliance on external links for primary results is a minor weakness for a standalone paper evaluation.
Reproducibility
The paper provides links to the GitHub repository and Hugging Face model weights, which significantly enhances reproducibility. The architectural details, such as the VAE compression ratios (40 latent vectors/sec for audio, 17 frames to 5 latent frames for video) and the specific attention mask strategies, are described in sufficient detail for implementation. The training data construction pipelines are mentioned but not fully detailed in the truncated text.
Limitations
The primary limitation is the lack of visible quantitative evaluation in the provided text. The system relies on a modular approach (LLM + Diffusion), which may introduce latency or synchronization challenges compared to unified end-to-end models, although the paper claims to address this with the unified timeline. The "dual rollout distillation" is computationally expensive as it requires generating rollouts during training. The paper is a "report" rather than a full conference paper, which may mean some theoretical justifications or ablation studies are omitted.
Broader impact
This work has significant potential impact on the field of human-computer interaction and embodied AI. By treating intelligence and physical reality as equally fundamental, it moves beyond passive video generation to active, reciprocal interaction. This could lead to more natural and engaging AI companions, virtual assistants, and interactive media. The full-duplex capability is a major step towards human-like conversational agents that can listen and speak simultaneously. The paper introduces a novel Reciprocal World Model architecture that tightly couples autoregressive intent reasoning with causal audiovisual generation to enable full-duplex, real-time interaction. The technical contribution is significant, particularly the dual rollout distillation and unified timeline synchronization, which address critical challenges in streaming multimodal generation. While the experimental section is truncated, the architectural innovation and open-source release suggest a high-impact contribution to the field of interactive world models.
Speech tokenizers serve as speech front ends, encoding input speech signals into representations for downstream models, such as automatic speech recognition (ASR) systems and large audio-language models (LALMs). In this paper, we find that speech tokenizers are fragile at the…
Abstract
Speech tokenizers serve as speech front ends, encoding input speech signals into representations for downstream models, such as automatic speech recognition (ASR) systems and large audio-language models (LALMs). In this paper, we find that speech tokenizers are fragile at the semantic level: small perturbations can disrupt their semantic encoding process and consequently cause severe degradation in downstream ASR and LALM performance. To study this vulnerability, we propose T-SemAttack, a transferable semantic attack based on a semantic encoder ensemble. By jointly drifting the representation spaces of multiple semantic encoders, T-SemAttack effectively disrupts the semantic content of speech while preserving perceptual quality, thereby inducing error features. We further analyze layer-wise representation drift and cross-model gradient alignment. Our findings show that perturbations can induce substantial shifts in deep semantic representations and that the input gradients of different surrogate encoders exhibit directional alignment. These observations suggest that shared adversarially sensitive directions in the input space may be a key factor underlying cross-model attack transferability. Building on these, we introduce ROSETok, a robust speech tokenizer that combines noisy training with robust semantic distillation to improve reconstruction fidelity and downstream SpeechLLM robustness. Extensive experiments on multiple datasets, 9 open-source and black-box ASR systems, and 4 LALMs demonstrate that T-SemAttack achieves strong transferable attack performance, while the proposed Robust Speech Tokenizer exhibits robustness under high-fidelity reconstruction and downstream tasks. Our code and demo are available at https://wxzyd123.github.io/t-semattack.
AI-generated analysis. Check claims against the paper.
Summary
The paper identifies the speech tokenizer as a critical security bottleneck and proposes a transferable semantic attack (T-SemAttack) along with a robust tokenizer (ROSETok) to mitigate these vulnerabilities. The technical contribution is significant, combining a novel attack mechanism based on multi-encoder representation drift with a principled defense strategy involving semantic distillation, supported by extensive experiments across multiple ASR and LALM systems and rigorous human perceptual evaluations.
Methodology
The paper proposes T-SemAttack, a transferable semantic attack targeting the front-end speech tokenizer rather than the entire ASR/LALM pipeline. The core innovation is the use of a semantic encoder ensemble (S3Tokenizer, HuBERT, Whisper, MFCC) to jointly optimize perturbations that drift the representation space while preserving perceptual quality via a psychoacoustic masking constraint. This approach is methodologically sound, leveraging the shared structural bottlenecks (e.g., Mel-spectrogram inputs) of different encoders to achieve cross-model transferability. The proposed defense, ROSETok, employs a two-stage training strategy: base pretraining with noisy inputs and semantic distillation from a Whisper teacher, followed by attack-aware fine-tuning. The choice to target encoder-level semantic preservation rather than token-level invariance is well-justified by the analysis of how semantic drift, rather than mere token flipping, causes downstream failure.
Results & evaluation
The experimental evaluation is extensive and rigorous. The authors test the attack on 9 ASR systems (including black-box commercial APIs like GPT-4o-mini and Gemini) and 4 LALMs, demonstrating strong transferability. The inclusion of a human study (MOS evaluation) to verify perceptual quality is a significant strength, confirming that the attacks remain "harmless" to human listeners while being devastating to models. The evaluation of ROSETok shows clear improvements in reconstruction fidelity (STOI) and downstream semantic robustness (FSC scores) compared to baselines like DAC and StableToken. The ablation studies effectively isolate the contributions of the semantic loss, psychoacoustic constraint, and distillation components.
Reproducibility
The paper provides high reproducibility. Code and a demo are available. Detailed hyperparameters for both the attack (SGD steps, learning rate, perturbation budget) and the tokenizer training (batch size, learning rate schedules, loss weights) are specified. The datasets used (LibriTTS, Seed-TTS, Emilia) are standard or publicly available. The specific configuration of the surrogate encoders and the psychoacoustic model implementation details are described sufficiently for replication.
Limitations
The primary limitation is the computational cost of the attack, with an RTF of ~21.8, which is high for real-time applications, though the authors argue for an offline threat model. The defense (ROSETok) is evaluated primarily on Gaussian noise and the specific T-SemAttack; its robustness against other types of adversarial attacks or distribution shifts is not fully explored. The human study, while valuable, uses a small sample size (20 participants) and a limited set of utterances (4 speakers), which may not generalize to all demographic groups or acoustic conditions.
Broader impact
This work has significant implications for the safety and security of deployed speech AI systems. By identifying the tokenizer as a critical vulnerability point, it shifts the focus of security research from end-to-end model hardening to front-end robustness. The dual-use nature of the attack (potential for evading transcription or moderation) is acknowledged, but the defensive contribution (ROSETok) provides a constructive path forward. The findings are directly applicable to industry systems relying on speech tokenizers for LLM integration, urging a re-evaluation of their security posture. The paper identifies the speech tokenizer as a critical security bottleneck and proposes a transferable semantic attack (T-SemAttack) along with a robust tokenizer (ROSETok) to mitigate these vulnerabilities. The technical contribution is significant, combining a novel attack mechanism based on multi-encoder representation drift with a principled defense strategy involving semantic distillation, supported by extensive experiments across multiple ASR and LALM systems and rigorous human perceptual evaluations.
Can a speech generator also serve as an effective voice encoder? We show that a pretrained autoregressive text-to-speech (TTS) backbone can be repurposed for reference compression, improving speaker similarity in subsequent synthesis. Jointly encoding reference speech and its…
Abstract
Can a speech generator also serve as an effective voice encoder? We show that a pretrained autoregressive text-to-speech (TTS) backbone can be repurposed for reference compression, improving speaker similarity in subsequent synthesis. Jointly encoding reference speech and its transcript further strengthens this capability. Controlled ablations and activation interventions trace the text benefit through intermediate acoustic representations into the compressed vectors that condition generation. Building on these findings, we introduce RSE-TTS, whose Reconstructive Speaker Encoder (RSE) shares the generation backbone and compresses paired reference text and speech into eight continuous vectors. Sharing reduces encoder-generator parameters by 48% relative to independent pretrained encoding. Large-scale English training yields a 118M-parameter generator that improves both word error rate and speaker similarity over the corresponding official MOSS-TTS-Nano cloning modes on all three English benchmarks. Combined with acoustic continuation, RSE-TTS approaches the speaker similarity of substantially larger systems while achieving lower WER than the evaluated CosyVoice 3 and MOSS-TTS-Local models. Audio samples are available at https://voxnexusai.github.io/RSE-TTS/.
AI-generated analysis. Check claims against the paper.
Summary
The paper demonstrates that a pretrained TTS backbone can be effectively repurposed as a text-aware reference encoder, significantly improving speaker similarity in compact models. By sharing the generation backbone and jointly encoding reference text and speech, RSE-TTS achieves a 48% reduction in parameters while outperforming baseline compact systems and approaching the quality of much larger models, supported by rigorous mechanistic analyses of the encoding process.
Methodology
The paper proposes RSE-TTS, a compact text-to-speech system that repurposes a pretrained autoregressive TTS backbone as a reference encoder. The core methodological contribution is the "Reconstructive Speaker Encoder" (RSE), which shares weights with the generation backbone to compress paired reference text and speech into eight continuous vectors. This parameter-sharing strategy reduces the encoder-generator parameter count by 48% compared to independent encoders. The authors provide a rigorous mechanistic analysis using activation patching and gradient interventions to demonstrate that the reference text influences intermediate acoustic representations, which are then captured by the query vectors. The approach effectively bridges the gap between generative modeling and speaker encoding, leveraging the text-speech alignment learned during pretraining to improve voice cloning fidelity in compact models.
Results & evaluation
The experimental evaluation is comprehensive, utilizing three major English benchmarks (Seed-TTS, CV3-eval, MiniMax). The authors conduct controlled ablations to isolate the effects of pretrained initialization, reference text inclusion, and parameter sharing. Results show that RSE-TTS outperforms the official MOSS-TTS-Nano in both Word Error Rate (WER) and Speaker Similarity (SIM). Furthermore, when combined with acoustic continuation, the system approaches the performance of much larger models like CosyVoice 3 and MOSS-TTS-Local while maintaining a significantly smaller footprint (129M vs. 859M+ parameters). The inclusion of mechanistic studies (e.g., reversing transcript order, patching activations) adds depth to the empirical validation, confirming the causal role of text-aware encoding in voice reconstruction.
Reproducibility
The paper provides detailed descriptions of the model architecture, training objectives, and evaluation protocols. However, the primary training data is an "in-house English speech corpus," which limits full reproducibility. The authors do not release the code or the specific checkpoint used for the final results, though they reference the base MOSS-TTS-Nano model. The detailed hyperparameters and loss weighting schemes in the appendix aid in partial reproduction, but the lack of open-source code and proprietary data remains a barrier.
Limitations
The system requires a reference transcript for optimal performance, which may not be available in all real-world scenarios. The paper focuses exclusively on English, leaving generalizability to other languages unverified. Additionally, the reliance on an in-house dataset for the final large-scale training limits the ability for the community to verify the scaling claims independently. The "acoustic continuation" mode, while effective, increases inference latency and context window requirements, partially negating the efficiency gains of the compact encoder.
Broader impact
This work contributes to the development of efficient, high-fidelity voice cloning systems suitable for edge devices or real-time applications. By demonstrating that generative capabilities can be transferred to encoding tasks, it offers a new paradigm for designing compact multimodal systems. The mechanistic insights into how text influences speaker representations could inform future architectures for zero-shot TTS and voice conversion. The paper demonstrates that a pretrained TTS backbone can be effectively repurposed as a text-aware reference encoder, significantly improving speaker similarity in compact models. By sharing the generation backbone and jointly encoding reference text and speech, RSE-TTS achieves a 48% reduction in parameters while outperforming baseline compact systems and approaching the quality of much larger models, supported by rigorous mechanistic analyses of the encoding process.
Generating long-form speech requires models to keep content, speaker identity, turn-taking, and expressive style consistent over time. Yet most open training corpora contain examples lasting only tens of seconds and open long-form benchmarks cover at most two speakers per…
Abstract
Generating long-form speech requires models to keep content, speaker identity, turn-taking, and expressive style consistent over time. Yet most open training corpora contain examples lasting only tens of seconds and open long-form benchmarks cover at most two speakers per recording. We introduce Emilia2, a 229K-hour multilingual dataset for controllable long-form speech generation. From in-the-wild data, Emilia2 constructs short utterances, single-speaker long-form speech, and multi-speaker dialogues, preserving transcripts, speaker identities, pauses, and acoustic transitions. Examples longer than five minutes account for 50.0% of single-speaker long-form hours and 66.8% of dialogue hours. To support expressive control, Emilia2 provides natural language instructions for 4.77M examples and nonverbal-vocalization tags for 933K examples. We also introduce Emilia2-testset, 23 hours of bilingual recordings of up to 30 minutes and four speakers, each speaker paired with reference audio. We train autoregressive (AR) and non-autoregressive (NAR) models on Emilia2. Our 0.6B-backbone NAR model achieves the lowest Chinese CER and English WER on TTSD-Eval (5.27%/9.93%) among compared systems, while our AR model is on par with Qwen3-TTS-12Hz-VD on InstructTTSEval (zh/en: 76.2/78.3 vs. 77.1/77.9). These results validate Emilia2 as a comprehensive data foundation for long-form speech generation and minute-scale expressive control. Demo: https://charlesnii.github.io/Emilia2-demo/.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces Emilia2, a 229K-hour multilingual dataset and benchmark for controllable long-form speech generation, demonstrating that models trained on this data achieve state-of-the-art performance in both long-form consistency and expressive control.
Methodology
The paper proposes a comprehensive data construction pipeline, Emilia2, that transforms in-the-wild audio into a structured dataset for long-form speech generation. The methodology is robust, utilizing a multi-stage process involving audio standardization, speaker diarization (DiariZen), VAD (FireRedVAD), and quality filtering (DNSMOS). A key methodological strength is the preservation of temporal continuity and acoustic transitions in long-form examples, rather than concatenating trimmed segments, which is crucial for modeling natural pauses and turn-taking. The integration of LLM-based annotation (Qwen3-Omni) for style instructions and specialized ASR for nonverbal vocalizations (NV) adds significant value for controllable generation. The training of both Autoregressive (AR) and Non-Autoregressive (NAR) models on this data validates the dataset's versatility.
Results & evaluation
The experimental evaluation is extensive, covering long-form generation (TTSD-Eval, Long-TTS-Eval, Emilia2-testset) and short-form controllable generation (InstructTTSEval, NV-Bench). The introduction of Emilia2-testset, featuring up to 30-minute recordings with up to 4 speakers and per-speaker reference audio, addresses a critical gap in existing benchmarks. The results show that the 0.6B backbone models trained on Emilia2 outperform larger baselines (e.g., 3B Higgs Audio v2) in objective metrics like WER/CER and speaker similarity. Subjective evaluations (MOS) further support the naturalness and control capabilities. The comparison against state-of-the-art systems like Qwen3-TTS and MOSS-TTSD is relevant and competitive.
Reproducibility
The paper provides detailed descriptions of the pipeline components (specific models used for diarization, ASR, etc.) and the training curriculum. However, the source data is described as "in-the-wild" from various internet sources, which may raise questions about licensing and exact reproducibility of the raw data collection. The release of the dataset and testset is a major positive for reproducibility. The code for the models is not explicitly linked in the provided text, but the demo page is available.
Limitations
The primary limitation is the reliance on in-the-wild data, which may contain inherent biases or quality issues despite filtering. The annotation of instructions and NV tags is limited to segments shorter than 5 minutes, which might restrict the granularity of control for very long, continuous expressive shifts. Additionally, the paper focuses heavily on Chinese and English for NV annotations, with other languages potentially having less expressive control support.
Broader impact
This dataset has significant potential to advance the field of long-form speech synthesis, enabling more realistic audiobooks, podcasts, and conversational agents. By providing a standardized benchmark with multi-speaker and long-duration support, it facilitates fair comparison of new architectures. The focus on controllable generation (style, NV) opens avenues for more expressive and human-like AI interactions. The paper introduces Emilia2, a 229K-hour multilingual dataset and benchmark for controllable long-form speech generation, demonstrating that models trained on this data achieve state-of-the-art performance in both long-form consistency and expressive control.
Full-duplex spoken dialogue, in which a system listens while it speaks and can backchannel, yield the floor, and be interrupted, is usually obtained by training on hundreds of thousands to millions of hours of speech. This is costly, rewrites the language model that supplies the…
Abstract
Full-duplex spoken dialogue, in which a system listens while it speaks and can backchannel, yield the floor, and be interrupted, is usually obtained by training on hundreds of thousands to millions of hours of speech. This is costly, rewrites the language model that supplies the system's knowledge, and still leaves turn-taking either implicit or decided coarsely rather than word by word. We show that a frozen, general-purpose multimodal LLM can instead be adapted into a word-level full-duplex system with little data. Our system is a thinker-talker cascade on an 80 ms frame clock. The thinker keeps the LLM frozen and trains only lightweight adapters and a few new token embeddings (97M parameters); at every frame it emits one token of agent text, with pauses, yields, and onsets represented as tokens of the same vocabulary and kept legal by a finite-state machine. A four-stage curriculum, each stage anchored to the model's own behavior under the previous format, performs the conversion in about 30 GPU-hours. The talker reads only the thinker's discrete tokens, so the two halves are trained independently. On Full-Duplex-Bench and InteractionBench, the system is competitive with open-source full-duplex systems built through large-scale speech training: among them, it is the least likely to take the floor when the user merely pauses, backchannels most often, and leads in spoken-task accuracy, groundedness, and coherence over long conversations.
AI-generated analysis. Check claims against the paper.
Summary
The paper presents a data-efficient framework for adapting frozen multimodal LLMs into full-duplex spoken dialogue systems using a thinker-talker cascade and finite-state turn-taking, achieving competitive performance with large-scale models at a fraction of the training cost.
Methodology
The paper proposes a highly efficient adaptation strategy for converting a frozen, general-purpose multimodal LLM (Qwen3-Omni) into a full-duplex spoken dialogue system. The core methodological contribution is the "thinker-talker" cascade operating on an 80ms frame clock. The "thinker" retains the LLM backbone frozen and trains only lightweight adapters (LoRA) and new token embeddings for timing controls (pauses, yields, onsets). A key innovation is the finite-state machine (FSM) that masks illegal tokens during sampling, ensuring that turn-taking decisions (represented as tokens in the same vocabulary as words) remain structurally valid. The "talker" is a separate streaming speech synthesis model that consumes only the discrete text tokens from the thinker, allowing independent training. The training curriculum is a four-stage process, each stage anchored to the model's previous behavior via self-distillation or KL divergence, requiring only ~30 GPU-hours. This approach effectively decouples the high-level dialogue logic (which remains in the frozen LLM) from the low-level acoustic generation and timing, avoiding the catastrophic forgetting and high data costs associated with full fine-tuning of speech models.
Results & evaluation
The system is evaluated on Full-Duplex-Bench (v1.0 and v1.5) and InteractionBench. The results show that the proposed system is competitive with large-scale open-source systems like Moshi and PersonaPlex, which require millions of hours of speech training. Specifically, the system excels in pause handling (least likely to take the floor when the user pauses) and backchannel frequency. It also leads in spoken-task accuracy, groundedness, and coherence on InteractionBench. However, it lags behind some baselines in responsiveness (latency) and whisper/volume control. The ablation study on noise robustness is particularly strong, demonstrating that training the channel-fusion layer and audio adapter is critical for handling real-world room tone, providing a clear mechanistic understanding of the model's robustness.
Reproducibility
The paper provides extensive implementation details, including specific hyperparameters, learning rates, batch sizes, and architectural dimensions for both the thinker and talker. The training data sources (synthetic dialogues, specific hours of audio) and the four-stage curriculum are described in detail. The use of standard components like Mimi codec and LoRA enhances reproducibility. However, the reliance on a specific proprietary multimodal LLM (Qwen3-Omni) and the specific synthetic data generation pipeline may limit immediate reproducibility for groups without access to these specific assets or the compute resources to generate similar synthetic data.
Limitations
The primary limitation is the dependence on a pre-trained, general-purpose multimodal LLM that already understands speech; the paper does not claim to learn full-duplex behavior from scratch with small data. The talker is currently trained only on synthesized speech and does not utilize the user's audio channel for voice conditioning (it is zeroed out), limiting its ability to adapt to user-specific acoustic characteristics or control volume/whispering effectively. The evaluation benchmarks, while standard, have relatively small sample sizes for some behaviors, leading to high variance in some metrics. The backchannel training data is small (701 unique dialogues), raising questions about generalization.
Broader impact
This work significantly lowers the barrier to entry for developing full-duplex spoken dialogue systems. By demonstrating that a frozen LLM can be adapted with minimal data and compute, it makes advanced conversational AI more accessible to smaller research groups and companies. The discrete token interface between thinker and talker offers a modular design that could be extended to other modalities or tasks. The explicit modeling of turn-taking via FSM provides a transparent and controllable mechanism for dialogue management, which is valuable for safety and user experience in real-world applications. The paper presents a data-efficient framework for adapting frozen multimodal LLMs into full-duplex spoken dialogue systems using a thinker-talker cascade and finite-state turn-taking, achieving competitive performance with large-scale models at a fraction of the training cost.
Answering spatial questions about multi-source scenes requires a model both to identify each sound and to locate it. Existing spatial audio-language models obtain spatial information from a dedicated encoder, which separates spatial and semantic processing into distinct feature…
Abstract
Answering spatial questions about multi-source scenes requires a model both to identify each sound and to locate it. Existing spatial audio-language models obtain spatial information from a dedicated encoder, which separates spatial and semantic processing into distinct feature spaces. We study whether a general-audio codec trained without spatial supervision can serve both purposes. A controlled layer-wise analysis shows that the frozen codec retains spatial information, with lateral cues carried by the difference between the two ear channels. This information, however, is not readily accessible: in dual-source scenes, a model trained on these features predicts left/right at chance and, on the remaining axes, assigns the detected direction to the queried and the non-queried source equally often. We attribute this failure to an audio--text information asymmetry, in which the audio contains both sources while the answer describes only the queried one. A controlled $2\times2$ experiment shows that either completing the scene description or exposing the inter-channel difference lifts left/right accuracy above chance, whereas binding directions to the queried source requires both. Building on these findings, AURA keeps the codec and the language model frozen and exceeds the strongest dedicated system on SpatialSoundQA by 10.31 points in dual-source direction accuracy, with about 40\% of its trainable parameters.
AI-generated analysis. Check claims against the paper.
Summary
The paper demonstrates that a frozen general-audio codec can support spatial reasoning by addressing audio-text information asymmetry through explicit inter-channel features and comprehensive scene supervision, achieving superior dual-source localization performance with reduced parameter count.
Methodology
The paper proposes AURA, a framework for spatial audio understanding that leverages a frozen general-audio codec (X-Codec) rather than a dedicated spatial encoder. The methodology is grounded in a rigorous layer-wise analysis of the codec's representations, identifying that spatial cues (specifically lateral cues) are preserved in deeper RVQ layers and are best accessed via inter-channel differences. The core innovation is the "information asymmetry" hypothesis, which posits that standard QA training fails because the audio contains multiple sources while the text target describes only one. To address this, AURA introduces two components: (1) a representation-side intervention adding explicit inter-channel difference slots to the audio tokens, and (2) a text-side intervention using "scene supervision" where the model must describe all sources before answering. The approach is technically sound, effectively bridging the gap between general audio representations and specific spatial reasoning tasks without retraining the heavy backbone models.
Results & evaluation
The experiments are well-controlled, particularly the 2x2 ablation study that isolates the effects of scene supervision and inter-channel differences. The results on SpatialSoundQA show a significant improvement (10.31 points) over the strongest dedicated system in dual-source direction accuracy, which is a strong claim given the parameter efficiency (40% of trainable parameters). The evaluation correctly focuses on the challenging dual-source scenarios where existing models fail. However, the reliance on a single benchmark (SpatialSoundQA) and the lack of evaluation on other spatial datasets (like BiDepth or real-world recordings) slightly limits the generalizability of the findings.
Reproducibility
The paper provides detailed descriptions of the architecture modifications (6-slot representation, LoRA modules) and the training strategy. However, specific hyperparameters, learning rates, and exact implementation details for the "geometric supervision" prompt construction are not fully detailed in the provided text. The use of standard components (Qwen3-8B, X-Codec) aids reproducibility, but the specific prompt engineering for scene supervision is a critical component that requires careful replication.
Limitations
The primary limitation is the dependence on simulated binaural data (SpatialSoundQA) for training and evaluation. It is unclear how well the model generalizes to real-world binaural recordings with complex noise and reverberation. Additionally, the model is frozen, which means it cannot adapt to new spatial concepts or room types beyond what the codec has seen. The "scene supervision" approach increases the length of the model's output, which may have computational implications for real-time applications.
Broader impact
This work has significant implications for the development of efficient spatial audio models. By demonstrating that general audio codecs retain spatial information, it challenges the necessity of dedicated spatial encoders, potentially simplifying the architecture of future audio-language models. The insights into information asymmetry in multi-source scenes are applicable to other multimodal tasks where the input contains more information than the target output. This could influence how datasets are constructed and how models are trained for complex scene understanding. The paper demonstrates that a frozen general-audio codec can support spatial reasoning by addressing audio-text information asymmetry through explicit inter-channel features and comprehensive scene supervision, achieving superior dual-source localization performance with reduced parameter count.
We present Walrus, a unified 8B speech-and-audio foundation model handling speech recognition (including code-switching and multi-talker speech), speech-to-text and speech-to-speech translation, text-to-speech, emotion recognition, audio captioning, and general audio…
Abstract
We present Walrus, a unified 8B speech-and-audio foundation model handling speech recognition (including code-switching and multi-talker speech), speech-to-text and speech-to-speech translation, text-to-speech, emotion recognition, audio captioning, and general audio understanding. Our main contribution is not a new architecture but a data-centric training recipe; we use a standard decoder-only backbone initialized from Gemma 4, so that gains can be traced to the data. For speech generation, Walrus predicts low-bitrate discrete speech tokens, and a lightweight token-to-waveform decoder converts them into high-fidelity audio. The recipe is a three-stage data curriculum that moves from broad multilingual data to multi-talker, code-switching, and domain-specific data. At each stage, task-aware and dataset-aware weighting balances tasks with very different token budgets and protects core skills from forgetting. Trained on over 600M samples covering more than 20 tasks and 100 languages, Walrus leads on most multilingual, multi-talker, and code-switching ASR, speech translation, and audio understanding benchmarks, outperforming both open speech language models of comparable size and dedicated task-specific models. On most benchmarks, it even matches or surpasses the $4\times$ larger Qwen3-Omni's performance. Results suggest that data engineering is a primary driver of quality in multi-task speech and audio modeling.
AI-generated analysis. Check claims against the paper.
Summary
The main contribution is a data-centric training recipe for a unified 8B speech-and-audio foundation model that achieves state-of-the-art results across diverse tasks without architectural novelty. By fixing the backbone and focusing on a three-stage data curriculum with task-aware weighting, the paper demonstrates that data engineering is a primary driver of quality in multi-task speech modeling, outperforming larger models and dedicated task-specific systems on key benchmarks for ASR, translation, and audio understanding.
Methodology
The paper proposes a data-centric approach to training a unified speech-and-audio foundation model, explicitly arguing that data engineering is more critical than architectural novelty. The methodology relies on a standard decoder-only Transformer backbone (Gemma 4 E4B) initialized with pre-trained weights, keeping the architecture fixed to isolate the impact of the data recipe. The core innovation lies in a three-stage data curriculum: Stage 1 establishes a broad multilingual foundation with high weights for ASR; Stage 2 introduces complex tasks like multi-talker ASR, code-switching, and timestamp prediction while reducing weights of foundation data to prevent catastrophic forgetting; Stage 3 focuses on domain-specific and specialized data. A key technical contribution is the use of task-aware and dataset-aware weighting to balance the disparate token budgets of generative tasks (TTS/S2ST) versus discriminative tasks (ASR), preventing generative tasks from dominating the loss. For speech generation, the model predicts low-bitrate discrete tokens (325 bps) using a VQ tokenizer trained on W2v-BERT features, which are then decoded into waveforms by a GAN-based vocoder conditioned on source speaker features for identity preservation.
Results & evaluation
The experimental evaluation is extensive, covering over 20 tasks and 100 languages. Walrus demonstrates state-of-the-art performance on multilingual ASR (VoxPopuli, FLEURS), code-switching ASR (CS-FLEURS), and multi-talker ASR (LibriMix, Daily Talk), significantly outperforming comparable open models and even matching or surpassing the much larger Qwen3-Omni (30B) on several benchmarks. Notably, it achieves a cpWER of 19.32 on LibriMix clean 2-mix, which is half the error rate of Qwen3-Omni. The paper also evaluates speech translation (S2ST) and audio understanding (MMSU, MMAU-Pro), showing strong results. The comparison between the Commercial-Ready (CR) and Research (R) variants provides insight into the impact of data licensing and quality, with the R variant showing consistent improvements across all benchmarks.
Reproducibility
The paper provides a high level of detail regarding the training setup, including optimizer hyperparameters, batch sizes, and the specific data weighting schemes for each stage. It lists the specific public datasets used (CommonVoice, VoxPopuli, LibriSpeech, etc.) and describes the synthetic data generation pipelines for code-switching and multi-talker scenarios. However, as is common with large-scale industrial models, the exact composition of the "internal data" and the specific implementation details of the proprietary data cleaning pipelines are not fully disclosed. The code and model weights are available via the provided GitHub link, which enhances reproducibility for those with access to the resources.
Limitations
The primary limitation is the reliance on a fixed backbone (Gemma 4 E4B), which may limit the model's scalability compared to architectures designed specifically for multimodal alignment. The paper does not provide a detailed ablation study isolating the contribution of each individual stage of the curriculum or the specific impact of the task-weighting strategy versus the curriculum order alone. Additionally, the evaluation of audio understanding is somewhat limited compared to the extensive ASR and translation results, and there is no subjective evaluation (e.g., MOS) for the generated speech, relying instead on objective metrics and comparisons to baselines.
Broader impact
This work has significant implications for the deployment of unified speech models in real-world applications, demonstrating that a single model can handle a wide range of tasks (transcription, translation, generation, understanding) efficiently. The emphasis on data engineering provides a roadmap for other researchers and companies to improve their speech models without necessarily increasing model size or complexity. The availability of a commercial-ready variant trained on licensed data is particularly valuable for industry adoption. The results suggest that careful data curation and mixing can yield performance gains comparable to or exceeding those from architectural changes, potentially shifting research focus towards data-centric methods in speech AI. The main contribution is a data-centric training recipe for a unified 8B speech-and-audio foundation model that achieves state-of-the-art results across diverse tasks without architectural novelty. By fixing the backbone and focusing on a three-stage data curriculum with task-aware weighting, the paper demonstrates that data engineering is a primary driver of quality in multi-task speech modeling, outperforming larger models and dedicated task-specific systems on key benchmarks for ASR, translation, and audio understanding.