In this report, we present Qwen-Audio-3.0-TTS, a production-oriented speech synthesis system that jointly advances content consistency, speaker similarity, prosodic naturalness, audio quality, controllability, multilingual coverage, efficiency, and robustness. It combines a 12.5~Hz low-frame-rate speech tokenizer for reduced inference latency with a five-stage progressive training paradigm for coordinated language model (LM) and flow-matching model (FM) optimization. The model provides production-level control through free-style natural-language instructions and fine-grained inline tags, while supporting 16 languages, 20 Chinese dialect regions, one-pass long-form synthesis up to 3 minutes, and robust generation from noisy, reverberant, or unclear reference speech. Across SEED-TTS-Eval, CV3-Eval, instruction-following, long-form, and acoustic-robustness evaluations, Qwen-Audio-3.0-TTS achieves state-of-the-art performance on many reported dimensions or the strongest aggregate results. It also ranks first on the independent Artificial Analysis Text-to-Speech Leaderboard. These results establish Qwen-Audio-3.0-TTS as a strong foundation for production-level speech synthesis.
Primary: Alibaba Group
All Institutions: Alibaba Group
[One sentence main contribution]. Qwen-Audio-3.0-TTS introduces a five-stage progressive training paradigm with joint LM-FM optimization and reinforcement learning to achieve state-of-the-art production-level speech synthesis with robustness and fine-grained control. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a significant technical contribution by systematically addressing the trade-offs between efficiency, fidelity, and controllability in TTS. The integration of GRPO for both LM and FM optimization is a novel application of RLHF techniques to speech generation, moving beyond simple supervised fine-tuning. The explicit focus on acoustic robustness through training-time augmentation is highly relevant for real-world deployment. The comprehensive evaluation suite, including new benchmarks for robustness and instruction following, sets a high standard for future TTS research. The results demonstrate that a unified model can outperform specialized systems in multiple dimensions, marking a step towards more general-purpose speech generation models.
The paper presents Qwen-Audio-3.0-TTS, a production-oriented text-to-speech system that integrates a 12.5 Hz speech tokenizer, a language model (LM), and a flow-matching model (FM). The core methodological contribution lies in a five-stage progressive training paradigm. Key innovations include: 1) Conditioning the FM on continuous LM hidden states rather than just discrete tokens to mitigate information bottleneck; 2) High-quality data annealing during joint LM-FM training; 3) Application of Group Relative Policy Optimization (GRPO) to the LM for content/prosody rewards and to the FM for speaker similarity/perceptual quality rewards; 4) Explicit acoustic robustness training using degraded prompts (noise, reverb, etc.) without separate denoising modules. The architecture combines autoregressive semantic planning with continuous acoustic generation, aiming to balance latency, fidelity, and controllability.
The evaluation is comprehensive and rigorous, covering multiple dimensions: zero-shot voice cloning (SEED-TTS-Eval, CV3-Eval), multilingual/cross-lingual synthesis, instruction following (VStyle), long-form generation, text normalization, and acoustic robustness. The paper provides extensive objective metrics (CER/WER, SIM, DNSMOS) and subjective human evaluations (Arena-style, dialect authenticity). Results indicate state-of-the-art performance on many reported dimensions, particularly in robustness and instruction following, while maintaining competitive content consistency and speaker similarity. The inclusion of a dedicated "hard-case" robustness benchmark and detailed ablation studies on tokenizer frame rates adds significant value.
The paper provides detailed descriptions of the architecture, training stages, and data types. It mentions specific models used for evaluation (Whisper, Paraformer, ERes2Net, DNSMOS) and provides a demo URL. However, full code and model weights are not explicitly stated as open-source in the provided text (though the project URL points to a GitHub page, the text itself focuses on the report). The detailed training recipe (5 stages, specific rewards) is described well enough for reproduction by experts in the field.
The paper acknowledges that pushing content accuracy lower can come at the expense of naturalness, suggesting a trade-off. The robustness training, while effective, relies on specific augmentation strategies that may not cover all real-world edge cases. The reliance on LLM-based evaluators (Gemini) for some subjective metrics introduces potential bias, although human evaluation is also provided. The "high-quality data annealing" requires careful curation, which may limit scalability if high-quality labeled data is scarce.
This work advances the field of speech synthesis by providing a robust, controllable, and efficient system suitable for production deployment. It addresses critical pain points like robustness to noisy prompts and fine-grained control, which are essential for real-world applications. The open-sourcing of such models (implied by the project URL) can democratize access to high-quality TTS technology. However, the ease of voice cloning also raises concerns about misuse for deepfakes, necessitating responsible AI practices. [One sentence main contribution]. Qwen-Audio-3.0-TTS introduces a five-stage progressive training paradigm with joint LM-FM optimization and reinforcement learning to achieve state-of-the-art production-level speech synthesis with robustness and fine-grained control. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a significant technical contribution by systematically addressing the trade-offs between efficiency, fidelity, and controllability in TTS. The integration of GRPO for both LM and FM optimization is a novel application of RLHF techniques to speech generation, moving beyond simple supervised fine-tuning. The explicit focus on acoustic robustness through training-time augmentation is highly relevant for real-world deployment. The comprehensive evaluation suite, including new benchmarks for robustness and instruction following, sets a high standard for future TTS research. The results demonstrate that a unified model can outperform specialized systems in multiple dimensions, marking a step towards more general-purpose speech generation models.
Automatic analysis of multimodal speech has shown strong potential for computationally detecting and monitoring a wide range of neurological, psychiatric, and respiratory conditions. However, progress in this field is limited by existing publicly accessible datasets, which are often small in scale, focused on a single condition or disease, and primarily speech focused. Moreover, if key confounding variables such as education, medication use, comorbidities, or mood state are insufficiently documented, the reliability and interpretability of computational analyses are further compromised. To address these limitations, we introduce CARE v1.0, a curated multimodal English dataset of approximately 144 hours of short video interviews collected from 612 individuals across 12 medical conditions plus a control cohort. For each video, a comprehensive set of clinically relevant multimodal descriptors is provided, alongside structured metadata covering factors such as medication, life impacts, and expressed emotions. The corpus's breadth and heterogeneity support a wide range of applications, including automatic disease and symptom detection, multimodal modelling of speech and non-verbal behaviour under emotionally charged contexts, and studies of disease trajectories and coping processes.
Primary: Universitat Politècnica de València
All Institutions: Universitat Politècnica de València, University of Oxford, Instituto de Telecomunicaçþes
This paper presents CARE v1.0, a comprehensive multimodal corpus of 144 hours of video interviews from 612 individuals across 12 medical conditions, enriched with automatically extracted multimodal descriptors and structured metadata, addressing critical gaps in scale, diversity, and clinical context in existing digital health datasets.
The paper introduces CARE v1.0, a large-scale multimodal corpus derived from the HEXI platform. The methodology focuses on data curation and feature engineering rather than novel algorithmic development. The authors employ a rigorous pipeline for extracting multimodal descriptors: audio features (eGeMAPS, DisVoice, XLS-R, TRILLsson), visual features (OpenFace for facial landmarks/AUs, gaze, head pose, MediaPipe for body keypoints), and linguistic features (WhisperX alignment, sentence embeddings). A key methodological contribution is the use of LLMs (gpt-oss-120b) to extract structured metadata (demographics, emotions, comorbidities) from unstructured narrative texts, validated against a manually annotated subset. This approach addresses a significant gap in existing datasets by providing rich, clinically relevant metadata alongside raw multimodal signals.
The experimental section is primarily descriptive, focusing on the statistical properties of the dataset (distribution of conditions, demographics, emotion frequencies) and the validation of the LLM-based metadata extraction pipeline. The authors compare five LLMs for metadata extraction, demonstrating that gpt-oss-120b performs best in terms of accuracy and adherence to constraints. They provide a baseline evaluation workflow for a binary classification task (Control vs. Parkinson's) using an SVM on aggregated features. While this demonstrates the utility of the dataset, the lack of extensive benchmarking against state-of-the-art multimodal models limits the assessment of the dataset's ability to drive performance improvements in specific tasks. The validation of the metadata extraction is solid, but the downstream ML performance is only illustrative.
The paper excels in reproducibility. The dataset is hosted on Hugging Face, and the code for feature extraction and the example evaluation workflow is publicly available. The detailed documentation of the feature extraction pipeline, including specific tools (OpenFace, DiariZen, WhisperX) and parameter settings, allows other researchers to replicate the feature computation. The use of deterministic settings for LLM inference (temperature=0) further ensures reproducibility of the metadata annotations.
The dataset is skewed towards White UK/Irish participants, limiting its generalizability to other ethnic groups. The control group is heterogeneous (caregivers, bereaved, professionals), which may introduce confounding variables if not carefully handled in downstream analyses. The reliance on LLMs for metadata extraction, while validated, introduces potential biases and hallucinations (though rare, as noted). The dataset is cross-sectional, limiting longitudinal studies of disease trajectories, although the authors mention this as a potential future application. The "in-the-wild" nature of some video clips (varying quality, lighting) may pose challenges for robust computer vision models.
CARE v1.0 has significant potential impact on the field of digital health and affective computing. By providing a large, multimodal, and clinically annotated dataset, it enables research into automated detection and monitoring of a wide range of neurological, psychiatric, and respiratory conditions. The inclusion of diverse medical conditions and rich metadata supports the development of more robust and interpretable computational models. The dataset can facilitate studies on the interplay between speech, non-verbal behavior, and emotional states in health contexts, contributing to a better understanding of disease manifestations and coping processes. This paper presents CARE v1.0, a comprehensive multimodal corpus of 144 hours of video interviews from 612 individuals across 12 medical conditions, enriched with automatically extracted multimodal descriptors and structured metadata, addressing critical gaps in scale, diversity, and clinical context in existing digital health datasets.
Recent generative models are moving beyond silent video or standalone audio synthesis toward the joint generation of synchronized audio and video. Despite this progress, jointly generating audio and video with fine-grained cross-modal correspondence remains challenging due to their fundamental structural differences. Most existing methods use audio and video VAEs trained separately. As a result, the two latent spaces lack cross-modal alignment, leaving the downstream generative model to learn cross-modal synchronization from scratch. We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations. Beyond reconstruction, OmniVAE uses a segment-level audio-video contrastive objective to capture temporal-semantic correspondence and align the two latent spaces. In parallel, it distills features from pretrained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces. Extensive experiments show that both objectives consistently improve the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation. These findings underscore the importance of learning unified representations as a foundation for omnimodal modeling.1
Primary: Fudan University
All Institutions: Fudan University, Shanghai Innovation Institute, MOSI Intelligence, Shanghai Jiao Tong University
OmniVAE introduces a novel jointly trained audio-video VAE with segment-level contrastive alignment and semantic distillation, significantly improving cross-modal synchronization and generation quality in text-to-audio-video tasks.
The paper proposes OmniVAE, a jointly trained audio-video Variational Autoencoder (VAE) designed to address the lack of cross-modal alignment in standard modality-specific tokenizers. The core methodological contribution lies in two training-only objectives added to standard reconstruction losses: (1) Segment-level Audio-Video Contrastive Learning (AVCLIP), which aligns audio and video latents at a fine temporal granularity using a hierarchical negative sampling strategy (intra-clip, sibling-clip, cross-video), and (2) Modality-Specific Semantic Distillation, which injects semantic structure from pretrained Qwen3-Omni encoders into each latent space independently. The approach is technically sound, leveraging established contrastive learning frameworks (InfoNCE) and distillation techniques (iREPA-style) but applying them specifically to the joint audio-video latent space. The three-stage training schedule (pretrain reconstruction -> joint contrastive/distillation -> adversarial refinement) is a practical engineering solution to handle the conflicting optimization dynamics of reconstruction and alignment.
The experimental evaluation is comprehensive and well-structured. The authors conduct extensive ablations to isolate the contributions of reconstruction, distillation, and contrastive learning. They employ a novel "sync probing" task (predicting temporal offset) as a proxy for alignment, which provides strong quantitative evidence that the latent spaces are indeed better aligned than baselines. Downstream text-to-audio-video generation experiments on Verse-Bench show consistent improvements in cross-modal alignment metrics (DeSync, LSE-C) and generation quality (AudioBox, IS, FAD) when using OmniVAE. The use of a large-scale, curated dataset (23M clips) with LLM-generated captions adds significant weight to the empirical claims. The reconstruction quality remains competitive with state-of-the-art independent VAEs, demonstrating that alignment does not come at the cost of fidelity.
The paper provides detailed architectural descriptions, loss formulations, and training schedules. The parameter counts are explicitly listed, distinguishing between inference and training-only modules. The dataset construction pipeline is described, though the specific filtering thresholds and LLM prompts might vary. The reliance on proprietary or large pretrained models (Qwen3-Omni, Wan2.2, HunyuanVideo-Foley) for initialization and distillation is noted, which aids reproducibility for those with access to these resources. The code is not explicitly linked in the text, but the methodology is described with sufficient detail for replication.
The primary limitation is that the alignment is learned during training only; the model does not enforce a shared latent distribution, meaning the latents are still modality-specific but aligned in a shared embedding space via the contrastive head. This might limit direct cross-modal generation without additional mapping layers. The sync probing metric, while useful, is a proxy and may not perfectly correlate with human perception of synchronization in complex scenes. The dataset, while large, is web-scraped and may contain noise despite filtering. The reliance on Qwen3-Omni for distillation assumes the availability of this specific large model, which might not be accessible to all researchers.
OmniVAE contributes to the field of unified multimodal generation by providing a stronger foundational representation for audio-video tasks. By decoupling the alignment learning from the generative model, it allows downstream generators to focus on content creation rather than learning synchronization from scratch. This could accelerate the development of high-quality, synchronized audio-video generative models for entertainment, education, and simulation. The emphasis on fine-grained temporal alignment addresses a key challenge in current generative AI, potentially leading to more realistic and usable synthetic media. OmniVAE introduces a novel jointly trained audio-video VAE with segment-level contrastive alignment and semantic distillation, significantly improving cross-modal synchronization and generation quality in text-to-audio-video tasks.
Spoken dialog systems are typically designed for clean, dyadic interactions in which a single user and an assistant take turns speaking. Real-world social conversations, however, are often more ambiguous: multiple speakers may participate in the same conversation amid irrelevant speech and background noise. Each utterance may be directed to the assistant, addressed to another speaker, or completely irrelevant. In such settings, the assistant must decide not only what to say, but also whether to speak at all. In this paper, we introduce Cocktail-Talker, a speech LLM framework for multi-speaker spoken dialog modeling in noisy social environments. We model the assistant's behavior with three action tokens: <|respond|>, <|listen|>, and <|ignore|>, placed before a response or silence. Cocktail-Talker is trained via supervised finetuning and reinforcement learning to generate the appropriate action token and, only in <|respond|> mode, a speech response. To prepare the training data, we develop Cocktail-DialogGen, an LLM-based data pipeline that simulates realistic multi-speaker dialogs with speaker roles across diverse social settings. Together, these components take a step toward spoken dialog systems that interact more naturally and selectively in complex social environments.
Primary: Columbia University
All Institutions: Columbia University, University of Illinois Urbana-Champaign
Cocktail-Talker introduces a novel framework for multi-speaker spoken dialog by modeling turn-taking as a discrete action selection problem, trained with GRPO on synthetically generated data, achieving state-of-the-art performance in noisy, multi-speaker environments. The paper makes a significant technical contribution by addressing the under-explored "speaking side" of multi-speaker conversation, moving beyond simple dyadic interactions to more complex social auditory scenes.
The paper proposes "Cocktail-Talker," a framework for multi-speaker spoken dialog in noisy environments. The core methodological contribution is the introduction of three discrete action tokens (`<|respond|>`, `<|listen|>`, `<|ignore|>`) to model turn-taking behavior, trained via Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO). The approach leverages a pre-existing speech LLM backbone (Qwen2.5-Omni) and freezes the audio encoder and TTS decoder, focusing training on the language model head and LoRA adapters. A significant portion of the work involves "Cocktail-DialogGen," an LLM-based pipeline to synthesize training data by simulating multi-speaker conversations with varying acoustic conditions (SNR, background noise) and semantic constraints (anonymous vs. named speakers). The use of GRPO with a reward function based on action accuracy and format integrity is a novel application of RLHF techniques to the specific problem of turn-taking decision-making in speech LLMs.
The evaluation is comprehensive, covering both seen and unseen environments. The paper compares Cocktail-Talker against several strong baselines, including Moshi, PersonaPlex, Step-Audio2, Kimi-Audio, Qwen2.5-Omni, and Qwen3-Omni. The metrics include binary response accuracy (Respond/Silent) and response quality (METEOR, ROUGE-L, BERTScore, SentCos). The results show that Cocktail-Talker significantly outperforms baselines in decision accuracy (macro F1 ~0.93) and response quality. The analysis breaks down performance by SNR, number of speakers, anonymity, and context length, providing robust insights into model behavior. The inclusion of "unseen environments" tests generalization, which is crucial for real-world applicability. The ablation studies (SFT vs. SFT+GRPO) demonstrate the benefit of the reinforcement learning stage.
The paper provides a GitHub link to the code. The data generation pipeline is described in detail, including the use of specific LLMs (Gemini 3 Pro, Qwen3-TTS) and noise datasets (DEMAND, Freesound). However, the reliance on proprietary models (Gemini, Qwen) for data generation and the specific random seeds or exact prompts used in the LLM-based data synthesis might pose reproducibility challenges for other researchers. The hyperparameters for SFT and GRPO are provided, which aids in reproducing the model training.
The paper acknowledges several limitations. The system is not streaming, which limits its real-time applicability. It assumes fixed turn boundaries in the simulation, which may not reflect the fluid nature of real-world interruptions. The model relies heavily on the simulated data, and while it generalizes to unseen environments, the distribution shift between synthetic and real-world cocktail party scenarios remains a potential issue. The performance degrades with increased speaker count and anonymity, indicating room for improvement in complex social reasoning.
This work contributes to the development of more natural and robust speech assistants that can operate in real-world social settings. By enabling selective participation, it reduces the annoyance of assistants interrupting irrelevant conversations or responding to noise. This has implications for the deployment of AI agents in homes, public spaces, and social robotics. However, the potential for misuse in surveillance or eavesdropping scenarios should be considered, although the model is designed to ignore irrelevant speech. Cocktail-Talker introduces a novel framework for multi-speaker spoken dialog by modeling turn-taking as a discrete action selection problem, trained with GRPO on synthetically generated data, achieving state-of-the-art performance in noisy, multi-speaker environments. The paper makes a significant technical contribution by addressing the under-explored "speaking side" of multi-speaker conversation, moving beyond simple dyadic interactions to more complex social auditory scenes.
Audio-capable foundation models enable end-to-end spoken interaction, but they also introduce safety risks beyond transcript content. It remains unclear how much jailbreak capability can arise from matched-text variation in speech delivery rather than from lexical rewriting or broader style transfer. We study this question by holding transcript content fixed and varying six speech-delivery presets whose acoustic attributes may co-vary. We present PJ-Break, a black-box evaluation protocol with presets targeting arousal, authority, and speaking rate, together with AdvAudio-Prosody, a 600-sample benchmark with acoustically verified attributes. On the exact post-QC Qwen2-Audio panel, the Q=1 Panic (38/95), Anger (35/95), and Fast (32/95) presets are all well above Neutral (4/95). The fixed six-query pool covers 44/95 Qwen2-Audio seeds and 15/95 GPT-4o seeds and exceeds a matched-budget StyleBreak reimplementation (27/95) on Qwen2-Audio. A same-voice pool excluding the confounded Commanding condition still reaches 40/95, and a retained-panel ablation shows emotional-delivery audio alone (44/95) is far more effective than emotional text alone (11/95). Exploratory surrogate diagnostics and pilot mitigation observations are secondary, non-core analyses. Overall, matched-text speech delivery should be treated as a first-class factor in Audio LLM safety evaluation
Primary: City University of Hong Kong
All Institutions: City University of Hong Kong, City University of Hong Kong (Dongguan)
The paper makes a valuable contribution to multimodal safety by empirically demonstrating that prosodic variations in speech delivery can significantly compromise the safety of Audio LLMs, independent of lexical content. Its controlled methodology and clear empirical findings provide a necessary foundation for future research into robust, prosody-aware safety mechanisms for spoken AI assistants.
The paper proposes a controlled experimental framework, PJ-Break, to isolate the impact of prosodic variations (arousal, authority, speaking rate) on the safety/jailbreak capabilities of Audio Large Language Models (Audio LLMs). By holding the lexical transcript constant and varying only the speech delivery via TTS presets, the authors address a specific and under-explored gap in multimodal safety: the influence of paralinguistic cues. The methodology is rigorous in its control of variables, utilizing a fixed TTS stack and explicit acoustic verification. The inclusion of an ablation study comparing emotional text vs. emotional audio is a strong methodological choice that directly supports the core hypothesis. The use of surrogate model diagnostics (activation patching) to hypothesize mechanisms, while clearly labeled as exploratory, adds depth to the analysis.
The evaluation is conducted on Qwen2-Audio and GPT-4o, using a benchmark of 600 samples (AdvAudio-Prosody). The results are compelling: prosodic jailbreaks significantly outperform neutral audio and even matched-budget style-transfer baselines (StyleBreak). The finding that "Panic" and "Anger" presets achieve high success rates (38/95 and 35/95 respectively) compared to Neutral (4/95) is a significant empirical contribution. The same-voice sensitivity analysis strengthens the claim that the effect is due to prosody rather than voice identity. However, the reliance on TTS-generated speech limits the ecological validity, although the authors do provide small-scale human-speech and OTA checks to mitigate this concern. The statistical significance (McNemar test) adds robustness to the main comparison.
The paper provides detailed implementation specifications, including the TTS engine (Azure Neural TTS), specific voices used, and the definition of the six presets. The acoustic verification metrics ($F_0$, RMS, etc.) are standard and reproducible. However, the dataset and code are explicitly *not* released due to safety concerns regarding the generation of harmful audio jailbreaks. This is a valid ethical stance for this domain but limits independent verification and extension. The description of the evaluation protocol and judge ensemble is sufficiently detailed for a competent researcher to replicate the core experiments, assuming access to the same models and TTS services.
The primary limitation is the reliance on TTS, which may not fully capture the nuances of human emotional speech or the specific acoustic artifacts that might arise in real-world over-the-air attacks. The "Commanding" condition is partially confounded by voice identity (different speaker), which the authors acknowledge but does not fully eliminate the confound. The study is limited to English and a single TTS provider, limiting generalizability to other languages and synthesis technologies. Furthermore, the surrogate analysis is restricted to one model (Qwen2-Audio), and the mechanistic insights (refusal direction) are hypothesis-generating rather than definitive proof of causality in closed-source models. The pilot mitigation results are preliminary and not robust.
This paper has significant implications for the safety evaluation of multimodal AI systems. It demonstrates that current safety filters, which often focus on text or content, may be bypassed by manipulating the *delivery* of the content. This necessitates a shift in safety research to include prosodic and paralinguistic analysis in red-teaming protocols. It also highlights potential vulnerabilities in crisis-response or assistance-oriented models that might be triggered by high-arousal speech. The work encourages the development of prosody-aware safety mechanisms and more robust multimodal safety benchmarks. The paper makes a valuable contribution to multimodal safety by empirically demonstrating that prosodic variations in speech delivery can significantly compromise the safety of Audio LLMs, independent of lexical content. Its controlled methodology and clear empirical findings provide a necessary foundation for future research into robust, prosody-aware safety mechanisms for spoken AI assistants.
Existing single-domain and multi-task audio systems remain limited in directly organizing speech, music, sound effects, ambience, and multiple roles into long-form temporal scenes. We present Qwen-Audio-3.0-Gen-Preview, a unified non-autoregressive framework that uses a Diffusion Transformer (DiT) and a shared variational autoencoder (VAE) to generate the complete mixed waveform. Prompt enhancement converts free-form requests into structured temporal records that are rendered as textual conditions, while a two-stage data curriculum and semantic conditional views train the proposed model to use these conditions across domains. A shared continuous VAE compresses 48kHz stereo waveforms into 25Hz latent sequences and incorporates semantic supervision, providing one representation for speech, music, sound effects, and their mixtures. On Seed-TTS-Eval, speaker similarity is the proposed model's clearest strength across all three subsets, and on the multi-speaker benchmark, the proposed model shows higher cross-turn consistency than Seed-Audio-1.0 in both languages. On AudioCaps, its advantages are concentrated in evaluations using large audio-language models and AudioBox. Relative to Seed-Audio-1.0, it achieves stronger temporal localization. Using approximately 10% music data of a dedicated in-house model, the proposed model remains close across all seven SongBench components and leads in three while retaining speech and general-audio capabilities. These results demonstrate the potential of unified generation for temporally structured, multi-domain audio.
Primary: Alibaba Token Foundry
All Institutions: Alibaba Token Foundry
This paper presents a significant engineering advancement in unified audio generation by introducing a structured, temporally-aware conditioning framework and a shared continuous latent space, demonstrating strong capabilities in multi-speaker dialogue and scene-level audio organization while maintaining competitive performance across diverse audio domains.
The paper proposes Qwen-Audio-3.0-Gen-Preview, a unified non-autoregressive (NAR) audio generation framework based on a Diffusion Transformer (DiT) and a shared continuous Variational Autoencoder (VAE). The core technical contribution lies in the data and conditioning pipeline rather than a radical architectural shift. The authors introduce a sophisticated "Prompt Enhancement" module that converts free-form text into structured temporal records (global scene, source profiles, time-localized events). This structured data is then rendered into textual conditions for the DiT. A key methodological component is the "Semantic Conditional View" training strategy, which uses conditional dropout on specific semantic units (e.g., hiding dialogue to force the model to rely on scene context) to improve robustness and controllability. The shared VAE is trained with auxiliary semantic supervision using a frozen LLM, aiming to create a latent space that is both acoustically high-fidelity and semantically rich. While the use of DiTs and continuous latents is established in recent audio generation literature (e.g., AudioGen, MusicGen), the specific integration of hierarchical temporal annotations and the two-stage curriculum for long-form scene generation represents a meaningful engineering contribution to the field of unified audio models.
The evaluation is comprehensive, covering Speech (Seed-TTS-Eval), General Audio (AudioCaps), and Music (SongBench). The paper demonstrates strong performance in speaker similarity and cross-turn consistency for multi-speaker dialogue, which is a significant challenge for unified models. On AudioCaps, the model leads in Large Audio-Language Model (LALM) based evaluations, suggesting better semantic alignment than competitors, although it does not lead in standard CLAP scores. The introduction of an in-house benchmark for temporal localization (mIoU) provides valuable insight into the model's ability to follow timing instructions, showing improvement over Seed-Audio-1.0. The comparison with a dedicated in-house music model highlights the efficiency of the unified approach, achieving competitive results with significantly less music data. However, the reliance on LALMs for evaluation introduces potential bias, as the judges are from the same ecosystem (Qwen/Gemini), and the lack of human MOS scores for the general audio/music tasks limits the perceptual validation.
The paper provides detailed descriptions of the data annotation pipeline, synthetic data construction (using Scaper-like recipes), and the training curriculum. The architecture of the VAE and DiT is described, including the latent dimensions (128-dim, 25Hz) and the semantic supervision objective. However, specific hyperparameters for the DiT training, the exact composition of the 200,000 hours of training data, and the code for the Prompt Enhancement module are not fully open-sourced or detailed to the extent required for exact replication. The use of proprietary in-house models for evaluation (Seed-Audio-1.0 baseline, in-house music model) also complicates direct comparison.
The paper acknowledges limitations regarding the synthetic data, noting residual context in source clips and mismatches in simulated acoustics. The reliance on LLMs for prompt enhancement and evaluation introduces a dependency on the capabilities of those specific models. The model's performance on music composition (melody, structure) lags behind dedicated autoregressive models, indicating that the NAR DiT architecture may struggle with long-range musical dependencies compared to sequential planners. The evaluation metrics, particularly the LALM-based ones, may not fully capture human perceptual quality or nuance.
This work advances the state of unified audio generation, moving towards systems that can produce complex, temporally structured scenes with multiple audio types. This has significant implications for content creation in gaming, film, and virtual reality, where coherent audio scenes are essential. By demonstrating that a single model can handle speech, music, and sound effects with temporal control, it reduces the complexity of audio production pipelines. However, the potential for misuse in generating realistic deepfakes or misleading audio content remains a concern, necessitating robust safety measures. This paper presents a significant engineering advancement in unified audio generation by introducing a structured, temporally-aware conditioning framework and a shared continuous latent space, demonstrating strong capabilities in multi-speaker dialogue and scene-level audio organization while maintaining competitive performance across diverse audio domains.
Cover song generation (CSG) should preserve the melodic and linguistic content of a reference song while recreating the remaining musical components. The state-of-the-art model SongEcho utilizes $F_0$ sequences and voiced/unvoiced (V/UV) tags for conditioning; however, implicit linguistic information from V/UV tags cannot guarantee lyric accuracy, leading to a high phoneme error rate (PER). Inspired by singing voice synthesis (SVS), we propose MPEcho, which integrates a phoneme encoder and a length regulator (LR) into the SongEcho framework. By providing explicit phoneme-level conditioning and precise temporal boundaries, MPEcho significantly reduces PER. To enable this, we developed Phonsa, a Whisper-based automatic transcription model that provides high-precision phoneme-level annotations for singing voices, overcoming the scarcity of high-quality audio-phoneme pairs. Experimental results validate the effectiveness of Phonsa for alignment and MPEcho for end-to-end CSG. The audio samples, code and weights can be accessed from https://lonian6.github.io/MPEcho.github.io/.
Primary: Taiwan AI Labs
All Institutions: Taiwan AI Labs, National Science and Technology Council of Taiwan, Ministry of Education (Taiwan)
MPEcho introduces a novel integration of SVS-style phoneme-level control into end-to-end cover song generation, significantly improving lyrical accuracy while maintaining melodic fidelity. This work demonstrates the value of cross-domain priors (SVS to CSG) and provides a robust framework for controllable music generation, advancing the state-of-the-art in both technical methodology and practical application.
The paper proposes MPEcho, a framework for Controllable Cover Song Generation (CSG) that integrates explicit phoneme-level conditioning into the SongEcho architecture. The core methodological contribution is the adaptation of Singing Voice Synthesis (SVS) techniquesâspecifically a length regulator and phoneme encoderâinto a diffusion-based full-song generation model. This addresses the limitation of previous models (SongEcho) that relied on coarse V/UV tags, leading to high phoneme error rates (PER). The authors also introduce Phonsa, a Whisper-based automatic transcription model optimized for singing voices, to provide the necessary high-precision phoneme-level annotations. The methodology involves a multi-condition guidance strategy using Adaptive Projection Guidance (APG) to disentangle text, lyrics, melody, and phoneme conditions. The approach is technically sound, leveraging established SVS priors to solve a specific controllability issue in music generation. While the integration of SVS components into LTS/CSG is not entirely new in concept, the specific implementation of a length regulator for end-to-end cover generation and the dedicated alignment model (Phonsa) for this domain represent a solid engineering contribution.
The experimental section is comprehensive, including objective metrics (Audiobox, CLAP, RPA, RCA, PER) and subjective evaluations (MOS). The results demonstrate that MPEcho significantly reduces PER compared to SongEcho (0.1865 vs 0.4562) while maintaining or improving musical quality metrics. Ablation studies effectively isolate the contributions of melody conditioning, phoneme conditioning, and the specific guidance strategies. The evaluation of Phonsa against MFA shows substantial improvements in alignment accuracy. However, the reliance on a Chinese-only dataset limits the generalizability of the findings. The objective metrics, particularly CLAP, show low scores across the board, which the authors attribute to domain mismatch, a valid but somewhat concerning observation for a generative model's performance assessment. The subjective evaluation adds valuable human-centric validation, confirming the objective trends.
The paper provides detailed implementation details, including dataset sizes, training hardware, hyperparameters, and model architectures. The code and weights are made available via a GitHub page, which significantly enhances reproducibility. The description of the Phonsa model and the MPEcho architecture is sufficiently detailed for replication. The use of standard datasets (M4Singer, Opencpop, GTsinger) for Phonsa training and a curated internal dataset for MPEcho allows for verification, although the internal dataset's exact composition is less transparent than public benchmarks.
The primary limitation is the restriction to single-singer scenarios and Mandarin language. The model's performance on multi-singer tracks or languages with different phonetic structures is untested. The reliance on Phonsa for alignment means that errors in Phonsa will propagate to MPEcho. Additionally, the internal dataset used for training MPEcho is not publicly available, which hinders direct comparison with other CSG models trained on different data. The low CLAP scores suggest that the generated audio may not always align well with text prompts in a broad semantic sense, although this is common in specialized music generation tasks.
This work contributes to the field of generative AI for music by improving the controllability of cover song generation. Accurate lyric rendering is crucial for user acceptance and creative utility in AI music tools. The development of Phonsa also advances the field of automatic speech recognition and alignment for singing, which has broader applications in music information retrieval and education. However, the ease of generating high-quality covers raises ethical concerns regarding copyright and artist likeness, which should be addressed in future deployments. MPEcho introduces a novel integration of SVS-style phoneme-level control into end-to-end cover song generation, significantly improving lyrical accuracy while maintaining melodic fidelity. This work demonstrates the value of cross-domain priors (SVS to CSG) and provides a robust framework for controllable music generation, advancing the state-of-the-art in both technical methodology and practical application.
The study of Target Speaker Extraction (TSE) aims to isolate a desired speaker from overlapping speech mixture given auxiliary cues. Existing systems are typically designed for specific cue types, limiting flexibility when cue availability varies across scenarios. We present WeSep, a unified framework that reformulates TSE as a heterogeneous cue-conditioned learning problem. In WeSep, cue modules and separator backbones are decoupled through standardized interfaces, enabling configurable cue injection and flexible integration of diverse modalities. The design enables systematic study of cue structure, intra- and cross-modal interaction, and dynamic cue availability within a shared optimization framework, facilitating adaptation to real-world conditions. Experiments across enrollment, spatial, visual, and textual cues reveal modality-dependent characteristics and demonstrate stable optimization under heterogeneous cue availability. The toolkit will be publicly available.
Primary: The Chinese University of Hong Kong, Shenzhen
All Institutions: The Chinese University of Hong Kong, Shenzhen, Nanjing University, Shenzhen Loop Area Institute
The paper presents WeSep, a modular framework for Target Speaker Extraction that standardizes cue injection and enables systematic study of heterogeneous cue availability. While it lacks algorithmic novelty, its contribution to software engineering and experimental standardization in the audio processing community is valuable and practically significant.
The paper proposes WeSep, a modular framework for Target Speaker Extraction (TSE) that decouples cue encoders from separator backbones. The core methodological contribution is architectural rather than algorithmic; it introduces a standardized interface for injecting diverse cues (enrollment, spatial, visual, textual) into a shared optimization framework. This allows for composable multi-cue integration and handles heterogeneous cue availability (missing cues) via zero-padding. While the decoupling strategy is sound and addresses a real pain point in TSE (fragmented codebases for different modalities), the novelty is limited. The concept of modular TSE is not new (e.g., ClearerVoice, various multi-modal baselines), and the paper explicitly states it does not propose a new separation architecture. The "innovation" lies in the software engineering and experimental design rather than novel deep learning mechanisms.
The evaluation is comprehensive in terms of breadth, covering four distinct cue modalities and their combinations on standard datasets (Libri2Mix, VoxCeleb2-mix, etc.). The experiments successfully demonstrate that the framework supports stable training with missing cues and that combining cues (e.g., speaker + spatial) yields performance gains. However, the results are largely confirmatory. The paper shows that existing components work well when plugged into this framework, rather than demonstrating that WeSep introduces a new state-of-the-art method. The performance metrics (SI-SDRi) are standard, and the improvements are attributed to the flexibility of composition rather than a breakthrough in extraction capability. The causal modeling experiment is a practical addition but shows expected degradation in performance for low latency.
The paper provides a GitHub URL for the toolkit, which is a significant positive for reproducibility. The experimental settings are described in reasonable detail, including backbone configurations (BSRNN, NBC2) and training hyperparameters (Adam, 150 epochs). The use of standard datasets enhances the ability of others to replicate the results. The modular design, if implemented cleanly as claimed, should facilitate reproduction by other researchers.
The primary limitation is the lack of algorithmic novelty. The paper admits to not proposing a new separator. The performance gains are marginal and stem from better engineering and composition rather than fundamental improvements in the extraction task. The reliance on zero-padding for missing cues is a simple heuristic that may not be optimal compared to learned masking or attention mechanisms, although it is effective for stability. The paper does not provide a detailed ablation study on the *optimal* fusion strategies beyond what is standard, limiting the theoretical contribution.
This work has positive broader impact by promoting modularity and standardization in the TSE field. By providing an open-source toolkit, it lowers the barrier to entry for researchers working on multi-modal TSE and encourages systematic comparison of cue types. It facilitates research into robust TSE systems that can handle dynamic cue availability, which is crucial for real-world deployment. The paper presents WeSep, a modular framework for Target Speaker Extraction that standardizes cue injection and enables systematic study of heterogeneous cue availability. While it lacks algorithmic novelty, its contribution to software engineering and experimental standardization in the audio processing community is valuable and practically significant.
Automatic analysis of multimodal speech has shown strong potential for computationally detecting and monitoring a wide range of neurological, psychiatric, and respiratory conditions. However, progress in this field is limited by existing publicly accessible datasets, which are often small in scale, focused on a single condition or disease, and primarily speech focused. Moreover, if key confounding variables such as education, medication use, comorbidities, or mood state are insufficiently documented, the reliability and interpretability of computational analyses are further compromised. To address these limitations, we introduce CARE v1.0, a curated multimodal English dataset of approximately 144 hours of short video interviews collected from 612 individuals across 12 medical conditions plus a control cohort. For each video, a comprehensive set of clinically relevant multimodal descriptors is provided, alongside structured metadata covering factors such as medication, life impacts, and expressed emotions. The corpus's breadth and heterogeneity support a wide range of applications, including automatic disease and symptom detection, multimodal modelling of speech and non-verbal behaviour under emotionally charged contexts, and studies of disease trajectories and coping processes.
Primary: Universitat Politècnica de València
All Institutions: Universitat Politècnica de València, University of Oxford, Instituto de Telecomunicaçþes
This paper presents CARE v1.0, a comprehensive multimodal corpus of 144 hours of video interviews from 612 individuals across 12 medical conditions, enriched with automatically extracted multimodal descriptors and structured metadata, addressing critical gaps in scale, diversity, and clinical context in existing digital health datasets.
The paper introduces CARE v1.0, a large-scale multimodal corpus derived from the HEXI platform. The methodology focuses on data curation and feature engineering rather than novel algorithmic development. The authors employ a rigorous pipeline for extracting multimodal descriptors: audio features (eGeMAPS, DisVoice, XLS-R, TRILLsson), visual features (OpenFace for facial landmarks/AUs, gaze, head pose, MediaPipe for body keypoints), and linguistic features (WhisperX alignment, sentence embeddings). A key methodological contribution is the use of LLMs (gpt-oss-120b) to extract structured metadata (demographics, emotions, comorbidities) from unstructured narrative texts, validated against a manually annotated subset. This approach addresses a significant gap in existing datasets by providing rich, clinically relevant metadata alongside raw multimodal signals.
The experimental section is primarily descriptive, focusing on the statistical properties of the dataset (distribution of conditions, demographics, emotion frequencies) and the validation of the LLM-based metadata extraction pipeline. The authors compare five LLMs for metadata extraction, demonstrating that gpt-oss-120b performs best in terms of accuracy and adherence to constraints. They provide a baseline evaluation workflow for a binary classification task (Control vs. Parkinson's) using an SVM on aggregated features. While this demonstrates the utility of the dataset, the lack of extensive benchmarking against state-of-the-art multimodal models limits the assessment of the dataset's ability to drive performance improvements in specific tasks. The validation of the metadata extraction is solid, but the downstream ML performance is only illustrative.
The paper excels in reproducibility. The dataset is hosted on Hugging Face, and the code for feature extraction and the example evaluation workflow is publicly available. The detailed documentation of the feature extraction pipeline, including specific tools (OpenFace, DiariZen, WhisperX) and parameter settings, allows other researchers to replicate the feature computation. The use of deterministic settings for LLM inference (temperature=0) further ensures reproducibility of the metadata annotations.
The dataset is skewed towards White UK/Irish participants, limiting its generalizability to other ethnic groups. The control group is heterogeneous (caregivers, bereaved, professionals), which may introduce confounding variables if not carefully handled in downstream analyses. The reliance on LLMs for metadata extraction, while validated, introduces potential biases and hallucinations (though rare, as noted). The dataset is cross-sectional, limiting longitudinal studies of disease trajectories, although the authors mention this as a potential future application. The "in-the-wild" nature of some video clips (varying quality, lighting) may pose challenges for robust computer vision models.
CARE v1.0 has significant potential impact on the field of digital health and affective computing. By providing a large, multimodal, and clinically annotated dataset, it enables research into automated detection and monitoring of a wide range of neurological, psychiatric, and respiratory conditions. The inclusion of diverse medical conditions and rich metadata supports the development of more robust and interpretable computational models. The dataset can facilitate studies on the interplay between speech, non-verbal behavior, and emotional states in health contexts, contributing to a better understanding of disease manifestations and coping processes. This paper presents CARE v1.0, a comprehensive multimodal corpus of 144 hours of video interviews from 612 individuals across 12 medical conditions, enriched with automatically extracted multimodal descriptors and structured metadata, addressing critical gaps in scale, diversity, and clinical context in existing digital health datasets.
Large audio-language models (LALMs) convey acoustic evidence to language decoders through native audio tokens, yet the internal roles of these tokens remain poorly understood. Using temporal audio grounding as a diagnostic setting, we examine how language-model fine-tuning affects the layerwise semantics, decoder accessibility, and temporal output alignment of native audio-token states through four complementary analyses: query-conditioned token semantics, calibrated token readout, temporal-window probes, and residual-delta erasure during generation. Alongside substantial improvements in temporal localization, semantic analysis of Qwen2.5-Omni shows that latent evidence for queried events is already present before fine-tuning and that the audio tokens most strongly aligned with the queried event appear at similar temporal positions before and after fine-tuning. After fine-tuning, event-related information in audio tokens becomes more accessible to the decoder, especially in early and middle layers, and a cross-checkpoint control shows that this improvement arises primarily from decoder adaptation. Temporal probes show that the base checkpoint already contains recoverable information about annotated windows and that fine-tuning mainly improves alignment with each checkpoint's own predicted temporal support. Residual-delta erasure further shows that removing audio-token updates within predicted windows harms timestamp generation more than removing the same number of randomly selected updates. The same broad improvements in decoder readability and prediction alignment also appear in Qwen2-Audio. Together, these results support a semantics-to-readout account in which grounding fine-tuning helps the decoder read existing event evidence and connect it more reliably to temporal outputs.
Primary: Unknown
All Institutions: Unknown
This paper provides a rigorous mechanistic analysis of how fine-tuning enhances temporal grounding in Large Audio-Language Models, demonstrating that improvements stem primarily from enhanced decoder accessibility to latent event evidence rather than the creation of new semantic representations.
The paper proposes a comprehensive mechanistic interpretability framework for Large Audio-Language Models (LALMs), specifically focusing on the "semantics-to-readout" hypothesis. The methodology is rigorous and multi-faceted, employing four distinct diagnostic tools: (1) Query-Conditioned Token Semantics to measure alignment between audio tokens and text event phrases; (2) Calibrated Token Readout using hidden-state insertion (patching) to assess decoder accessibility; (3) Output-Facing Temporal Diagnostics using linear probes on both annotated and predicted windows; and (4) Residual-Delta Erasure to test the functional necessity of specific token updates during generation. This combination of probing, patching, and causal intervention provides a robust separation of representation quality from decoder utility. The approach is well-suited for the domain, addressing the "black box" nature of multimodal alignment in LALMs.
The experiments are conducted on two prominent LALMs, Qwen2.5-Omni and Qwen2-Audio, fine-tuned on a newly constructed dataset, AudioGrounding-QA, derived from the AudioGrounding corpus. The evaluation demonstrates substantial improvements in temporal grounding metrics (mIoU, F1) post-fine-tuning. The core findings are supported by detailed layer-wise analyses. Key results include the discovery that latent event evidence exists in base models but is not effectively read out, and that fine-tuning primarily enhances decoder accessibility (particularly in early/middle layers) rather than creating new semantic representations. The cross-checkpoint controls and residual erasure results strongly support the authors' claims about the mechanism of improvement. The replication on Qwen2-Audio adds significant weight to the generalizability of the findings.
The paper provides detailed descriptions of the diagnostic pipelines, including the construction of the semantic bank, the specific prompts for readout, and the implementation of linear probes and erasure interventions. The dataset construction is clearly explained, and the fine-tuning setup (LoRA on LM blocks only) is specified. While the full code is not explicitly linked in the text provided, the methodological detail is sufficient for reproduction by experts in the field. The use of standard metrics (mIoU, F1) and established embedding models (SBERT) further aids reproducibility.
The study focuses primarily on temporal grounding, a specific task, which may limit the generalizability of the "semantics-to-readout" account to other multimodal tasks (e.g., audio captioning or open-ended dialogue). The analysis is largely confined to Qwen-based architectures; while Qwen2-Audio is used for replication, the broader landscape of LALMs (e.g., Whisper-based or other proprietary models) is not explored. Additionally, the "calibrated readout" relies on SBERT embeddings, which may not perfectly capture the semantic nuances required for complex event reasoning, potentially underestimating or overestimating readout gains. The residual erasure is limited to layer-level interventions, which might miss more complex, distributed circuits.
This work contributes significantly to the field of mechanistic interpretability for multimodal models. By demystifying how fine-tuning affects internal representations, it provides actionable insights for improving LALM training strategies, particularly for tasks requiring precise temporal or spatial grounding. It challenges the assumption that fine-tuning creates new knowledge from scratch, suggesting instead that it optimizes access to pre-existing latent structures. This has implications for efficient fine-tuning, model compression, and the development of more reliable multimodal AI systems. This paper provides a rigorous mechanistic analysis of how fine-tuning enhances temporal grounding in Large Audio-Language Models, demonstrating that improvements stem primarily from enhanced decoder accessibility to latent event evidence rather than the creation of new semantic representations.
This is an implementation and measurement study of what it costs to run a streaming speech enhancer on a CPU. We port FastEnhancer-Medium at 48 kHz to faster-enhancer.c, a C runtime with six int8 GEMM tiers selected at initialization, leaving architecture and weights untouched. One Apple M2 core reaches 0.069 real-time factor, against 0.230 for the fp32 ONNX Runtime graph on the same machine, a 3.3x speedup. A Galaxy S23+ (Snapdragon 8 Gen 2) reaches 0.096. The speedup comes from specializing every layer of the runtime around one fixed model. Activation ranges are recomputed per frame, so no calibration set is needed; the k=3 convolutions use Winograd F(2,3); cross-stage state is fp16; the GRU and the dequantization epilogues are fused; and nothing is allocated after startup. Over 824 VoiceBank-DEMAND utterances the engine tracks fp32 to within -0.006 PESQ and -0.08 dB SNR. Speed alone does not settle deployment cost. The enhancer holds a fraction of a core for as long as the microphone is open, so its real-time factor is a duty cycle. A benchmark races through a file; an audio callback does not. Pacing to the 6.67 ms deadline costs 4.2x per frame, saves 49% of the energy, and leaves the cheapest core placement missing 96% of its deadlines. All SIMD tiers within an architecture family emit byte-identical output. The runtime is released as a dependency-free library.
Primary: Hanyang University
All Institutions: Hanyang University
This paper delivers a highly optimized, dependency-free C runtime for streaming speech enhancement, achieving significant speedups and energy savings on commodity CPUs through specialized int8 quantization and SIMD kernel fusion, while maintaining bit-identical output and negligible quality loss.
The paper presents a highly specialized, dependency-free C runtime (`faster-enhancer.c`) designed to deploy the FastEnhancer-Medium speech enhancement model on commodity CPUs. The methodology focuses on extreme engineering optimization rather than algorithmic novelty. Key technical strategies include: (1) Training-free int8 quantization with per-frame activation range recomputation, eliminating the need for calibration datasets; (2) End-to-end runtime specialization with six SIMD tiers (ARM NEON, DOTPROD, I8MM, x86 AVX2, AVX-VNNI, AVX-512 VNNI) selected at initialization; (3) Use of Winograd F(2,3) for $k=3$ convolutions to reduce multiply-accumulate counts; (4) Fusion of GRU gates and dequantization epilogues to minimize memory traffic; (5) Strict byte-identical output guarantees across different compilers and architectures by addressing floating-point non-determinism in the Hann window generation and integer overflow edge cases (clamping to [-127, 127]). The approach is a masterclass in low-level systems engineering for ML deployment.
The evaluation is rigorous and addresses the specific constraints of audio processing. The authors measure Real-Time Factor (RTF) and energy efficiency (joules per second) on two distinct platforms: Apple M2 (macOS) and Galaxy S23+ (Android). They demonstrate a 3.3x speedup over fp32 ONNX Runtime on the M2 (0.069 RTF vs 0.230 RTF). Crucially, they introduce a "duty-cycled" benchmarking protocol that accounts for the 6.67 ms frame deadline, revealing that "race-to-idle" strategies are energy-inefficient compared to pacing. They also evaluate quality using PESQ, STOI, SNR, SI-SDR, and SIGMOS, showing negligible degradation (-0.006 PESQ, -0.08 dB SNR) compared to the fp32 baseline. The analysis of long-term stability (30-minute runs) highlights scheduler-induced jitter, a critical insight for real-world deployment.
The paper provides a high degree of reproducibility. The code is released as a dependency-free library on GitHub. The authors provide detailed descriptions of the quantization scheme, kernel implementations, and the specific steps taken to ensure bit-identical outputs across platforms (e.g., double-precision window generation). The evaluation protocol is clearly defined, including the use of VoiceBank-DEMAND and specific timing methodologies. The inclusion of hash comparisons for output verification adds a layer of rigor often missing in systems papers.
The primary limitation is the scope of the model: it is a port of an existing architecture (FastEnhancer-Medium) and does not propose a new model. The x86 tiers are verified for bit-equality but not timed due to lack of hardware, limiting the generality of the performance claims. The runtime is single-threaded and CPU-only, excluding GPU/TPU acceleration which is common in other deployment stacks. The quality metrics, while robust, rely on standard objective measures; subjective listening tests are not included, though the objective deltas are argued to be inaudible. The "cheapest core" placement on Android was found to miss deadlines, limiting the flexibility of core placement in heterogeneous systems.
This work has significant impact for on-device AI, particularly in privacy-sensitive applications like real-time communication, hearing aids, and voice assistants where latency and power consumption are critical. By providing a lightweight, dependency-free runtime, it lowers the barrier to deploying high-quality neural speech enhancement on resource-constrained devices. It challenges the industry's reliance on heavy general-purpose inference engines (like ONNX Runtime) for specific, fixed models, advocating for specialized, hand-tuned kernels. The emphasis on energy efficiency and deadline adherence provides a valuable framework for evaluating the true cost of ML deployment in audio pipelines. This paper delivers a highly optimized, dependency-free C runtime for streaming speech enhancement, achieving significant speedups and energy savings on commodity CPUs through specialized int8 quantization and SIMD kernel fusion, while maintaining bit-identical output and negligible quality loss.
Voice activity detection (VAD) triggers downstream speech processing in always-on systems under strict memory, latency, and compute constraints. Recent compact models report strong accuracy but rely on components that are not widely supported: learnable filterbanks, recurrent layers, or non-causal post-processing. We propose kiloVAD, designed for embedded inference using standard Mel features, CNN-only layers, and tunable context/spectral parameters. We introduce per-layer structured pruning with self-distillation and angle-based quantization-aware training (QAT) that outperforms standard QAT by 1-4%. Evaluated per-frame under causal conditions, kiloVAD achieves 0.850 AUC on AVA-Speech with 2.1 k parameters and 200 ms context, establishing a new state of the art for causal, deployment-ready VAD.
Primary: Analog Devices, Inc.
All Institutions: Analog Devices, Inc., University of California, Los Angeles (UCLA)
The paper presents a well-engineered, deployment-focused VAD model that achieves a strong balance between accuracy, latency, and model size through effective structured pruning and angle-aware quantization.
The paper proposes kiloVAD, a compact Voice Activity Detection (VAD) model designed specifically for edge deployment. The core methodological contributions are architectural and training-related rather than novel deep learning architectures. The architecture is a standard CNN backbone (depthwise separable convolutions) with specific engineering choices: an adapter layer for channel decoupling, global average pooling for context flexibility, and amplitude-agnostic preprocessing. The primary technical novelty lies in the compression pipeline: (1) Per-layer structured pruning optimized via multi-objective search (FPR vs. parameter count) combined with self-distillation, and (2) Angle-based Quantization-Aware Training (QAT). The angle-based QAT modifies the loss function to align feature vectors with class weight vectors (prototypes) and repel them from non-target prototypes, aiming to mitigate quantization-induced angular errors. While the application of angle-based margin losses to QAT is a known technique in classification, its specific adaptation for ultra-low-bitwidth VAD on microcontrollers is a valid engineering contribution. However, the underlying model architecture is not novel; it relies on standard, well-established components (Mel spectrograms, CNNs, Global Average Pooling).
The evaluation is rigorous in the context of edge deployment constraints. The authors evaluate on the AVA-Speech dataset, which is appropriate for testing generalization to unseen environments. They provide a detailed comparison with prior compact VAD models (MarbleNet, TinyVAD, SincQDR, ResectNet, AtomicVAD). A key strength is the strict causal evaluation protocol (200ms context), which highlights the latency advantages of their design over competitors that use non-causal windows or larger contexts. The results show that kiloVAD achieves competitive AUC (0.850) with significantly fewer parameters (2.1k) and lower latency than most competitors, while maintaining full causal compatibility and standard operator support. The ablation studies on pruning strategies and quantization methods are clear and demonstrate the efficacy of the proposed techniques. The claim of "state of the art" is qualified by the specific constraints (causal, standard ops, low latency), which is honest and accurate.
The paper provides sufficient detail for reproduction. The architecture is described with specific layer types, kernel sizes, and channel dimensions. The training protocol includes optimizer settings, learning rate schedules, and loss functions. The authors provide pretrained weights and a web-based demo. The use of standard libraries (torch-pruning, Optuna) and standard datasets (LibriSpeech, AVA-Speech) facilitates reproducibility. The only minor gap is the exact code implementation of the angle-based QAT loss, though the mathematical formulation is provided.
The paper focuses heavily on parameter count and latency, but does not provide detailed energy consumption measurements (e.g., mJ/inference) on specific hardware, which is crucial for "always-on" edge devices. The model's performance on highly non-stationary noise or very low SNR conditions is not deeply explored beyond the standard AVA-Speech evaluation. The reliance on Mel spectrograms, while standard, means it does not explore raw audio end-to-end, which some recent works argue is beneficial for extreme compression. The "angle-based" QAT, while effective, adds complexity to the training loop compared to standard STE QAT.
This work has significant practical impact for the deployment of speech processing systems on resource-constrained IoT devices, smart speakers, and hearables. By providing a model that is compatible with standard embedded toolchains (TFLM) and requires no custom DSP kernels, it lowers the barrier to entry for developers. The emphasis on causal, low-latency inference is critical for user experience in real-time applications. The work contributes to the broader trend of efficient AI, demonstrating that high performance can be achieved through careful engineering and compression techniques rather than just larger models. The paper presents a well-engineered, deployment-focused VAD model that achieves a strong balance between accuracy, latency, and model size through effective structured pruning and angle-aware quantization.
Some text-to-speech systems ship a synthesis model and preset style vectors but not the reference encoder that turns audio into such a vector. The model still accepts a style vector; a user with a voice of their own cannot produce one. We solve for that input directly, inverting the released pipeline by gradient descent: every weight stays frozen and only the style vector is optimized, against time-pooled WavLM statistics of one recording. Because the objective discards the time axis, the synthesized text may differ from the recording, so no transcript and no alignment are needed. On 154 speakers from two corpora, ECAPA-TDNN similarity rises from 0.132 to 0.413 and ResNet from 0.099 to 0.401, improving for every speaker; a verifier at its equal-error point accepts 53% of the recovered voices as the target, against 1% for the presets they start from.
Primary: Hanyang University
All Institutions: Hanyang University
This paper presents a novel gradient-based method for extracting voice styles from frozen TTS models without reference encoders, demonstrating that withheld encoders do not fully secure voice identity and offering a reproducible framework for analyzing TTS latent spaces.
The paper proposes a gradient-based inverse optimization method to extract style vectors from frozen Text-to-Speech (TTS) models when the reference encoder is withheld. The core innovation lies in bypassing the need for transcribed reference audio or forced alignment by using a content-independent loss function. Specifically, it optimizes the style vector to match the time-pooled mean and standard deviation of hidden states from an early layer (Layer 4) of a frozen WavLM model. This approach effectively decouples speaker identity from linguistic content, allowing the system to work with untranscribed, arbitrary audio recordings. The method is technically sound, leveraging existing differentiable wrappers for ONNX models and standard optimization techniques (Adam, learning rate scheduling). It draws strong parallels to GAN inversion and Textual Inversion in the image domain, adapting these concepts to the audio TTS space.
The evaluation is conducted on 154 speakers from two corpora (VCTK and Common Voice subset of Seed-TTS Eval). The results demonstrate a significant improvement in speaker similarity metrics (ECAPA-TDNN and ResNet) compared to using the nearest preset style vector. The improvement is universal across all tested speakers. The paper provides a thorough analysis of the stopping criterion, characterizing the "content floor" to prevent overfitting into unrealistic style regions. It also evaluates intelligibility (WER) and naturalness (UTMOS), showing that the extracted voices remain intelligible and natural-sounding. The use of independent verification metrics (ECAPA, ResNet) to validate the optimization objective (WavLM statistics) strengthens the claim of successful voice cloning.
The authors release code and provide detailed hyperparameters, including the optimization setup, loss function definition, and stopping criteria. The use of public models (SupertonicTTS, WavLM, ECAPA-TDNN) and public datasets (VCTK, Common Voice) ensures that the experiments are fully reproducible. The description of the gradient flow through the ONNX modules is sufficiently detailed for implementation.
The method is currently validated on a single TTS model (SupertonicTTS). While the authors argue it should generalize to other models with continuous style inputs, this has not been empirically demonstrated. The optimization is sensitive to the choice of the WavLM layer and the stopping threshold, which are somewhat heuristic. The method does not recover duration/prosody styles, inheriting them from the initialization preset. Furthermore, the evaluation prompts are the same ones used during optimization, so generalization to unseen text is not fully tested. The paper acknowledges that the extracted voices might still be detectable by spoofing countermeasures, even if they pass speaker verification.
This work has significant dual-use implications. On one hand, it highlights a security vulnerability in TTS systems that withhold reference encoders, demonstrating that partial releases do not fully protect voice identity. This can inform better security practices for TTS providers. On the other hand, it provides a powerful tool for voice cloning, which can be misused for impersonation and fraud. The authors address this by releasing code with ethical warnings and not packaging it as a turnkey cloning tool. The work contributes to the broader understanding of latent spaces in generative models and the transferability of self-supervised speech representations. This paper presents a novel gradient-based method for extracting voice styles from frozen TTS models without reference encoders, demonstrating that withheld encoders do not fully secure voice identity and offering a reproducible framework for analyzing TTS latent spaces.
In this report, we present Qwen-Audio-3.0-TTS, a production-oriented speech synthesis system that jointly advances content consistency, speaker similarity, prosodic naturalness, audio quality, controllability, multilingual coverage, efficiency, and robustness. It combines a 12.5~Hz low-frame-rate speech tokenizer for reduced inference latency with a five-stage progressive training paradigm for coordinated language model (LM) and flow-matching model (FM) optimization. The model provides production-level control through free-style natural-language instructions and fine-grained inline tags, while supporting 16 languages, 20 Chinese dialect regions, one-pass long-form synthesis up to 3 minutes, and robust generation from noisy, reverberant, or unclear reference speech. Across SEED-TTS-Eval, CV3-Eval, instruction-following, long-form, and acoustic-robustness evaluations, Qwen-Audio-3.0-TTS achieves state-of-the-art performance on many reported dimensions or the strongest aggregate results. It also ranks first on the independent Artificial Analysis Text-to-Speech Leaderboard. These results establish Qwen-Audio-3.0-TTS as a strong foundation for production-level speech synthesis.
Primary: Alibaba Group
All Institutions: Alibaba Group
[One sentence main contribution]. Qwen-Audio-3.0-TTS introduces a five-stage progressive training paradigm with joint LM-FM optimization and reinforcement learning to achieve state-of-the-art production-level speech synthesis with robustness and fine-grained control. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a significant technical contribution by systematically addressing the trade-offs between efficiency, fidelity, and controllability in TTS. The integration of GRPO for both LM and FM optimization is a novel application of RLHF techniques to speech generation, moving beyond simple supervised fine-tuning. The explicit focus on acoustic robustness through training-time augmentation is highly relevant for real-world deployment. The comprehensive evaluation suite, including new benchmarks for robustness and instruction following, sets a high standard for future TTS research. The results demonstrate that a unified model can outperform specialized systems in multiple dimensions, marking a step towards more general-purpose speech generation models.
The paper presents Qwen-Audio-3.0-TTS, a production-oriented text-to-speech system that integrates a 12.5 Hz speech tokenizer, a language model (LM), and a flow-matching model (FM). The core methodological contribution lies in a five-stage progressive training paradigm. Key innovations include: 1) Conditioning the FM on continuous LM hidden states rather than just discrete tokens to mitigate information bottleneck; 2) High-quality data annealing during joint LM-FM training; 3) Application of Group Relative Policy Optimization (GRPO) to the LM for content/prosody rewards and to the FM for speaker similarity/perceptual quality rewards; 4) Explicit acoustic robustness training using degraded prompts (noise, reverb, etc.) without separate denoising modules. The architecture combines autoregressive semantic planning with continuous acoustic generation, aiming to balance latency, fidelity, and controllability.
The evaluation is comprehensive and rigorous, covering multiple dimensions: zero-shot voice cloning (SEED-TTS-Eval, CV3-Eval), multilingual/cross-lingual synthesis, instruction following (VStyle), long-form generation, text normalization, and acoustic robustness. The paper provides extensive objective metrics (CER/WER, SIM, DNSMOS) and subjective human evaluations (Arena-style, dialect authenticity). Results indicate state-of-the-art performance on many reported dimensions, particularly in robustness and instruction following, while maintaining competitive content consistency and speaker similarity. The inclusion of a dedicated "hard-case" robustness benchmark and detailed ablation studies on tokenizer frame rates adds significant value.
The paper provides detailed descriptions of the architecture, training stages, and data types. It mentions specific models used for evaluation (Whisper, Paraformer, ERes2Net, DNSMOS) and provides a demo URL. However, full code and model weights are not explicitly stated as open-source in the provided text (though the project URL points to a GitHub page, the text itself focuses on the report). The detailed training recipe (5 stages, specific rewards) is described well enough for reproduction by experts in the field.
The paper acknowledges that pushing content accuracy lower can come at the expense of naturalness, suggesting a trade-off. The robustness training, while effective, relies on specific augmentation strategies that may not cover all real-world edge cases. The reliance on LLM-based evaluators (Gemini) for some subjective metrics introduces potential bias, although human evaluation is also provided. The "high-quality data annealing" requires careful curation, which may limit scalability if high-quality labeled data is scarce.
This work advances the field of speech synthesis by providing a robust, controllable, and efficient system suitable for production deployment. It addresses critical pain points like robustness to noisy prompts and fine-grained control, which are essential for real-world applications. The open-sourcing of such models (implied by the project URL) can democratize access to high-quality TTS technology. However, the ease of voice cloning also raises concerns about misuse for deepfakes, necessitating responsible AI practices. [One sentence main contribution]. Qwen-Audio-3.0-TTS introduces a five-stage progressive training paradigm with joint LM-FM optimization and reinforcement learning to achieve state-of-the-art production-level speech synthesis with robustness and fine-grained control. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a significant technical contribution by systematically addressing the trade-offs between efficiency, fidelity, and controllability in TTS. The integration of GRPO for both LM and FM optimization is a novel application of RLHF techniques to speech generation, moving beyond simple supervised fine-tuning. The explicit focus on acoustic robustness through training-time augmentation is highly relevant for real-world deployment. The comprehensive evaluation suite, including new benchmarks for robustness and instruction following, sets a high standard for future TTS research. The results demonstrate that a unified model can outperform specialized systems in multiple dimensions, marking a step towards more general-purpose speech generation models.
Contrastive Language-Audio Pretraining (CLAP) learns aligned text and audio representations in a shared embedding space. However, independent encoding of each modality limits its ability to model cross-modal semantics in complex audio understanding and retrieval tasks. To address this limitation, this paper proposes Text-Prompted CLAP (TP-CLAP), a parameter-efficient extension of CLAP that introduces a cross-attention-based fusion module to incorporate textual prompts into audio features. TP-CLAP is trained using an audio multiple-choice question (audio-MCQ) framework, where it learns to align query-conditioned audio representations with text embeddings of correct answer choices via contrastive learning. Experiments demonstrate that TP-CLAP achieves competitive performance on audio question answering (audio-QA) with substantially larger audio-LLMs, while also improving the base CLAP model on audio-text retrieval and zero-shot classification benchmarks. The learned representations are further fine-tuned for attribute-focused audio-to-audio retrieval, showing that TP-CLAP consistently outperforms the standard CLAP baseline in music retrieval tasks.
Primary: University of Cambridge
All Institutions: University of Cambridge, Toshiba Cambridge Research Laboratory
This paper presents a method for learning query-conditioned audio representations by integrating a cross-attention fusion module into the CLAP framework, trained via an audio multiple-choice question paradigm, demonstrating significant improvements in attribute-focused retrieval and competitive performance in audio question answering tasks.
The paper proposes TP-CLAP, a parameter-efficient extension of the CLAP model that introduces a cross-attention-based fusion module to condition audio representations on textual prompts. The core methodological contribution lies in the training framework: it leverages AudioMCQ data to train the model to align query-conditioned audio embeddings with the text embeddings of correct multiple-choice answers. This is combined with standard CLAP contrastive loss and auxiliary losses to maintain general alignment. The approach is technically sound, addressing a specific limitation of static embeddings in CLAP (inability to handle query-dependent tasks like QA or attribute-focused retrieval). The use of cross-attention to inject textual context into audio features is a known pattern (similar to Q-Former in LLaVA or cross-attention in BLIP), but its application specifically to enable attribute-focused audio retrieval via an MCQ framework is a distinct and practical contribution. The derivation of the loss functions, including the specific handling of multi-label attributes in the fine-tuning stage, is rigorous.
The experimental evaluation is comprehensive, covering audio-text retrieval, zero-shot classification, audio question answering (QA), and attribute-focused audio-to-audio retrieval. The results show consistent improvements over the base CLAP model and the CLAP+AudioMCQ baseline (which lacks the fusion module). Notably, TP-CLAP achieves competitive performance on audio QA against much larger audio-LLMs, which is a strong result highlighting the efficiency of the approach. The attribute-focused retrieval experiments are particularly compelling, demonstrating that the model can isolate specific attributes (e.g., instrument, genre) rather than relying on global similarity. The use of standard benchmarks (AudioCaps, Clotho, ESC50, FSD50K, NSynth, MagnaTagATune) adds credibility. However, the improvement in standard retrieval tasks is modest, suggesting the primary value is in the conditional/attribute-specific capabilities.
The paper provides detailed implementation details, including encoder architectures (CED-Base, bert-base-uncased), hyperparameters (learning rates, batch sizes, warmup epochs), and loss function weights. The data sources (AudioMCQ, AudioCaps, Clotho, WavCaps, MACS, NSynth, MagnaTagATune) are well-known and accessible. The training stages are clearly delineated. This level of detail allows for reasonable reproducibility, assuming access to the pre-trained CLAP weights and the specified datasets.
The paper acknowledges that the model is smaller than audio-LLMs like Audio Flamingo 3, which still outperform it in QA. The improvements on standard retrieval tasks are marginal. The method relies heavily on the quality of the AudioMCQ dataset; if the questions/answers are noisy or biased, the model may learn spurious correlations. Furthermore, the cross-attention module adds computational overhead during inference compared to the original CLAP, although it remains lightweight compared to full LLMs. The paper does not discuss potential biases in the attribute-focused retrieval (e.g., gender or cultural biases in instrument/genre associations).
This work contributes to the field of multimodal audio understanding by enabling more flexible and query-dependent audio representation learning. It facilitates applications such as advanced music information retrieval (MIR), where users might search for "sad piano music" or "jazz with saxophone," rather than just general genre or mood. It also improves accessibility tools that need to understand specific audio contexts. The parameter-efficient nature makes it more accessible for deployment on edge devices or with limited compute compared to large multimodal LLMs. This paper presents a method for learning query-conditioned audio representations by integrating a cross-attention fusion module into the CLAP framework, trained via an audio multiple-choice question paradigm, demonstrating significant improvements in attribute-focused retrieval and competitive performance in audio question answering tasks.
Siri Expressive Voices synthesize rich, configurable speech in real time and entirely on device, powered by AFM 3 Core Advanced, Apple's most powerful on-device foundation model. This work presents the memory-efficient audio synthesis architecture behind that capability: a detokenizer that converts the semantic audio tokens emitted by the foundation model into high-fidelity audio within the tight compute and memory budget of the Apple Matrix Coprocessor (AMX). We convert semantic audio tokens to a residual vector quantization (RVQ) representation with a three-component design, a streaming encoder, a temporal decoder, and a depth decoder, that systematically decouples temporal and depth processing. A single reusable depth decoder with Diffusion Transformer (DiT)-style stage conditioning generates all RVQ levels autoregressively, replacing the dedicated per-level decoders of prior multi-decoder architectures, while causal sliding window attention with fixed-window key-value caching yields constant memory complexity independent of sequence length. Deployed on the AMX, the detokenizer sustains roughly 10 ms per generation step, about 16x faster than real time, with a peak runtime memory of only 21 MB and 329 MB of on-device assets, enabling continuous streaming synthesis of 20-320 seconds of audio. This constant, small footprint replaces the linear and quadratic memory scaling of conventional transformer- and GAN-based approaches. Ablation studies validate the key architectural components, and audio quality assessment confirms that the architecture maintains synthesis fidelity while achieving efficiency gains over existing methods. Operating at a 1-billion-parameter activation size within AFM 3 Core Advanced, it improves Mean Opinion Score by +0.28 overall (4.15 vs. 3.87) and by +0.42 on conversational speech (4.24 vs. 3.82) over the prior on-device text-to-speech system.
Primary: Apple Inc.
All Institutions: Apple Inc.
The paper presents a significant engineering achievement in deploying efficient, high-quality audio synthesis on-device, introducing architectural innovations that decouple temporal and depth processing to achieve constant memory complexity and superior efficiency compared to prior methods.
The paper proposes a memory-efficient audio synthesis architecture designed for on-device deployment, specifically targeting the Apple Matrix Coprocessor (AMX). The core methodological contribution is a three-component design: a streaming encoder, a temporal decoder, and a depth decoder. The key innovation lies in the "depth decoder," which replaces the per-level input/output projections found in prior works like Moshi with a single, fully shared transformer body conditioned on the RVQ level index via DiT-style rotary positional encoding (RoPE). This eliminates parameter overhead associated with the number of quantization levels. Additionally, the architecture employs causal sliding window attention with a fixed-window key-value cache to ensure constant memory complexity independent of sequence length, a critical feature for real-time streaming. The temporal decoder bridges semantic tokens and RVQ tokens, incorporating a temporal lookahead mechanism and autoregressive feedback from the mean RVQ embeddings. The approach effectively decouples temporal and depth processing, allowing for efficient generation of high-fidelity audio from semantic tokens.
The evaluation is comprehensive, covering both algorithmic scaling and production deployment metrics. On a common GPU (NVIDIA H100), the proposed model demonstrates constant memory and generation time scaling with sequence length (up to 320 seconds), whereas baselines (Transformer Decoder, Autoregressive GAN) scale linearly or quadratically. On-device, the model achieves a peak runtime memory of 21 MB and sustains ~10 ms per generation step (16x real-time). Audio quality is assessed using objective metrics (UTMOS, SI-SNR, VisQOL, MOSNet) and subjective Mean Opinion Score (MOS) evaluations. The model outperforms the prior on-device system (+0.28 overall MOS, +0.42 on conversational speech) and shows competitive or superior performance against baselines like Moshi/Mimi and internal GAN/Transformer models on UTMOS and SI-SNR. Ablation studies validate the contributions of the unified depth decoder, DiT conditioning, fixed KV cache, and temporal lookahead.
The paper provides detailed architectural descriptions, including layer configurations (4-6-2), attention mechanisms, and conditioning strategies. However, reproducibility is significantly hindered by the use of a proprietary audio corpus ("internally recorded voice assistant interactions") and the lack of open-source code. The hyperparameters and training details are described, but the specific data distribution and preprocessing pipelines are not disclosed due to confidentiality protocols. The comparison with baselines is conducted on a common GPU, which helps, but the lack of access to the training data and code limits independent verification of the exact performance claims.
The primary limitation is the reliance on proprietary data and closed-source implementation, preventing independent replication and broader community adoption. The model is specifically optimized for Apple's AMX hardware, and while algorithmic scaling is reported on GPUs, the real-world performance on other platforms may vary. The "temporal lookahead" mechanism, while effective, introduces a slight latency overhead and requires access to future semantic tokens, which might be a constraint in strictly causal, low-latency streaming scenarios where future context is not immediately available. The comparison with Moshi is somewhat indirect, as Moshi's on-device performance metrics are not provided for direct comparison, although algorithmic scaling is compared.
This work demonstrates the feasibility of high-quality, real-time, on-device audio synthesis with minimal resource overhead, enabling privacy-preserving voice assistants and immersive audio applications without cloud dependency. The architectural innovations, particularly the decoupled temporal-depth processing with shared parameters and fixed-window attention, offer valuable design patterns for efficient transformer-based generative models in resource-constrained environments. It pushes the boundary of what is possible with on-device AI, potentially reducing reliance on cloud infrastructure for audio generation tasks. The paper presents a significant engineering achievement in deploying efficient, high-quality audio synthesis on-device, introducing architectural innovations that decouple temporal and depth processing to achieve constant memory complexity and superior efficiency compared to prior methods.
Recent generative models are moving beyond silent video or standalone audio synthesis toward the joint generation of synchronized audio and video. Despite this progress, jointly generating audio and video with fine-grained cross-modal correspondence remains challenging due to their fundamental structural differences. Most existing methods use audio and video VAEs trained separately. As a result, the two latent spaces lack cross-modal alignment, leaving the downstream generative model to learn cross-modal synchronization from scratch. We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations. Beyond reconstruction, OmniVAE uses a segment-level audio-video contrastive objective to capture temporal-semantic correspondence and align the two latent spaces. In parallel, it distills features from pretrained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces. Extensive experiments show that both objectives consistently improve the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation. These findings underscore the importance of learning unified representations as a foundation for omnimodal modeling.1
Primary: Fudan University
All Institutions: Fudan University, Shanghai Innovation Institute, MOSI Intelligence, Shanghai Jiao Tong University
OmniVAE introduces a novel jointly trained audio-video VAE with segment-level contrastive alignment and semantic distillation, significantly improving cross-modal synchronization and generation quality in text-to-audio-video tasks.
The paper proposes OmniVAE, a jointly trained audio-video Variational Autoencoder (VAE) designed to address the lack of cross-modal alignment in standard modality-specific tokenizers. The core methodological contribution lies in two training-only objectives added to standard reconstruction losses: (1) Segment-level Audio-Video Contrastive Learning (AVCLIP), which aligns audio and video latents at a fine temporal granularity using a hierarchical negative sampling strategy (intra-clip, sibling-clip, cross-video), and (2) Modality-Specific Semantic Distillation, which injects semantic structure from pretrained Qwen3-Omni encoders into each latent space independently. The approach is technically sound, leveraging established contrastive learning frameworks (InfoNCE) and distillation techniques (iREPA-style) but applying them specifically to the joint audio-video latent space. The three-stage training schedule (pretrain reconstruction -> joint contrastive/distillation -> adversarial refinement) is a practical engineering solution to handle the conflicting optimization dynamics of reconstruction and alignment.
The experimental evaluation is comprehensive and well-structured. The authors conduct extensive ablations to isolate the contributions of reconstruction, distillation, and contrastive learning. They employ a novel "sync probing" task (predicting temporal offset) as a proxy for alignment, which provides strong quantitative evidence that the latent spaces are indeed better aligned than baselines. Downstream text-to-audio-video generation experiments on Verse-Bench show consistent improvements in cross-modal alignment metrics (DeSync, LSE-C) and generation quality (AudioBox, IS, FAD) when using OmniVAE. The use of a large-scale, curated dataset (23M clips) with LLM-generated captions adds significant weight to the empirical claims. The reconstruction quality remains competitive with state-of-the-art independent VAEs, demonstrating that alignment does not come at the cost of fidelity.
The paper provides detailed architectural descriptions, loss formulations, and training schedules. The parameter counts are explicitly listed, distinguishing between inference and training-only modules. The dataset construction pipeline is described, though the specific filtering thresholds and LLM prompts might vary. The reliance on proprietary or large pretrained models (Qwen3-Omni, Wan2.2, HunyuanVideo-Foley) for initialization and distillation is noted, which aids reproducibility for those with access to these resources. The code is not explicitly linked in the text, but the methodology is described with sufficient detail for replication.
The primary limitation is that the alignment is learned during training only; the model does not enforce a shared latent distribution, meaning the latents are still modality-specific but aligned in a shared embedding space via the contrastive head. This might limit direct cross-modal generation without additional mapping layers. The sync probing metric, while useful, is a proxy and may not perfectly correlate with human perception of synchronization in complex scenes. The dataset, while large, is web-scraped and may contain noise despite filtering. The reliance on Qwen3-Omni for distillation assumes the availability of this specific large model, which might not be accessible to all researchers.
OmniVAE contributes to the field of unified multimodal generation by providing a stronger foundational representation for audio-video tasks. By decoupling the alignment learning from the generative model, it allows downstream generators to focus on content creation rather than learning synchronization from scratch. This could accelerate the development of high-quality, synchronized audio-video generative models for entertainment, education, and simulation. The emphasis on fine-grained temporal alignment addresses a key challenge in current generative AI, potentially leading to more realistic and usable synthetic media. OmniVAE introduces a novel jointly trained audio-video VAE with segment-level contrastive alignment and semantic distillation, significantly improving cross-modal synchronization and generation quality in text-to-audio-video tasks.
Joint Embedding Predictive Architectures (JEPA) have recently emerged as a paradigm for learning world models by predicting latent representations, offering a promising direction for self-supervised learning. While initial attempts have applied JEPA to the music domain, it remains unclear how such frameworks can naturally support the formation of a world model for music. In this work, we propose to learn a world model of piano sound using JEPA by framing music as an action-conditioned system: the audio is treated as the state, and the pianoroll as the instrument action. Given a current audio state and an action, the model predicts the resulting future audio state, mirroring how humans learn musical sound through interaction. The model is trained in a fully offline setting using paired audio-pianoroll data, without environment interaction. Experiments show that the learned model captures the relationships between musical actions and their resulting sound. The resulting representations support downstream tasks, including beat tracking, composer identification, and key estimation, and enable piano transcription via planning, by searching for actions that best explain a target sound.
Primary: McGill University
All Institutions: McGill University, Advanced Machine Intelligence (AMI Labs), Centre for Interdisciplinary Research in Music Media and Technology (CIRMMT), Mohamed bin Zayid University of Artificial Intelligence (MBZUAI), Schulich School of Music
[One sentence main contribution]. The paper introduces Music-JEPA, an action-conditioned world model that learns latent dynamics of piano music by predicting future audio states from current states and pianoroll actions. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. This work makes a significant contribution to self-supervised learning in audio by moving beyond passive representation learning to active, action-conditioned modeling. The technical execution is solid, with clear demonstrations of learned dynamics and competitive performance on downstream tasks. While the novelty is incremental (applying known JEPA techniques to a new domain with a specific conditioning mechanism), the significance lies in the rigorous evaluation of action-conditioned latent spaces for music, providing a strong baseline for future research in controllable music generation and understanding. The limitation of requiring paired data is a notable constraint, but the results validate the potential of this paradigm.
The paper proposes Music-JEPA, an action-conditioned Joint Embedding Predictive Architecture for piano music. The core innovation lies in framing music as a dynamical system where audio spectrograms are states and pianoroll/pedal signals are actions. The model predicts future latent states given current states and actions, leveraging the EMA-based training scheme to prevent collapse. The methodology is technically sound, adapting recent vision JEPA techniques to the audio domain with a specific focus on action-conditioning. The use of ViT backbones for both spectrogram and pianoroll inputs is standard but effective. The introduction of an inverse predictor for transcription via planning is a notable conceptual extension, although the implementation relies on amortized optimization rather than direct latent-space planning, which is a pragmatic choice given the high dimensionality of the action space.
The experimental evaluation is comprehensive, covering temporal dynamics analysis, downstream MIR tasks (beat tracking, composer identification, key recognition), and transcription via planning. The results show that Music-JEPA outperforms a passive audio-only JEPA baseline and remains competitive with MERT, a much larger model, on several tasks. The ablation studies on perturbation sensitivity provide strong evidence that the model learns meaningful action-conditioned dynamics rather than just temporal proximity. The transcription results are mixed; while pedal estimation is strong, note-level transcription lags behind supervised methods, which is expected for a generative/planning approach but highlights current limitations. The comparison with MERT is fair and demonstrates the efficiency of the proposed approach.
The paper provides detailed hyperparameters, dataset splits (MAESTRO v3.0.0), and architectural details. The training algorithm is clearly outlined. However, the code and pretrained models are stated to be "released soon," which currently hinders immediate reproducibility. The specific patch sizes and embedding dimensions are provided, allowing for architectural replication.
The primary limitation is the reliance on paired audio-MIDI data, which restricts the model to domains with precise action annotations (like piano) and prevents direct application to unannotated audio. The transcription performance, particularly for notes, is inferior to supervised methods, indicating that the latent space may not be perfectly invertible or that the planning objective is not yet optimized for high-fidelity reconstruction. The model's ability to generalize to other instruments or genres is untested. Additionally, the "planning" aspect is implemented via an inverse model (amortized planning) rather than search-based planning, which limits the interpretability of the "world model" aspect in terms of explicit trajectory optimization.
This work contributes to the growing field of world models for audio and music information retrieval. By demonstrating that action-conditioned latent dynamics can capture musical structure, it opens avenues for more interactive and controllable music AI systems. The approach could inspire similar frameworks for other domains where actions influence observations, such as robotics or speech synthesis. The focus on self-supervised learning with minimal labeled data aligns with broader trends in efficient AI. [One sentence main contribution]. The paper introduces Music-JEPA, an action-conditioned world model that learns latent dynamics of piano music by predicting future audio states from current states and pianoroll actions. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. This work makes a significant contribution to self-supervised learning in audio by moving beyond passive representation learning to active, action-conditioned modeling. The technical execution is solid, with clear demonstrations of learned dynamics and competitive performance on downstream tasks. While the novelty is incremental (applying known JEPA techniques to a new domain with a specific conditioning mechanism), the significance lies in the rigorous evaluation of action-conditioned latent spaces for music, providing a strong baseline for future research in controllable music generation and understanding. The limitation of requiring paired data is a notable constraint, but the results validate the potential of this paradigm.
Real-time score following from sheet images remains chal- lenging because the model must process streaming au- dio while resolving highly repetitive visual patterns un- der strict latency constraints. Recent image-based meth- ods have attempted to use multi-resolution prediction by simultaneously predicting the positions of the active sys- tem, bar, and note. However, their predictions across these different levels of notation are independent, which makes the predictions unstable and introduces unnecessary ex- tra search space for bar- and note-level predictions. Most existing methods also lack mechanisms to recover from score discontinuities, such as repeats, da capo (D.C.), or coda jumps. This paper proposes CODA, to the best of our knowledge, the first real-time score following system that addresses both gaps. CODA explicitly exploits the cascaded structure of music scores: it first selects the ac- tive system, then the active bar within it, and finally the active note within the selected bar. This enforces pre- diction consistency across resolutions. A silence-driven break mode enables recovery from arbitrary score discon- tinuities without requiring knowledge of the repeat struc- ture. Evaluated on the Multimodal Sheet Music Dataset (MSMD) piano benchmarks, CODA achieves state-of-the- art tracking accuracy and discontinuity-recovery perfor- mance under real-time throughput. Code is available at https://github.com/ValleyC/CODA.
Primary: University of Alberta
All Institutions: University of Alberta, Western University
This paper presents a significant methodological advancement in real-time image-based score following by introducing a cascaded selection architecture and a novel silence-driven recovery mechanism, effectively addressing the critical challenges of geometric consistency and score discontinuity handling.
The paper proposes CODA, a cascaded online alignment framework for real-time image-based score following. The core methodological innovation is the reformulation of the tracking problem from independent multi-resolution object detection (as seen in prior work like CYOLO-SB) to a cascaded selection-and-regression task. By explicitly predicting the system, then the bar within that system, and finally the note within that bar, the model enforces geometric consistency and significantly reduces the search space at each stage. The architecture utilizes a causal Mamba encoder for audio processing, which efficiently handles streaming data with linear complexity, paired with a CNN visual backbone. A key technical contribution is the "silence-driven break mode," which leverages the observation that score discontinuities (repeats, da capo, coda jumps) are often preceded by silence. This allows the system to reset its temporal priors and re-localize without requiring explicit symbolic knowledge of the score's repeat structure. The use of beam search with learned temporal priors further stabilizes the predictions.
The evaluation is conducted on the Multimodal Sheet Music Dataset (MSMD), a standard benchmark for this task. The authors report state-of-the-art results in tracking accuracy on both synthetic and real-audio settings compared to strong baselines like CYOLO-SB. Crucially, they introduce a new "repeat-aware jump test benchmark" with manually annotated repeat structures for the 94 test pieces, addressing a significant gap in the literature where discontinuity handling was previously unstandardized. The results show substantial improvements in system and bar accuracy, and notably, CODA demonstrates robust recovery from jumps, whereas baselines fail. The ablation studies effectively isolate the contributions of the cascade structure, beam search, and cross-attention mechanisms.
The paper provides detailed implementation details, including audio preprocessing (log-filterbank, hop size), model architecture dimensions, training schedules (two-phase curriculum with scheduled sampling), and hyperparameters. The code is made publicly available on GitHub. The dataset (MSMD) is publicly accessible. The inclusion of a specific benchmark for jump recovery enhances reproducibility for future work in this niche area.
The method is evaluated exclusively on solo piano music from the MSMD dataset. The performance on more complex musical textures (chamber music, orchestral) or different instruments is not reported. The reliance on synthesized audio for the primary evaluation setting (Setting I) may not fully capture the challenges of real-world acoustic recordings, although Setting II provides some real-audio validation. The "silence" assumption for jump recovery, while effective for many classical pieces, might fail in continuous musical passages where performers execute jumps without significant pauses.
This work advances the field of Human-Computer Interaction (HCI) in music, specifically interactive accompaniment and automated page-turning systems. By enabling robust real-time score following from images without requiring symbolic scores, it lowers the barrier to entry for using digital sheet music in performance settings. The standardized benchmark for discontinuity handling will facilitate fairer comparisons in future research. This paper presents a significant methodological advancement in real-time image-based score following by introducing a cascaded selection architecture and a novel silence-driven recovery mechanism, effectively addressing the critical challenges of geometric consistency and score discontinuity handling.
Omni models transcribe clean, single-speaker speech well, but their accuracy drops sharply when speakers overlap and the scene is noisy, exactly where knowing who said what matters most. A natural fix is a short scene description. We show why this is risky: answer-bearing text lets the model copy instead of listen, so the score rises although nothing has been heard; a silent test exposes this shortcut at once. We call this failure mode perception bypass and address it with Audio-Grounded Scaffold Context (AGSC). AGSC links three steps: first, we build clues from audio to guide listening without giving the answer; second, answer-overlap and silence tests probe them for leakage and audio dependence; finally, those clues scaffold training but vanish at test time, yielding no-clue capability. Across three heterogeneous Omni models, training on AGSC lowers no-clue capped mean permutation word error rate (mpWER) on overlapping, noisy speech from 25%-71% to 9%-15%. For streaming control, we formulate a joint GDPO task in which the model learns when to use a clue and how to produce a speaker-attributed transcript from separately normalized format, gate, and transcript rewards. After internalization, AGSC adds almost no inference overhead.
Primary: unknown
All Institutions: unknown
This paper introduces AGSC, a training framework that uses audio-grounded scaffolds and a novel "silent test" evaluation to prevent perception bypass in Omni models, significantly improving robustness in overlapping and noisy speech scenarios through internalized learning.
The paper proposes Audio-Grounded Scaffold Context (AGSC), a framework designed to mitigate "perception bypass" in Omni modelsâwhere models rely on textual context rather than audio signals. The methodology involves synthesizing "clues" (partial, shuffled words, time windows) from audio using off-the-shelf front-ends (diarization, ASR, VAD). These clues are used during Supervised Fine-Tuning (SFT) and Group Reward-Decoupled Policy Optimization (GDPO). The core innovation lies in the training protocol: using a "silent test" (silencing the audio input) to detect if the model is copying text, and ensuring that performance gains persist even when the scaffold clues are removed at inference time (internalization). The GDPO component introduces a novel reward structure with separate normalization for format, gate, and transcript rewards, and a conditional reward mechanism to prevent gate collapse. While the concept of using scaffolds is not entirely new, the specific application to preventing perception bypass in Omni models and the rigorous "silent test" evaluation metric constitute a meaningful methodological contribution to robust multimodal training.
The authors evaluate their approach on three heterogeneous Omni models (Qwen3-Omni, MiniCPM-o, Ming-flash-omni) using a custom benchmark, Context-Speech Bench (CSB), which includes overlapping and noisy speech scenarios. The results show significant improvements in mean permutation Word Error Rate (mpWER) for the smaller models (Ming, MiniCPM) after SFT, reducing errors from ~25-68% to ~8-15%. The paper provides a thorough ablation study, including the "silent test" to prove that gains are not due to text copying. The GDPO experiments demonstrate the ability to learn gating strategies, although the full-chain joint training shows some instability (gate collapse) which is addressed with a secondary supervised step. The evaluation is comprehensive, covering both offline transcription and streaming control, and the use of multiple models strengthens the generalizability claims.
The paper states that code, benchmark manifests, and result evidence will be open-sourced upon acceptance. The methodology describes the pipeline components (Pyannote, Silero, Sepformer) and the training hyperparameters (LoRA ranks, learning rates). However, the reliance on specific versions of large base models (Qwen3, MiniCPM, Ming) and the complex synthesis pipeline for clues may pose reproducibility challenges. The description of the GDPO reward normalization and the specific "silent test" protocol is detailed enough to allow replication by researchers familiar with RLHF techniques.
The paper acknowledges that the external detector for gating has lower performance than the learned gate but notes the learned gate can suffer from collapse. The improvements are less pronounced for the strongest baseline model (Qwen3), suggesting diminishing returns for already high-performing systems. The "clue" generation relies on imperfect off-the-shelf tools, which might introduce noise or errors that the model must learn to ignore, potentially limiting the ceiling of performance. Furthermore, the "silent test" is an aggregate control and may not catch all forms of shortcut learning. The paper also notes that unconditional clue usage at test time can hurt performance, highlighting the necessity of the learned gate.
This work addresses a critical failure mode in multimodal AI systems, enhancing their reliability in real-world scenarios like meetings and call centers where audio is often noisy or overlapping. By preventing "perception bypass," the method ensures that models are genuinely listening, which is crucial for trust and safety in automated transcription and assistance systems. The Context-Speech Bench provides a valuable resource for evaluating robustness in challenging acoustic conditions. This paper introduces AGSC, a training framework that uses audio-grounded scaffolds and a novel "silent test" evaluation to prevent perception bypass in Omni models, significantly improving robustness in overlapping and noisy speech scenarios through internalized learning.