In this report, we present Qwen-Audio-3.0-TTS, a production-oriented speech synthesis system that jointly advances content consistency, speaker similarity, prosodic naturalness, audio quality, controllability, multilingual coverage, efficiency, and robustness. It combines a 12.5~Hz low-frame-rate speech tokenizer for reduced inference latency with a five-stage progressive training paradigm for coordinated language model (LM) and flow-matching model (FM) optimization. The model provides production-level control through free-style natural-language instructions and fine-grained inline tags, while supporting 16 languages, 20 Chinese dialect regions, one-pass long-form synthesis up to 3 minutes, and robust generation from noisy, reverberant, or unclear reference speech. Across SEED-TTS-Eval, CV3-Eval, instruction-following, long-form, and acoustic-robustness evaluations, Qwen-Audio-3.0-TTS achieves state-of-the-art performance on many reported dimensions or the strongest aggregate results. It also ranks first on the independent Artificial Analysis Text-to-Speech Leaderboard. These results establish Qwen-Audio-3.0-TTS as a strong foundation for production-level speech synthesis.
Primary: Alibaba Group
All Institutions: Alibaba Group
[One sentence main contribution]. Qwen-Audio-3.0-TTS introduces a five-stage progressive training paradigm with joint LM-FM optimization and reinforcement learning to achieve state-of-the-art production-level speech synthesis with robustness and fine-grained control. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a significant technical contribution by systematically addressing the trade-offs between efficiency, fidelity, and controllability in TTS. The integration of GRPO for both LM and FM optimization is a novel application of RLHF techniques to speech generation, moving beyond simple supervised fine-tuning. The explicit focus on acoustic robustness through training-time augmentation is highly relevant for real-world deployment. The comprehensive evaluation suite, including new benchmarks for robustness and instruction following, sets a high standard for future TTS research. The results demonstrate that a unified model can outperform specialized systems in multiple dimensions, marking a step towards more general-purpose speech generation models.
The paper presents Qwen-Audio-3.0-TTS, a production-oriented text-to-speech system that integrates a 12.5 Hz speech tokenizer, a language model (LM), and a flow-matching model (FM). The core methodological contribution lies in a five-stage progressive training paradigm. Key innovations include: 1) Conditioning the FM on continuous LM hidden states rather than just discrete tokens to mitigate information bottleneck; 2) High-quality data annealing during joint LM-FM training; 3) Application of Group Relative Policy Optimization (GRPO) to the LM for content/prosody rewards and to the FM for speaker similarity/perceptual quality rewards; 4) Explicit acoustic robustness training using degraded prompts (noise, reverb, etc.) without separate denoising modules. The architecture combines autoregressive semantic planning with continuous acoustic generation, aiming to balance latency, fidelity, and controllability.
The evaluation is comprehensive and rigorous, covering multiple dimensions: zero-shot voice cloning (SEED-TTS-Eval, CV3-Eval), multilingual/cross-lingual synthesis, instruction following (VStyle), long-form generation, text normalization, and acoustic robustness. The paper provides extensive objective metrics (CER/WER, SIM, DNSMOS) and subjective human evaluations (Arena-style, dialect authenticity). Results indicate state-of-the-art performance on many reported dimensions, particularly in robustness and instruction following, while maintaining competitive content consistency and speaker similarity. The inclusion of a dedicated "hard-case" robustness benchmark and detailed ablation studies on tokenizer frame rates adds significant value.
The paper provides detailed descriptions of the architecture, training stages, and data types. It mentions specific models used for evaluation (Whisper, Paraformer, ERes2Net, DNSMOS) and provides a demo URL. However, full code and model weights are not explicitly stated as open-source in the provided text (though the project URL points to a GitHub page, the text itself focuses on the report). The detailed training recipe (5 stages, specific rewards) is described well enough for reproduction by experts in the field.
The paper acknowledges that pushing content accuracy lower can come at the expense of naturalness, suggesting a trade-off. The robustness training, while effective, relies on specific augmentation strategies that may not cover all real-world edge cases. The reliance on LLM-based evaluators (Gemini) for some subjective metrics introduces potential bias, although human evaluation is also provided. The "high-quality data annealing" requires careful curation, which may limit scalability if high-quality labeled data is scarce.
This work advances the field of speech synthesis by providing a robust, controllable, and efficient system suitable for production deployment. It addresses critical pain points like robustness to noisy prompts and fine-grained control, which are essential for real-world applications. The open-sourcing of such models (implied by the project URL) can democratize access to high-quality TTS technology. However, the ease of voice cloning also raises concerns about misuse for deepfakes, necessitating responsible AI practices. [One sentence main contribution]. Qwen-Audio-3.0-TTS introduces a five-stage progressive training paradigm with joint LM-FM optimization and reinforcement learning to achieve state-of-the-art production-level speech synthesis with robustness and fine-grained control. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a significant technical contribution by systematically addressing the trade-offs between efficiency, fidelity, and controllability in TTS. The integration of GRPO for both LM and FM optimization is a novel application of RLHF techniques to speech generation, moving beyond simple supervised fine-tuning. The explicit focus on acoustic robustness through training-time augmentation is highly relevant for real-world deployment. The comprehensive evaluation suite, including new benchmarks for robustness and instruction following, sets a high standard for future TTS research. The results demonstrate that a unified model can outperform specialized systems in multiple dimensions, marking a step towards more general-purpose speech generation models.
Automatic analysis of multimodal speech has shown strong potential for computationally detecting and monitoring a wide range of neurological, psychiatric, and respiratory conditions. However, progress in this field is limited by existing publicly accessible datasets, which are often small in scale, focused on a single condition or disease, and primarily speech focused. Moreover, if key confounding variables such as education, medication use, comorbidities, or mood state are insufficiently documented, the reliability and interpretability of computational analyses are further compromised. To address these limitations, we introduce CARE v1.0, a curated multimodal English dataset of approximately 144 hours of short video interviews collected from 612 individuals across 12 medical conditions plus a control cohort. For each video, a comprehensive set of clinically relevant multimodal descriptors is provided, alongside structured metadata covering factors such as medication, life impacts, and expressed emotions. The corpus's breadth and heterogeneity support a wide range of applications, including automatic disease and symptom detection, multimodal modelling of speech and non-verbal behaviour under emotionally charged contexts, and studies of disease trajectories and coping processes.
Primary: Universitat Politècnica de València
All Institutions: Universitat Politècnica de València, University of Oxford, Instituto de Telecomunicações
This paper presents CARE v1.0, a comprehensive multimodal corpus of 144 hours of video interviews from 612 individuals across 12 medical conditions, enriched with automatically extracted multimodal descriptors and structured metadata, addressing critical gaps in scale, diversity, and clinical context in existing digital health datasets.
The paper introduces CARE v1.0, a large-scale multimodal corpus derived from the HEXI platform. The methodology focuses on data curation and feature engineering rather than novel algorithmic development. The authors employ a rigorous pipeline for extracting multimodal descriptors: audio features (eGeMAPS, DisVoice, XLS-R, TRILLsson), visual features (OpenFace for facial landmarks/AUs, gaze, head pose, MediaPipe for body keypoints), and linguistic features (WhisperX alignment, sentence embeddings). A key methodological contribution is the use of LLMs (gpt-oss-120b) to extract structured metadata (demographics, emotions, comorbidities) from unstructured narrative texts, validated against a manually annotated subset. This approach addresses a significant gap in existing datasets by providing rich, clinically relevant metadata alongside raw multimodal signals.
The experimental section is primarily descriptive, focusing on the statistical properties of the dataset (distribution of conditions, demographics, emotion frequencies) and the validation of the LLM-based metadata extraction pipeline. The authors compare five LLMs for metadata extraction, demonstrating that gpt-oss-120b performs best in terms of accuracy and adherence to constraints. They provide a baseline evaluation workflow for a binary classification task (Control vs. Parkinson's) using an SVM on aggregated features. While this demonstrates the utility of the dataset, the lack of extensive benchmarking against state-of-the-art multimodal models limits the assessment of the dataset's ability to drive performance improvements in specific tasks. The validation of the metadata extraction is solid, but the downstream ML performance is only illustrative.
The paper excels in reproducibility. The dataset is hosted on Hugging Face, and the code for feature extraction and the example evaluation workflow is publicly available. The detailed documentation of the feature extraction pipeline, including specific tools (OpenFace, DiariZen, WhisperX) and parameter settings, allows other researchers to replicate the feature computation. The use of deterministic settings for LLM inference (temperature=0) further ensures reproducibility of the metadata annotations.
The dataset is skewed towards White UK/Irish participants, limiting its generalizability to other ethnic groups. The control group is heterogeneous (caregivers, bereaved, professionals), which may introduce confounding variables if not carefully handled in downstream analyses. The reliance on LLMs for metadata extraction, while validated, introduces potential biases and hallucinations (though rare, as noted). The dataset is cross-sectional, limiting longitudinal studies of disease trajectories, although the authors mention this as a potential future application. The "in-the-wild" nature of some video clips (varying quality, lighting) may pose challenges for robust computer vision models.
CARE v1.0 has significant potential impact on the field of digital health and affective computing. By providing a large, multimodal, and clinically annotated dataset, it enables research into automated detection and monitoring of a wide range of neurological, psychiatric, and respiratory conditions. The inclusion of diverse medical conditions and rich metadata supports the development of more robust and interpretable computational models. The dataset can facilitate studies on the interplay between speech, non-verbal behavior, and emotional states in health contexts, contributing to a better understanding of disease manifestations and coping processes. This paper presents CARE v1.0, a comprehensive multimodal corpus of 144 hours of video interviews from 612 individuals across 12 medical conditions, enriched with automatically extracted multimodal descriptors and structured metadata, addressing critical gaps in scale, diversity, and clinical context in existing digital health datasets.
Audio-encoder-LLM-decoder architectures have become the dominant paradigm for modern automatic speech recognition (ASR), improving transcription quality through large-scale language modeling. However, the cost of autoregressive decoding scales with decoder size, creating a fundamental trade-off between recognition quality and serving latency. We argue this trade-off is not inherent: unlike open-ended text generation, ASR outputs are strongly anchored to the input speech signal, providing a natural inductive bias toward high-parallelism decoding. Building on this, we introduce ParaASR, an ASR system that leverages Multi-Token Prediction (MTP) to let a 4B LLM decoder emit multiple tokens per forward step. Starting from a publicly available audio-language foundation, the model first establishes a robust autoregressive recognizer and then aligns five future-token branches through a staged optimization recipe. At inference, it proposes a six-token continuation per step and admits only the verified prefix into the transcript, preserving the safety of standard autoregressive decoding. The average accepted length reaches 5.0 out of 6 proposed tokens, confirming that the deterministic structure of speech makes ASR an especially natural setting for multi-token decoding. ParaASR further retains a native 32K-context window and transcribes up to 30 minutes of audio in a single pass. Across diverse benchmarks, it attains average error rates of 2.97%, 3.68%, and 3.70% on Chinese, English, and long-form evaluations, respectively, while reaching a real-time factor (RTF) as low as 0.0053. These results show that decoder scaling, low-latency inference, and long-context transcription need not be competing goals when future-token proposals are anchored by the acoustic signal and guarded by autoregressive verification.
Primary: StepFun
All Institutions: StepFun, NTU, PKU, UNSW, SJTU, USTC
ParaASR effectively adapts multi-token prediction to ASR, achieving a compelling balance between state-of-the-art accuracy and low-latency inference, thereby challenging the perceived trade-off between model scale and serving efficiency in LLM-based speech recognition.
The paper proposes ParaASR, an LLM-based Automatic Speech Recognition (ASR) system that integrates Multi-Token Prediction (MTP) to accelerate inference. The core methodological contribution is the application of speculative decoding principles (specifically MTP) to the ASR domain, leveraging the deterministic nature of speech-to-text mapping to propose multiple future tokens per forward pass. The architecture consists of a frozen audio encoder, a linear adapter, and a 4B LLM decoder with five auxiliary MTP branches. The training recipe is staged: audio-language pretraining, ASR supervised fine-tuning (SFT), and finally MTP alignment. The MTP training involves a frozen-branch alignment stage followed by joint calibration. The approach is technically sound and logically motivated; the insight that ASR outputs are more predictable than open-ended text generation is valid and well-supported by the high acceptance rates reported. However, the specific architectural modifications (adding MTP heads) are incremental adaptations of existing speculative decoding techniques rather than a fundamentally new algorithmic breakthrough.
The evaluation covers a wide range of benchmarks including Chinese (AISHELL-1, AISHELL-2, WenetSpeech), English (LibriSpeech, Common Voice, VoxPopuli, Earnings22), and long-form ASR tasks. The results demonstrate state-of-the-art accuracy for an LLM-based ASR system, with low CER/WER scores. Crucially, the paper reports a Real-Time Factor (RTF) of 0.0053 on a single H800 GPU, which is a significant efficiency improvement over standard autoregressive decoding. The ablation studies confirm that MTP does not degrade accuracy and that the staged training is effective. The comparison against baselines like VibeVoice-ASR, FunASR-Nano, and Qwen3-ASR-1.7B is appropriate. The long-form evaluation using a constructed Wenet testnet set adds value, though the construction method (concatenating segments) is a simplification of true long-form challenges.
The paper provides detailed descriptions of the training stages, data mixture (100K hours short-form, 50K hours long-form pseudo-labeled), and hyperparameters (learning rates, batch sizes). However, the use of proprietary datasets and the "publicly available audio-language foundation" (likely StepAudio, which is not fully open-weight or documented in sufficient detail for exact reproduction) limits reproducibility. The specific details of the long-form data curation pipeline (VAD, ROVER, LLM refinement) are described but the code for this pipeline is not provided. The model weights are not mentioned as open-sourced.
The paper does not extensively discuss the computational overhead of the MTP branches during training or the memory footprint. The reliance on pseudo-labeled data for long-form training introduces potential error propagation, although the filtering mechanism ($e > 0.05$) mitigates this. The performance on code-switching and highly noisy environments is mentioned but specific metrics are not detailed in the abstract/text provided. The generalization of the high acceptance rate (5.0/6) to out-of-distribution domains (e.g., heavy accents, different languages not seen in training) is not fully explored.
This work demonstrates that large language models can be efficiently deployed for real-time speech recognition, potentially lowering the barrier for high-quality ASR services. It highlights the synergy between acoustic grounding and language modeling efficiency. The ability to transcribe long audio in a single pass has significant implications for meeting transcription, broadcast monitoring, and archival processing. ParaASR effectively adapts multi-token prediction to ASR, achieving a compelling balance between state-of-the-art accuracy and low-latency inference, thereby challenging the perceived trade-off between model scale and serving efficiency in LLM-based speech recognition.
In this paper, we introduce noise-aware self-supervised learning (NA-SSL) models for noise-aware anomalous sound detection (NA-ASD). NA-ASD is an ASD task with two-channel audio recordings, where one microphone is located close to the target machine and the other is located farther away to capture noise. For this task, we simulate two-channel recordings using diverse audio datasets and train NA-SSL models to extract clean SSL representations of the close-microphone signal by using the far-microphone recording dominated by background noise as auxiliary information. The NA-SSL models are then used as frontends in the standard ASD framework. Our experimental evaluation on the DCASE 2026 Challenge Task 2 development dataset demonstrates the effectiveness of the NA-SSL framework across three base SSL models (BEATs, EAT, and Dasheng), both with and without discriminative fine-tuning. Furthermore, the challenge results proved the effectiveness of the proposed approach, where the NA-BEATs system won the challenge by a large margin, achieving an official score of 70.24%, while the second-place system achieved 65.46%.
Primary: Mitsubishi Electric Research Laboratories (MERL)
All Institutions: Mitsubishi Electric Research Laboratories (MERL), Nagoya University, Information Technology R&D Center, Mitsubishi Electric Corporation
The paper presents a robust and effective Noise-Aware Self-Supervised Learning framework that successfully leverages auxiliary noise information to improve Anomalous Sound Detection, achieving first place in the DCASE 2026 Challenge.
The paper proposes a Noise-Aware Self-Supervised Learning (NA-SSL) framework tailored for the Noise-Aware Anomalous Sound Detection (NA-ASD) task. The core methodology involves adapting existing Self-Supervised Learning (SSL) models (BEATs, EAT, Dasheng) by inserting trainable Noise-Aware (NA) layers that utilize cross-attention to condition the representation of the close-microphone signal on the far-microphone noise signal. This approach leverages the physical setup of the NA-ASD task, where a distant microphone captures ambient noise. The authors simulate two-channel recordings using diverse datasets (FSD50K, WHAM!, DEMAND, QUT-NOISE) to pre-train these NA-SSL models. The method is then integrated into a standard ASD pipeline with discriminative fine-tuning and various backend aggregation techniques (BEAM, RDP). While the application of cross-attention for noise conditioning is not entirely new in speech processing, its systematic adaptation and evaluation across multiple modern, large-scale audio SSL backbones for the specific NA-ASD challenge context represents a solid engineering contribution. The novelty lies in the specific integration strategy and the demonstration of its efficacy across different architectural families (masked prediction, masked bootstrapping, masked autoencoder).
The experimental evaluation is comprehensive and rigorous, centered on the DCASE 2026 Challenge Task 2. The authors evaluate their method on the development set with detailed ablation studies across different base models, fine-tuning strategies (with/without discriminative fine-tuning), and backend techniques. The results show consistent improvements over the original SSL models. Crucially, the paper reports the official challenge results, where the proposed NA-BEATs system won the challenge with a significant margin (70.24% vs 65.46% for second place). This real-world validation on a blind test set provides strong evidence of the method's effectiveness and robustness. The inclusion of variance analysis and ensemble results further strengthens the empirical claims. The comparison against other top-performing systems highlights the efficiency of the proposed approach, which achieves state-of-the-art performance using a single model rather than large ensembles.
The paper provides substantial implementation details, including the architecture of the NA layers (number of heads, hidden size), training hyperparameters (optimizer, learning rate, batch size, EMA decay), data simulation parameters (room impulse response settings, SNR ranges), and backend configurations. The use of standard libraries (Pyroomacoustics) and well-known base models facilitates reproduction. However, the specific random seeds for the "three trials" are mentioned but not explicitly listed in the text, which might introduce slight variability in exact replication. The code for the NA layers is not explicitly linked, though the architecture is described in sufficient detail for implementation.
The primary limitation is the reliance on simulated data for pre-training the NA-SSL models. While the simulation uses diverse datasets, the gap between simulated room acoustics and real-world factory environments may still exist. The method is specifically designed for the NA-ASD task with two-microphone setups; its applicability to single-microphone scenarios or other noise-aware tasks is not explored. Additionally, the performance gain, while significant, comes with the computational overhead of the additional NA layers and the need for the far-microphone signal, which may not always be available or feasible to deploy in all industrial settings. The paper does not extensively discuss the failure cases or scenarios where the far-microphone signal might be corrupted or uninformative.
This work contributes to the field of industrial anomaly detection, which has significant implications for predictive maintenance, safety, and efficiency in manufacturing. By improving the robustness of ASD systems in noisy environments, it enables more reliable deployment of automated monitoring systems. The noise-aware SSL framework could potentially be adapted to other audio processing tasks requiring robustness to environmental noise, such as speech recognition or audio event detection in challenging acoustic conditions. The paper presents a robust and effective Noise-Aware Self-Supervised Learning framework that successfully leverages auxiliary noise information to improve Anomalous Sound Detection, achieving first place in the DCASE 2026 Challenge.
Bandwidth extension (BWE) aims to recover missing high-frequency content from band-limited speech. Existing methods often formulate BWE as a fixed or predefined bandwidth conversion problem, potentially requiring cutoff-specific models or retraining when the input bandwidth changes. This assumption limits their applicability to practical scenarios where speech may arrive with diverse cutoff frequencies. We propose AnyBand, a unified BWE framework that recasts bandwidth extension as in-context spectral infilling. Motivated by prompt-based zero-shot speech generation, AnyBand conditions high-frequency generation on the observed low-frequency spectrum, using the available band as a frequency-domain prompt that conveys content, speaker, prosodic, and spectral-envelope cues. This formulation enables a single model to perform cutoff-conditioned generation over a continuous range of input bandwidths. AnyBand is trained with missing-band conditional flow matching and an Easy-to-Balanced cutoff curriculum over continuously sampled cutoff frequencies. To better exploit the spectral prompt, we introduce a frequency-aware Diffusion Transformer that models cross-frequency interactions and long-range temporal dependencies, followed by a physically motivated multi-view adversarial refinement stage to enhance spectral realism, envelope coherence, and harmonic consistency. Experiments on multiple datasets and bandwidth settings show that AnyBand consistently improves spectral reconstruction over existing baselines while achieving competitive perceptual quality across both standard and irregular input cutoffs. Audio samples are available.
Primary: National University of Singapore
All Institutions: National University of Singapore, Unknown
[One sentence main contribution]. AnyBand introduces a unified, in-context spectral infilling framework for speech bandwidth extension using a frequency-aware Diffusion Transformer and a novel cutoff curriculum. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a well-motivated and technically sophisticated approach to a long-standing problem in audio processing. By reframing BWE as a conditional generation task conditioned on a spectral prompt, it elegantly handles continuous bandwidth variations. The integration of frequency-aware modeling within a DiT architecture is a strong technical contribution that addresses the specific inductive biases required for spectral data. While the novelty is incremental in the context of the rapid evolution of diffusion models, the specific application to unified BWE with the proposed curriculum and adversarial refinement is valuable. The work is significant for its practical utility in creating flexible, single-model solutions for diverse bandwidth scenarios, potentially simplifying deployment in real-world applications.
The paper proposes AnyBand, a unified framework for speech bandwidth extension (BWE) that treats the task as in-context spectral infilling. The core innovation lies in conditioning a Diffusion Transformer (DiT) on the observed low-frequency spectrum as a "frequency-domain prompt" to generate the missing high-frequency content. The architecture augments a temporal DiT backbone with explicit frequency encoders/decoders to model cross-frequency interactions. The training strategy employs a novel "Easy-to-Balanced" cutoff curriculum to handle continuously varying cutoff frequencies, avoiding the need for separate models per bandwidth. Additionally, a multi-view adversarial refinement stage is introduced to enhance spectral realism, envelope coherence, and harmonic consistency. The approach is technically sound, leveraging recent advances in flow matching and diffusion transformers, and applies the concept of in-context learning (typically seen in LLMs) to the spectral domain.
The authors evaluate AnyBand on VCTK and EARS datasets, comparing against baselines like NU-Wave 2, AudioSR, FLowHigh, and Fre-Painter. Metrics include LSD, LF-LSD, HF-LSD, NISQA, COL, and STOI. The results show consistent improvements in spectral reconstruction (lower LSD) and competitive perceptual quality (NISQA) across various input cutoffs (2kHz, 4kHz, etc.). The experiments are comprehensive, covering both in-domain and out-of-domain settings. However, the reliance on objective metrics like NISQA and LSD, while standard, lacks extensive subjective human evaluation (MOS/MUSHRA) which is crucial for perceptual quality claims in audio restoration. The comparison is fair, but the marginal gains in some perceptual metrics over strong baselines like AudioSR warrant caution in claiming a transformative leap.
The paper provides detailed architectural descriptions, loss functions, and training procedures. The inclusion of a demo page suggests code or at least pre-trained models might be available, though no GitHub URL is explicitly provided in the text. The "Easy-to-Balanced" curriculum and specific discriminator designs are described with sufficient mathematical detail for reproduction. The use of standard solvers (Heun) and common datasets enhances reproducibility.
The method assumes the input low-frequency spectrum is clean and correctly aligned; it does not address blind BWE where the cutoff frequency is unknown or the input is degraded by noise/compression beyond simple band-limiting (though BABE is cited as related work, AnyBand itself requires the mask/cutoff extent). The computational cost of the DiT backbone and adversarial refinement may be higher than lightweight CNN-based BWE systems. The paper does not provide extensive ablation studies on the contribution of each discriminator component or the specific impact of the frequency encoder/decoder versus a standard DiT.
AnyBand has significant potential for improving speech quality in legacy communication systems, audio restoration, and voice conversion where bandwidth constraints are dynamic. By unifying multiple bandwidth settings into a single model, it reduces the engineering overhead of deploying multiple specialized models. However, like all generative audio tools, it carries risks of misuse in deepfake generation or misrepresentation, although BWE is generally considered a benign enhancement task. [One sentence main contribution]. AnyBand introduces a unified, in-context spectral infilling framework for speech bandwidth extension using a frequency-aware Diffusion Transformer and a novel cutoff curriculum. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a well-motivated and technically sophisticated approach to a long-standing problem in audio processing. By reframing BWE as a conditional generation task conditioned on a spectral prompt, it elegantly handles continuous bandwidth variations. The integration of frequency-aware modeling within a DiT architecture is a strong technical contribution that addresses the specific inductive biases required for spectral data. While the novelty is incremental in the context of the rapid evolution of diffusion models, the specific application to unified BWE with the proposed curriculum and adversarial refinement is valuable. The work is significant for its practical utility in creating flexible, single-model solutions for diverse bandwidth scenarios, potentially simplifying deployment in real-world applications.
Audio-encoder-LLM-decoder architectures have become the dominant paradigm for modern automatic speech recognition (ASR), improving transcription quality through large-scale language modeling. However, the cost of autoregressive decoding scales with decoder size, creating a fundamental trade-off between recognition quality and serving latency. We argue this trade-off is not inherent: unlike open-ended text generation, ASR outputs are strongly anchored to the input speech signal, providing a natural inductive bias toward high-parallelism decoding. Building on this, we introduce ParaASR, an ASR system that leverages Multi-Token Prediction (MTP) to let a 4B LLM decoder emit multiple tokens per forward step. Starting from a publicly available audio-language foundation, the model first establishes a robust autoregressive recognizer and then aligns five future-token branches through a staged optimization recipe. At inference, it proposes a six-token continuation per step and admits only the verified prefix into the transcript, preserving the safety of standard autoregressive decoding. The average accepted length reaches 5.0 out of 6 proposed tokens, confirming that the deterministic structure of speech makes ASR an especially natural setting for multi-token decoding. ParaASR further retains a native 32K-context window and transcribes up to 30 minutes of audio in a single pass. Across diverse benchmarks, it attains average error rates of 2.97%, 3.68%, and 3.70% on Chinese, English, and long-form evaluations, respectively, while reaching a real-time factor (RTF) as low as 0.0053. These results show that decoder scaling, low-latency inference, and long-context transcription need not be competing goals when future-token proposals are anchored by the acoustic signal and guarded by autoregressive verification.
Primary: StepFun
All Institutions: StepFun, NTU, PKU, UNSW, SJTU, USTC
ParaASR effectively adapts multi-token prediction to ASR, achieving a compelling balance between state-of-the-art accuracy and low-latency inference, thereby challenging the perceived trade-off between model scale and serving efficiency in LLM-based speech recognition.
The paper proposes ParaASR, an LLM-based Automatic Speech Recognition (ASR) system that integrates Multi-Token Prediction (MTP) to accelerate inference. The core methodological contribution is the application of speculative decoding principles (specifically MTP) to the ASR domain, leveraging the deterministic nature of speech-to-text mapping to propose multiple future tokens per forward pass. The architecture consists of a frozen audio encoder, a linear adapter, and a 4B LLM decoder with five auxiliary MTP branches. The training recipe is staged: audio-language pretraining, ASR supervised fine-tuning (SFT), and finally MTP alignment. The MTP training involves a frozen-branch alignment stage followed by joint calibration. The approach is technically sound and logically motivated; the insight that ASR outputs are more predictable than open-ended text generation is valid and well-supported by the high acceptance rates reported. However, the specific architectural modifications (adding MTP heads) are incremental adaptations of existing speculative decoding techniques rather than a fundamentally new algorithmic breakthrough.
The evaluation covers a wide range of benchmarks including Chinese (AISHELL-1, AISHELL-2, WenetSpeech), English (LibriSpeech, Common Voice, VoxPopuli, Earnings22), and long-form ASR tasks. The results demonstrate state-of-the-art accuracy for an LLM-based ASR system, with low CER/WER scores. Crucially, the paper reports a Real-Time Factor (RTF) of 0.0053 on a single H800 GPU, which is a significant efficiency improvement over standard autoregressive decoding. The ablation studies confirm that MTP does not degrade accuracy and that the staged training is effective. The comparison against baselines like VibeVoice-ASR, FunASR-Nano, and Qwen3-ASR-1.7B is appropriate. The long-form evaluation using a constructed Wenet testnet set adds value, though the construction method (concatenating segments) is a simplification of true long-form challenges.
The paper provides detailed descriptions of the training stages, data mixture (100K hours short-form, 50K hours long-form pseudo-labeled), and hyperparameters (learning rates, batch sizes). However, the use of proprietary datasets and the "publicly available audio-language foundation" (likely StepAudio, which is not fully open-weight or documented in sufficient detail for exact reproduction) limits reproducibility. The specific details of the long-form data curation pipeline (VAD, ROVER, LLM refinement) are described but the code for this pipeline is not provided. The model weights are not mentioned as open-sourced.
The paper does not extensively discuss the computational overhead of the MTP branches during training or the memory footprint. The reliance on pseudo-labeled data for long-form training introduces potential error propagation, although the filtering mechanism ($e > 0.05$) mitigates this. The performance on code-switching and highly noisy environments is mentioned but specific metrics are not detailed in the abstract/text provided. The generalization of the high acceptance rate (5.0/6) to out-of-distribution domains (e.g., heavy accents, different languages not seen in training) is not fully explored.
This work demonstrates that large language models can be efficiently deployed for real-time speech recognition, potentially lowering the barrier for high-quality ASR services. It highlights the synergy between acoustic grounding and language modeling efficiency. The ability to transcribe long audio in a single pass has significant implications for meeting transcription, broadcast monitoring, and archival processing. ParaASR effectively adapts multi-token prediction to ASR, achieving a compelling balance between state-of-the-art accuracy and low-latency inference, thereby challenging the perceived trade-off between model scale and serving efficiency in LLM-based speech recognition.
Balancing sequence length, representational capacity, and long-horizon stability is a central problem in autoregressive (AR) speech and audio generation. Representations with higher frame rates or greater capacity can preserve more signal detail, but they also make streaming generation more vulnerable to distribution drift and AR error accumulation. Conversely, shorter and more compressed representations simplify AR modeling, but their limited bandwidth may discard important components and constrain the upper bound of reconstruction fidelity and generation quality. We ask whether a low-frame-rate, high-dimensional, high-bandwidth continuous representation can be co-designed with a streaming generation framework to support robust high-fidelity reconstruction, strong single-token predictability, and superior long-horizon stability. We decompose this goal into two coupled problems: what geometric and statistical properties a high-dimensional representation space should have, and how an AR continuous-token generator should be structured to resist error accumulation. Accordingly, we propose Locodec, a locally encoded codec that shapes its representation space to improve the interpolatability of a lower-dimensional core manifold and the identifiability of the native high-dimensional coordinates, thereby improving the predictability of high-dimensional high-bandwidth tokens. We also propose MP-ELD, a single-token AR flow-matching framework that uses multi-path information routing and residual classifier-free guidance to mitigate error accumulation. Experiments with 8-Hz, 768-dimensional tokens show that our design preserves reconstruction quality, improves single-token predictability, achieves competitive WER, and maintains stable long-form synthesis, without using external SSL/ASR models, pretrained text language models, or post-training stages.
Primary: ByteDance Seed
All Institutions: ByteDance Seed
The paper presents a theoretically grounded and methodologically novel approach to stable autoregressive speech generation by co-designing a spherical, manifold-constrained tokenizer with a multi-path flow-matching generator. The geometric justification for spherical tokens and the specific mechanisms for inducing interpolatability and identifiability are strong contributions, although the empirical validation appears limited by the lack of detailed comparative metrics in the provided text.
The paper proposes a joint design for a tokenizer (Locodec) and an autoregressive generative model (MP-ELD) aimed at solving the trade-off between high-fidelity reconstruction and long-horizon stability in speech generation. The core technical contributions are: 1) A spherical continuous tokenization scheme where the latent space is shaped by a lower-dimensional interpolatable manifold via orthogonal projection/lifting, combined with a "postfix dimension dropout" mechanism to induce an energy hierarchy (identifiability) among token dimensions. 2) A multi-path encoder-LM-decoder framework for AR flow matching that uses path dropout to separate acoustic and alignment information, allowing for independent classifier-free guidance (CFG) scales to mitigate error accumulation. The theoretical derivation regarding spherical cap covering and the equivalence of x-pred and v-pred in spherical flow matching is mathematically sound and provides a strong geometric justification for the design choices. The approach is novel in its specific combination of spherical geometry constraints, manifold shaping via orthogonal projections, and multi-path guidance for stability.
The paper reports experiments on the Seed-TTS-eval dataset, claiming competitive Word Error Rate (WER) and stable long-form synthesis without external SSL/ASR models or post-training. However, the provided text is truncated before the full experimental results section. Based on the abstract and introduction, the evaluation focuses on WER, reconstruction quality, and stability. The claim of "competitive WER" is significant, but without specific numbers or comparison to state-of-the-art baselines (like VALL-E, AudioLM, or recent flow-matching speech models), the strength of the empirical claim is hard to fully assess. The methodology suggests a strong theoretical basis, but the experimental validation appears to rely on standard metrics which may not fully capture the "stability" improvements claimed (e.g., drift in timbre/loudness over long horizons).
The paper provides detailed mathematical formulations for the spherical corruption, orthogonal projection/lifting, and the flow matching bridge. The description of the "postfix dimension dropout" and the multi-path routing is specific. However, the truncation of the text means the full architectural details of the MP-ELD model and the exact hyperparameters for the training objectives are not fully visible. Assuming the full paper contains these details, reproducibility seems high due to the explicit mathematical definitions. The lack of code release (implied by "none" for project URL) is a barrier.
The primary limitation is the reliance on a specific geometric structure (spherical manifold) which may not generalize to all audio types (e.g., complex music with non-stationary characteristics). The "postfix dimension dropout" introduces an arbitrary ordering of dimensions, which might be suboptimal if the learned energy hierarchy does not align with semantic importance. The claim of stability without external semantic guidance is ambitious; semantic drift is a known issue in purely acoustic autoregressive models, and the paper's mitigation via multi-path CFG may not fully resolve it for complex linguistic content. The truncation of the text prevents a full assessment of the ablation studies and failure cases.
This work contributes to the field of efficient and stable audio generation, potentially enabling more robust streaming speech systems. By reducing reliance on large external SSL models for tokenization, it may lower the barrier to entry for high-quality speech generation. The geometric insights into spherical latent spaces could influence future work in continuous tokenization for other modalities. The paper presents a theoretically grounded and methodologically novel approach to stable autoregressive speech generation by co-designing a spherical, manifold-constrained tokenizer with a multi-path flow-matching generator. The geometric justification for spherical tokens and the specific mechanisms for inducing interpolatability and identifiability are strong contributions, although the empirical validation appears limited by the lack of detailed comparative metrics in the provided text.
Spoken dialog systems are typically designed for clean, dyadic interactions in which a single user and an assistant take turns speaking. Real-world social conversations, however, are often more ambiguous: multiple speakers may participate in the same conversation amid irrelevant speech and background noise. Each utterance may be directed to the assistant, addressed to another speaker, or completely irrelevant. In such settings, the assistant must decide not only what to say, but also whether to speak at all. In this paper, we introduce Cocktail-Talker, a speech LLM framework for multi-speaker spoken dialog modeling in noisy social environments. We model the assistant's behavior with three action tokens: <|respond|>, <|listen|>, and <|ignore|>, placed before a response or silence. Cocktail-Talker is trained via supervised finetuning and reinforcement learning to generate the appropriate action token and, only in <|respond|> mode, a speech response. To prepare the training data, we develop Cocktail-DialogGen, an LLM-based data pipeline that simulates realistic multi-speaker dialogs with speaker roles across diverse social settings. Together, these components take a step toward spoken dialog systems that interact more naturally and selectively in complex social environments.
Primary: Columbia University
All Institutions: Columbia University, University of Illinois Urbana-Champaign
Cocktail-Talker introduces a novel framework for multi-speaker spoken dialog by modeling turn-taking as a discrete action selection problem, trained with GRPO on synthetically generated data, achieving state-of-the-art performance in noisy, multi-speaker environments. The paper makes a significant technical contribution by addressing the under-explored "speaking side" of multi-speaker conversation, moving beyond simple dyadic interactions to more complex social auditory scenes.
The paper proposes "Cocktail-Talker," a framework for multi-speaker spoken dialog in noisy environments. The core methodological contribution is the introduction of three discrete action tokens (`<|respond|>`, `<|listen|>`, `<|ignore|>`) to model turn-taking behavior, trained via Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO). The approach leverages a pre-existing speech LLM backbone (Qwen2.5-Omni) and freezes the audio encoder and TTS decoder, focusing training on the language model head and LoRA adapters. A significant portion of the work involves "Cocktail-DialogGen," an LLM-based pipeline to synthesize training data by simulating multi-speaker conversations with varying acoustic conditions (SNR, background noise) and semantic constraints (anonymous vs. named speakers). The use of GRPO with a reward function based on action accuracy and format integrity is a novel application of RLHF techniques to the specific problem of turn-taking decision-making in speech LLMs.
The evaluation is comprehensive, covering both seen and unseen environments. The paper compares Cocktail-Talker against several strong baselines, including Moshi, PersonaPlex, Step-Audio2, Kimi-Audio, Qwen2.5-Omni, and Qwen3-Omni. The metrics include binary response accuracy (Respond/Silent) and response quality (METEOR, ROUGE-L, BERTScore, SentCos). The results show that Cocktail-Talker significantly outperforms baselines in decision accuracy (macro F1 ~0.93) and response quality. The analysis breaks down performance by SNR, number of speakers, anonymity, and context length, providing robust insights into model behavior. The inclusion of "unseen environments" tests generalization, which is crucial for real-world applicability. The ablation studies (SFT vs. SFT+GRPO) demonstrate the benefit of the reinforcement learning stage.
The paper provides a GitHub link to the code. The data generation pipeline is described in detail, including the use of specific LLMs (Gemini 3 Pro, Qwen3-TTS) and noise datasets (DEMAND, Freesound). However, the reliance on proprietary models (Gemini, Qwen) for data generation and the specific random seeds or exact prompts used in the LLM-based data synthesis might pose reproducibility challenges for other researchers. The hyperparameters for SFT and GRPO are provided, which aids in reproducing the model training.
The paper acknowledges several limitations. The system is not streaming, which limits its real-time applicability. It assumes fixed turn boundaries in the simulation, which may not reflect the fluid nature of real-world interruptions. The model relies heavily on the simulated data, and while it generalizes to unseen environments, the distribution shift between synthetic and real-world cocktail party scenarios remains a potential issue. The performance degrades with increased speaker count and anonymity, indicating room for improvement in complex social reasoning.
This work contributes to the development of more natural and robust speech assistants that can operate in real-world social settings. By enabling selective participation, it reduces the annoyance of assistants interrupting irrelevant conversations or responding to noise. This has implications for the deployment of AI agents in homes, public spaces, and social robotics. However, the potential for misuse in surveillance or eavesdropping scenarios should be considered, although the model is designed to ignore irrelevant speech. Cocktail-Talker introduces a novel framework for multi-speaker spoken dialog by modeling turn-taking as a discrete action selection problem, trained with GRPO on synthetically generated data, achieving state-of-the-art performance in noisy, multi-speaker environments. The paper makes a significant technical contribution by addressing the under-explored "speaking side" of multi-speaker conversation, moving beyond simple dyadic interactions to more complex social auditory scenes.
Audio-capable foundation models enable end-to-end spoken interaction, but they also introduce safety risks beyond transcript content. It remains unclear how much jailbreak capability can arise from matched-text variation in speech delivery rather than from lexical rewriting or broader style transfer. We study this question by holding transcript content fixed and varying six speech-delivery presets whose acoustic attributes may co-vary. We present PJ-Break, a black-box evaluation protocol with presets targeting arousal, authority, and speaking rate, together with AdvAudio-Prosody, a 600-sample benchmark with acoustically verified attributes. On the exact post-QC Qwen2-Audio panel, the Q=1 Panic (38/95), Anger (35/95), and Fast (32/95) presets are all well above Neutral (4/95). The fixed six-query pool covers 44/95 Qwen2-Audio seeds and 15/95 GPT-4o seeds and exceeds a matched-budget StyleBreak reimplementation (27/95) on Qwen2-Audio. A same-voice pool excluding the confounded Commanding condition still reaches 40/95, and a retained-panel ablation shows emotional-delivery audio alone (44/95) is far more effective than emotional text alone (11/95). Exploratory surrogate diagnostics and pilot mitigation observations are secondary, non-core analyses. Overall, matched-text speech delivery should be treated as a first-class factor in Audio LLM safety evaluation
Primary: City University of Hong Kong
All Institutions: City University of Hong Kong, City University of Hong Kong (Dongguan)
The paper makes a valuable contribution to multimodal safety by empirically demonstrating that prosodic variations in speech delivery can significantly compromise the safety of Audio LLMs, independent of lexical content. Its controlled methodology and clear empirical findings provide a necessary foundation for future research into robust, prosody-aware safety mechanisms for spoken AI assistants.
The paper proposes a controlled experimental framework, PJ-Break, to isolate the impact of prosodic variations (arousal, authority, speaking rate) on the safety/jailbreak capabilities of Audio Large Language Models (Audio LLMs). By holding the lexical transcript constant and varying only the speech delivery via TTS presets, the authors address a specific and under-explored gap in multimodal safety: the influence of paralinguistic cues. The methodology is rigorous in its control of variables, utilizing a fixed TTS stack and explicit acoustic verification. The inclusion of an ablation study comparing emotional text vs. emotional audio is a strong methodological choice that directly supports the core hypothesis. The use of surrogate model diagnostics (activation patching) to hypothesize mechanisms, while clearly labeled as exploratory, adds depth to the analysis.
The evaluation is conducted on Qwen2-Audio and GPT-4o, using a benchmark of 600 samples (AdvAudio-Prosody). The results are compelling: prosodic jailbreaks significantly outperform neutral audio and even matched-budget style-transfer baselines (StyleBreak). The finding that "Panic" and "Anger" presets achieve high success rates (38/95 and 35/95 respectively) compared to Neutral (4/95) is a significant empirical contribution. The same-voice sensitivity analysis strengthens the claim that the effect is due to prosody rather than voice identity. However, the reliance on TTS-generated speech limits the ecological validity, although the authors do provide small-scale human-speech and OTA checks to mitigate this concern. The statistical significance (McNemar test) adds robustness to the main comparison.
The paper provides detailed implementation specifications, including the TTS engine (Azure Neural TTS), specific voices used, and the definition of the six presets. The acoustic verification metrics ($F_0$, RMS, etc.) are standard and reproducible. However, the dataset and code are explicitly *not* released due to safety concerns regarding the generation of harmful audio jailbreaks. This is a valid ethical stance for this domain but limits independent verification and extension. The description of the evaluation protocol and judge ensemble is sufficiently detailed for a competent researcher to replicate the core experiments, assuming access to the same models and TTS services.
The primary limitation is the reliance on TTS, which may not fully capture the nuances of human emotional speech or the specific acoustic artifacts that might arise in real-world over-the-air attacks. The "Commanding" condition is partially confounded by voice identity (different speaker), which the authors acknowledge but does not fully eliminate the confound. The study is limited to English and a single TTS provider, limiting generalizability to other languages and synthesis technologies. Furthermore, the surrogate analysis is restricted to one model (Qwen2-Audio), and the mechanistic insights (refusal direction) are hypothesis-generating rather than definitive proof of causality in closed-source models. The pilot mitigation results are preliminary and not robust.
This paper has significant implications for the safety evaluation of multimodal AI systems. It demonstrates that current safety filters, which often focus on text or content, may be bypassed by manipulating the *delivery* of the content. This necessitates a shift in safety research to include prosodic and paralinguistic analysis in red-teaming protocols. It also highlights potential vulnerabilities in crisis-response or assistance-oriented models that might be triggered by high-arousal speech. The work encourages the development of prosody-aware safety mechanisms and more robust multimodal safety benchmarks. The paper makes a valuable contribution to multimodal safety by empirically demonstrating that prosodic variations in speech delivery can significantly compromise the safety of Audio LLMs, independent of lexical content. Its controlled methodology and clear empirical findings provide a necessary foundation for future research into robust, prosody-aware safety mechanisms for spoken AI assistants.
Existing single-domain and multi-task audio systems remain limited in directly organizing speech, music, sound effects, ambience, and multiple roles into long-form temporal scenes. We present Qwen-Audio-3.0-Gen-Preview, a unified non-autoregressive framework that uses a Diffusion Transformer (DiT) and a shared variational autoencoder (VAE) to generate the complete mixed waveform. Prompt enhancement converts free-form requests into structured temporal records that are rendered as textual conditions, while a two-stage data curriculum and semantic conditional views train the proposed model to use these conditions across domains. A shared continuous VAE compresses 48kHz stereo waveforms into 25Hz latent sequences and incorporates semantic supervision, providing one representation for speech, music, sound effects, and their mixtures. On Seed-TTS-Eval, speaker similarity is the proposed model's clearest strength across all three subsets, and on the multi-speaker benchmark, the proposed model shows higher cross-turn consistency than Seed-Audio-1.0 in both languages. On AudioCaps, its advantages are concentrated in evaluations using large audio-language models and AudioBox. Relative to Seed-Audio-1.0, it achieves stronger temporal localization. Using approximately 10% music data of a dedicated in-house model, the proposed model remains close across all seven SongBench components and leads in three while retaining speech and general-audio capabilities. These results demonstrate the potential of unified generation for temporally structured, multi-domain audio.
Primary: Alibaba Token Foundry
All Institutions: Alibaba Token Foundry
This paper presents a significant engineering advancement in unified audio generation by introducing a structured, temporally-aware conditioning framework and a shared continuous latent space, demonstrating strong capabilities in multi-speaker dialogue and scene-level audio organization while maintaining competitive performance across diverse audio domains.
The paper proposes Qwen-Audio-3.0-Gen-Preview, a unified non-autoregressive (NAR) audio generation framework based on a Diffusion Transformer (DiT) and a shared continuous Variational Autoencoder (VAE). The core technical contribution lies in the data and conditioning pipeline rather than a radical architectural shift. The authors introduce a sophisticated "Prompt Enhancement" module that converts free-form text into structured temporal records (global scene, source profiles, time-localized events). This structured data is then rendered into textual conditions for the DiT. A key methodological component is the "Semantic Conditional View" training strategy, which uses conditional dropout on specific semantic units (e.g., hiding dialogue to force the model to rely on scene context) to improve robustness and controllability. The shared VAE is trained with auxiliary semantic supervision using a frozen LLM, aiming to create a latent space that is both acoustically high-fidelity and semantically rich. While the use of DiTs and continuous latents is established in recent audio generation literature (e.g., AudioGen, MusicGen), the specific integration of hierarchical temporal annotations and the two-stage curriculum for long-form scene generation represents a meaningful engineering contribution to the field of unified audio models.
The evaluation is comprehensive, covering Speech (Seed-TTS-Eval), General Audio (AudioCaps), and Music (SongBench). The paper demonstrates strong performance in speaker similarity and cross-turn consistency for multi-speaker dialogue, which is a significant challenge for unified models. On AudioCaps, the model leads in Large Audio-Language Model (LALM) based evaluations, suggesting better semantic alignment than competitors, although it does not lead in standard CLAP scores. The introduction of an in-house benchmark for temporal localization (mIoU) provides valuable insight into the model's ability to follow timing instructions, showing improvement over Seed-Audio-1.0. The comparison with a dedicated in-house music model highlights the efficiency of the unified approach, achieving competitive results with significantly less music data. However, the reliance on LALMs for evaluation introduces potential bias, as the judges are from the same ecosystem (Qwen/Gemini), and the lack of human MOS scores for the general audio/music tasks limits the perceptual validation.
The paper provides detailed descriptions of the data annotation pipeline, synthetic data construction (using Scaper-like recipes), and the training curriculum. The architecture of the VAE and DiT is described, including the latent dimensions (128-dim, 25Hz) and the semantic supervision objective. However, specific hyperparameters for the DiT training, the exact composition of the 200,000 hours of training data, and the code for the Prompt Enhancement module are not fully open-sourced or detailed to the extent required for exact replication. The use of proprietary in-house models for evaluation (Seed-Audio-1.0 baseline, in-house music model) also complicates direct comparison.
The paper acknowledges limitations regarding the synthetic data, noting residual context in source clips and mismatches in simulated acoustics. The reliance on LLMs for prompt enhancement and evaluation introduces a dependency on the capabilities of those specific models. The model's performance on music composition (melody, structure) lags behind dedicated autoregressive models, indicating that the NAR DiT architecture may struggle with long-range musical dependencies compared to sequential planners. The evaluation metrics, particularly the LALM-based ones, may not fully capture human perceptual quality or nuance.
This work advances the state of unified audio generation, moving towards systems that can produce complex, temporally structured scenes with multiple audio types. This has significant implications for content creation in gaming, film, and virtual reality, where coherent audio scenes are essential. By demonstrating that a single model can handle speech, music, and sound effects with temporal control, it reduces the complexity of audio production pipelines. However, the potential for misuse in generating realistic deepfakes or misleading audio content remains a concern, necessitating robust safety measures. This paper presents a significant engineering advancement in unified audio generation by introducing a structured, temporally-aware conditioning framework and a shared continuous latent space, demonstrating strong capabilities in multi-speaker dialogue and scene-level audio organization while maintaining competitive performance across diverse audio domains.
Cover song generation (CSG) should preserve the melodic and linguistic content of a reference song while recreating the remaining musical components. The state-of-the-art model SongEcho utilizes $F_0$ sequences and voiced/unvoiced (V/UV) tags for conditioning; however, implicit linguistic information from V/UV tags cannot guarantee lyric accuracy, leading to a high phoneme error rate (PER). Inspired by singing voice synthesis (SVS), we propose MPEcho, which integrates a phoneme encoder and a length regulator (LR) into the SongEcho framework. By providing explicit phoneme-level conditioning and precise temporal boundaries, MPEcho significantly reduces PER. To enable this, we developed Phonsa, a Whisper-based automatic transcription model that provides high-precision phoneme-level annotations for singing voices, overcoming the scarcity of high-quality audio-phoneme pairs. Experimental results validate the effectiveness of Phonsa for alignment and MPEcho for end-to-end CSG. The audio samples, code and weights can be accessed from https://lonian6.github.io/MPEcho.github.io/.
Primary: Taiwan AI Labs
All Institutions: Taiwan AI Labs, National Science and Technology Council of Taiwan, Ministry of Education (Taiwan)
MPEcho introduces a novel integration of SVS-style phoneme-level control into end-to-end cover song generation, significantly improving lyrical accuracy while maintaining melodic fidelity. This work demonstrates the value of cross-domain priors (SVS to CSG) and provides a robust framework for controllable music generation, advancing the state-of-the-art in both technical methodology and practical application.
The paper proposes MPEcho, a framework for Controllable Cover Song Generation (CSG) that integrates explicit phoneme-level conditioning into the SongEcho architecture. The core methodological contribution is the adaptation of Singing Voice Synthesis (SVS) techniques—specifically a length regulator and phoneme encoder—into a diffusion-based full-song generation model. This addresses the limitation of previous models (SongEcho) that relied on coarse V/UV tags, leading to high phoneme error rates (PER). The authors also introduce Phonsa, a Whisper-based automatic transcription model optimized for singing voices, to provide the necessary high-precision phoneme-level annotations. The methodology involves a multi-condition guidance strategy using Adaptive Projection Guidance (APG) to disentangle text, lyrics, melody, and phoneme conditions. The approach is technically sound, leveraging established SVS priors to solve a specific controllability issue in music generation. While the integration of SVS components into LTS/CSG is not entirely new in concept, the specific implementation of a length regulator for end-to-end cover generation and the dedicated alignment model (Phonsa) for this domain represent a solid engineering contribution.
The experimental section is comprehensive, including objective metrics (Audiobox, CLAP, RPA, RCA, PER) and subjective evaluations (MOS). The results demonstrate that MPEcho significantly reduces PER compared to SongEcho (0.1865 vs 0.4562) while maintaining or improving musical quality metrics. Ablation studies effectively isolate the contributions of melody conditioning, phoneme conditioning, and the specific guidance strategies. The evaluation of Phonsa against MFA shows substantial improvements in alignment accuracy. However, the reliance on a Chinese-only dataset limits the generalizability of the findings. The objective metrics, particularly CLAP, show low scores across the board, which the authors attribute to domain mismatch, a valid but somewhat concerning observation for a generative model's performance assessment. The subjective evaluation adds valuable human-centric validation, confirming the objective trends.
The paper provides detailed implementation details, including dataset sizes, training hardware, hyperparameters, and model architectures. The code and weights are made available via a GitHub page, which significantly enhances reproducibility. The description of the Phonsa model and the MPEcho architecture is sufficiently detailed for replication. The use of standard datasets (M4Singer, Opencpop, GTsinger) for Phonsa training and a curated internal dataset for MPEcho allows for verification, although the internal dataset's exact composition is less transparent than public benchmarks.
The primary limitation is the restriction to single-singer scenarios and Mandarin language. The model's performance on multi-singer tracks or languages with different phonetic structures is untested. The reliance on Phonsa for alignment means that errors in Phonsa will propagate to MPEcho. Additionally, the internal dataset used for training MPEcho is not publicly available, which hinders direct comparison with other CSG models trained on different data. The low CLAP scores suggest that the generated audio may not always align well with text prompts in a broad semantic sense, although this is common in specialized music generation tasks.
This work contributes to the field of generative AI for music by improving the controllability of cover song generation. Accurate lyric rendering is crucial for user acceptance and creative utility in AI music tools. The development of Phonsa also advances the field of automatic speech recognition and alignment for singing, which has broader applications in music information retrieval and education. However, the ease of generating high-quality covers raises ethical concerns regarding copyright and artist likeness, which should be addressed in future deployments. MPEcho introduces a novel integration of SVS-style phoneme-level control into end-to-end cover song generation, significantly improving lyrical accuracy while maintaining melodic fidelity. This work demonstrates the value of cross-domain priors (SVS to CSG) and provides a robust framework for controllable music generation, advancing the state-of-the-art in both technical methodology and practical application.
The study of Target Speaker Extraction (TSE) aims to isolate a desired speaker from overlapping speech mixture given auxiliary cues. Existing systems are typically designed for specific cue types, limiting flexibility when cue availability varies across scenarios. We present WeSep, a unified framework that reformulates TSE as a heterogeneous cue-conditioned learning problem. In WeSep, cue modules and separator backbones are decoupled through standardized interfaces, enabling configurable cue injection and flexible integration of diverse modalities. The design enables systematic study of cue structure, intra- and cross-modal interaction, and dynamic cue availability within a shared optimization framework, facilitating adaptation to real-world conditions. Experiments across enrollment, spatial, visual, and textual cues reveal modality-dependent characteristics and demonstrate stable optimization under heterogeneous cue availability. The toolkit will be publicly available.
Primary: The Chinese University of Hong Kong, Shenzhen
All Institutions: The Chinese University of Hong Kong, Shenzhen, Nanjing University, Shenzhen Loop Area Institute
The paper presents WeSep, a modular framework for Target Speaker Extraction that standardizes cue injection and enables systematic study of heterogeneous cue availability. While it lacks algorithmic novelty, its contribution to software engineering and experimental standardization in the audio processing community is valuable and practically significant.
The paper proposes WeSep, a modular framework for Target Speaker Extraction (TSE) that decouples cue encoders from separator backbones. The core methodological contribution is architectural rather than algorithmic; it introduces a standardized interface for injecting diverse cues (enrollment, spatial, visual, textual) into a shared optimization framework. This allows for composable multi-cue integration and handles heterogeneous cue availability (missing cues) via zero-padding. While the decoupling strategy is sound and addresses a real pain point in TSE (fragmented codebases for different modalities), the novelty is limited. The concept of modular TSE is not new (e.g., ClearerVoice, various multi-modal baselines), and the paper explicitly states it does not propose a new separation architecture. The "innovation" lies in the software engineering and experimental design rather than novel deep learning mechanisms.
The evaluation is comprehensive in terms of breadth, covering four distinct cue modalities and their combinations on standard datasets (Libri2Mix, VoxCeleb2-mix, etc.). The experiments successfully demonstrate that the framework supports stable training with missing cues and that combining cues (e.g., speaker + spatial) yields performance gains. However, the results are largely confirmatory. The paper shows that existing components work well when plugged into this framework, rather than demonstrating that WeSep introduces a new state-of-the-art method. The performance metrics (SI-SDRi) are standard, and the improvements are attributed to the flexibility of composition rather than a breakthrough in extraction capability. The causal modeling experiment is a practical addition but shows expected degradation in performance for low latency.
The paper provides a GitHub URL for the toolkit, which is a significant positive for reproducibility. The experimental settings are described in reasonable detail, including backbone configurations (BSRNN, NBC2) and training hyperparameters (Adam, 150 epochs). The use of standard datasets enhances the ability of others to replicate the results. The modular design, if implemented cleanly as claimed, should facilitate reproduction by other researchers.
The primary limitation is the lack of algorithmic novelty. The paper admits to not proposing a new separator. The performance gains are marginal and stem from better engineering and composition rather than fundamental improvements in the extraction task. The reliance on zero-padding for missing cues is a simple heuristic that may not be optimal compared to learned masking or attention mechanisms, although it is effective for stability. The paper does not provide a detailed ablation study on the *optimal* fusion strategies beyond what is standard, limiting the theoretical contribution.
This work has positive broader impact by promoting modularity and standardization in the TSE field. By providing an open-source toolkit, it lowers the barrier to entry for researchers working on multi-modal TSE and encourages systematic comparison of cue types. It facilitates research into robust TSE systems that can handle dynamic cue availability, which is crucial for real-world deployment. The paper presents WeSep, a modular framework for Target Speaker Extraction that standardizes cue injection and enables systematic study of heterogeneous cue availability. While it lacks algorithmic novelty, its contribution to software engineering and experimental standardization in the audio processing community is valuable and practically significant.
Automatic analysis of multimodal speech has shown strong potential for computationally detecting and monitoring a wide range of neurological, psychiatric, and respiratory conditions. However, progress in this field is limited by existing publicly accessible datasets, which are often small in scale, focused on a single condition or disease, and primarily speech focused. Moreover, if key confounding variables such as education, medication use, comorbidities, or mood state are insufficiently documented, the reliability and interpretability of computational analyses are further compromised. To address these limitations, we introduce CARE v1.0, a curated multimodal English dataset of approximately 144 hours of short video interviews collected from 612 individuals across 12 medical conditions plus a control cohort. For each video, a comprehensive set of clinically relevant multimodal descriptors is provided, alongside structured metadata covering factors such as medication, life impacts, and expressed emotions. The corpus's breadth and heterogeneity support a wide range of applications, including automatic disease and symptom detection, multimodal modelling of speech and non-verbal behaviour under emotionally charged contexts, and studies of disease trajectories and coping processes.
Primary: Universitat Politècnica de València
All Institutions: Universitat Politècnica de València, University of Oxford, Instituto de Telecomunicações
This paper presents CARE v1.0, a comprehensive multimodal corpus of 144 hours of video interviews from 612 individuals across 12 medical conditions, enriched with automatically extracted multimodal descriptors and structured metadata, addressing critical gaps in scale, diversity, and clinical context in existing digital health datasets.
The paper introduces CARE v1.0, a large-scale multimodal corpus derived from the HEXI platform. The methodology focuses on data curation and feature engineering rather than novel algorithmic development. The authors employ a rigorous pipeline for extracting multimodal descriptors: audio features (eGeMAPS, DisVoice, XLS-R, TRILLsson), visual features (OpenFace for facial landmarks/AUs, gaze, head pose, MediaPipe for body keypoints), and linguistic features (WhisperX alignment, sentence embeddings). A key methodological contribution is the use of LLMs (gpt-oss-120b) to extract structured metadata (demographics, emotions, comorbidities) from unstructured narrative texts, validated against a manually annotated subset. This approach addresses a significant gap in existing datasets by providing rich, clinically relevant metadata alongside raw multimodal signals.
The experimental section is primarily descriptive, focusing on the statistical properties of the dataset (distribution of conditions, demographics, emotion frequencies) and the validation of the LLM-based metadata extraction pipeline. The authors compare five LLMs for metadata extraction, demonstrating that gpt-oss-120b performs best in terms of accuracy and adherence to constraints. They provide a baseline evaluation workflow for a binary classification task (Control vs. Parkinson's) using an SVM on aggregated features. While this demonstrates the utility of the dataset, the lack of extensive benchmarking against state-of-the-art multimodal models limits the assessment of the dataset's ability to drive performance improvements in specific tasks. The validation of the metadata extraction is solid, but the downstream ML performance is only illustrative.
The paper excels in reproducibility. The dataset is hosted on Hugging Face, and the code for feature extraction and the example evaluation workflow is publicly available. The detailed documentation of the feature extraction pipeline, including specific tools (OpenFace, DiariZen, WhisperX) and parameter settings, allows other researchers to replicate the feature computation. The use of deterministic settings for LLM inference (temperature=0) further ensures reproducibility of the metadata annotations.
The dataset is skewed towards White UK/Irish participants, limiting its generalizability to other ethnic groups. The control group is heterogeneous (caregivers, bereaved, professionals), which may introduce confounding variables if not carefully handled in downstream analyses. The reliance on LLMs for metadata extraction, while validated, introduces potential biases and hallucinations (though rare, as noted). The dataset is cross-sectional, limiting longitudinal studies of disease trajectories, although the authors mention this as a potential future application. The "in-the-wild" nature of some video clips (varying quality, lighting) may pose challenges for robust computer vision models.
CARE v1.0 has significant potential impact on the field of digital health and affective computing. By providing a large, multimodal, and clinically annotated dataset, it enables research into automated detection and monitoring of a wide range of neurological, psychiatric, and respiratory conditions. The inclusion of diverse medical conditions and rich metadata supports the development of more robust and interpretable computational models. The dataset can facilitate studies on the interplay between speech, non-verbal behavior, and emotional states in health contexts, contributing to a better understanding of disease manifestations and coping processes. This paper presents CARE v1.0, a comprehensive multimodal corpus of 144 hours of video interviews from 612 individuals across 12 medical conditions, enriched with automatically extracted multimodal descriptors and structured metadata, addressing critical gaps in scale, diversity, and clinical context in existing digital health datasets.
Large audio-language models (LALMs) convey acoustic evidence to language decoders through native audio tokens, yet the internal roles of these tokens remain poorly understood. Using temporal audio grounding as a diagnostic setting, we examine how language-model fine-tuning affects the layerwise semantics, decoder accessibility, and temporal output alignment of native audio-token states through four complementary analyses: query-conditioned token semantics, calibrated token readout, temporal-window probes, and residual-delta erasure during generation. Alongside substantial improvements in temporal localization, semantic analysis of Qwen2.5-Omni shows that latent evidence for queried events is already present before fine-tuning and that the audio tokens most strongly aligned with the queried event appear at similar temporal positions before and after fine-tuning. After fine-tuning, event-related information in audio tokens becomes more accessible to the decoder, especially in early and middle layers, and a cross-checkpoint control shows that this improvement arises primarily from decoder adaptation. Temporal probes show that the base checkpoint already contains recoverable information about annotated windows and that fine-tuning mainly improves alignment with each checkpoint's own predicted temporal support. Residual-delta erasure further shows that removing audio-token updates within predicted windows harms timestamp generation more than removing the same number of randomly selected updates. The same broad improvements in decoder readability and prediction alignment also appear in Qwen2-Audio. Together, these results support a semantics-to-readout account in which grounding fine-tuning helps the decoder read existing event evidence and connect it more reliably to temporal outputs.
Primary: Unknown
All Institutions: Unknown
This paper provides a rigorous mechanistic analysis of how fine-tuning enhances temporal grounding in Large Audio-Language Models, demonstrating that improvements stem primarily from enhanced decoder accessibility to latent event evidence rather than the creation of new semantic representations.
The paper proposes a comprehensive mechanistic interpretability framework for Large Audio-Language Models (LALMs), specifically focusing on the "semantics-to-readout" hypothesis. The methodology is rigorous and multi-faceted, employing four distinct diagnostic tools: (1) Query-Conditioned Token Semantics to measure alignment between audio tokens and text event phrases; (2) Calibrated Token Readout using hidden-state insertion (patching) to assess decoder accessibility; (3) Output-Facing Temporal Diagnostics using linear probes on both annotated and predicted windows; and (4) Residual-Delta Erasure to test the functional necessity of specific token updates during generation. This combination of probing, patching, and causal intervention provides a robust separation of representation quality from decoder utility. The approach is well-suited for the domain, addressing the "black box" nature of multimodal alignment in LALMs.
The experiments are conducted on two prominent LALMs, Qwen2.5-Omni and Qwen2-Audio, fine-tuned on a newly constructed dataset, AudioGrounding-QA, derived from the AudioGrounding corpus. The evaluation demonstrates substantial improvements in temporal grounding metrics (mIoU, F1) post-fine-tuning. The core findings are supported by detailed layer-wise analyses. Key results include the discovery that latent event evidence exists in base models but is not effectively read out, and that fine-tuning primarily enhances decoder accessibility (particularly in early/middle layers) rather than creating new semantic representations. The cross-checkpoint controls and residual erasure results strongly support the authors' claims about the mechanism of improvement. The replication on Qwen2-Audio adds significant weight to the generalizability of the findings.
The paper provides detailed descriptions of the diagnostic pipelines, including the construction of the semantic bank, the specific prompts for readout, and the implementation of linear probes and erasure interventions. The dataset construction is clearly explained, and the fine-tuning setup (LoRA on LM blocks only) is specified. While the full code is not explicitly linked in the text provided, the methodological detail is sufficient for reproduction by experts in the field. The use of standard metrics (mIoU, F1) and established embedding models (SBERT) further aids reproducibility.
The study focuses primarily on temporal grounding, a specific task, which may limit the generalizability of the "semantics-to-readout" account to other multimodal tasks (e.g., audio captioning or open-ended dialogue). The analysis is largely confined to Qwen-based architectures; while Qwen2-Audio is used for replication, the broader landscape of LALMs (e.g., Whisper-based or other proprietary models) is not explored. Additionally, the "calibrated readout" relies on SBERT embeddings, which may not perfectly capture the semantic nuances required for complex event reasoning, potentially underestimating or overestimating readout gains. The residual erasure is limited to layer-level interventions, which might miss more complex, distributed circuits.
This work contributes significantly to the field of mechanistic interpretability for multimodal models. By demystifying how fine-tuning affects internal representations, it provides actionable insights for improving LALM training strategies, particularly for tasks requiring precise temporal or spatial grounding. It challenges the assumption that fine-tuning creates new knowledge from scratch, suggesting instead that it optimizes access to pre-existing latent structures. This has implications for efficient fine-tuning, model compression, and the development of more reliable multimodal AI systems. This paper provides a rigorous mechanistic analysis of how fine-tuning enhances temporal grounding in Large Audio-Language Models, demonstrating that improvements stem primarily from enhanced decoder accessibility to latent event evidence rather than the creation of new semantic representations.
This is an implementation and measurement study of what it costs to run a streaming speech enhancer on a CPU. We port FastEnhancer-Medium at 48 kHz to faster-enhancer.c, a C runtime with six int8 GEMM tiers selected at initialization, leaving architecture and weights untouched. One Apple M2 core reaches 0.069 real-time factor, against 0.230 for the fp32 ONNX Runtime graph on the same machine, a 3.3x speedup. A Galaxy S23+ (Snapdragon 8 Gen 2) reaches 0.096. The speedup comes from specializing every layer of the runtime around one fixed model. Activation ranges are recomputed per frame, so no calibration set is needed; the k=3 convolutions use Winograd F(2,3); cross-stage state is fp16; the GRU and the dequantization epilogues are fused; and nothing is allocated after startup. Over 824 VoiceBank-DEMAND utterances the engine tracks fp32 to within -0.006 PESQ and -0.08 dB SNR. Speed alone does not settle deployment cost. The enhancer holds a fraction of a core for as long as the microphone is open, so its real-time factor is a duty cycle. A benchmark races through a file; an audio callback does not. Pacing to the 6.67 ms deadline costs 4.2x per frame, saves 49% of the energy, and leaves the cheapest core placement missing 96% of its deadlines. All SIMD tiers within an architecture family emit byte-identical output. The runtime is released as a dependency-free library.
Primary: Hanyang University
All Institutions: Hanyang University
This paper delivers a highly optimized, dependency-free C runtime for streaming speech enhancement, achieving significant speedups and energy savings on commodity CPUs through specialized int8 quantization and SIMD kernel fusion, while maintaining bit-identical output and negligible quality loss.
The paper presents a highly specialized, dependency-free C runtime (`faster-enhancer.c`) designed to deploy the FastEnhancer-Medium speech enhancement model on commodity CPUs. The methodology focuses on extreme engineering optimization rather than algorithmic novelty. Key technical strategies include: (1) Training-free int8 quantization with per-frame activation range recomputation, eliminating the need for calibration datasets; (2) End-to-end runtime specialization with six SIMD tiers (ARM NEON, DOTPROD, I8MM, x86 AVX2, AVX-VNNI, AVX-512 VNNI) selected at initialization; (3) Use of Winograd F(2,3) for $k=3$ convolutions to reduce multiply-accumulate counts; (4) Fusion of GRU gates and dequantization epilogues to minimize memory traffic; (5) Strict byte-identical output guarantees across different compilers and architectures by addressing floating-point non-determinism in the Hann window generation and integer overflow edge cases (clamping to [-127, 127]). The approach is a masterclass in low-level systems engineering for ML deployment.
The evaluation is rigorous and addresses the specific constraints of audio processing. The authors measure Real-Time Factor (RTF) and energy efficiency (joules per second) on two distinct platforms: Apple M2 (macOS) and Galaxy S23+ (Android). They demonstrate a 3.3x speedup over fp32 ONNX Runtime on the M2 (0.069 RTF vs 0.230 RTF). Crucially, they introduce a "duty-cycled" benchmarking protocol that accounts for the 6.67 ms frame deadline, revealing that "race-to-idle" strategies are energy-inefficient compared to pacing. They also evaluate quality using PESQ, STOI, SNR, SI-SDR, and SIGMOS, showing negligible degradation (-0.006 PESQ, -0.08 dB SNR) compared to the fp32 baseline. The analysis of long-term stability (30-minute runs) highlights scheduler-induced jitter, a critical insight for real-world deployment.
The paper provides a high degree of reproducibility. The code is released as a dependency-free library on GitHub. The authors provide detailed descriptions of the quantization scheme, kernel implementations, and the specific steps taken to ensure bit-identical outputs across platforms (e.g., double-precision window generation). The evaluation protocol is clearly defined, including the use of VoiceBank-DEMAND and specific timing methodologies. The inclusion of hash comparisons for output verification adds a layer of rigor often missing in systems papers.
The primary limitation is the scope of the model: it is a port of an existing architecture (FastEnhancer-Medium) and does not propose a new model. The x86 tiers are verified for bit-equality but not timed due to lack of hardware, limiting the generality of the performance claims. The runtime is single-threaded and CPU-only, excluding GPU/TPU acceleration which is common in other deployment stacks. The quality metrics, while robust, rely on standard objective measures; subjective listening tests are not included, though the objective deltas are argued to be inaudible. The "cheapest core" placement on Android was found to miss deadlines, limiting the flexibility of core placement in heterogeneous systems.
This work has significant impact for on-device AI, particularly in privacy-sensitive applications like real-time communication, hearing aids, and voice assistants where latency and power consumption are critical. By providing a lightweight, dependency-free runtime, it lowers the barrier to deploying high-quality neural speech enhancement on resource-constrained devices. It challenges the industry's reliance on heavy general-purpose inference engines (like ONNX Runtime) for specific, fixed models, advocating for specialized, hand-tuned kernels. The emphasis on energy efficiency and deadline adherence provides a valuable framework for evaluating the true cost of ML deployment in audio pipelines. This paper delivers a highly optimized, dependency-free C runtime for streaming speech enhancement, achieving significant speedups and energy savings on commodity CPUs through specialized int8 quantization and SIMD kernel fusion, while maintaining bit-identical output and negligible quality loss.
Voice activity detection (VAD) triggers downstream speech processing in always-on systems under strict memory, latency, and compute constraints. Recent compact models report strong accuracy but rely on components that are not widely supported: learnable filterbanks, recurrent layers, or non-causal post-processing. We propose kiloVAD, designed for embedded inference using standard Mel features, CNN-only layers, and tunable context/spectral parameters. We introduce per-layer structured pruning with self-distillation and angle-based quantization-aware training (QAT) that outperforms standard QAT by 1-4%. Evaluated per-frame under causal conditions, kiloVAD achieves 0.850 AUC on AVA-Speech with 2.1 k parameters and 200 ms context, establishing a new state of the art for causal, deployment-ready VAD.
Primary: Analog Devices, Inc.
All Institutions: Analog Devices, Inc., University of California, Los Angeles (UCLA)
The paper presents a well-engineered, deployment-focused VAD model that achieves a strong balance between accuracy, latency, and model size through effective structured pruning and angle-aware quantization.
The paper proposes kiloVAD, a compact Voice Activity Detection (VAD) model designed specifically for edge deployment. The core methodological contributions are architectural and training-related rather than novel deep learning architectures. The architecture is a standard CNN backbone (depthwise separable convolutions) with specific engineering choices: an adapter layer for channel decoupling, global average pooling for context flexibility, and amplitude-agnostic preprocessing. The primary technical novelty lies in the compression pipeline: (1) Per-layer structured pruning optimized via multi-objective search (FPR vs. parameter count) combined with self-distillation, and (2) Angle-based Quantization-Aware Training (QAT). The angle-based QAT modifies the loss function to align feature vectors with class weight vectors (prototypes) and repel them from non-target prototypes, aiming to mitigate quantization-induced angular errors. While the application of angle-based margin losses to QAT is a known technique in classification, its specific adaptation for ultra-low-bitwidth VAD on microcontrollers is a valid engineering contribution. However, the underlying model architecture is not novel; it relies on standard, well-established components (Mel spectrograms, CNNs, Global Average Pooling).
The evaluation is rigorous in the context of edge deployment constraints. The authors evaluate on the AVA-Speech dataset, which is appropriate for testing generalization to unseen environments. They provide a detailed comparison with prior compact VAD models (MarbleNet, TinyVAD, SincQDR, ResectNet, AtomicVAD). A key strength is the strict causal evaluation protocol (200ms context), which highlights the latency advantages of their design over competitors that use non-causal windows or larger contexts. The results show that kiloVAD achieves competitive AUC (0.850) with significantly fewer parameters (2.1k) and lower latency than most competitors, while maintaining full causal compatibility and standard operator support. The ablation studies on pruning strategies and quantization methods are clear and demonstrate the efficacy of the proposed techniques. The claim of "state of the art" is qualified by the specific constraints (causal, standard ops, low latency), which is honest and accurate.
The paper provides sufficient detail for reproduction. The architecture is described with specific layer types, kernel sizes, and channel dimensions. The training protocol includes optimizer settings, learning rate schedules, and loss functions. The authors provide pretrained weights and a web-based demo. The use of standard libraries (torch-pruning, Optuna) and standard datasets (LibriSpeech, AVA-Speech) facilitates reproducibility. The only minor gap is the exact code implementation of the angle-based QAT loss, though the mathematical formulation is provided.
The paper focuses heavily on parameter count and latency, but does not provide detailed energy consumption measurements (e.g., mJ/inference) on specific hardware, which is crucial for "always-on" edge devices. The model's performance on highly non-stationary noise or very low SNR conditions is not deeply explored beyond the standard AVA-Speech evaluation. The reliance on Mel spectrograms, while standard, means it does not explore raw audio end-to-end, which some recent works argue is beneficial for extreme compression. The "angle-based" QAT, while effective, adds complexity to the training loop compared to standard STE QAT.
This work has significant practical impact for the deployment of speech processing systems on resource-constrained IoT devices, smart speakers, and hearables. By providing a model that is compatible with standard embedded toolchains (TFLM) and requires no custom DSP kernels, it lowers the barrier to entry for developers. The emphasis on causal, low-latency inference is critical for user experience in real-time applications. The work contributes to the broader trend of efficient AI, demonstrating that high performance can be achieved through careful engineering and compression techniques rather than just larger models. The paper presents a well-engineered, deployment-focused VAD model that achieves a strong balance between accuracy, latency, and model size through effective structured pruning and angle-aware quantization.
Some text-to-speech systems ship a synthesis model and preset style vectors but not the reference encoder that turns audio into such a vector. The model still accepts a style vector; a user with a voice of their own cannot produce one. We solve for that input directly, inverting the released pipeline by gradient descent: every weight stays frozen and only the style vector is optimized, against time-pooled WavLM statistics of one recording. Because the objective discards the time axis, the synthesized text may differ from the recording, so no transcript and no alignment are needed. On 154 speakers from two corpora, ECAPA-TDNN similarity rises from 0.132 to 0.413 and ResNet from 0.099 to 0.401, improving for every speaker; a verifier at its equal-error point accepts 53% of the recovered voices as the target, against 1% for the presets they start from.
Primary: Hanyang University
All Institutions: Hanyang University
This paper presents a novel gradient-based method for extracting voice styles from frozen TTS models without reference encoders, demonstrating that withheld encoders do not fully secure voice identity and offering a reproducible framework for analyzing TTS latent spaces.
The paper proposes a gradient-based inverse optimization method to extract style vectors from frozen Text-to-Speech (TTS) models when the reference encoder is withheld. The core innovation lies in bypassing the need for transcribed reference audio or forced alignment by using a content-independent loss function. Specifically, it optimizes the style vector to match the time-pooled mean and standard deviation of hidden states from an early layer (Layer 4) of a frozen WavLM model. This approach effectively decouples speaker identity from linguistic content, allowing the system to work with untranscribed, arbitrary audio recordings. The method is technically sound, leveraging existing differentiable wrappers for ONNX models and standard optimization techniques (Adam, learning rate scheduling). It draws strong parallels to GAN inversion and Textual Inversion in the image domain, adapting these concepts to the audio TTS space.
The evaluation is conducted on 154 speakers from two corpora (VCTK and Common Voice subset of Seed-TTS Eval). The results demonstrate a significant improvement in speaker similarity metrics (ECAPA-TDNN and ResNet) compared to using the nearest preset style vector. The improvement is universal across all tested speakers. The paper provides a thorough analysis of the stopping criterion, characterizing the "content floor" to prevent overfitting into unrealistic style regions. It also evaluates intelligibility (WER) and naturalness (UTMOS), showing that the extracted voices remain intelligible and natural-sounding. The use of independent verification metrics (ECAPA, ResNet) to validate the optimization objective (WavLM statistics) strengthens the claim of successful voice cloning.
The authors release code and provide detailed hyperparameters, including the optimization setup, loss function definition, and stopping criteria. The use of public models (SupertonicTTS, WavLM, ECAPA-TDNN) and public datasets (VCTK, Common Voice) ensures that the experiments are fully reproducible. The description of the gradient flow through the ONNX modules is sufficiently detailed for implementation.
The method is currently validated on a single TTS model (SupertonicTTS). While the authors argue it should generalize to other models with continuous style inputs, this has not been empirically demonstrated. The optimization is sensitive to the choice of the WavLM layer and the stopping threshold, which are somewhat heuristic. The method does not recover duration/prosody styles, inheriting them from the initialization preset. Furthermore, the evaluation prompts are the same ones used during optimization, so generalization to unseen text is not fully tested. The paper acknowledges that the extracted voices might still be detectable by spoofing countermeasures, even if they pass speaker verification.
This work has significant dual-use implications. On one hand, it highlights a security vulnerability in TTS systems that withhold reference encoders, demonstrating that partial releases do not fully protect voice identity. This can inform better security practices for TTS providers. On the other hand, it provides a powerful tool for voice cloning, which can be misused for impersonation and fraud. The authors address this by releasing code with ethical warnings and not packaging it as a turnkey cloning tool. The work contributes to the broader understanding of latent spaces in generative models and the transferability of self-supervised speech representations. This paper presents a novel gradient-based method for extracting voice styles from frozen TTS models without reference encoders, demonstrating that withheld encoders do not fully secure voice identity and offering a reproducible framework for analyzing TTS latent spaces.
In this report, we present Qwen-Audio-3.0-TTS, a production-oriented speech synthesis system that jointly advances content consistency, speaker similarity, prosodic naturalness, audio quality, controllability, multilingual coverage, efficiency, and robustness. It combines a 12.5~Hz low-frame-rate speech tokenizer for reduced inference latency with a five-stage progressive training paradigm for coordinated language model (LM) and flow-matching model (FM) optimization. The model provides production-level control through free-style natural-language instructions and fine-grained inline tags, while supporting 16 languages, 20 Chinese dialect regions, one-pass long-form synthesis up to 3 minutes, and robust generation from noisy, reverberant, or unclear reference speech. Across SEED-TTS-Eval, CV3-Eval, instruction-following, long-form, and acoustic-robustness evaluations, Qwen-Audio-3.0-TTS achieves state-of-the-art performance on many reported dimensions or the strongest aggregate results. It also ranks first on the independent Artificial Analysis Text-to-Speech Leaderboard. These results establish Qwen-Audio-3.0-TTS as a strong foundation for production-level speech synthesis.
Primary: Alibaba Group
All Institutions: Alibaba Group
[One sentence main contribution]. Qwen-Audio-3.0-TTS introduces a five-stage progressive training paradigm with joint LM-FM optimization and reinforcement learning to achieve state-of-the-art production-level speech synthesis with robustness and fine-grained control. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a significant technical contribution by systematically addressing the trade-offs between efficiency, fidelity, and controllability in TTS. The integration of GRPO for both LM and FM optimization is a novel application of RLHF techniques to speech generation, moving beyond simple supervised fine-tuning. The explicit focus on acoustic robustness through training-time augmentation is highly relevant for real-world deployment. The comprehensive evaluation suite, including new benchmarks for robustness and instruction following, sets a high standard for future TTS research. The results demonstrate that a unified model can outperform specialized systems in multiple dimensions, marking a step towards more general-purpose speech generation models.
The paper presents Qwen-Audio-3.0-TTS, a production-oriented text-to-speech system that integrates a 12.5 Hz speech tokenizer, a language model (LM), and a flow-matching model (FM). The core methodological contribution lies in a five-stage progressive training paradigm. Key innovations include: 1) Conditioning the FM on continuous LM hidden states rather than just discrete tokens to mitigate information bottleneck; 2) High-quality data annealing during joint LM-FM training; 3) Application of Group Relative Policy Optimization (GRPO) to the LM for content/prosody rewards and to the FM for speaker similarity/perceptual quality rewards; 4) Explicit acoustic robustness training using degraded prompts (noise, reverb, etc.) without separate denoising modules. The architecture combines autoregressive semantic planning with continuous acoustic generation, aiming to balance latency, fidelity, and controllability.
The evaluation is comprehensive and rigorous, covering multiple dimensions: zero-shot voice cloning (SEED-TTS-Eval, CV3-Eval), multilingual/cross-lingual synthesis, instruction following (VStyle), long-form generation, text normalization, and acoustic robustness. The paper provides extensive objective metrics (CER/WER, SIM, DNSMOS) and subjective human evaluations (Arena-style, dialect authenticity). Results indicate state-of-the-art performance on many reported dimensions, particularly in robustness and instruction following, while maintaining competitive content consistency and speaker similarity. The inclusion of a dedicated "hard-case" robustness benchmark and detailed ablation studies on tokenizer frame rates adds significant value.
The paper provides detailed descriptions of the architecture, training stages, and data types. It mentions specific models used for evaluation (Whisper, Paraformer, ERes2Net, DNSMOS) and provides a demo URL. However, full code and model weights are not explicitly stated as open-source in the provided text (though the project URL points to a GitHub page, the text itself focuses on the report). The detailed training recipe (5 stages, specific rewards) is described well enough for reproduction by experts in the field.
The paper acknowledges that pushing content accuracy lower can come at the expense of naturalness, suggesting a trade-off. The robustness training, while effective, relies on specific augmentation strategies that may not cover all real-world edge cases. The reliance on LLM-based evaluators (Gemini) for some subjective metrics introduces potential bias, although human evaluation is also provided. The "high-quality data annealing" requires careful curation, which may limit scalability if high-quality labeled data is scarce.
This work advances the field of speech synthesis by providing a robust, controllable, and efficient system suitable for production deployment. It addresses critical pain points like robustness to noisy prompts and fine-grained control, which are essential for real-world applications. The open-sourcing of such models (implied by the project URL) can democratize access to high-quality TTS technology. However, the ease of voice cloning also raises concerns about misuse for deepfakes, necessitating responsible AI practices. [One sentence main contribution]. Qwen-Audio-3.0-TTS introduces a five-stage progressive training paradigm with joint LM-FM optimization and reinforcement learning to achieve state-of-the-art production-level speech synthesis with robustness and fine-grained control. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a significant technical contribution by systematically addressing the trade-offs between efficiency, fidelity, and controllability in TTS. The integration of GRPO for both LM and FM optimization is a novel application of RLHF techniques to speech generation, moving beyond simple supervised fine-tuning. The explicit focus on acoustic robustness through training-time augmentation is highly relevant for real-world deployment. The comprehensive evaluation suite, including new benchmarks for robustness and instruction following, sets a high standard for future TTS research. The results demonstrate that a unified model can outperform specialized systems in multiple dimensions, marking a step towards more general-purpose speech generation models.
Contrastive Language-Audio Pretraining (CLAP) learns aligned text and audio representations in a shared embedding space. However, independent encoding of each modality limits its ability to model cross-modal semantics in complex audio understanding and retrieval tasks. To address this limitation, this paper proposes Text-Prompted CLAP (TP-CLAP), a parameter-efficient extension of CLAP that introduces a cross-attention-based fusion module to incorporate textual prompts into audio features. TP-CLAP is trained using an audio multiple-choice question (audio-MCQ) framework, where it learns to align query-conditioned audio representations with text embeddings of correct answer choices via contrastive learning. Experiments demonstrate that TP-CLAP achieves competitive performance on audio question answering (audio-QA) with substantially larger audio-LLMs, while also improving the base CLAP model on audio-text retrieval and zero-shot classification benchmarks. The learned representations are further fine-tuned for attribute-focused audio-to-audio retrieval, showing that TP-CLAP consistently outperforms the standard CLAP baseline in music retrieval tasks.
Primary: University of Cambridge
All Institutions: University of Cambridge, Toshiba Cambridge Research Laboratory
This paper presents a method for learning query-conditioned audio representations by integrating a cross-attention fusion module into the CLAP framework, trained via an audio multiple-choice question paradigm, demonstrating significant improvements in attribute-focused retrieval and competitive performance in audio question answering tasks.
The paper proposes TP-CLAP, a parameter-efficient extension of the CLAP model that introduces a cross-attention-based fusion module to condition audio representations on textual prompts. The core methodological contribution lies in the training framework: it leverages AudioMCQ data to train the model to align query-conditioned audio embeddings with the text embeddings of correct multiple-choice answers. This is combined with standard CLAP contrastive loss and auxiliary losses to maintain general alignment. The approach is technically sound, addressing a specific limitation of static embeddings in CLAP (inability to handle query-dependent tasks like QA or attribute-focused retrieval). The use of cross-attention to inject textual context into audio features is a known pattern (similar to Q-Former in LLaVA or cross-attention in BLIP), but its application specifically to enable attribute-focused audio retrieval via an MCQ framework is a distinct and practical contribution. The derivation of the loss functions, including the specific handling of multi-label attributes in the fine-tuning stage, is rigorous.
The experimental evaluation is comprehensive, covering audio-text retrieval, zero-shot classification, audio question answering (QA), and attribute-focused audio-to-audio retrieval. The results show consistent improvements over the base CLAP model and the CLAP+AudioMCQ baseline (which lacks the fusion module). Notably, TP-CLAP achieves competitive performance on audio QA against much larger audio-LLMs, which is a strong result highlighting the efficiency of the approach. The attribute-focused retrieval experiments are particularly compelling, demonstrating that the model can isolate specific attributes (e.g., instrument, genre) rather than relying on global similarity. The use of standard benchmarks (AudioCaps, Clotho, ESC50, FSD50K, NSynth, MagnaTagATune) adds credibility. However, the improvement in standard retrieval tasks is modest, suggesting the primary value is in the conditional/attribute-specific capabilities.
The paper provides detailed implementation details, including encoder architectures (CED-Base, bert-base-uncased), hyperparameters (learning rates, batch sizes, warmup epochs), and loss function weights. The data sources (AudioMCQ, AudioCaps, Clotho, WavCaps, MACS, NSynth, MagnaTagATune) are well-known and accessible. The training stages are clearly delineated. This level of detail allows for reasonable reproducibility, assuming access to the pre-trained CLAP weights and the specified datasets.
The paper acknowledges that the model is smaller than audio-LLMs like Audio Flamingo 3, which still outperform it in QA. The improvements on standard retrieval tasks are marginal. The method relies heavily on the quality of the AudioMCQ dataset; if the questions/answers are noisy or biased, the model may learn spurious correlations. Furthermore, the cross-attention module adds computational overhead during inference compared to the original CLAP, although it remains lightweight compared to full LLMs. The paper does not discuss potential biases in the attribute-focused retrieval (e.g., gender or cultural biases in instrument/genre associations).
This work contributes to the field of multimodal audio understanding by enabling more flexible and query-dependent audio representation learning. It facilitates applications such as advanced music information retrieval (MIR), where users might search for "sad piano music" or "jazz with saxophone," rather than just general genre or mood. It also improves accessibility tools that need to understand specific audio contexts. The parameter-efficient nature makes it more accessible for deployment on edge devices or with limited compute compared to large multimodal LLMs. This paper presents a method for learning query-conditioned audio representations by integrating a cross-attention fusion module into the CLAP framework, trained via an audio multiple-choice question paradigm, demonstrating significant improvements in attribute-focused retrieval and competitive performance in audio question answering tasks.
Siri Expressive Voices synthesize rich, configurable speech in real time and entirely on device, powered by AFM 3 Core Advanced, Apple's most powerful on-device foundation model. This work presents the memory-efficient audio synthesis architecture behind that capability: a detokenizer that converts the semantic audio tokens emitted by the foundation model into high-fidelity audio within the tight compute and memory budget of the Apple Matrix Coprocessor (AMX). We convert semantic audio tokens to a residual vector quantization (RVQ) representation with a three-component design, a streaming encoder, a temporal decoder, and a depth decoder, that systematically decouples temporal and depth processing. A single reusable depth decoder with Diffusion Transformer (DiT)-style stage conditioning generates all RVQ levels autoregressively, replacing the dedicated per-level decoders of prior multi-decoder architectures, while causal sliding window attention with fixed-window key-value caching yields constant memory complexity independent of sequence length. Deployed on the AMX, the detokenizer sustains roughly 10 ms per generation step, about 16x faster than real time, with a peak runtime memory of only 21 MB and 329 MB of on-device assets, enabling continuous streaming synthesis of 20-320 seconds of audio. This constant, small footprint replaces the linear and quadratic memory scaling of conventional transformer- and GAN-based approaches. Ablation studies validate the key architectural components, and audio quality assessment confirms that the architecture maintains synthesis fidelity while achieving efficiency gains over existing methods. Operating at a 1-billion-parameter activation size within AFM 3 Core Advanced, it improves Mean Opinion Score by +0.28 overall (4.15 vs. 3.87) and by +0.42 on conversational speech (4.24 vs. 3.82) over the prior on-device text-to-speech system.
Primary: Apple Inc.
All Institutions: Apple Inc.
The paper presents a significant engineering achievement in deploying efficient, high-quality audio synthesis on-device, introducing architectural innovations that decouple temporal and depth processing to achieve constant memory complexity and superior efficiency compared to prior methods.
The paper proposes a memory-efficient audio synthesis architecture designed for on-device deployment, specifically targeting the Apple Matrix Coprocessor (AMX). The core methodological contribution is a three-component design: a streaming encoder, a temporal decoder, and a depth decoder. The key innovation lies in the "depth decoder," which replaces the per-level input/output projections found in prior works like Moshi with a single, fully shared transformer body conditioned on the RVQ level index via DiT-style rotary positional encoding (RoPE). This eliminates parameter overhead associated with the number of quantization levels. Additionally, the architecture employs causal sliding window attention with a fixed-window key-value cache to ensure constant memory complexity independent of sequence length, a critical feature for real-time streaming. The temporal decoder bridges semantic tokens and RVQ tokens, incorporating a temporal lookahead mechanism and autoregressive feedback from the mean RVQ embeddings. The approach effectively decouples temporal and depth processing, allowing for efficient generation of high-fidelity audio from semantic tokens.
The evaluation is comprehensive, covering both algorithmic scaling and production deployment metrics. On a common GPU (NVIDIA H100), the proposed model demonstrates constant memory and generation time scaling with sequence length (up to 320 seconds), whereas baselines (Transformer Decoder, Autoregressive GAN) scale linearly or quadratically. On-device, the model achieves a peak runtime memory of 21 MB and sustains ~10 ms per generation step (16x real-time). Audio quality is assessed using objective metrics (UTMOS, SI-SNR, VisQOL, MOSNet) and subjective Mean Opinion Score (MOS) evaluations. The model outperforms the prior on-device system (+0.28 overall MOS, +0.42 on conversational speech) and shows competitive or superior performance against baselines like Moshi/Mimi and internal GAN/Transformer models on UTMOS and SI-SNR. Ablation studies validate the contributions of the unified depth decoder, DiT conditioning, fixed KV cache, and temporal lookahead.
The paper provides detailed architectural descriptions, including layer configurations (4-6-2), attention mechanisms, and conditioning strategies. However, reproducibility is significantly hindered by the use of a proprietary audio corpus ("internally recorded voice assistant interactions") and the lack of open-source code. The hyperparameters and training details are described, but the specific data distribution and preprocessing pipelines are not disclosed due to confidentiality protocols. The comparison with baselines is conducted on a common GPU, which helps, but the lack of access to the training data and code limits independent verification of the exact performance claims.
The primary limitation is the reliance on proprietary data and closed-source implementation, preventing independent replication and broader community adoption. The model is specifically optimized for Apple's AMX hardware, and while algorithmic scaling is reported on GPUs, the real-world performance on other platforms may vary. The "temporal lookahead" mechanism, while effective, introduces a slight latency overhead and requires access to future semantic tokens, which might be a constraint in strictly causal, low-latency streaming scenarios where future context is not immediately available. The comparison with Moshi is somewhat indirect, as Moshi's on-device performance metrics are not provided for direct comparison, although algorithmic scaling is compared.
This work demonstrates the feasibility of high-quality, real-time, on-device audio synthesis with minimal resource overhead, enabling privacy-preserving voice assistants and immersive audio applications without cloud dependency. The architectural innovations, particularly the decoupled temporal-depth processing with shared parameters and fixed-window attention, offer valuable design patterns for efficient transformer-based generative models in resource-constrained environments. It pushes the boundary of what is possible with on-device AI, potentially reducing reliance on cloud infrastructure for audio generation tasks. The paper presents a significant engineering achievement in deploying efficient, high-quality audio synthesis on-device, introducing architectural innovations that decouple temporal and depth processing to achieve constant memory complexity and superior efficiency compared to prior methods.
Recent generative models are moving beyond silent video or standalone audio synthesis toward the joint generation of synchronized audio and video. Despite this progress, jointly generating audio and video with fine-grained cross-modal correspondence remains challenging due to their fundamental structural differences. Most existing methods use audio and video VAEs trained separately. As a result, the two latent spaces lack cross-modal alignment, leaving the downstream generative model to learn cross-modal synchronization from scratch. We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations. Beyond reconstruction, OmniVAE uses a segment-level audio-video contrastive objective to capture temporal-semantic correspondence and align the two latent spaces. In parallel, it distills features from pretrained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces. Extensive experiments show that both objectives consistently improve the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation. These findings underscore the importance of learning unified representations as a foundation for omnimodal modeling.1
Primary: Fudan University
All Institutions: Fudan University, Shanghai Innovation Institute, MOSI Intelligence, Shanghai Jiao Tong University
OmniVAE introduces a novel jointly trained audio-video VAE with segment-level contrastive alignment and semantic distillation, significantly improving cross-modal synchronization and generation quality in text-to-audio-video tasks.
The paper proposes OmniVAE, a jointly trained audio-video Variational Autoencoder (VAE) designed to address the lack of cross-modal alignment in standard modality-specific tokenizers. The core methodological contribution lies in two training-only objectives added to standard reconstruction losses: (1) Segment-level Audio-Video Contrastive Learning (AVCLIP), which aligns audio and video latents at a fine temporal granularity using a hierarchical negative sampling strategy (intra-clip, sibling-clip, cross-video), and (2) Modality-Specific Semantic Distillation, which injects semantic structure from pretrained Qwen3-Omni encoders into each latent space independently. The approach is technically sound, leveraging established contrastive learning frameworks (InfoNCE) and distillation techniques (iREPA-style) but applying them specifically to the joint audio-video latent space. The three-stage training schedule (pretrain reconstruction -> joint contrastive/distillation -> adversarial refinement) is a practical engineering solution to handle the conflicting optimization dynamics of reconstruction and alignment.
The experimental evaluation is comprehensive and well-structured. The authors conduct extensive ablations to isolate the contributions of reconstruction, distillation, and contrastive learning. They employ a novel "sync probing" task (predicting temporal offset) as a proxy for alignment, which provides strong quantitative evidence that the latent spaces are indeed better aligned than baselines. Downstream text-to-audio-video generation experiments on Verse-Bench show consistent improvements in cross-modal alignment metrics (DeSync, LSE-C) and generation quality (AudioBox, IS, FAD) when using OmniVAE. The use of a large-scale, curated dataset (23M clips) with LLM-generated captions adds significant weight to the empirical claims. The reconstruction quality remains competitive with state-of-the-art independent VAEs, demonstrating that alignment does not come at the cost of fidelity.
The paper provides detailed architectural descriptions, loss formulations, and training schedules. The parameter counts are explicitly listed, distinguishing between inference and training-only modules. The dataset construction pipeline is described, though the specific filtering thresholds and LLM prompts might vary. The reliance on proprietary or large pretrained models (Qwen3-Omni, Wan2.2, HunyuanVideo-Foley) for initialization and distillation is noted, which aids reproducibility for those with access to these resources. The code is not explicitly linked in the text, but the methodology is described with sufficient detail for replication.
The primary limitation is that the alignment is learned during training only; the model does not enforce a shared latent distribution, meaning the latents are still modality-specific but aligned in a shared embedding space via the contrastive head. This might limit direct cross-modal generation without additional mapping layers. The sync probing metric, while useful, is a proxy and may not perfectly correlate with human perception of synchronization in complex scenes. The dataset, while large, is web-scraped and may contain noise despite filtering. The reliance on Qwen3-Omni for distillation assumes the availability of this specific large model, which might not be accessible to all researchers.
OmniVAE contributes to the field of unified multimodal generation by providing a stronger foundational representation for audio-video tasks. By decoupling the alignment learning from the generative model, it allows downstream generators to focus on content creation rather than learning synchronization from scratch. This could accelerate the development of high-quality, synchronized audio-video generative models for entertainment, education, and simulation. The emphasis on fine-grained temporal alignment addresses a key challenge in current generative AI, potentially leading to more realistic and usable synthetic media. OmniVAE introduces a novel jointly trained audio-video VAE with segment-level contrastive alignment and semantic distillation, significantly improving cross-modal synchronization and generation quality in text-to-audio-video tasks.