Audio-encoder-LLM-decoder architectures have become the dominant paradigm for modern automatic speech recognition (ASR), improving transcription quality through large-scale language modeling. However, the cost of autoregressive decoding scales with decoder size, creating a fundamental trade-off between recognition quality and serving latency. We argue this trade-off is not inherent: unlike open-ended text generation, ASR outputs are strongly anchored to the input speech signal, providing a natural inductive bias toward high-parallelism decoding. Building on this, we introduce ParaASR, an ASR system that leverages Multi-Token Prediction (MTP) to let a 4B LLM decoder emit multiple tokens per forward step. Starting from a publicly available audio-language foundation, the model first establishes a robust autoregressive recognizer and then aligns five future-token branches through a staged optimization recipe. At inference, it proposes a six-token continuation per step and admits only the verified prefix into the transcript, preserving the safety of standard autoregressive decoding. The average accepted length reaches 5.0 out of 6 proposed tokens, confirming that the deterministic structure of speech makes ASR an especially natural setting for multi-token decoding. ParaASR further retains a native 32K-context window and transcribes up to 30 minutes of audio in a single pass. Across diverse benchmarks, it attains average error rates of 2.97%, 3.68%, and 3.70% on Chinese, English, and long-form evaluations, respectively, while reaching a real-time factor (RTF) as low as 0.0053. These results show that decoder scaling, low-latency inference, and long-context transcription need not be competing goals when future-token proposals are anchored by the acoustic signal and guarded by autoregressive verification.
Primary: StepFun
All Institutions: StepFun, NTU, PKU, UNSW, SJTU, USTC
ParaASR effectively adapts multi-token prediction to ASR, achieving a compelling balance between state-of-the-art accuracy and low-latency inference, thereby challenging the perceived trade-off between model scale and serving efficiency in LLM-based speech recognition.
The paper proposes ParaASR, an LLM-based Automatic Speech Recognition (ASR) system that integrates Multi-Token Prediction (MTP) to accelerate inference. The core methodological contribution is the application of speculative decoding principles (specifically MTP) to the ASR domain, leveraging the deterministic nature of speech-to-text mapping to propose multiple future tokens per forward pass. The architecture consists of a frozen audio encoder, a linear adapter, and a 4B LLM decoder with five auxiliary MTP branches. The training recipe is staged: audio-language pretraining, ASR supervised fine-tuning (SFT), and finally MTP alignment. The MTP training involves a frozen-branch alignment stage followed by joint calibration. The approach is technically sound and logically motivated; the insight that ASR outputs are more predictable than open-ended text generation is valid and well-supported by the high acceptance rates reported. However, the specific architectural modifications (adding MTP heads) are incremental adaptations of existing speculative decoding techniques rather than a fundamentally new algorithmic breakthrough.
The evaluation covers a wide range of benchmarks including Chinese (AISHELL-1, AISHELL-2, WenetSpeech), English (LibriSpeech, Common Voice, VoxPopuli, Earnings22), and long-form ASR tasks. The results demonstrate state-of-the-art accuracy for an LLM-based ASR system, with low CER/WER scores. Crucially, the paper reports a Real-Time Factor (RTF) of 0.0053 on a single H800 GPU, which is a significant efficiency improvement over standard autoregressive decoding. The ablation studies confirm that MTP does not degrade accuracy and that the staged training is effective. The comparison against baselines like VibeVoice-ASR, FunASR-Nano, and Qwen3-ASR-1.7B is appropriate. The long-form evaluation using a constructed Wenet testnet set adds value, though the construction method (concatenating segments) is a simplification of true long-form challenges.
The paper provides detailed descriptions of the training stages, data mixture (100K hours short-form, 50K hours long-form pseudo-labeled), and hyperparameters (learning rates, batch sizes). However, the use of proprietary datasets and the "publicly available audio-language foundation" (likely StepAudio, which is not fully open-weight or documented in sufficient detail for exact reproduction) limits reproducibility. The specific details of the long-form data curation pipeline (VAD, ROVER, LLM refinement) are described but the code for this pipeline is not provided. The model weights are not mentioned as open-sourced.
The paper does not extensively discuss the computational overhead of the MTP branches during training or the memory footprint. The reliance on pseudo-labeled data for long-form training introduces potential error propagation, although the filtering mechanism ($e > 0.05$) mitigates this. The performance on code-switching and highly noisy environments is mentioned but specific metrics are not detailed in the abstract/text provided. The generalization of the high acceptance rate (5.0/6) to out-of-distribution domains (e.g., heavy accents, different languages not seen in training) is not fully explored.
This work demonstrates that large language models can be efficiently deployed for real-time speech recognition, potentially lowering the barrier for high-quality ASR services. It highlights the synergy between acoustic grounding and language modeling efficiency. The ability to transcribe long audio in a single pass has significant implications for meeting transcription, broadcast monitoring, and archival processing. ParaASR effectively adapts multi-token prediction to ASR, achieving a compelling balance between state-of-the-art accuracy and low-latency inference, thereby challenging the perceived trade-off between model scale and serving efficiency in LLM-based speech recognition.
Speech and audio generation is often needed in animation dubbing, audio drama, movies, advertising, games, podcasts, and short-video production. In these scenarios, creators may need to design voices without reference recordings, control speaker styles with natural language, support acoustic scenes with environments and audio effects, and later reuse the designed voices. Therefore, it is important to support multi-speaker speech and audio generation for both instruct and zero-shot tasks. The instruct task requires a caption of the environment, speaker styles, and fine-grained content, while the zero-shot task uses reference audio together with the same fine-grained content. We address these tasks from both the data and model sides. First, we propose SwanData-Caption, which cleans raw speech and audio data, adds targeted synthetic coverage, and annotates diverse and accurate multi-level captions. Then, we propose SwanTale, a multi-speaker expressive speech and audio generation model that supports both zero-shot and instruct tasks. We introduce SwanVAE to support high-quality multi-audio-modality generation. Then, we adopt reward-conditioned quality control and Engram conditioning, along with Unified MoE for multi-task and multi-audio-modality modeling. In addition, we use curriculum learning and GRPO post-training to let the model progressively learn and strengthen its capabilities. Experimental results show that SwanTale leads on multiple key zero-shot and instruct metrics, achieves the best expressiveness scores in both tasks, and supports complex instruct generation involving multi-speaker speech and audio. Demos can be found at https://swanaigc.github.io/#swantale.
Primary: ByteDance
All Institutions: ByteDance, Zhejiang University
SwanTale presents a robust and unified framework for multi-speaker speech and audio generation, effectively bridging the gap between zero-shot cloning and instruct-based creative control through a sophisticated data pipeline and a novel VAE-MoE architecture, significantly advancing the state-of-the-art in expressive audio synthesis.
The paper proposes SwanTale, a unified model for multi-speaker speech and audio generation supporting both zero-shot (reference-based) and instruct (caption-based) tasks. The methodology is comprehensive, addressing data, model architecture, and training. Key technical contributions include: 1) SwanData-Caption, a pipeline for generating fine-grained multi-level captions using an LLM (Seed2.0 Lite) with a persona library to guide annotation. 2) SwanVAE, a custom VAE with a Gaussian bottleneck and Transformer-based decoder for 48kHz audio, designed to balance reconstruction quality and latent learnability. 3) A flow-based Transformer prior with Unified MoE (Mixture of Experts) to handle multiple tasks and audio modalities (speech, environment, effects, singing). 4) Reward-conditioned quality control and GRPO (Group Relative Policy Optimization) post-training to improve pronunciation, stability, and caption adherence. The approach to unify zero-shot and instruct generation in a single model with complex audio scene modeling is technically sound and represents a significant engineering effort. The use of Engram conditioning and curriculum learning is also noted.
The evaluation covers zero-shot TTS, instruct TTS, and acoustic quality. Benchmarks used include SwanBench-Speech, InstructTTSEval, SwanBench-Scene, and SwanBench-Caption. The paper claims leadership on multiple key metrics. The data pipeline includes rigorous filtering (DNSMOS, SQUIM) and human auditing (best-worst comparison for expressiveness). The inclusion of synthetic data for underrepresented demographics (elderly) and challenging pronunciations is a strong point for robustness. The evaluation of multi-speaker dialogue and complex instruct generation adds depth beyond standard single-speaker TTS.
The paper provides detailed descriptions of the data pipeline, model architecture (SwanVAE, Unified MoE), and training procedures (curriculum, GRPO). However, the code is not publicly available (Project URL is none), and some components (e.g., Seed2.0 Lite, SwanAligner, SwanVerifier) are referenced as internal or proprietary tools. While the methodology is described well, full reproducibility might be challenging without access to the specific internal models and the full dataset (70M records).
The reliance on internal data and proprietary models (Seed2.0, SwanAligner) limits immediate reproducibility. The paper mentions "SwanBench" which appears to be a new benchmark, potentially lacking the long-term community validation of established benchmarks like LibriSpeech or VCTK. The complexity of the model (Unified MoE, GRPO) might make inference and fine-tuning resource-intensive. The "Engram conditioning" is mentioned but not fully detailed in the provided text snippet, which might obscure its specific novelty or implementation details.
This work has significant potential for creative industries (animation, gaming, film) by enabling more expressive and controllable audio generation. The ability to generate complex audio scenes with speech and effects from natural language instructions is a powerful tool for content creation. However, the ease of voice cloning and impersonation raises ethical concerns regarding deepfakes and misinformation, which the authors should address in terms of safety guardrails. SwanTale presents a robust and unified framework for multi-speaker speech and audio generation, effectively bridging the gap between zero-shot cloning and instruct-based creative control through a sophisticated data pipeline and a novel VAE-MoE architecture, significantly advancing the state-of-the-art in expressive audio synthesis.
Audio reasoning is essential for machine understanding of the acoustic world. Reinforcement learning with verifiable rewards can elicit such reasoning, yet existing reward designs are complementary in their limitations: outcome-based rewards supervise only the final answer and let the model reach it without attending to the audio, whereas process-based rewards score the reasoning itself but rely on coarse, hand-crafted, and fixed criteria that neither adapt to each question nor stay grounded in the acoustic evidence. Moreover, questions differ in what they demand, with some hinging on perception and others on multi-step reasoning, and any static criterion weakens as the policy improves. Supervising the reasoning process with fine-grained, audio-grounded, and adaptive rewards is therefore crucial, yet challenging since such rewards are impractical to design by hand for every sample. To this end, we introduce AudioRubrics, a reinforcement learning framework that supervises audio reasoning with self-evolving, audio-grounded rubric rewards. AudioRubrics synthesizes per-sample rubrics from the raw waveform and, conditioned on the model's own rollouts, regenerates and reweights criteria per group, supplying a continuous learning signal that keeps targeting the current policy's weaknesses as static criteria saturate. Comprehensive evaluations across three audio reasoning benchmarks reveal that AudioRubrics substantially outperforms a wide range of open-source and training-based baselines. Furthermore, our analysis shows that the gains scale with the capability of the rubric generator and judge, and AudioRubrics converges to a stable reasoning length that avoids both degenerate collapse and unbounded growth. The improvement in audio perception further demonstrates the effectiveness of anchoring supervision in the acoustic evidence. Our project page is available at https://audiorubrics.github.io.
Primary: Microsoft Research
All Institutions: University of Maryland, College Park, University of Illinois Urbana-Champaign, University of Illinois Chicago, Microsoft Research, MBZUAI
AudioRubrics introduces a novel reinforcement learning framework that utilizes self-evolving, audio-grounded rubrics to provide dense, adaptive process-level rewards for audio reasoning, significantly outperforming static reward baselines on multiple benchmarks.
The paper proposes "AudioRubrics," a reinforcement learning framework designed to enhance the reasoning capabilities of Large Audio-Language Models (LALMs). The core innovation lies in replacing static, hand-crafted process rewards with self-evolving, audio-grounded rubrics. The methodology involves three key steps: (1) Initializing rubrics from raw audio waveforms using an audio-capable model to ensure grounding; (2) Evolving these rubrics during training by eliciting new criteria from the model's own rollouts, pruning non-discriminative ones (those with zero variance in judgments across a group), and re-weighting the survivors; (3) Combining these evolving rubric rewards with outcome accuracy and an overthinking penalty (to prevent verbose, hallucinated reasoning) within a Group Relative Policy Optimization (GRPO) framework. The approach addresses the saturation of static rewards and the lack of audio grounding in text-based process rewards. The technical design is sound, leveraging the heterogeneity of rollouts to generate adaptive evaluation criteria.
The authors evaluate AudioRubrics on three benchmarks: MMAU, MMAR, and MMSU. They compare against a wide range of proprietary (GPT-4o-Audio, Gemini) and open-source baselines (Qwen2-Audio, Phi-4-Multimodal, etc.), as well as other RL-based reasoning methods. The results indicate substantial improvements over baselines, particularly in multi-step reasoning tasks. The paper includes analysis showing that the rubrics evolve to become more sophisticated as training progresses, and that the gains scale with the capability of the rubric generator. The inclusion of an overthinking penalty is shown to stabilize reasoning length. The evaluation is comprehensive and covers diverse audio domains (speech, sound, music).
The paper provides a GitHub repository link and a Hugging Face collection for model weights and datasets. The methodology is described in detail, including the GRPO objective and the specific steps for rubric elicitation and filtering. However, the exact hyperparameters for the rubric generation model, the specific taxonomy of initial dimensions, and the precise implementation details of the "negative criteria" elicitation might require careful reading of the appendix (not fully provided here, but implied). The code availability significantly aids reproducibility.
The approach relies heavily on the quality of the rubric generator/judge model. If this model is biased or lacks audio understanding, the evolving rubrics may propagate errors. The method introduces additional computational overhead during training due to the continuous generation and evaluation of rubrics. The paper mentions that static criteria saturate, but does not extensively discuss the potential for "rubric collapse" where the model optimizes for a narrow set of evolving criteria that do not generalize. The reliance on an external judge model for rubric generation and judging introduces a dependency that might not be available in all settings.
This work contributes to the development of more robust and interpretable audio reasoning systems. By grounding reasoning in audio evidence and providing dense process-level feedback, it reduces hallucinations and improves the reliability of AI systems in critical applications like healthcare and multimedia analysis. The self-evolving nature of the rewards could inspire similar adaptive reward mechanisms in other multimodal reasoning tasks. AudioRubrics introduces a novel reinforcement learning framework that utilizes self-evolving, audio-grounded rubrics to provide dense, adaptive process-level rewards for audio reasoning, significantly outperforming static reward baselines on multiple benchmarks.
Reinforcement learning for flow-matching text-to-speech is complicated by deterministic ODE sampling: trajectory-level policy-gradient methods typically convert the ODE into an SDE and track per-step likelihood ratios, introducing stochastic perturbations and substantial overhead. We propose GROW, a group-relative advantage-weighted on-policy RL method that acts directly on the standard flow-matching objective. For each prompt, GROW samples a group of on-policy utterances, separately standardizes intelligibility and speaker-similarity rewards within the group, and combines them to reweight flow-matching regression. A Wasserstein-2 velocity penalty anchors the updated model to a frozen pretrained reference. A group-mean reward baseline is introduced to convert reward weighting into advantage weighting. For strong pretrained TTS models with concentrated rewards, positive exponential weighting is dominated by reward-agnostic self-imitation, whereas a zero-mean signed advantage preserves effective within-group credit assignment. Instantiated on DiTAR and evaluated on LibriSpeech and Seed-TTS EN/ZH, GROW reduces average WER from 2.016 to 1.558 and raises speaker similarity from 0.676 to 0.715 while keeping UTMOS. With 10-NFE training rollouts and 32-NFE evaluation, GROW retains comparable performance while training 2.9x faster than 32-NFE DiTAR-GRPO. We will open-source complete GROW codes, faithful DiTAR reproduction, and all model checkpoints.
Primary: Shanghai Innovation Institute
All Institutions: Shanghai Innovation Institute, Shanghai Jiao Tong University
GROW introduces a novel, efficient, and theoretically motivated advantage-weighted regression framework for flow-matching TTS, demonstrating superior performance and training efficiency over trajectory-based RL methods. The paper makes a significant technical contribution by bridging the gap between discrete-token RL alignment techniques and continuous flow-matching generators, offering a scalable solution for post-training autoregressive-diffusion models.
The paper proposes GROW, a reinforcement learning method tailored for autoregressive-diffusion Text-to-Speech (TTS) models, specifically instantiated on DiTAR. The core innovation lies in addressing the inefficiency and instability of converting deterministic ODE sampling (used in flow-matching) into Stochastic Differential Equations (SDEs) for policy gradient methods. Instead, GROW applies advantage-weighted regression directly to the flow-matching objective. It samples a group of on-policy rollouts, computes group-relative advantages for intelligibility and speaker similarity rewards, and uses these signed, normalized advantages to weight the standard flow-matching loss. A Wasserstein-2 velocity penalty is used to anchor the model to the pretrained reference, preventing divergence. The methodology is theoretically grounded, drawing on Taylor expansions to justify why linear advantage weighting outperforms exponential weighting in the high-reward, concentrated regime typical of strong pretrained TTS models. The approach is elegant in its simplicity, avoiding the overhead of trajectory storage and per-step likelihood ratio calculations required by methods like GRPO or PPO in continuous spaces.
The experimental evaluation is comprehensive and robust. The authors compare GROW against a strong baseline (DiTAR-GRPO) and the pretrained model across three benchmarks: LibriSpeech-PC, Seed-TTS EN, and Seed-TTS ZH. Results show consistent improvements in Word Error Rate (WER) and Speaker Similarity (SIM) while maintaining or slightly improving UTMOS. Notably, GROW achieves these gains with significantly higher training efficiency, being 2.9x faster than the 32-NFE DiTAR-GRPO baseline when using 10-NFE rollouts. The ablation studies are particularly strong, providing detailed analysis on the impact of rollout NFE, CFG settings, learning rates, W2 anchor strength, and reward design. The decomposition of self-imitation vs. reward-contrast provides valuable insight into why the proposed weighting scheme works better than exponential alternatives.
The paper provides extensive implementation details, including model architecture (Qwen3-0.6B backbone, LocDiT), hyperparameters (learning rates, group size, NFEs), and dataset descriptions. The authors explicitly state they will open-source the code, reproduction, and checkpoints, which significantly enhances reproducibility. The algorithm is clearly defined with pseudocode. The use of standard metrics (WER, SIM, UTMOS) and public datasets further supports reproducibility.
The method is currently evaluated primarily on zero-shot TTS tasks using specific reward models (ASR for WER, Speaker Encoder for SIM). The generalizability to other TTS objectives (e.g., emotion, prosody) or other flow-matching architectures (e.g., F5-TTS) is suggested but not extensively demonstrated. The reliance on frozen reward models introduces potential biases inherent in those models (e.g., ASR errors affecting WER rewards). Additionally, the performance gain, while consistent, is modest in absolute terms (e.g., WER reduction of ~0.45), though significant in the context of already high-performing models. The method's effectiveness on very short or very long utterances is not explicitly analyzed.
This work contributes to the efficient alignment of generative audio models, potentially lowering the computational cost of post-training for high-quality TTS systems. This could democratize access to advanced speech synthesis technologies. However, as with any TTS advancement, there are risks associated with improved voice cloning capabilities, including potential misuse for deepfakes or misinformation. The authors' open-source commitment promotes transparency and further research in this area. GROW introduces a novel, efficient, and theoretically motivated advantage-weighted regression framework for flow-matching TTS, demonstrating superior performance and training efficiency over trajectory-based RL methods. The paper makes a significant technical contribution by bridging the gap between discrete-token RL alignment techniques and continuous flow-matching generators, offering a scalable solution for post-training autoregressive-diffusion models.
What is music style? Though often described using text labels such as "swing," "classical," or "emotional," the real style remains implicit and hidden in concrete music examples. In this paper, we introduce a cross-modal framework that learns implicit music styles from raw audio and applies them to symbolic music generation. Inspired by BLIP-2, our model leverages a Querying Transformer (Q-Former) to extract style representations from a large, pre-trained audio language model (LM), and further applies them to condition a symbolic LM for generating piano arrangements. We adopt a two-stage training strategy: contrastive learning to align auditory style with symbolic expression, followed by generative modeling for music arrangement. Our model generates piano performances jointly conditioned on a lead sheet (content) and a reference audio example (style), enabling controllable and stylistically faithful arrangement. Experiments demonstrate the effectiveness of our approach in piano cover generation, style transfer, and audio-to-MIDI retrieval, achieving substantial improvements in style-aware alignment and music quality.
Primary: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
All Institutions: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), National University of Singapore, School of Computing
The paper presents a novel and technically sound approach to cross-modal music generation, effectively applying vision-language bootstrapping techniques to the audio-symbolic domain for piano arrangement.
The paper proposes a cross-modal framework for piano arrangement by leveraging a Querying Transformer (Q-Former) to bridge audio and symbolic domains. The core innovation lies in using the Q-Former to extract implicit style representations from a frozen audio language model (MusicGen) and conditioning a symbolic language model (MuseCoco) with these representations, alongside a lead sheet for content. The two-stage training processâcontrastive/matching learning for style alignment followed by generative modelingâis a logical extension of vision-language bootstrapping methods (like BLIP-2) to the audio-symbolic domain. The approach effectively disentangles content (lead sheet) from style (audio reference), addressing a significant gap in current music generation systems that often rely on text prompts or explicit symbolic controls. The use of LoRA adapters to condition the frozen symbolic LM is a practical and efficient design choice.
The experimental evaluation is comprehensive, covering objective metrics (MCA, CA, GPC, VCC, TA) and subjective listening tests. The inclusion of out-of-distribution datasets (Ballroom, GTZAN) strengthens the claim of generalization. The results show competitive performance against baselines like PiCoGen2 and A2M, particularly in style coherence metrics. The audio-to-MIDI retrieval task further validates the quality of the learned cross-modal representations. However, the lower content preservation scores on the in-distribution POP909 set compared to PCG2 are a notable weakness, attributed to errors in the upstream Sheetsage transcription. The subjective evaluation, while well-designed with ANOVA analysis, relies on a small sample size (21 participants), which limits the statistical power of the findings.
The paper provides detailed model configurations, including parameter counts, training epochs, batch sizes, and optimizer settings. The use of open-source or widely available backbones (MusicGen, MuseCoco, Sheetsage) enhances reproducibility. The data pairing strategy and specific metric definitions are clearly described. The lack of a public code repository is a minor drawback, but the detailed appendix and demo page mitigate this to some extent.
The authors acknowledge that the style representation is segment-level (4-bar), which may not capture long-term stylistic evolution or transitions. The reliance on Sheetsage for lead sheet extraction introduces potential error propagation. The method is currently limited to piano arrangement, and generalizing to multi-track or more complex instrumental arrangements is identified as future work. The subjective evaluation sample size is small.
This work contributes to the field of AI-assisted music creation by providing a tool for expressive piano arrangement. It enables musicians to reinterpret existing audio with specific stylistic nuances while maintaining harmonic and melodic integrity. The cross-modal representation learning technique has broader applications in audio-symbolic retrieval and style transfer. The potential for misuse in generating copyrighted music arrangements without permission is a consideration, though the focus on arrangement rather than direct cloning mitigates some risks. The paper presents a novel and technically sound approach to cross-modal music generation, effectively applying vision-language bootstrapping techniques to the audio-symbolic domain for piano arrangement.
Reference-conditioned melody-preserving lyric editing replaces words while retaining a performance's timing, singer identity, and naturalness. Continuous-latent autoregression avoids finite codebooks and offers stepwise generation with learned stopping. Editing creates a conflict absent from ordinary reconstruction: training pairs reference cues with original lyrics, whereas inference asks revised lyrics to override source-lyric-correlated cues; one source-following patch can propagate through AR history. We introduce CLASVS. Its State-Control-Transition (SCT) routing keeps target-lyric and reference-melody controls persistent, returns semantic feedback on phonetic progress to the causal planner, and confines the previous latent patch to the local Transition. Progressive State-Control Grounding (PSCG) learns this contract through paired-edit-free, content-consistent Mandarin reconstruction. On two Mandarin benchmarks, CLASVS improves all four operations over discrete-AR Vevo2 and reduces macro-PER by 46.2%, while maintaining melody, singer similarity, and perceptual quality. Together, these results establish a strong continuous-AR operating point for score-annotation-free lyric edits and a basis for broader stepwise control. Audio demonstrations are available on our project page: https://piedpiperg.github.io/Liyric-SVS/.
Primary: Tencent AI Lab
All Institutions: Tencent AI Lab, Tsinghua University, Peking University
CLASVS introduces a novel continuous-latent autoregressive framework with a State-Control-Transition routing mechanism that effectively addresses the conflict between melody preservation and lyric editing in singing voice synthesis, achieving state-of-the-art performance in Mandarin lyric editing tasks.
The paper proposes CLASVS, a continuous-latent autoregressive model for melody-preserving lyric editing in singing voice synthesis. The core methodological contribution is the State-Control-Transition (SCT) routing mechanism, which attempts to resolve the conflict between preserving reference melody/timing and generating new lyrics. It separates the control signals (target lyrics, reference melody) from the generative state (semantic feedback, previous latent patch). The training strategy, Progressive State-Control Grounding (PSCG), relies on content-consistent reconstruction (training on the same audio/transcript) rather than paired counterfactual edits, which is a significant theoretical challenge. The use of a frozen causal semantic encoder for phonetic progress feedback is a novel architectural choice for maintaining lyric alignment in an autoregressive setting. However, the reliance on existing components (AudioVAE, Qwen3-0.6B, Flow-DiT) limits the novelty of the underlying generative primitives, focusing innovation primarily on the routing and training protocol.
The evaluation is comprehensive, comparing CLASVS against discrete-AR (Vevo2) and continuous-NAR (YingMusic-Singer-Plus) baselines. The paper reports significant improvements in macro-Phone Error Rate (PER) and perceptual metrics (MOS) over Vevo2. The inclusion of a "target-preference margin" metric to audit whether the model actually follows target lyrics rather than reverting to source lyrics is a strong methodological addition to the evaluation suite. The ablation studies effectively isolate the contributions of the SCT components. However, the evaluation is restricted to Mandarin, and the comparison with YingMusic+ shows mixed results (CLASVS is better at deletion/insertion, but YingMusic+ is better at substitution), suggesting that the autoregressive approach may have limitations in handling large lexical changes compared to parallel generation.
The paper provides detailed descriptions of the architecture, training stages, and hyperparameters. It mentions the release of code and checkpoints upon publication, which supports reproducibility. The use of standard datasets (Emilia, curated singing) and public baselines facilitates comparison. The specific "CLA-LyricEdit-320" benchmark is described in detail, allowing for independent verification.
The primary limitation is the lack of paired counterfactual training data, which forces the model to generalize from reconstruction tasks. This may limit its ability to handle complex edits or non-Mandarin languages. The autoregressive nature of the model results in slower inference speeds compared to NAR baselines like YingMusic+. The evaluation is limited to Mandarin, and cross-lingual generalization is not tested. Additionally, the reliance on automatic melody extraction (chroma-derived tokens) may introduce errors that propagate through the generation process.
This work contributes to the field of controllable singing voice synthesis, enabling more natural and flexible lyric editing without the need for manual score annotations. This has applications in music production, karaoke, and accessibility. However, the ability to edit lyrics while preserving singer identity raises ethical concerns regarding voice impersonation and copyright, which the authors acknowledge and address with proposed safety measures. CLASVS introduces a novel continuous-latent autoregressive framework with a State-Control-Transition routing mechanism that effectively addresses the conflict between melody preservation and lyric editing in singing voice synthesis, achieving state-of-the-art performance in Mandarin lyric editing tasks.
Spoken emotional dialogue requires a model to understand a user's spoken input and generate a response that is both semantically appropriate and emotionally expressive. This is challenging because communicative intent may be stated explicitly in lexical content or conveyed more implicitly through paralinguistic cues, which can complement or diverge from the words themselves. However, two limitations constrain progress in this area: the scarcity of benchmarks that distinguish these intent expressions, and the lack of reinforcement learning objectives that jointly account for response quality and emotional expression. To address the lack of suitable benchmarks, we introduce ParaIntent, a Chinese benchmark comprising 14 intent categories with balanced explicit and implicit samples, together with a multidimensional evaluation protocol covering intent fulfillment, response quality, and emotional expression. For policy optimization, existing approaches either use a shared objective for text and speech or apply reinforcement learning to only one modality, leaving modality-specific learning signals entangled within policy optimization. Motivated by this, we propose Acoustic-Lexical Decoupled Policy Optimization (ALPO), which computes independent textual and acoustic advantages and routes them to the corresponding text and speech tokens within a unified rollout. Under identical reward functions and training budgets, ALPO improves over standard GRPO on most automatic metrics and achieves the best subjective results among the fine-tuned variants, with particularly clear gains in emotional expressiveness on both the synthetic and human-recorded test sets.
Primary: StepFun (inferred from "Step-Audio-2-mini" and author list context, though explicitly listed as Step in affiliations)
All Institutions: StepFun
The paper presents a practical and effective method for improving spoken LLMs by decoupling textual and acoustic advantages during reinforcement learning, supported by a new benchmark and rigorous evaluation.
The paper proposes Acoustic-Lexical Decoupled Policy Optimization (ALPO), a reinforcement learning method designed for spoken dialogue models that interleaves text and speech tokens. The core innovation is the decoupling of advantages: instead of applying a single sequence-level reward (as in standard GRPO) to all tokens, ALPO computes separate textual and acoustic advantages and routes them only to their corresponding token types. This addresses the "credit assignment" problem where shared rewards create conflicting gradients for text quality versus emotional expression. The method is theoretically sound and directly addresses a known limitation in current speech-LLM post-training strategies (like ParaS2S and WavAlign). However, the mechanism is a relatively straightforward extension of existing RLHF/RLVR techniques (specifically GRPO) applied to a multimodal token structure. It lacks deep architectural novelty, relying instead on a clever engineering solution to a specific training instability.
The experimental evaluation is comprehensive and robust. The authors introduce ParaIntent, a new Chinese benchmark with 14 intent categories, balancing explicit and implicit intent. They provide both synthetic and human-recorded test sets. The results show consistent improvements over SFT, DPO, GRPO, and Hybrid SFT-GRPO baselines across automatic metrics (Intent Fulfillment, Response Quality, Emotion Accuracy, Emotion Similarity) and subjective evaluations (LLM-based pairwise and human ratings). The ablation studies effectively isolate the contribution of token-specific routing versus independent normalization. The analysis of gradient norms provides strong empirical evidence for the method's effectiveness in balancing text and speech learning signals. The use of held-out evaluators (DeepSeek-V4-Pro) for some metrics adds credibility.
The paper provides sufficient detail for reproduction. The dataset construction pipeline (GLM-5 for text, IndexTTS2 for speech) is described. The reward functions are defined. The backbone model (Step-Audio-2-mini) and training setup are specified. The code for the baseline methods is likely available or easily implementable given the references. The primary barrier to reproducibility is the proprietary nature of the Step-Audio-2-mini model and the specific LLM evaluators used, though the methodology itself is generalizable.
The paper acknowledges several limitations. The benchmark is in Chinese, limiting immediate generalizability to other languages. The human-recorded test set, while valuable, is small (4.2K) and collected under controlled protocols, not spontaneous interaction. The method is evaluated on a single-turn setting. The reliance on TTS synthesis for training data (140K samples) means the model may not fully capture the nuances of real human speech generation, although the human-recorded test set mitigates this somewhat. The "implicit intent" definition relies on paralinguistic cues, which can be subjective and difficult to ground objectively.
This work contributes to the field of spoken dialogue systems by providing a more nuanced benchmark for emotional intent and a training method that improves the balance between semantic content and emotional expression. This can lead to more natural and empathetic AI assistants. The benchmark ParaIntent could serve as a standard for evaluating future spoken LLMs in Chinese. The decoupled policy optimization approach could be adapted to other multimodal generation tasks where distinct modalities serve different functions (e.g., video generation with text and visual tokens). The paper presents a practical and effective method for improving spoken LLMs by decoupling textual and acoustic advantages during reinforcement learning, supported by a new benchmark and rigorous evaluation.
Recent zero-shot text-to-speech (TTS) systems achieve remarkable naturalness and speaker similarity but typically require high-quality speaker prompts and either strip away or entangle the acoustic environment with speaker characteristics, limiting their real-world applicability. We present an extended DAIEN-TTS, an environment-aware zero-shot TTS framework that disentangles and jointly models speech, background noise, and reverberation, enabling independent control over timbre and acoustic environment through separate speaker and environment prompts. Built upon the flow-matching-based F5-TTS, it uses a speech-environment separation module to decompose environmental speech into speech, noise, and reverberation components, which are injected into the Diffusion Transformer for environment-aware generation. Training uses simulated data constructed by mixing clean speech with noise and room impulse responses, together with a cross-speaker conditioning strategy that suppresses speaker information leakage from the environment branch. When real-world data are available, the system can be further fine-tuned to bridge the simulated-to-real domain gap.At inference, a triple classifier-free guidance mechanism enables fine-grained control over speech, noise, and reverberation, and a signal-to-noise-ratio adaptation strategy aligns the synthesized speech with the environment prompt. Experiments on simulated and real-world test sets show that DAIEN-TTS generates environmental personalized speech with high naturalness, strong speaker similarity, and faithful noise and reverberation reproduction, while offering controllability beyond prior environment-aware TTS systems.
Primary: University of Science and Technology of China
All Institutions: University of Science and Technology of China, National Institute of Informatics
This paper presents a technically robust extension of environment-aware zero-shot TTS, effectively disentangling speech, noise, and reverberation to enable independent control, thereby enhancing the realism and applicability of synthesized speech in diverse acoustic scenarios.
The paper proposes an extension of the DAIEN-TTS framework, building upon the flow-matching-based F5-TTS architecture. The core methodological contribution is the integration of a Speech-Environment Separation (SES) module that disentangles environmental speech into clean speech, background noise, and reverberation components. This allows for independent control over timbre (via speaker prompt) and acoustic environment (via environment prompt). The authors introduce a two-stage separation process using masking networks and a cross-speaker conditioning strategy during training to prevent speaker identity leakage into the environment branch. At inference, a Triple Classifier-Free Guidance (TCFG) mechanism and SNR adaptation strategy are employed to align the synthesized speech with the environment prompt. The approach is technically sound, leveraging established techniques (masking, flow matching, DiT) in a novel configuration for the specific problem of disentangled environment-aware TTS. However, the novelty is incremental, primarily extending a previous conference paper by adding reverberation modeling and real-world fine-tuning capabilities.
The evaluation covers both simulated and real-world test sets. The authors demonstrate that the system can generate environmental personalized speech with high naturalness, strong speaker similarity, and faithful noise/reverberation reproduction. The inclusion of a simulated-to-real domain gap bridging strategy via fine-tuning on real-world data is a significant practical contribution. The experiments appear comprehensive, addressing the limitations of prior work that either discarded environmental factors or entangled them with speaker characteristics. The use of both objective metrics (implied by the context of TTS evaluation) and subjective evaluations (naturalness, similarity) is standard and appropriate. The results support the claim of improved controllability over prior environment-aware TTS systems.
The paper provides detailed descriptions of the SES module, the TTS module architecture, and the training strategies. The use of open-source components like F5-TTS and ECAPA-TDNN aids reproducibility. However, the specific implementation details of the masking networks and the exact hyperparameters for the TCFG and SNR adaptation are crucial for full reproducibility and are likely included in the full text (though truncated here). The reliance on simulated data for pretraining and the specific recipe for mixing noise and RIRs should be well-defined. The paper mentions it is an extended version of a conference paper, suggesting the core methodology is established.
A key limitation is the reliance on the SES module's accuracy; errors in separating noise and reverberation will propagate to the TTS generation. The assumption that reverberation can be effectively encoded as a single utterance-level embedding may oversimplify complex, spatially varying acoustic environments. The performance on real-world data depends on the quality of the separation module when applied to unseen real-world recordings, which may differ from the simulated distribution. The paper does not explicitly discuss the computational cost or latency of the two-stage separation plus generation pipeline, which could be a barrier for real-time applications.
This work has significant potential impact on AI audiobook production, virtual reality, and the generation of synthetic training data for ASR and ASV systems. By enabling realistic reproduction of diverse acoustic environments, it addresses a critical gap in current TTS systems. The ability to control environmental factors independently opens new avenues for creative applications and robust model training. However, the potential for misuse in generating realistic-sounding fake audio in specific environments (e.g., deepfakes with specific background noise) remains a concern, consistent with broader audio generation risks. This paper presents a technically robust extension of environment-aware zero-shot TTS, effectively disentangling speech, noise, and reverberation to enable independent control, thereby enhancing the realism and applicability of synthesized speech in diverse acoustic scenarios.
Speech and audio generation is often needed in animation dubbing, audio drama, movies, advertising, games, podcasts, and short-video production. In these scenarios, creators may need to design voices without reference recordings, control speaker styles with natural language, support acoustic scenes with environments and audio effects, and later reuse the designed voices. Therefore, it is important to support multi-speaker speech and audio generation for both instruct and zero-shot tasks. The instruct task requires a caption of the environment, speaker styles, and fine-grained content, while the zero-shot task uses reference audio together with the same fine-grained content. We address these tasks from both the data and model sides. First, we propose SwanData-Caption, which cleans raw speech and audio data, adds targeted synthetic coverage, and annotates diverse and accurate multi-level captions. Then, we propose SwanTale, a multi-speaker expressive speech and audio generation model that supports both zero-shot and instruct tasks. We introduce SwanVAE to support high-quality multi-audio-modality generation. Then, we adopt reward-conditioned quality control and Engram conditioning, along with Unified MoE for multi-task and multi-audio-modality modeling. In addition, we use curriculum learning and GRPO post-training to let the model progressively learn and strengthen its capabilities. Experimental results show that SwanTale leads on multiple key zero-shot and instruct metrics, achieves the best expressiveness scores in both tasks, and supports complex instruct generation involving multi-speaker speech and audio. Demos can be found at https://swanaigc.github.io/#swantale.
Primary: ByteDance
All Institutions: ByteDance, Zhejiang University
SwanTale presents a robust and unified framework for multi-speaker speech and audio generation, effectively bridging the gap between zero-shot cloning and instruct-based creative control through a sophisticated data pipeline and a novel VAE-MoE architecture, significantly advancing the state-of-the-art in expressive audio synthesis.
The paper proposes SwanTale, a unified model for multi-speaker speech and audio generation supporting both zero-shot (reference-based) and instruct (caption-based) tasks. The methodology is comprehensive, addressing data, model architecture, and training. Key technical contributions include: 1) SwanData-Caption, a pipeline for generating fine-grained multi-level captions using an LLM (Seed2.0 Lite) with a persona library to guide annotation. 2) SwanVAE, a custom VAE with a Gaussian bottleneck and Transformer-based decoder for 48kHz audio, designed to balance reconstruction quality and latent learnability. 3) A flow-based Transformer prior with Unified MoE (Mixture of Experts) to handle multiple tasks and audio modalities (speech, environment, effects, singing). 4) Reward-conditioned quality control and GRPO (Group Relative Policy Optimization) post-training to improve pronunciation, stability, and caption adherence. The approach to unify zero-shot and instruct generation in a single model with complex audio scene modeling is technically sound and represents a significant engineering effort. The use of Engram conditioning and curriculum learning is also noted.
The evaluation covers zero-shot TTS, instruct TTS, and acoustic quality. Benchmarks used include SwanBench-Speech, InstructTTSEval, SwanBench-Scene, and SwanBench-Caption. The paper claims leadership on multiple key metrics. The data pipeline includes rigorous filtering (DNSMOS, SQUIM) and human auditing (best-worst comparison for expressiveness). The inclusion of synthetic data for underrepresented demographics (elderly) and challenging pronunciations is a strong point for robustness. The evaluation of multi-speaker dialogue and complex instruct generation adds depth beyond standard single-speaker TTS.
The paper provides detailed descriptions of the data pipeline, model architecture (SwanVAE, Unified MoE), and training procedures (curriculum, GRPO). However, the code is not publicly available (Project URL is none), and some components (e.g., Seed2.0 Lite, SwanAligner, SwanVerifier) are referenced as internal or proprietary tools. While the methodology is described well, full reproducibility might be challenging without access to the specific internal models and the full dataset (70M records).
The reliance on internal data and proprietary models (Seed2.0, SwanAligner) limits immediate reproducibility. The paper mentions "SwanBench" which appears to be a new benchmark, potentially lacking the long-term community validation of established benchmarks like LibriSpeech or VCTK. The complexity of the model (Unified MoE, GRPO) might make inference and fine-tuning resource-intensive. The "Engram conditioning" is mentioned but not fully detailed in the provided text snippet, which might obscure its specific novelty or implementation details.
This work has significant potential for creative industries (animation, gaming, film) by enabling more expressive and controllable audio generation. The ability to generate complex audio scenes with speech and effects from natural language instructions is a powerful tool for content creation. However, the ease of voice cloning and impersonation raises ethical concerns regarding deepfakes and misinformation, which the authors should address in terms of safety guardrails. SwanTale presents a robust and unified framework for multi-speaker speech and audio generation, effectively bridging the gap between zero-shot cloning and instruct-based creative control through a sophisticated data pipeline and a novel VAE-MoE architecture, significantly advancing the state-of-the-art in expressive audio synthesis.
Audio reasoning is essential for machine understanding of the acoustic world. Reinforcement learning with verifiable rewards can elicit such reasoning, yet existing reward designs are complementary in their limitations: outcome-based rewards supervise only the final answer and let the model reach it without attending to the audio, whereas process-based rewards score the reasoning itself but rely on coarse, hand-crafted, and fixed criteria that neither adapt to each question nor stay grounded in the acoustic evidence. Moreover, questions differ in what they demand, with some hinging on perception and others on multi-step reasoning, and any static criterion weakens as the policy improves. Supervising the reasoning process with fine-grained, audio-grounded, and adaptive rewards is therefore crucial, yet challenging since such rewards are impractical to design by hand for every sample. To this end, we introduce AudioRubrics, a reinforcement learning framework that supervises audio reasoning with self-evolving, audio-grounded rubric rewards. AudioRubrics synthesizes per-sample rubrics from the raw waveform and, conditioned on the model's own rollouts, regenerates and reweights criteria per group, supplying a continuous learning signal that keeps targeting the current policy's weaknesses as static criteria saturate. Comprehensive evaluations across three audio reasoning benchmarks reveal that AudioRubrics substantially outperforms a wide range of open-source and training-based baselines. Furthermore, our analysis shows that the gains scale with the capability of the rubric generator and judge, and AudioRubrics converges to a stable reasoning length that avoids both degenerate collapse and unbounded growth. The improvement in audio perception further demonstrates the effectiveness of anchoring supervision in the acoustic evidence. Our project page is available at https://audiorubrics.github.io.
Primary: Microsoft Research
All Institutions: University of Maryland, College Park, University of Illinois Urbana-Champaign, University of Illinois Chicago, Microsoft Research, MBZUAI
AudioRubrics introduces a novel reinforcement learning framework that utilizes self-evolving, audio-grounded rubrics to provide dense, adaptive process-level rewards for audio reasoning, significantly outperforming static reward baselines on multiple benchmarks.
The paper proposes "AudioRubrics," a reinforcement learning framework designed to enhance the reasoning capabilities of Large Audio-Language Models (LALMs). The core innovation lies in replacing static, hand-crafted process rewards with self-evolving, audio-grounded rubrics. The methodology involves three key steps: (1) Initializing rubrics from raw audio waveforms using an audio-capable model to ensure grounding; (2) Evolving these rubrics during training by eliciting new criteria from the model's own rollouts, pruning non-discriminative ones (those with zero variance in judgments across a group), and re-weighting the survivors; (3) Combining these evolving rubric rewards with outcome accuracy and an overthinking penalty (to prevent verbose, hallucinated reasoning) within a Group Relative Policy Optimization (GRPO) framework. The approach addresses the saturation of static rewards and the lack of audio grounding in text-based process rewards. The technical design is sound, leveraging the heterogeneity of rollouts to generate adaptive evaluation criteria.
The authors evaluate AudioRubrics on three benchmarks: MMAU, MMAR, and MMSU. They compare against a wide range of proprietary (GPT-4o-Audio, Gemini) and open-source baselines (Qwen2-Audio, Phi-4-Multimodal, etc.), as well as other RL-based reasoning methods. The results indicate substantial improvements over baselines, particularly in multi-step reasoning tasks. The paper includes analysis showing that the rubrics evolve to become more sophisticated as training progresses, and that the gains scale with the capability of the rubric generator. The inclusion of an overthinking penalty is shown to stabilize reasoning length. The evaluation is comprehensive and covers diverse audio domains (speech, sound, music).
The paper provides a GitHub repository link and a Hugging Face collection for model weights and datasets. The methodology is described in detail, including the GRPO objective and the specific steps for rubric elicitation and filtering. However, the exact hyperparameters for the rubric generation model, the specific taxonomy of initial dimensions, and the precise implementation details of the "negative criteria" elicitation might require careful reading of the appendix (not fully provided here, but implied). The code availability significantly aids reproducibility.
The approach relies heavily on the quality of the rubric generator/judge model. If this model is biased or lacks audio understanding, the evolving rubrics may propagate errors. The method introduces additional computational overhead during training due to the continuous generation and evaluation of rubrics. The paper mentions that static criteria saturate, but does not extensively discuss the potential for "rubric collapse" where the model optimizes for a narrow set of evolving criteria that do not generalize. The reliance on an external judge model for rubric generation and judging introduces a dependency that might not be available in all settings.
This work contributes to the development of more robust and interpretable audio reasoning systems. By grounding reasoning in audio evidence and providing dense process-level feedback, it reduces hallucinations and improves the reliability of AI systems in critical applications like healthcare and multimedia analysis. The self-evolving nature of the rewards could inspire similar adaptive reward mechanisms in other multimodal reasoning tasks. AudioRubrics introduces a novel reinforcement learning framework that utilizes self-evolving, audio-grounded rubrics to provide dense, adaptive process-level rewards for audio reasoning, significantly outperforming static reward baselines on multiple benchmarks.
Token compression in OmniLLMs is typically posed as a single saliency-ranking problem: score each multimodal token, keep the top-K. We argue this abstraction is mis-specified. The same attention score simultaneously decides two things: how much retained capacity each modality receives, and which tokens within a modality are kept. A shared top-K rule therefore inherits this audio-favoring allocation prior, spending retained capacity on audio before video tokens have a chance to compete. We propose Macer, a training-free compressor that first assigns explicit audio and video budgets, then performs allocation-normalized ranking within each modality at modality-specific shallow layers. Macer significantly reduces token cost while preserving accuracy across audio-grounded, audio--video joint, visual-dominant, and video-centric benchmarks. At 25 % retention, Macer preserves 98.7 % of full-token performance on Qwen2.5-Omni-7B and 97.3 % on Qwen2.5-Omni-3B. On Qwen2.5-Omni-7B, this 25 % setting reaches OmniZip-level performance at 45 % retention while using lower FLOPs. On OmniVinci-9B, the same allocation-before-ranking principle improves over shared top-K ranking by up to 12.9 points.
Primary: University of Houston
All Institutions: University of Houston, The University of Texas at Arlington
The paper makes a significant contribution to multimodal efficiency by identifying and correcting a fundamental mis-specification in how shared-attention mechanisms handle token compression, proposing a theoretically grounded, training-free solution that significantly improves the efficiency-accuracy trade-off for OmniLLMs.
The paper proposes "Macer," a training-free token compression method for OmniLLMs (Omnimodal Large Language Models). The core theoretical contribution is the identification of a structural flaw in existing shared-decoder compression methods: the coupling of cross-modal capacity allocation and within-modality token selection via a single shared attention score. The authors argue that raw shared attention inherently favors audio tokens due to architectural priors, leading to suboptimal video retention. Macer addresses this by decoupling these two decisions. It first explicitly allocates a budget for audio and video tokens based on a user-defined ratio. Then, it performs modality-local ranking using "Allocation-Normalized Modality Scoring" (ANMS), which normalizes attention scores within each modality to remove the cross-modal mass bias. Furthermore, it leverages the observation that audio and video saliency become readable at different shallow layers, applying pruning at modality-specific depths ($L_a$ for audio, $L_v$ for video). A lightweight temporal coverage term is added for video to prevent temporal clustering. The approach is inference-time only, requiring no training or model modification.
The evaluation is conducted on three OmniLLM backbones: Qwen2.5-Omni-7B, Qwen2.5-Omni-3B, and OmniVinci-9B. Benchmarks cover audio-grounded, audio-video joint, visual-dominant, and video-centric tasks. The results claim significant performance preservation at low retention ratios (e.g., 98.7% of full performance at 25% retention on Qwen2.5-Omni-7B). The paper compares Macer against ToMe-style merging and random dropping, demonstrating that explicit budget allocation outperforms implicit budgeting induced by shared ranking. The ablation studies support the three key observations: (1) shared attention is not neutral, (2) budgeting is the primary driver of cross-modal balance, and (3) modality-specific readout layers are beneficial. The results are strong and directly validate the proposed mechanism.
The paper provides detailed mathematical formulations for the ANMS scores and the budget allocation logic. It specifies the hyperparameters used (readout layers, budget ratios). Since the method is training-free and applied to open-source models (Qwen2.5-Omni, OmniVinci), reproduction is feasible for researchers with access to the necessary compute. The paper mentions an appendix with additional details, which is standard for reproducibility.
The method relies on specific architectural assumptions of shared-decoder OmniLLMs where audio and video tokens are interleaved and processed by the same attention mechanism. Its applicability to encoder-decoder architectures or models with separate modality-specific encoders that feed into a unified decoder might require adaptation. The "training-free" nature means it cannot learn task-specific compression patterns that a trained compressor might, potentially capping its maximum efficiency compared to learned methods. The paper notes that the temporal coverage term is lightweight but does not fully address complex long-range video dependencies if the budget is extremely tight.
This work addresses a critical bottleneck in deploying OmniLLMs: inference cost and latency. By enabling efficient compression without retraining, it lowers the barrier to using these models in resource-constrained environments. The insight into modality bias in shared attention mechanisms is broadly applicable to the design and analysis of multimodal models, potentially influencing future model architectures to be more modality-agnostic or to handle allocation explicitly. The paper makes a significant contribution to multimodal efficiency by identifying and correcting a fundamental mis-specification in how shared-attention mechanisms handle token compression, proposing a theoretically grounded, training-free solution that significantly improves the efficiency-accuracy trade-off for OmniLLMs.
We present JoyAI-Talker, a full-duplex speech dialogue system that delivers robust foundation model capabilities while empowering empathetic interaction and voice agent intelligence. JoyAI-Talker adopts a modular Thinker-Talker architecture and further implements a unified speech-text joint training pipeline to mitigate the common "cognitive degradation" bottleneck, thereby largely preserving the model's core textual reasoning, STEM, and logical capabilities while extending them to speech-based interaction. For expressive speech synthesis, the Talker module employs a text-controllable generation paradigm that enables natural-language instructions to flexibly control vocal attributes and localized paralinguistic events, such as laughter and sighs, supporting more expressive and fine-grained speech responses. To enhance conversational empathy, we introduce the Persona-Adaptive Empathetic Response (PAER) framework. PAER employs a hierarchical cognitive pipeline to extract non-verbal speaker cues, such as gender, age, and emotional state, from raw input audio, incorporate them into the Thinker's CoT reasoning, and generate context-adaptive responses that align semantically appropriate text with fine-grained control over utterance-level expressiveness and localized paralinguistic events, including sighs, speaking rate, and volume. We further integrate Joy-Duplex, a state-driven, plug-and-play full-duplex framework that functions as an efficient gating engine for real-time turn control. Extensive evaluations show that JoyAI-Talker achieves highly competitive performance on foundational T2T and S2T benchmarks. In full-duplex evaluation, the system reaches a high response rate of 0.88 under user interruptions while maintaining an extremely low false-trigger rate under background speech, demonstrating its readiness for fluid and natural speech dialogue.
Primary: JD.com
All Institutions: JD.com
[One sentence main contribution]. [The paper presents JoyAI-Talker, a modular full-duplex speech dialogue system that integrates a large MoE language model with an expressive speech generator, utilizing a novel state-driven gating mechanism and a persona-adaptive empathy framework to achieve robust, low-latency, and emotionally intelligent voice interactions while preserving the reasoning capabilities of the underlying LLM.]
The paper proposes JoyAI-Talker, a full-duplex speech dialogue system built on a decoupled "Thinker-Talker" architecture. The Thinker is a 48.9B parameter MoE LLM (JoyAI-LLM Flash) adapted for speech via a projector, while the Talker is a speech generation model based on JoyVoice. Key methodological contributions include: 1) A unified speech-text joint training pipeline starting from mid-training to mitigate "cognitive degradation" in LLMs; 2) The Persona-Adaptive Empathetic Response (PAER) framework, which uses Chain-of-Thought reasoning to extract speaker attributes (gender, age, emotion) from audio and inject them into the response generation; 3) A state-driven full-duplex framework (Joy-Duplex) using a lightweight 1.7B decoder to predict interleaved text and state tokens (e.g., `<|accept|>`, `<|reject|>`) for turn-taking control, rather than relying on energy-based VAD. The approach is technically sound and addresses significant pain points in current LSLM (Large Speech Language Model) systems, particularly the trade-off between reasoning capability and acoustic alignment, and the fragility of full-duplex interaction. However, the core components (MoE LLMs, DiT-based TTS, CoT for empathy) are incremental combinations of existing state-of-the-art techniques rather than fundamentally new algorithmic breakthroughs.
The evaluation is extensive and covers multiple dimensions: T2T benchmarks (to verify reasoning retention), S2T benchmarks (ASR, translation, QA), empathy perception (AIR-Bench, MER2025), empathetic dialogue generation (EchoMind), and full-duplex interaction (Full-Duplex-Bench v1.5). The results show competitive performance on T2T and S2T tasks, demonstrating that the joint training did not catastrophically forget textual capabilities. The full-duplex evaluation highlights a strong balance between interruption response (0.88) and low false-trigger rates, which is a practical and significant achievement. The inclusion of a commercial baseline (Gemini 3.1 Live) adds credibility. However, the paper relies heavily on LLM-as-a-Judge for empathy metrics, which can be biased. The specific architectural novelty of the "Thinker-Talker" interface is standard in many recent systems (e.g., Qwen2-Audio, Moshi), though the specific implementation of the state-token gating is a notable engineering contribution.
The paper provides detailed descriptions of the training stages (Mid-training, Context Extension, SFT, DPO), the model architecture (MoE, MLA), and the data construction strategies (e.g., voice conversion for Talker adaptation). The use of standard frameworks (Megatron-Core) and public benchmarks enhances reproducibility. However, the specific "JoyAI-LLM Flash" backbone and the internal data mixes are proprietary to JD.com, making exact replication impossible without access to their internal models and data. The description of the Joy-Duplex state machine is clear enough for implementation.
The decoupled architecture introduces latency overhead compared to unified end-to-end models, as noted by the authors. The reliance on a separate 1.7B model for duplex control adds computational complexity. The empathy framework relies on the Thinker's ability to correctly infer attributes via CoT, which can fail if the audio is noisy or ambiguous, potentially leading to hallucinated persona traits. The "cognitive degradation" mitigation is empirical; while results are good, the theoretical guarantee is absent. Furthermore, the system's performance is heavily dependent on the quality of the ASR transcription passed to the Thinker, creating a potential error propagation bottleneck in the decoupled design.
This work contributes to the development of more natural and robust voice assistants, which has significant implications for human-computer interaction, accessibility, and companion AI. The focus on empathy and full-duplex interaction moves the field closer to seamless human-like dialogue. However, the deployment of such systems raises concerns about privacy (continuous audio processing), bias in empathetic responses (stereotyping based on inferred gender/age), and the potential for deceptive anthropomorphism. The open-weight nature of the underlying components (if released) could accelerate research in this area. [One sentence main contribution]. [The paper presents JoyAI-Talker, a modular full-duplex speech dialogue system that integrates a large MoE language model with an expressive speech generator, utilizing a novel state-driven gating mechanism and a persona-adaptive empathy framework to achieve robust, low-latency, and emotionally intelligent voice interactions while preserving the reasoning capabilities of the underlying LLM.]
Automatic Drum Transcription (ADT) is commonly formulated as a direct mapping from a music mixture to symbolic drum events. While effective for transcription, this formulation discards the acoustic stems that are useful for editing, remixing, and production. We revisit an alternative separate-and-detect formulation, where a drum source separation front end first produces five editable drum stems, and a fixed onset detector then converts each stem into symbolic events. The separator is built on a five-stem latent diffusion model that jointly generates kick, snare, toms, hi-hats, and cymbals in a compact VAE latent space. We further study two training-only auxiliary branches--an onset branch (OB) and a timbre branch (TB)--which shape the separator during learning but are discarded at inference. Trained on synthetic drum multitracks and evaluated on MDB Drums and ENST-Drums, the proposed pipeline consistently improves over a strong U-Net-based drum separation baseline in overall transcription F1. It also outperforms a representative end-to-end ADT system on kick and snare F1 under our evaluation protocol, while additionally providing separated audio stems. The ablation results show that OB gives the most stable transcription gains, whereas TB changes the trade-off between reconstruction, acoustic stem quality, and onset detection. These results suggest that generative drum demixing can serve not only as a source separation model, but also as a practical front end for interpretable drum transcription.
Primary: National Taiwan University
All Institutions: National Taiwan University, AI Center of Research Excellence (AI-CoRE), Data Science Degree Program, Institute of Information Science, Academia Sinica, Rhythm Culture Corporation
The paper presents a novel "separate-and-detect" framework for automatic drum transcription that leverages a latent diffusion model with training-only auxiliary branches to optimize for both audio separation quality and transcription accuracy, demonstrating significant improvements over existing baselines on standard datasets.
The paper proposes a "separate-and-detect" pipeline for Automatic Drum Transcription (ADT). Instead of direct event prediction, it uses a five-stem latent diffusion model (based on MSG-LD) to separate drum stems (kick, snare, toms, hi-hats, cymbals) from a drum-dominant mixture, followed by a fixed onset detector. The core methodological contribution is the introduction of two training-only auxiliary branches: an Onset Branch (OB) that encourages sparse percussive structure, and a Timbre Branch (TB) that regularizes acoustic features using DrumGAN descriptors. This design aims to optimize the generative model not just for audio reconstruction, but for downstream transcription utility. The approach is technically sound, leveraging the strengths of diffusion models for high-fidelity separation while addressing the specific needs of rhythmic transcription through auxiliary supervision.
The evaluation is conducted on MDB Drums and ENST-Drums datasets. The authors compare their method against a strong U-Net baseline (LarsNet) and an end-to-end ADT system (ADTOF). Results show that the proposed method, particularly with the +OB auxiliary branch, significantly improves overall transcription F1 over LarsNet and outperforms ADTOF on kick and snare classes. The ablation study effectively isolates the impact of the auxiliary branches, showing that OB provides stable transcription gains, while TB influences the trade-off between reconstruction quality and timbral accuracy. The use of FrĂŠchet Audio Distance (FAD) and mel-MSE provides a comprehensive view of separation quality, complementing the transcription metrics.
The paper provides clear descriptions of the dataset preprocessing, model architecture modifications, and training details. The code and demo are publicly available on GitHub and a dedicated website, which significantly enhances reproducibility. The use of standard datasets (MDB, ENST) and established baselines (ADTOF, LarsNet) allows for fair comparison. The specific hyperparameters for the onset detector tuning are mentioned, though the exact random seeds are not explicitly detailed in the text, the public code likely contains them.
The authors acknowledge several limitations. The system operates at 16 kHz, which limits high-frequency content recovery, affecting hi-hats and cymbals. The iterative nature of diffusion sampling makes inference slower (approx. 25 seconds for 10 seconds of audio) compared to single-pass models. The method struggles with dense, broadband textures like cymbals due to inter-class confusion and VAE latent compression. Additionally, the reliance on a fixed upstream source separation model (Demucs) means errors in the initial drum extraction cannot be corrected by the downstream components.
This work contributes to the field of audio signal processing and music information retrieval by demonstrating that generative models can be optimized for symbolic transcription tasks. The ability to provide both editable stems and symbolic events is valuable for music production, education, and analysis. The "separate-and-detect" paradigm offers a new perspective on ADT, potentially inspiring similar hybrid approaches in other domains where both audio fidelity and symbolic interpretation are required. The paper presents a novel "separate-and-detect" framework for automatic drum transcription that leverages a latent diffusion model with training-only auxiliary branches to optimize for both audio separation quality and transcription accuracy, demonstrating significant improvements over existing baselines on standard datasets.
In this paper, we introduce noise-aware self-supervised learning (NA-SSL) models for noise-aware anomalous sound detection (NA-ASD). NA-ASD is an ASD task with two-channel audio recordings, where one microphone is located close to the target machine and the other is located farther away to capture noise. For this task, we simulate two-channel recordings using diverse audio datasets and train NA-SSL models to extract clean SSL representations of the close-microphone signal by using the far-microphone recording dominated by background noise as auxiliary information. The NA-SSL models are then used as frontends in the standard ASD framework. Our experimental evaluation on the DCASE 2026 Challenge Task 2 development dataset demonstrates the effectiveness of the NA-SSL framework across three base SSL models (BEATs, EAT, and Dasheng), both with and without discriminative fine-tuning. Furthermore, the challenge results proved the effectiveness of the proposed approach, where the NA-BEATs system won the challenge by a large margin, achieving an official score of 70.24%, while the second-place system achieved 65.46%.
Primary: Mitsubishi Electric Research Laboratories (MERL)
All Institutions: Mitsubishi Electric Research Laboratories (MERL), Nagoya University, Information Technology R&D Center, Mitsubishi Electric Corporation
The paper presents a robust and effective Noise-Aware Self-Supervised Learning framework that successfully leverages auxiliary noise information to improve Anomalous Sound Detection, achieving first place in the DCASE 2026 Challenge.
The paper proposes a Noise-Aware Self-Supervised Learning (NA-SSL) framework tailored for the Noise-Aware Anomalous Sound Detection (NA-ASD) task. The core methodology involves adapting existing Self-Supervised Learning (SSL) models (BEATs, EAT, Dasheng) by inserting trainable Noise-Aware (NA) layers that utilize cross-attention to condition the representation of the close-microphone signal on the far-microphone noise signal. This approach leverages the physical setup of the NA-ASD task, where a distant microphone captures ambient noise. The authors simulate two-channel recordings using diverse datasets (FSD50K, WHAM!, DEMAND, QUT-NOISE) to pre-train these NA-SSL models. The method is then integrated into a standard ASD pipeline with discriminative fine-tuning and various backend aggregation techniques (BEAM, RDP). While the application of cross-attention for noise conditioning is not entirely new in speech processing, its systematic adaptation and evaluation across multiple modern, large-scale audio SSL backbones for the specific NA-ASD challenge context represents a solid engineering contribution. The novelty lies in the specific integration strategy and the demonstration of its efficacy across different architectural families (masked prediction, masked bootstrapping, masked autoencoder).
The experimental evaluation is comprehensive and rigorous, centered on the DCASE 2026 Challenge Task 2. The authors evaluate their method on the development set with detailed ablation studies across different base models, fine-tuning strategies (with/without discriminative fine-tuning), and backend techniques. The results show consistent improvements over the original SSL models. Crucially, the paper reports the official challenge results, where the proposed NA-BEATs system won the challenge with a significant margin (70.24% vs 65.46% for second place). This real-world validation on a blind test set provides strong evidence of the method's effectiveness and robustness. The inclusion of variance analysis and ensemble results further strengthens the empirical claims. The comparison against other top-performing systems highlights the efficiency of the proposed approach, which achieves state-of-the-art performance using a single model rather than large ensembles.
The paper provides substantial implementation details, including the architecture of the NA layers (number of heads, hidden size), training hyperparameters (optimizer, learning rate, batch size, EMA decay), data simulation parameters (room impulse response settings, SNR ranges), and backend configurations. The use of standard libraries (Pyroomacoustics) and well-known base models facilitates reproduction. However, the specific random seeds for the "three trials" are mentioned but not explicitly listed in the text, which might introduce slight variability in exact replication. The code for the NA layers is not explicitly linked, though the architecture is described in sufficient detail for implementation.
The primary limitation is the reliance on simulated data for pre-training the NA-SSL models. While the simulation uses diverse datasets, the gap between simulated room acoustics and real-world factory environments may still exist. The method is specifically designed for the NA-ASD task with two-microphone setups; its applicability to single-microphone scenarios or other noise-aware tasks is not explored. Additionally, the performance gain, while significant, comes with the computational overhead of the additional NA layers and the need for the far-microphone signal, which may not always be available or feasible to deploy in all industrial settings. The paper does not extensively discuss the failure cases or scenarios where the far-microphone signal might be corrupted or uninformative.
This work contributes to the field of industrial anomaly detection, which has significant implications for predictive maintenance, safety, and efficiency in manufacturing. By improving the robustness of ASD systems in noisy environments, it enables more reliable deployment of automated monitoring systems. The noise-aware SSL framework could potentially be adapted to other audio processing tasks requiring robustness to environmental noise, such as speech recognition or audio event detection in challenging acoustic conditions. The paper presents a robust and effective Noise-Aware Self-Supervised Learning framework that successfully leverages auxiliary noise information to improve Anomalous Sound Detection, achieving first place in the DCASE 2026 Challenge.
Bandwidth extension (BWE) aims to recover missing high-frequency content from band-limited speech. Existing methods often formulate BWE as a fixed or predefined bandwidth conversion problem, potentially requiring cutoff-specific models or retraining when the input bandwidth changes. This assumption limits their applicability to practical scenarios where speech may arrive with diverse cutoff frequencies. We propose AnyBand, a unified BWE framework that recasts bandwidth extension as in-context spectral infilling. Motivated by prompt-based zero-shot speech generation, AnyBand conditions high-frequency generation on the observed low-frequency spectrum, using the available band as a frequency-domain prompt that conveys content, speaker, prosodic, and spectral-envelope cues. This formulation enables a single model to perform cutoff-conditioned generation over a continuous range of input bandwidths. AnyBand is trained with missing-band conditional flow matching and an Easy-to-Balanced cutoff curriculum over continuously sampled cutoff frequencies. To better exploit the spectral prompt, we introduce a frequency-aware Diffusion Transformer that models cross-frequency interactions and long-range temporal dependencies, followed by a physically motivated multi-view adversarial refinement stage to enhance spectral realism, envelope coherence, and harmonic consistency. Experiments on multiple datasets and bandwidth settings show that AnyBand consistently improves spectral reconstruction over existing baselines while achieving competitive perceptual quality across both standard and irregular input cutoffs. Audio samples are available.
Primary: National University of Singapore
All Institutions: National University of Singapore, Unknown
[One sentence main contribution]. AnyBand introduces a unified, in-context spectral infilling framework for speech bandwidth extension using a frequency-aware Diffusion Transformer and a novel cutoff curriculum. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a well-motivated and technically sophisticated approach to a long-standing problem in audio processing. By reframing BWE as a conditional generation task conditioned on a spectral prompt, it elegantly handles continuous bandwidth variations. The integration of frequency-aware modeling within a DiT architecture is a strong technical contribution that addresses the specific inductive biases required for spectral data. While the novelty is incremental in the context of the rapid evolution of diffusion models, the specific application to unified BWE with the proposed curriculum and adversarial refinement is valuable. The work is significant for its practical utility in creating flexible, single-model solutions for diverse bandwidth scenarios, potentially simplifying deployment in real-world applications.
The paper proposes AnyBand, a unified framework for speech bandwidth extension (BWE) that treats the task as in-context spectral infilling. The core innovation lies in conditioning a Diffusion Transformer (DiT) on the observed low-frequency spectrum as a "frequency-domain prompt" to generate the missing high-frequency content. The architecture augments a temporal DiT backbone with explicit frequency encoders/decoders to model cross-frequency interactions. The training strategy employs a novel "Easy-to-Balanced" cutoff curriculum to handle continuously varying cutoff frequencies, avoiding the need for separate models per bandwidth. Additionally, a multi-view adversarial refinement stage is introduced to enhance spectral realism, envelope coherence, and harmonic consistency. The approach is technically sound, leveraging recent advances in flow matching and diffusion transformers, and applies the concept of in-context learning (typically seen in LLMs) to the spectral domain.
The authors evaluate AnyBand on VCTK and EARS datasets, comparing against baselines like NU-Wave 2, AudioSR, FLowHigh, and Fre-Painter. Metrics include LSD, LF-LSD, HF-LSD, NISQA, COL, and STOI. The results show consistent improvements in spectral reconstruction (lower LSD) and competitive perceptual quality (NISQA) across various input cutoffs (2kHz, 4kHz, etc.). The experiments are comprehensive, covering both in-domain and out-of-domain settings. However, the reliance on objective metrics like NISQA and LSD, while standard, lacks extensive subjective human evaluation (MOS/MUSHRA) which is crucial for perceptual quality claims in audio restoration. The comparison is fair, but the marginal gains in some perceptual metrics over strong baselines like AudioSR warrant caution in claiming a transformative leap.
The paper provides detailed architectural descriptions, loss functions, and training procedures. The inclusion of a demo page suggests code or at least pre-trained models might be available, though no GitHub URL is explicitly provided in the text. The "Easy-to-Balanced" curriculum and specific discriminator designs are described with sufficient mathematical detail for reproduction. The use of standard solvers (Heun) and common datasets enhances reproducibility.
The method assumes the input low-frequency spectrum is clean and correctly aligned; it does not address blind BWE where the cutoff frequency is unknown or the input is degraded by noise/compression beyond simple band-limiting (though BABE is cited as related work, AnyBand itself requires the mask/cutoff extent). The computational cost of the DiT backbone and adversarial refinement may be higher than lightweight CNN-based BWE systems. The paper does not provide extensive ablation studies on the contribution of each discriminator component or the specific impact of the frequency encoder/decoder versus a standard DiT.
AnyBand has significant potential for improving speech quality in legacy communication systems, audio restoration, and voice conversion where bandwidth constraints are dynamic. By unifying multiple bandwidth settings into a single model, it reduces the engineering overhead of deploying multiple specialized models. However, like all generative audio tools, it carries risks of misuse in deepfake generation or misrepresentation, although BWE is generally considered a benign enhancement task. [One sentence main contribution]. AnyBand introduces a unified, in-context spectral infilling framework for speech bandwidth extension using a frequency-aware Diffusion Transformer and a novel cutoff curriculum. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a well-motivated and technically sophisticated approach to a long-standing problem in audio processing. By reframing BWE as a conditional generation task conditioned on a spectral prompt, it elegantly handles continuous bandwidth variations. The integration of frequency-aware modeling within a DiT architecture is a strong technical contribution that addresses the specific inductive biases required for spectral data. While the novelty is incremental in the context of the rapid evolution of diffusion models, the specific application to unified BWE with the proposed curriculum and adversarial refinement is valuable. The work is significant for its practical utility in creating flexible, single-model solutions for diverse bandwidth scenarios, potentially simplifying deployment in real-world applications.
Audio-encoder-LLM-decoder architectures have become the dominant paradigm for modern automatic speech recognition (ASR), improving transcription quality through large-scale language modeling. However, the cost of autoregressive decoding scales with decoder size, creating a fundamental trade-off between recognition quality and serving latency. We argue this trade-off is not inherent: unlike open-ended text generation, ASR outputs are strongly anchored to the input speech signal, providing a natural inductive bias toward high-parallelism decoding. Building on this, we introduce ParaASR, an ASR system that leverages Multi-Token Prediction (MTP) to let a 4B LLM decoder emit multiple tokens per forward step. Starting from a publicly available audio-language foundation, the model first establishes a robust autoregressive recognizer and then aligns five future-token branches through a staged optimization recipe. At inference, it proposes a six-token continuation per step and admits only the verified prefix into the transcript, preserving the safety of standard autoregressive decoding. The average accepted length reaches 5.0 out of 6 proposed tokens, confirming that the deterministic structure of speech makes ASR an especially natural setting for multi-token decoding. ParaASR further retains a native 32K-context window and transcribes up to 30 minutes of audio in a single pass. Across diverse benchmarks, it attains average error rates of 2.97%, 3.68%, and 3.70% on Chinese, English, and long-form evaluations, respectively, while reaching a real-time factor (RTF) as low as 0.0053. These results show that decoder scaling, low-latency inference, and long-context transcription need not be competing goals when future-token proposals are anchored by the acoustic signal and guarded by autoregressive verification.
Primary: StepFun
All Institutions: StepFun, NTU, PKU, UNSW, SJTU, USTC
ParaASR effectively adapts multi-token prediction to ASR, achieving a compelling balance between state-of-the-art accuracy and low-latency inference, thereby challenging the perceived trade-off between model scale and serving efficiency in LLM-based speech recognition.
The paper proposes ParaASR, an LLM-based Automatic Speech Recognition (ASR) system that integrates Multi-Token Prediction (MTP) to accelerate inference. The core methodological contribution is the application of speculative decoding principles (specifically MTP) to the ASR domain, leveraging the deterministic nature of speech-to-text mapping to propose multiple future tokens per forward pass. The architecture consists of a frozen audio encoder, a linear adapter, and a 4B LLM decoder with five auxiliary MTP branches. The training recipe is staged: audio-language pretraining, ASR supervised fine-tuning (SFT), and finally MTP alignment. The MTP training involves a frozen-branch alignment stage followed by joint calibration. The approach is technically sound and logically motivated; the insight that ASR outputs are more predictable than open-ended text generation is valid and well-supported by the high acceptance rates reported. However, the specific architectural modifications (adding MTP heads) are incremental adaptations of existing speculative decoding techniques rather than a fundamentally new algorithmic breakthrough.
The evaluation covers a wide range of benchmarks including Chinese (AISHELL-1, AISHELL-2, WenetSpeech), English (LibriSpeech, Common Voice, VoxPopuli, Earnings22), and long-form ASR tasks. The results demonstrate state-of-the-art accuracy for an LLM-based ASR system, with low CER/WER scores. Crucially, the paper reports a Real-Time Factor (RTF) of 0.0053 on a single H800 GPU, which is a significant efficiency improvement over standard autoregressive decoding. The ablation studies confirm that MTP does not degrade accuracy and that the staged training is effective. The comparison against baselines like VibeVoice-ASR, FunASR-Nano, and Qwen3-ASR-1.7B is appropriate. The long-form evaluation using a constructed Wenet testnet set adds value, though the construction method (concatenating segments) is a simplification of true long-form challenges.
The paper provides detailed descriptions of the training stages, data mixture (100K hours short-form, 50K hours long-form pseudo-labeled), and hyperparameters (learning rates, batch sizes). However, the use of proprietary datasets and the "publicly available audio-language foundation" (likely StepAudio, which is not fully open-weight or documented in sufficient detail for exact reproduction) limits reproducibility. The specific details of the long-form data curation pipeline (VAD, ROVER, LLM refinement) are described but the code for this pipeline is not provided. The model weights are not mentioned as open-sourced.
The paper does not extensively discuss the computational overhead of the MTP branches during training or the memory footprint. The reliance on pseudo-labeled data for long-form training introduces potential error propagation, although the filtering mechanism ($e > 0.05$) mitigates this. The performance on code-switching and highly noisy environments is mentioned but specific metrics are not detailed in the abstract/text provided. The generalization of the high acceptance rate (5.0/6) to out-of-distribution domains (e.g., heavy accents, different languages not seen in training) is not fully explored.
This work demonstrates that large language models can be efficiently deployed for real-time speech recognition, potentially lowering the barrier for high-quality ASR services. It highlights the synergy between acoustic grounding and language modeling efficiency. The ability to transcribe long audio in a single pass has significant implications for meeting transcription, broadcast monitoring, and archival processing. ParaASR effectively adapts multi-token prediction to ASR, achieving a compelling balance between state-of-the-art accuracy and low-latency inference, thereby challenging the perceived trade-off between model scale and serving efficiency in LLM-based speech recognition.
Balancing sequence length, representational capacity, and long-horizon stability is a central problem in autoregressive (AR) speech and audio generation. Representations with higher frame rates or greater capacity can preserve more signal detail, but they also make streaming generation more vulnerable to distribution drift and AR error accumulation. Conversely, shorter and more compressed representations simplify AR modeling, but their limited bandwidth may discard important components and constrain the upper bound of reconstruction fidelity and generation quality. We ask whether a low-frame-rate, high-dimensional, high-bandwidth continuous representation can be co-designed with a streaming generation framework to support robust high-fidelity reconstruction, strong single-token predictability, and superior long-horizon stability. We decompose this goal into two coupled problems: what geometric and statistical properties a high-dimensional representation space should have, and how an AR continuous-token generator should be structured to resist error accumulation. Accordingly, we propose Locodec, a locally encoded codec that shapes its representation space to improve the interpolatability of a lower-dimensional core manifold and the identifiability of the native high-dimensional coordinates, thereby improving the predictability of high-dimensional high-bandwidth tokens. We also propose MP-ELD, a single-token AR flow-matching framework that uses multi-path information routing and residual classifier-free guidance to mitigate error accumulation. Experiments with 8-Hz, 768-dimensional tokens show that our design preserves reconstruction quality, improves single-token predictability, achieves competitive WER, and maintains stable long-form synthesis, without using external SSL/ASR models, pretrained text language models, or post-training stages.
Primary: ByteDance Seed
All Institutions: ByteDance Seed
The paper presents a theoretically grounded and methodologically novel approach to stable autoregressive speech generation by co-designing a spherical, manifold-constrained tokenizer with a multi-path flow-matching generator. The geometric justification for spherical tokens and the specific mechanisms for inducing interpolatability and identifiability are strong contributions, although the empirical validation appears limited by the lack of detailed comparative metrics in the provided text.
The paper proposes a joint design for a tokenizer (Locodec) and an autoregressive generative model (MP-ELD) aimed at solving the trade-off between high-fidelity reconstruction and long-horizon stability in speech generation. The core technical contributions are: 1) A spherical continuous tokenization scheme where the latent space is shaped by a lower-dimensional interpolatable manifold via orthogonal projection/lifting, combined with a "postfix dimension dropout" mechanism to induce an energy hierarchy (identifiability) among token dimensions. 2) A multi-path encoder-LM-decoder framework for AR flow matching that uses path dropout to separate acoustic and alignment information, allowing for independent classifier-free guidance (CFG) scales to mitigate error accumulation. The theoretical derivation regarding spherical cap covering and the equivalence of x-pred and v-pred in spherical flow matching is mathematically sound and provides a strong geometric justification for the design choices. The approach is novel in its specific combination of spherical geometry constraints, manifold shaping via orthogonal projections, and multi-path guidance for stability.
The paper reports experiments on the Seed-TTS-eval dataset, claiming competitive Word Error Rate (WER) and stable long-form synthesis without external SSL/ASR models or post-training. However, the provided text is truncated before the full experimental results section. Based on the abstract and introduction, the evaluation focuses on WER, reconstruction quality, and stability. The claim of "competitive WER" is significant, but without specific numbers or comparison to state-of-the-art baselines (like VALL-E, AudioLM, or recent flow-matching speech models), the strength of the empirical claim is hard to fully assess. The methodology suggests a strong theoretical basis, but the experimental validation appears to rely on standard metrics which may not fully capture the "stability" improvements claimed (e.g., drift in timbre/loudness over long horizons).
The paper provides detailed mathematical formulations for the spherical corruption, orthogonal projection/lifting, and the flow matching bridge. The description of the "postfix dimension dropout" and the multi-path routing is specific. However, the truncation of the text means the full architectural details of the MP-ELD model and the exact hyperparameters for the training objectives are not fully visible. Assuming the full paper contains these details, reproducibility seems high due to the explicit mathematical definitions. The lack of code release (implied by "none" for project URL) is a barrier.
The primary limitation is the reliance on a specific geometric structure (spherical manifold) which may not generalize to all audio types (e.g., complex music with non-stationary characteristics). The "postfix dimension dropout" introduces an arbitrary ordering of dimensions, which might be suboptimal if the learned energy hierarchy does not align with semantic importance. The claim of stability without external semantic guidance is ambitious; semantic drift is a known issue in purely acoustic autoregressive models, and the paper's mitigation via multi-path CFG may not fully resolve it for complex linguistic content. The truncation of the text prevents a full assessment of the ablation studies and failure cases.
This work contributes to the field of efficient and stable audio generation, potentially enabling more robust streaming speech systems. By reducing reliance on large external SSL models for tokenization, it may lower the barrier to entry for high-quality speech generation. The geometric insights into spherical latent spaces could influence future work in continuous tokenization for other modalities. The paper presents a theoretically grounded and methodologically novel approach to stable autoregressive speech generation by co-designing a spherical, manifold-constrained tokenizer with a multi-path flow-matching generator. The geometric justification for spherical tokens and the specific mechanisms for inducing interpolatability and identifiability are strong contributions, although the empirical validation appears limited by the lack of detailed comparative metrics in the provided text.
Spoken dialog systems are typically designed for clean, dyadic interactions in which a single user and an assistant take turns speaking. Real-world social conversations, however, are often more ambiguous: multiple speakers may participate in the same conversation amid irrelevant speech and background noise. Each utterance may be directed to the assistant, addressed to another speaker, or completely irrelevant. In such settings, the assistant must decide not only what to say, but also whether to speak at all. In this paper, we introduce Cocktail-Talker, a speech LLM framework for multi-speaker spoken dialog modeling in noisy social environments. We model the assistant's behavior with three action tokens: <|respond|>, <|listen|>, and <|ignore|>, placed before a response or silence. Cocktail-Talker is trained via supervised finetuning and reinforcement learning to generate the appropriate action token and, only in <|respond|> mode, a speech response. To prepare the training data, we develop Cocktail-DialogGen, an LLM-based data pipeline that simulates realistic multi-speaker dialogs with speaker roles across diverse social settings. Together, these components take a step toward spoken dialog systems that interact more naturally and selectively in complex social environments.
Primary: Columbia University
All Institutions: Columbia University, University of Illinois Urbana-Champaign
Cocktail-Talker introduces a novel framework for multi-speaker spoken dialog by modeling turn-taking as a discrete action selection problem, trained with GRPO on synthetically generated data, achieving state-of-the-art performance in noisy, multi-speaker environments. The paper makes a significant technical contribution by addressing the under-explored "speaking side" of multi-speaker conversation, moving beyond simple dyadic interactions to more complex social auditory scenes.
The paper proposes "Cocktail-Talker," a framework for multi-speaker spoken dialog in noisy environments. The core methodological contribution is the introduction of three discrete action tokens (`<|respond|>`, `<|listen|>`, `<|ignore|>`) to model turn-taking behavior, trained via Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO). The approach leverages a pre-existing speech LLM backbone (Qwen2.5-Omni) and freezes the audio encoder and TTS decoder, focusing training on the language model head and LoRA adapters. A significant portion of the work involves "Cocktail-DialogGen," an LLM-based pipeline to synthesize training data by simulating multi-speaker conversations with varying acoustic conditions (SNR, background noise) and semantic constraints (anonymous vs. named speakers). The use of GRPO with a reward function based on action accuracy and format integrity is a novel application of RLHF techniques to the specific problem of turn-taking decision-making in speech LLMs.
The evaluation is comprehensive, covering both seen and unseen environments. The paper compares Cocktail-Talker against several strong baselines, including Moshi, PersonaPlex, Step-Audio2, Kimi-Audio, Qwen2.5-Omni, and Qwen3-Omni. The metrics include binary response accuracy (Respond/Silent) and response quality (METEOR, ROUGE-L, BERTScore, SentCos). The results show that Cocktail-Talker significantly outperforms baselines in decision accuracy (macro F1 ~0.93) and response quality. The analysis breaks down performance by SNR, number of speakers, anonymity, and context length, providing robust insights into model behavior. The inclusion of "unseen environments" tests generalization, which is crucial for real-world applicability. The ablation studies (SFT vs. SFT+GRPO) demonstrate the benefit of the reinforcement learning stage.
The paper provides a GitHub link to the code. The data generation pipeline is described in detail, including the use of specific LLMs (Gemini 3 Pro, Qwen3-TTS) and noise datasets (DEMAND, Freesound). However, the reliance on proprietary models (Gemini, Qwen) for data generation and the specific random seeds or exact prompts used in the LLM-based data synthesis might pose reproducibility challenges for other researchers. The hyperparameters for SFT and GRPO are provided, which aids in reproducing the model training.
The paper acknowledges several limitations. The system is not streaming, which limits its real-time applicability. It assumes fixed turn boundaries in the simulation, which may not reflect the fluid nature of real-world interruptions. The model relies heavily on the simulated data, and while it generalizes to unseen environments, the distribution shift between synthetic and real-world cocktail party scenarios remains a potential issue. The performance degrades with increased speaker count and anonymity, indicating room for improvement in complex social reasoning.
This work contributes to the development of more natural and robust speech assistants that can operate in real-world social settings. By enabling selective participation, it reduces the annoyance of assistants interrupting irrelevant conversations or responding to noise. This has implications for the deployment of AI agents in homes, public spaces, and social robotics. However, the potential for misuse in surveillance or eavesdropping scenarios should be considered, although the model is designed to ignore irrelevant speech. Cocktail-Talker introduces a novel framework for multi-speaker spoken dialog by modeling turn-taking as a discrete action selection problem, trained with GRPO on synthetically generated data, achieving state-of-the-art performance in noisy, multi-speaker environments. The paper makes a significant technical contribution by addressing the under-explored "speaking side" of multi-speaker conversation, moving beyond simple dyadic interactions to more complex social auditory scenes.
Audio-capable foundation models enable end-to-end spoken interaction, but they also introduce safety risks beyond transcript content. It remains unclear how much jailbreak capability can arise from matched-text variation in speech delivery rather than from lexical rewriting or broader style transfer. We study this question by holding transcript content fixed and varying six speech-delivery presets whose acoustic attributes may co-vary. We present PJ-Break, a black-box evaluation protocol with presets targeting arousal, authority, and speaking rate, together with AdvAudio-Prosody, a 600-sample benchmark with acoustically verified attributes. On the exact post-QC Qwen2-Audio panel, the Q=1 Panic (38/95), Anger (35/95), and Fast (32/95) presets are all well above Neutral (4/95). The fixed six-query pool covers 44/95 Qwen2-Audio seeds and 15/95 GPT-4o seeds and exceeds a matched-budget StyleBreak reimplementation (27/95) on Qwen2-Audio. A same-voice pool excluding the confounded Commanding condition still reaches 40/95, and a retained-panel ablation shows emotional-delivery audio alone (44/95) is far more effective than emotional text alone (11/95). Exploratory surrogate diagnostics and pilot mitigation observations are secondary, non-core analyses. Overall, matched-text speech delivery should be treated as a first-class factor in Audio LLM safety evaluation
Primary: City University of Hong Kong
All Institutions: City University of Hong Kong, City University of Hong Kong (Dongguan)
The paper makes a valuable contribution to multimodal safety by empirically demonstrating that prosodic variations in speech delivery can significantly compromise the safety of Audio LLMs, independent of lexical content. Its controlled methodology and clear empirical findings provide a necessary foundation for future research into robust, prosody-aware safety mechanisms for spoken AI assistants.
The paper proposes a controlled experimental framework, PJ-Break, to isolate the impact of prosodic variations (arousal, authority, speaking rate) on the safety/jailbreak capabilities of Audio Large Language Models (Audio LLMs). By holding the lexical transcript constant and varying only the speech delivery via TTS presets, the authors address a specific and under-explored gap in multimodal safety: the influence of paralinguistic cues. The methodology is rigorous in its control of variables, utilizing a fixed TTS stack and explicit acoustic verification. The inclusion of an ablation study comparing emotional text vs. emotional audio is a strong methodological choice that directly supports the core hypothesis. The use of surrogate model diagnostics (activation patching) to hypothesize mechanisms, while clearly labeled as exploratory, adds depth to the analysis.
The evaluation is conducted on Qwen2-Audio and GPT-4o, using a benchmark of 600 samples (AdvAudio-Prosody). The results are compelling: prosodic jailbreaks significantly outperform neutral audio and even matched-budget style-transfer baselines (StyleBreak). The finding that "Panic" and "Anger" presets achieve high success rates (38/95 and 35/95 respectively) compared to Neutral (4/95) is a significant empirical contribution. The same-voice sensitivity analysis strengthens the claim that the effect is due to prosody rather than voice identity. However, the reliance on TTS-generated speech limits the ecological validity, although the authors do provide small-scale human-speech and OTA checks to mitigate this concern. The statistical significance (McNemar test) adds robustness to the main comparison.
The paper provides detailed implementation specifications, including the TTS engine (Azure Neural TTS), specific voices used, and the definition of the six presets. The acoustic verification metrics ($F_0$, RMS, etc.) are standard and reproducible. However, the dataset and code are explicitly *not* released due to safety concerns regarding the generation of harmful audio jailbreaks. This is a valid ethical stance for this domain but limits independent verification and extension. The description of the evaluation protocol and judge ensemble is sufficiently detailed for a competent researcher to replicate the core experiments, assuming access to the same models and TTS services.
The primary limitation is the reliance on TTS, which may not fully capture the nuances of human emotional speech or the specific acoustic artifacts that might arise in real-world over-the-air attacks. The "Commanding" condition is partially confounded by voice identity (different speaker), which the authors acknowledge but does not fully eliminate the confound. The study is limited to English and a single TTS provider, limiting generalizability to other languages and synthesis technologies. Furthermore, the surrogate analysis is restricted to one model (Qwen2-Audio), and the mechanistic insights (refusal direction) are hypothesis-generating rather than definitive proof of causality in closed-source models. The pilot mitigation results are preliminary and not robust.
This paper has significant implications for the safety evaluation of multimodal AI systems. It demonstrates that current safety filters, which often focus on text or content, may be bypassed by manipulating the *delivery* of the content. This necessitates a shift in safety research to include prosodic and paralinguistic analysis in red-teaming protocols. It also highlights potential vulnerabilities in crisis-response or assistance-oriented models that might be triggered by high-arousal speech. The work encourages the development of prosody-aware safety mechanisms and more robust multimodal safety benchmarks. The paper makes a valuable contribution to multimodal safety by empirically demonstrating that prosodic variations in speech delivery can significantly compromise the safety of Audio LLMs, independent of lexical content. Its controlled methodology and clear empirical findings provide a necessary foundation for future research into robust, prosody-aware safety mechanisms for spoken AI assistants.
Existing single-domain and multi-task audio systems remain limited in directly organizing speech, music, sound effects, ambience, and multiple roles into long-form temporal scenes. We present Qwen-Audio-3.0-Gen-Preview, a unified non-autoregressive framework that uses a Diffusion Transformer (DiT) and a shared variational autoencoder (VAE) to generate the complete mixed waveform. Prompt enhancement converts free-form requests into structured temporal records that are rendered as textual conditions, while a two-stage data curriculum and semantic conditional views train the proposed model to use these conditions across domains. A shared continuous VAE compresses 48kHz stereo waveforms into 25Hz latent sequences and incorporates semantic supervision, providing one representation for speech, music, sound effects, and their mixtures. On Seed-TTS-Eval, speaker similarity is the proposed model's clearest strength across all three subsets, and on the multi-speaker benchmark, the proposed model shows higher cross-turn consistency than Seed-Audio-1.0 in both languages. On AudioCaps, its advantages are concentrated in evaluations using large audio-language models and AudioBox. Relative to Seed-Audio-1.0, it achieves stronger temporal localization. Using approximately 10% music data of a dedicated in-house model, the proposed model remains close across all seven SongBench components and leads in three while retaining speech and general-audio capabilities. These results demonstrate the potential of unified generation for temporally structured, multi-domain audio.
Primary: Alibaba Token Foundry
All Institutions: Alibaba Token Foundry
This paper presents a significant engineering advancement in unified audio generation by introducing a structured, temporally-aware conditioning framework and a shared continuous latent space, demonstrating strong capabilities in multi-speaker dialogue and scene-level audio organization while maintaining competitive performance across diverse audio domains.
The paper proposes Qwen-Audio-3.0-Gen-Preview, a unified non-autoregressive (NAR) audio generation framework based on a Diffusion Transformer (DiT) and a shared continuous Variational Autoencoder (VAE). The core technical contribution lies in the data and conditioning pipeline rather than a radical architectural shift. The authors introduce a sophisticated "Prompt Enhancement" module that converts free-form text into structured temporal records (global scene, source profiles, time-localized events). This structured data is then rendered into textual conditions for the DiT. A key methodological component is the "Semantic Conditional View" training strategy, which uses conditional dropout on specific semantic units (e.g., hiding dialogue to force the model to rely on scene context) to improve robustness and controllability. The shared VAE is trained with auxiliary semantic supervision using a frozen LLM, aiming to create a latent space that is both acoustically high-fidelity and semantically rich. While the use of DiTs and continuous latents is established in recent audio generation literature (e.g., AudioGen, MusicGen), the specific integration of hierarchical temporal annotations and the two-stage curriculum for long-form scene generation represents a meaningful engineering contribution to the field of unified audio models.
The evaluation is comprehensive, covering Speech (Seed-TTS-Eval), General Audio (AudioCaps), and Music (SongBench). The paper demonstrates strong performance in speaker similarity and cross-turn consistency for multi-speaker dialogue, which is a significant challenge for unified models. On AudioCaps, the model leads in Large Audio-Language Model (LALM) based evaluations, suggesting better semantic alignment than competitors, although it does not lead in standard CLAP scores. The introduction of an in-house benchmark for temporal localization (mIoU) provides valuable insight into the model's ability to follow timing instructions, showing improvement over Seed-Audio-1.0. The comparison with a dedicated in-house music model highlights the efficiency of the unified approach, achieving competitive results with significantly less music data. However, the reliance on LALMs for evaluation introduces potential bias, as the judges are from the same ecosystem (Qwen/Gemini), and the lack of human MOS scores for the general audio/music tasks limits the perceptual validation.
The paper provides detailed descriptions of the data annotation pipeline, synthetic data construction (using Scaper-like recipes), and the training curriculum. The architecture of the VAE and DiT is described, including the latent dimensions (128-dim, 25Hz) and the semantic supervision objective. However, specific hyperparameters for the DiT training, the exact composition of the 200,000 hours of training data, and the code for the Prompt Enhancement module are not fully open-sourced or detailed to the extent required for exact replication. The use of proprietary in-house models for evaluation (Seed-Audio-1.0 baseline, in-house music model) also complicates direct comparison.
The paper acknowledges limitations regarding the synthetic data, noting residual context in source clips and mismatches in simulated acoustics. The reliance on LLMs for prompt enhancement and evaluation introduces a dependency on the capabilities of those specific models. The model's performance on music composition (melody, structure) lags behind dedicated autoregressive models, indicating that the NAR DiT architecture may struggle with long-range musical dependencies compared to sequential planners. The evaluation metrics, particularly the LALM-based ones, may not fully capture human perceptual quality or nuance.
This work advances the state of unified audio generation, moving towards systems that can produce complex, temporally structured scenes with multiple audio types. This has significant implications for content creation in gaming, film, and virtual reality, where coherent audio scenes are essential. By demonstrating that a single model can handle speech, music, and sound effects with temporal control, it reduces the complexity of audio production pipelines. However, the potential for misuse in generating realistic deepfakes or misleading audio content remains a concern, necessitating robust safety measures. This paper presents a significant engineering advancement in unified audio generation by introducing a structured, temporally-aware conditioning framework and a shared continuous latent space, demonstrating strong capabilities in multi-speaker dialogue and scene-level audio organization while maintaining competitive performance across diverse audio domains.
Cover song generation (CSG) should preserve the melodic and linguistic content of a reference song while recreating the remaining musical components. The state-of-the-art model SongEcho utilizes $F_0$ sequences and voiced/unvoiced (V/UV) tags for conditioning; however, implicit linguistic information from V/UV tags cannot guarantee lyric accuracy, leading to a high phoneme error rate (PER). Inspired by singing voice synthesis (SVS), we propose MPEcho, which integrates a phoneme encoder and a length regulator (LR) into the SongEcho framework. By providing explicit phoneme-level conditioning and precise temporal boundaries, MPEcho significantly reduces PER. To enable this, we developed Phonsa, a Whisper-based automatic transcription model that provides high-precision phoneme-level annotations for singing voices, overcoming the scarcity of high-quality audio-phoneme pairs. Experimental results validate the effectiveness of Phonsa for alignment and MPEcho for end-to-end CSG. The audio samples, code and weights can be accessed from https://lonian6.github.io/MPEcho.github.io/.
Primary: Taiwan AI Labs
All Institutions: Taiwan AI Labs, National Science and Technology Council of Taiwan, Ministry of Education (Taiwan)
MPEcho introduces a novel integration of SVS-style phoneme-level control into end-to-end cover song generation, significantly improving lyrical accuracy while maintaining melodic fidelity. This work demonstrates the value of cross-domain priors (SVS to CSG) and provides a robust framework for controllable music generation, advancing the state-of-the-art in both technical methodology and practical application.
The paper proposes MPEcho, a framework for Controllable Cover Song Generation (CSG) that integrates explicit phoneme-level conditioning into the SongEcho architecture. The core methodological contribution is the adaptation of Singing Voice Synthesis (SVS) techniquesâspecifically a length regulator and phoneme encoderâinto a diffusion-based full-song generation model. This addresses the limitation of previous models (SongEcho) that relied on coarse V/UV tags, leading to high phoneme error rates (PER). The authors also introduce Phonsa, a Whisper-based automatic transcription model optimized for singing voices, to provide the necessary high-precision phoneme-level annotations. The methodology involves a multi-condition guidance strategy using Adaptive Projection Guidance (APG) to disentangle text, lyrics, melody, and phoneme conditions. The approach is technically sound, leveraging established SVS priors to solve a specific controllability issue in music generation. While the integration of SVS components into LTS/CSG is not entirely new in concept, the specific implementation of a length regulator for end-to-end cover generation and the dedicated alignment model (Phonsa) for this domain represent a solid engineering contribution.
The experimental section is comprehensive, including objective metrics (Audiobox, CLAP, RPA, RCA, PER) and subjective evaluations (MOS). The results demonstrate that MPEcho significantly reduces PER compared to SongEcho (0.1865 vs 0.4562) while maintaining or improving musical quality metrics. Ablation studies effectively isolate the contributions of melody conditioning, phoneme conditioning, and the specific guidance strategies. The evaluation of Phonsa against MFA shows substantial improvements in alignment accuracy. However, the reliance on a Chinese-only dataset limits the generalizability of the findings. The objective metrics, particularly CLAP, show low scores across the board, which the authors attribute to domain mismatch, a valid but somewhat concerning observation for a generative model's performance assessment. The subjective evaluation adds valuable human-centric validation, confirming the objective trends.
The paper provides detailed implementation details, including dataset sizes, training hardware, hyperparameters, and model architectures. The code and weights are made available via a GitHub page, which significantly enhances reproducibility. The description of the Phonsa model and the MPEcho architecture is sufficiently detailed for replication. The use of standard datasets (M4Singer, Opencpop, GTsinger) for Phonsa training and a curated internal dataset for MPEcho allows for verification, although the internal dataset's exact composition is less transparent than public benchmarks.
The primary limitation is the restriction to single-singer scenarios and Mandarin language. The model's performance on multi-singer tracks or languages with different phonetic structures is untested. The reliance on Phonsa for alignment means that errors in Phonsa will propagate to MPEcho. Additionally, the internal dataset used for training MPEcho is not publicly available, which hinders direct comparison with other CSG models trained on different data. The low CLAP scores suggest that the generated audio may not always align well with text prompts in a broad semantic sense, although this is common in specialized music generation tasks.
This work contributes to the field of generative AI for music by improving the controllability of cover song generation. Accurate lyric rendering is crucial for user acceptance and creative utility in AI music tools. The development of Phonsa also advances the field of automatic speech recognition and alignment for singing, which has broader applications in music information retrieval and education. However, the ease of generating high-quality covers raises ethical concerns regarding copyright and artist likeness, which should be addressed in future deployments. MPEcho introduces a novel integration of SVS-style phoneme-level control into end-to-end cover song generation, significantly improving lyrical accuracy while maintaining melodic fidelity. This work demonstrates the value of cross-domain priors (SVS to CSG) and provides a robust framework for controllable music generation, advancing the state-of-the-art in both technical methodology and practical application.
The study of Target Speaker Extraction (TSE) aims to isolate a desired speaker from overlapping speech mixture given auxiliary cues. Existing systems are typically designed for specific cue types, limiting flexibility when cue availability varies across scenarios. We present WeSep, a unified framework that reformulates TSE as a heterogeneous cue-conditioned learning problem. In WeSep, cue modules and separator backbones are decoupled through standardized interfaces, enabling configurable cue injection and flexible integration of diverse modalities. The design enables systematic study of cue structure, intra- and cross-modal interaction, and dynamic cue availability within a shared optimization framework, facilitating adaptation to real-world conditions. Experiments across enrollment, spatial, visual, and textual cues reveal modality-dependent characteristics and demonstrate stable optimization under heterogeneous cue availability. The toolkit will be publicly available.
Primary: The Chinese University of Hong Kong, Shenzhen
All Institutions: The Chinese University of Hong Kong, Shenzhen, Nanjing University, Shenzhen Loop Area Institute
The paper presents WeSep, a modular framework for Target Speaker Extraction that standardizes cue injection and enables systematic study of heterogeneous cue availability. While it lacks algorithmic novelty, its contribution to software engineering and experimental standardization in the audio processing community is valuable and practically significant.
The paper proposes WeSep, a modular framework for Target Speaker Extraction (TSE) that decouples cue encoders from separator backbones. The core methodological contribution is architectural rather than algorithmic; it introduces a standardized interface for injecting diverse cues (enrollment, spatial, visual, textual) into a shared optimization framework. This allows for composable multi-cue integration and handles heterogeneous cue availability (missing cues) via zero-padding. While the decoupling strategy is sound and addresses a real pain point in TSE (fragmented codebases for different modalities), the novelty is limited. The concept of modular TSE is not new (e.g., ClearerVoice, various multi-modal baselines), and the paper explicitly states it does not propose a new separation architecture. The "innovation" lies in the software engineering and experimental design rather than novel deep learning mechanisms.
The evaluation is comprehensive in terms of breadth, covering four distinct cue modalities and their combinations on standard datasets (Libri2Mix, VoxCeleb2-mix, etc.). The experiments successfully demonstrate that the framework supports stable training with missing cues and that combining cues (e.g., speaker + spatial) yields performance gains. However, the results are largely confirmatory. The paper shows that existing components work well when plugged into this framework, rather than demonstrating that WeSep introduces a new state-of-the-art method. The performance metrics (SI-SDRi) are standard, and the improvements are attributed to the flexibility of composition rather than a breakthrough in extraction capability. The causal modeling experiment is a practical addition but shows expected degradation in performance for low latency.
The paper provides a GitHub URL for the toolkit, which is a significant positive for reproducibility. The experimental settings are described in reasonable detail, including backbone configurations (BSRNN, NBC2) and training hyperparameters (Adam, 150 epochs). The use of standard datasets enhances the ability of others to replicate the results. The modular design, if implemented cleanly as claimed, should facilitate reproduction by other researchers.
The primary limitation is the lack of algorithmic novelty. The paper admits to not proposing a new separator. The performance gains are marginal and stem from better engineering and composition rather than fundamental improvements in the extraction task. The reliance on zero-padding for missing cues is a simple heuristic that may not be optimal compared to learned masking or attention mechanisms, although it is effective for stability. The paper does not provide a detailed ablation study on the *optimal* fusion strategies beyond what is standard, limiting the theoretical contribution.
This work has positive broader impact by promoting modularity and standardization in the TSE field. By providing an open-source toolkit, it lowers the barrier to entry for researchers working on multi-modal TSE and encourages systematic comparison of cue types. It facilitates research into robust TSE systems that can handle dynamic cue availability, which is crucial for real-world deployment. The paper presents WeSep, a modular framework for Target Speaker Extraction that standardizes cue injection and enables systematic study of heterogeneous cue availability. While it lacks algorithmic novelty, its contribution to software engineering and experimental standardization in the audio processing community is valuable and practically significant.