Speech and audio generation is often needed in animation dubbing, audio drama, movies, advertising, games, podcasts, and short-video production. In these scenarios, creators may need to design voices without reference recordings, control speaker styles with natural language, support acoustic scenes with environments and audio effects, and later reuse the designed voices. Therefore, it is important to support multi-speaker speech and audio generation for both instruct and zero-shot tasks. The instruct task requires a caption of the environment, speaker styles, and fine-grained content, while the zero-shot task uses reference audio together with the same fine-grained content. We address these tasks from both the data and model sides. First, we propose SwanData-Caption, which cleans raw speech and audio data, adds targeted synthetic coverage, and annotates diverse and accurate multi-level captions. Then, we propose SwanTale, a multi-speaker expressive speech and audio generation model that supports both zero-shot and instruct tasks. We introduce SwanVAE to support high-quality multi-audio-modality generation. Then, we adopt reward-conditioned quality control and Engram conditioning, along with Unified MoE for multi-task and multi-audio-modality modeling. In addition, we use curriculum learning and GRPO post-training to let the model progressively learn and strengthen its capabilities. Experimental results show that SwanTale leads on multiple key zero-shot and instruct metrics, achieves the best expressiveness scores in both tasks, and supports complex instruct generation involving multi-speaker speech and audio. Demos can be found at https://swanaigc.github.io/#swantale.
Primary: ByteDance
All Institutions: ByteDance, Zhejiang University
SwanTale presents a robust and unified framework for multi-speaker speech and audio generation, effectively bridging the gap between zero-shot cloning and instruct-based creative control through a sophisticated data pipeline and a novel VAE-MoE architecture, significantly advancing the state-of-the-art in expressive audio synthesis.
The paper proposes SwanTale, a unified model for multi-speaker speech and audio generation supporting both zero-shot (reference-based) and instruct (caption-based) tasks. The methodology is comprehensive, addressing data, model architecture, and training. Key technical contributions include: 1) SwanData-Caption, a pipeline for generating fine-grained multi-level captions using an LLM (Seed2.0 Lite) with a persona library to guide annotation. 2) SwanVAE, a custom VAE with a Gaussian bottleneck and Transformer-based decoder for 48kHz audio, designed to balance reconstruction quality and latent learnability. 3) A flow-based Transformer prior with Unified MoE (Mixture of Experts) to handle multiple tasks and audio modalities (speech, environment, effects, singing). 4) Reward-conditioned quality control and GRPO (Group Relative Policy Optimization) post-training to improve pronunciation, stability, and caption adherence. The approach to unify zero-shot and instruct generation in a single model with complex audio scene modeling is technically sound and represents a significant engineering effort. The use of Engram conditioning and curriculum learning is also noted.
The evaluation covers zero-shot TTS, instruct TTS, and acoustic quality. Benchmarks used include SwanBench-Speech, InstructTTSEval, SwanBench-Scene, and SwanBench-Caption. The paper claims leadership on multiple key metrics. The data pipeline includes rigorous filtering (DNSMOS, SQUIM) and human auditing (best-worst comparison for expressiveness). The inclusion of synthetic data for underrepresented demographics (elderly) and challenging pronunciations is a strong point for robustness. The evaluation of multi-speaker dialogue and complex instruct generation adds depth beyond standard single-speaker TTS.
The paper provides detailed descriptions of the data pipeline, model architecture (SwanVAE, Unified MoE), and training procedures (curriculum, GRPO). However, the code is not publicly available (Project URL is none), and some components (e.g., Seed2.0 Lite, SwanAligner, SwanVerifier) are referenced as internal or proprietary tools. While the methodology is described well, full reproducibility might be challenging without access to the specific internal models and the full dataset (70M records).
The reliance on internal data and proprietary models (Seed2.0, SwanAligner) limits immediate reproducibility. The paper mentions "SwanBench" which appears to be a new benchmark, potentially lacking the long-term community validation of established benchmarks like LibriSpeech or VCTK. The complexity of the model (Unified MoE, GRPO) might make inference and fine-tuning resource-intensive. The "Engram conditioning" is mentioned but not fully detailed in the provided text snippet, which might obscure its specific novelty or implementation details.
This work has significant potential for creative industries (animation, gaming, film) by enabling more expressive and controllable audio generation. The ability to generate complex audio scenes with speech and effects from natural language instructions is a powerful tool for content creation. However, the ease of voice cloning and impersonation raises ethical concerns regarding deepfakes and misinformation, which the authors should address in terms of safety guardrails. SwanTale presents a robust and unified framework for multi-speaker speech and audio generation, effectively bridging the gap between zero-shot cloning and instruct-based creative control through a sophisticated data pipeline and a novel VAE-MoE architecture, significantly advancing the state-of-the-art in expressive audio synthesis.
Audio reasoning is essential for machine understanding of the acoustic world. Reinforcement learning with verifiable rewards can elicit such reasoning, yet existing reward designs are complementary in their limitations: outcome-based rewards supervise only the final answer and let the model reach it without attending to the audio, whereas process-based rewards score the reasoning itself but rely on coarse, hand-crafted, and fixed criteria that neither adapt to each question nor stay grounded in the acoustic evidence. Moreover, questions differ in what they demand, with some hinging on perception and others on multi-step reasoning, and any static criterion weakens as the policy improves. Supervising the reasoning process with fine-grained, audio-grounded, and adaptive rewards is therefore crucial, yet challenging since such rewards are impractical to design by hand for every sample. To this end, we introduce AudioRubrics, a reinforcement learning framework that supervises audio reasoning with self-evolving, audio-grounded rubric rewards. AudioRubrics synthesizes per-sample rubrics from the raw waveform and, conditioned on the model's own rollouts, regenerates and reweights criteria per group, supplying a continuous learning signal that keeps targeting the current policy's weaknesses as static criteria saturate. Comprehensive evaluations across three audio reasoning benchmarks reveal that AudioRubrics substantially outperforms a wide range of open-source and training-based baselines. Furthermore, our analysis shows that the gains scale with the capability of the rubric generator and judge, and AudioRubrics converges to a stable reasoning length that avoids both degenerate collapse and unbounded growth. The improvement in audio perception further demonstrates the effectiveness of anchoring supervision in the acoustic evidence. Our project page is available at https://audiorubrics.github.io.
Primary: Microsoft Research
All Institutions: University of Maryland, College Park, University of Illinois Urbana-Champaign, University of Illinois Chicago, Microsoft Research, MBZUAI
AudioRubrics introduces a novel reinforcement learning framework that utilizes self-evolving, audio-grounded rubrics to provide dense, adaptive process-level rewards for audio reasoning, significantly outperforming static reward baselines on multiple benchmarks.
The paper proposes "AudioRubrics," a reinforcement learning framework designed to enhance the reasoning capabilities of Large Audio-Language Models (LALMs). The core innovation lies in replacing static, hand-crafted process rewards with self-evolving, audio-grounded rubrics. The methodology involves three key steps: (1) Initializing rubrics from raw audio waveforms using an audio-capable model to ensure grounding; (2) Evolving these rubrics during training by eliciting new criteria from the model's own rollouts, pruning non-discriminative ones (those with zero variance in judgments across a group), and re-weighting the survivors; (3) Combining these evolving rubric rewards with outcome accuracy and an overthinking penalty (to prevent verbose, hallucinated reasoning) within a Group Relative Policy Optimization (GRPO) framework. The approach addresses the saturation of static rewards and the lack of audio grounding in text-based process rewards. The technical design is sound, leveraging the heterogeneity of rollouts to generate adaptive evaluation criteria.
The authors evaluate AudioRubrics on three benchmarks: MMAU, MMAR, and MMSU. They compare against a wide range of proprietary (GPT-4o-Audio, Gemini) and open-source baselines (Qwen2-Audio, Phi-4-Multimodal, etc.), as well as other RL-based reasoning methods. The results indicate substantial improvements over baselines, particularly in multi-step reasoning tasks. The paper includes analysis showing that the rubrics evolve to become more sophisticated as training progresses, and that the gains scale with the capability of the rubric generator. The inclusion of an overthinking penalty is shown to stabilize reasoning length. The evaluation is comprehensive and covers diverse audio domains (speech, sound, music).
The paper provides a GitHub repository link and a Hugging Face collection for model weights and datasets. The methodology is described in detail, including the GRPO objective and the specific steps for rubric elicitation and filtering. However, the exact hyperparameters for the rubric generation model, the specific taxonomy of initial dimensions, and the precise implementation details of the "negative criteria" elicitation might require careful reading of the appendix (not fully provided here, but implied). The code availability significantly aids reproducibility.
The approach relies heavily on the quality of the rubric generator/judge model. If this model is biased or lacks audio understanding, the evolving rubrics may propagate errors. The method introduces additional computational overhead during training due to the continuous generation and evaluation of rubrics. The paper mentions that static criteria saturate, but does not extensively discuss the potential for "rubric collapse" where the model optimizes for a narrow set of evolving criteria that do not generalize. The reliance on an external judge model for rubric generation and judging introduces a dependency that might not be available in all settings.
This work contributes to the development of more robust and interpretable audio reasoning systems. By grounding reasoning in audio evidence and providing dense process-level feedback, it reduces hallucinations and improves the reliability of AI systems in critical applications like healthcare and multimedia analysis. The self-evolving nature of the rewards could inspire similar adaptive reward mechanisms in other multimodal reasoning tasks. AudioRubrics introduces a novel reinforcement learning framework that utilizes self-evolving, audio-grounded rubrics to provide dense, adaptive process-level rewards for audio reasoning, significantly outperforming static reward baselines on multiple benchmarks.
Reinforcement learning for flow-matching text-to-speech is complicated by deterministic ODE sampling: trajectory-level policy-gradient methods typically convert the ODE into an SDE and track per-step likelihood ratios, introducing stochastic perturbations and substantial overhead. We propose GROW, a group-relative advantage-weighted on-policy RL method that acts directly on the standard flow-matching objective. For each prompt, GROW samples a group of on-policy utterances, separately standardizes intelligibility and speaker-similarity rewards within the group, and combines them to reweight flow-matching regression. A Wasserstein-2 velocity penalty anchors the updated model to a frozen pretrained reference. A group-mean reward baseline is introduced to convert reward weighting into advantage weighting. For strong pretrained TTS models with concentrated rewards, positive exponential weighting is dominated by reward-agnostic self-imitation, whereas a zero-mean signed advantage preserves effective within-group credit assignment. Instantiated on DiTAR and evaluated on LibriSpeech and Seed-TTS EN/ZH, GROW reduces average WER from 2.016 to 1.558 and raises speaker similarity from 0.676 to 0.715 while keeping UTMOS. With 10-NFE training rollouts and 32-NFE evaluation, GROW retains comparable performance while training 2.9x faster than 32-NFE DiTAR-GRPO. We will open-source complete GROW codes, faithful DiTAR reproduction, and all model checkpoints.
Primary: Shanghai Innovation Institute
All Institutions: Shanghai Innovation Institute, Shanghai Jiao Tong University
GROW introduces a novel, efficient, and theoretically motivated advantage-weighted regression framework for flow-matching TTS, demonstrating superior performance and training efficiency over trajectory-based RL methods. The paper makes a significant technical contribution by bridging the gap between discrete-token RL alignment techniques and continuous flow-matching generators, offering a scalable solution for post-training autoregressive-diffusion models.
The paper proposes GROW, a reinforcement learning method tailored for autoregressive-diffusion Text-to-Speech (TTS) models, specifically instantiated on DiTAR. The core innovation lies in addressing the inefficiency and instability of converting deterministic ODE sampling (used in flow-matching) into Stochastic Differential Equations (SDEs) for policy gradient methods. Instead, GROW applies advantage-weighted regression directly to the flow-matching objective. It samples a group of on-policy rollouts, computes group-relative advantages for intelligibility and speaker similarity rewards, and uses these signed, normalized advantages to weight the standard flow-matching loss. A Wasserstein-2 velocity penalty is used to anchor the model to the pretrained reference, preventing divergence. The methodology is theoretically grounded, drawing on Taylor expansions to justify why linear advantage weighting outperforms exponential weighting in the high-reward, concentrated regime typical of strong pretrained TTS models. The approach is elegant in its simplicity, avoiding the overhead of trajectory storage and per-step likelihood ratio calculations required by methods like GRPO or PPO in continuous spaces.
The experimental evaluation is comprehensive and robust. The authors compare GROW against a strong baseline (DiTAR-GRPO) and the pretrained model across three benchmarks: LibriSpeech-PC, Seed-TTS EN, and Seed-TTS ZH. Results show consistent improvements in Word Error Rate (WER) and Speaker Similarity (SIM) while maintaining or slightly improving UTMOS. Notably, GROW achieves these gains with significantly higher training efficiency, being 2.9x faster than the 32-NFE DiTAR-GRPO baseline when using 10-NFE rollouts. The ablation studies are particularly strong, providing detailed analysis on the impact of rollout NFE, CFG settings, learning rates, W2 anchor strength, and reward design. The decomposition of self-imitation vs. reward-contrast provides valuable insight into why the proposed weighting scheme works better than exponential alternatives.
The paper provides extensive implementation details, including model architecture (Qwen3-0.6B backbone, LocDiT), hyperparameters (learning rates, group size, NFEs), and dataset descriptions. The authors explicitly state they will open-source the code, reproduction, and checkpoints, which significantly enhances reproducibility. The algorithm is clearly defined with pseudocode. The use of standard metrics (WER, SIM, UTMOS) and public datasets further supports reproducibility.
The method is currently evaluated primarily on zero-shot TTS tasks using specific reward models (ASR for WER, Speaker Encoder for SIM). The generalizability to other TTS objectives (e.g., emotion, prosody) or other flow-matching architectures (e.g., F5-TTS) is suggested but not extensively demonstrated. The reliance on frozen reward models introduces potential biases inherent in those models (e.g., ASR errors affecting WER rewards). Additionally, the performance gain, while consistent, is modest in absolute terms (e.g., WER reduction of ~0.45), though significant in the context of already high-performing models. The method's effectiveness on very short or very long utterances is not explicitly analyzed.
This work contributes to the efficient alignment of generative audio models, potentially lowering the computational cost of post-training for high-quality TTS systems. This could democratize access to advanced speech synthesis technologies. However, as with any TTS advancement, there are risks associated with improved voice cloning capabilities, including potential misuse for deepfakes or misinformation. The authors' open-source commitment promotes transparency and further research in this area. GROW introduces a novel, efficient, and theoretically motivated advantage-weighted regression framework for flow-matching TTS, demonstrating superior performance and training efficiency over trajectory-based RL methods. The paper makes a significant technical contribution by bridging the gap between discrete-token RL alignment techniques and continuous flow-matching generators, offering a scalable solution for post-training autoregressive-diffusion models.
Continuous-latent autoregressive speech generation has emerged as a promising alternative to discrete-token modeling by avoiding quantization loss and preserving richer acoustic information. However, continuous acoustic targets do not ex- pose linguistic structure as explicit token-level prediction tar- gets. Consequently, the autoregressive language model (LM) must acquire linguistic structure indirectly through acous- tic prediction, which can compromise the content fidelity of generated speech. We propose SemBridge, a training-only semantic-token anchoring framework for continuous-latent autoregressive speech generation. SemBridge uses discrete se- mantic tokens to directly supervise autoregressive LM states and employs a Semantic-Aligned Acoustic VAE to organize the continuous target space under the same semantic refer- ence. The semantic supervision is used only during train- ing, while inference remains entirely continuous. We evalu- ate SemBridge on zero-shot text-to-speech (TTS) and score- conditioned singing voice synthesis (SVS). Across multi- ple benchmarks, SemBridge improves content accuracy, as measured by word and character error rates (WER/CER), while maintaining competitive speaker similarity and percep- tual quality. Experimental results demonstrate that explicit semantic-token supervision for autoregressive state learning is an effective and general direction for continuous speech generation. Speech samples are available.1 The model code and checkpoints will be available at https://github.com/ASLP- lab/SemBridge
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University, Soul AI Lab
SemBridge introduces a training-only semantic-token anchoring framework that improves content fidelity in continuous-latent autoregressive speech generation by supervising LM states with discrete semantic tokens, demonstrating significant gains in WER/CER across TTS and SVS benchmarks while maintaining competitive acoustic quality.
The paper proposes SemBridge, a framework for continuous-latent autoregressive speech generation. The core innovation is "semantic-token anchoring," which uses discrete semantic tokens (from a frozen GLM-4 tokenizer) to supervise the hidden states of the autoregressive language model during training. This is complemented by a Semantic-Aligned Acoustic VAE (SA-VAE) that aligns continuous acoustic latents with semantic embeddings. The approach is technically sound and addresses a real gap in continuous speech generation: the lack of explicit linguistic structure in continuous targets. However, the novelty is moderate; using discrete tokens to guide continuous generation is a known strategy (e.g., in VALL-E X, SpeechTokenizer), and the specific implementation of anchoring LM states with classification loss is a straightforward adaptation of existing multi-task learning principles. The method is essentially a training-time regularization technique that does not alter the inference architecture, which is a pragmatic but not groundbreaking design choice.
The evaluation is comprehensive, covering zero-shot TTS on Seed-TTS-Eval and CV3-Eval, as well as score-conditioned SVS on GMO-SVS. The results show consistent improvements in content accuracy (WER/CER) compared to baselines like MELA-TTS and VoxCPM, while maintaining competitive speaker similarity and perceptual quality. The ablation studies are well-conducted, isolating the effects of target-space alignment, state anchoring, anchoring depth, and anchoring strength. The inclusion of SVS results adds value by demonstrating the method's transferability to a more challenging task with explicit musical constraints. The comparison with a wide range of baselines (discrete, continuous, and hybrid) provides a strong context for the results. The paper effectively demonstrates that explicit semantic supervision improves content fidelity without sacrificing acoustic quality.
The paper provides detailed descriptions of the model architecture, training stages, hyperparameters, and datasets. It specifies the use of the GLM-4 tokenizer and the SA-VAE architecture. The training configurations, including batch sizes, learning rates, and loss weights, are clearly listed. The authors state that code and checkpoints will be available, and a demo link is provided. The detailed ablation studies and clear separation of Stage I and Stage II training procedures enhance reproducibility. However, the reliance on specific internal datasets (Soul AI Lab's 20K-hour singing data) for the joint TTS-SVS model limits full reproducibility for that specific configuration, though the TTS-only experiments are likely reproducible with the public VoxBox corpus.
The primary limitation is that the semantic supervision is only applied during training. While this keeps inference efficient, it means the model relies entirely on the learned alignment between continuous latents and discrete tokens, which might not be as robust as end-to-end discrete modeling in some edge cases. The paper notes a trade-off between content accuracy and synthesis quality depending on the anchoring depth and weight, suggesting that hyperparameter tuning is critical. Additionally, the use of a frozen semantic tokenizer means the model cannot adapt the semantic representation to the specific domain of speech generation, potentially missing out on speech-specific semantic nuances that could be learned jointly. The improvement in WER/CER, while statistically significant, is not massive, and the gains in perceptual quality (UTMOS) are marginal.
SemBridge contributes to the field of speech synthesis by providing a robust method for improving content fidelity in continuous-latent models. This can lead to more reliable and accurate text-to-speech and singing voice synthesis systems, which have broad applications in accessibility, entertainment, and human-computer interaction. The method's ability to improve lyric intelligibility in SVS is particularly relevant for the music industry. However, the potential for misuse in generating deepfakes or misleading audio content remains a concern, as with any advanced speech synthesis technology. The paper does not explicitly discuss safety mitigations, which is a common omission in technical research but important for broader impact assessment. SemBridge introduces a training-only semantic-token anchoring framework that improves content fidelity in continuous-latent autoregressive speech generation by supervising LM states with discrete semantic tokens, demonstrating significant gains in WER/CER across TTS and SVS benchmarks while maintaining competitive acoustic quality.
Mechanistic interpretability of music generation has concentrated on audio models, leaving symbolic models largely unexplored. We analyze two public text-to-MIDI systems of contrasting design: the purpose-built encoder--decoder text2midi and MIDI-LLM, a Llama~3.2~1B model extended with MIDI tokens using linear probing, the logit and tuned lenses, activation patching and difference-in-means steering. Across these methods, we recover musically meaningful structure and show how architecture shapes its formation and control. Pitch, instrumentation, harmony and texture are linearly decodable in both models. text2midi refines predictions gradually across depth, whereas MIDI-LLM works largely in its inherited textual basis before a sharp late rotation into the musical vocabulary; patching identifies a matching late attenuation of prompt-driven instrument transfer. Steering produces bidirectional changes in register and polyphony in both systems, and in tempo/energy in MIDI-LLM. Our two-orientation protocol isolates directional control and shows that all-layer interventions are robust in text2midi but accumulate disruptively in MIDI-LLM. Together, the results provide a practical toolkit for tracing and controlling musical concepts in symbolic generators. Audio examples are available on a demo website.
Primary: Warsaw University of Technology
All Institutions: Warsaw University of Technology
This paper makes a significant contribution to the field of mechanistic interpretability by extending its methods to symbolic music generation, offering a detailed comparative analysis of two distinct architectures and providing a robust protocol for causal intervention and control.
The paper employs a rigorous mechanistic interpretability toolkit, including linear probing, logit/tuned lenses, activation patching, and difference-in-means steering. The methodology is sound and well-executed, particularly the novel "bidirectional evaluation protocol" for steering, which effectively decomposes directional control from symmetric drift. The comparative analysis between a purpose-built encoder-decoder (text2midi) and a repurposed LLM (MIDI-LLM) provides valuable insights into how architecture influences the formation of symbolic representations. The use of the tuned lens to disentangle basis changes from genuine prediction formation in the LLM is a strong technical contribution.
The experimental setup is comprehensive, covering two distinct models and multiple interpretability techniques. The results are clearly presented and support the claims of architectural differences in prediction formation (gradual vs. late binding). The steering experiments demonstrate robust control over register and polyphony, with quantifiable metrics for tempo/energy in the LLM. The controlled probing on SynTheory adds necessary isolation to the ecological validity of full-generation probing. The statistical treatment, including confidence intervals and seed-clustered standard errors, is appropriate.
The paper provides detailed descriptions of the models, datasets (MIDICaps, SynTheory), and experimental protocols. The authors state that code will be released, and audio examples are available online. The specific hyperparameters for probing and steering are reported. However, the exact codebase for the "tuned lens" implementation and the specific steering vector computation details could benefit from more explicit pseudocode or references to specific libraries to ensure perfect reproducibility.
The study is limited to two models, which restricts the generalizability of the architectural conclusions. The metrics for tempo and polyphony are proxies (note density, mean pitch) and may not perfectly capture the musical concepts, especially in multi-track MIDI where interactions are complex. The probing labels for full generations are heuristic estimates, introducing potential noise. The steering directions are derived from text contrasts that vary lexically and musically, making it difficult to isolate the pure musical concept vector.
This work bridges the gap between mechanistic interpretability in language models and symbolic music generation. It provides a practical toolkit for researchers and developers to trace and control musical concepts in generative models, potentially leading to more controllable and interpretable music AI systems. The findings on architecture-dependent intervention strategies are directly applicable to the design of future symbolic music models. This paper makes a significant contribution to the field of mechanistic interpretability by extending its methods to symbolic music generation, offering a detailed comparative analysis of two distinct architectures and providing a robust protocol for causal intervention and control.
Speech recordings often contain missing, corrupted, or incorrect regions that must be reconstructed or modified without re-synthesizing the entire utterance. Speech inpainting restores missing segments, whereas speech editing replaces spoken content according to an edited transcript. Both tasks require the generated speech to express the intended words while remaining consistent with the surrounding speaker identity, prosody, timing, and recording conditions. Discrete diffusion is particularly well suited to these tasks because it can iteratively refine masked tokens while jointly conditioning on both left and right acoustic context. We introduce SIEDD, a discrete diffusion framework for text-guided speech inpainting and editing over hierarchical codec tokens. Its core architecture, HiCoDD, follows the RVQ generation order by representing previously generated codebooks as clean, committed acoustic context and applying diffusion only to the current refinement codebook. This separation enables leakage-free joint training while matching sequential coarse-to-fine inference. The model further combines phoneme-level conditioning, span-localized classifier-free guidance, and duration prediction to support both fixed-duration inpainting and variable-duration text edits. On the RealEdit benchmark, SIEDD achieves the best overall speech-editing performance among the evaluated methods. It also outperforms the evaluated autoregressive baselines across all speech-inpainting settings, on both single and multiple gaps. These results demonstrate that explicitly modeling the codec hierarchy substantially improves context-preserving speech reconstruction and editing. See our full code at https://github.com/iftachShoham/SIEDD.
Primary: Ben-Gurion University of the Negev
All Institutions: Ben-Gurion University of the Negev, University of Haifa
The paper presents SIEDD, a novel discrete diffusion framework that leverages the hierarchical structure of neural codecs for superior text-guided speech inpainting and editing. By introducing HiCoDD, which enforces coarse-to-fine generation and leakage-free conditioning, the authors address key limitations of existing autoregressive and naive diffusion approaches, achieving state-of-the-art performance on the RealEdit benchmark.
The paper proposes SIEDD, a discrete diffusion framework for text-guided speech inpainting and editing. The core methodological contribution is HiCoDD, a hierarchical diffusion process that respects the Residual Vector Quantization (RVQ) structure of neural codecs (like EnCodec). Instead of flattening all codebooks into a single sequence, HiCoDD generates codebooks sequentially from coarse to fine, conditioning the diffusion on previously committed (clean) lower-level codebooks. This addresses the issue of "leakage" and ensures that the hierarchical dependency of the codec is preserved during generation. The paper also introduces span-localized classifier-free guidance (CFG) adapted for categorical scores, which randomizes phonemes only within the edited span to preserve surrounding context. A duration predictor is included to handle variable-length edits. The methodology is technically sound, well-motivated by the limitations of autoregressive models in non-causal generation tasks, and correctly applies discrete diffusion theory (DWDSE) to the hierarchical token space.
The evaluation is conducted on the RealEdit benchmark, comparing SIEDD against VoiceCraft, SSR-Speech, and a TTS baseline. Metrics include WER, Speaker Similarity (SIM), Mel Cepstral Distortion (MCD), F0/Energy distance, and UTMOS. SIEDD achieves state-of-the-art results in overall editing performance (lowest WER, highest SIM, lowest MCD) and significantly outperforms baselines in speech inpainting, particularly in multi-gap scenarios where autoregressive models struggle with error propagation. The ablation studies effectively isolate the contributions of the hierarchical structure, duration predictor, and localized CFG. The results are consistent with the claims, showing that modeling the codec hierarchy improves context preservation. The use of standard, widely accepted metrics strengthens the validity of the claims.
The paper provides a GitHub link to the code and configuration files. The appendix details the architecture, training hyperparameters, and dataset construction for the duration predictor. The training setup (550k steps on two A6000 GPUs) is specified. The use of standard datasets (LibriTTS, GigaSpeech) and tokenizers (EnCodec) aids reproducibility. The detailed ablation study configurations further support reproducibility.
The method relies on the quality of the underlying neural codec (EnCodec); errors in tokenization may propagate. The diffusion process, while effective, is computationally more expensive than autoregressive inference, although inference time is not the primary focus of this work. The duration predictor, while better than heuristics, may still struggle with extreme prosodic variations or non-speech content. The evaluation is limited to English speech.
This work contributes to the field of generative audio, specifically in robust speech editing and restoration. Applications include audio repair, content creation, and accessibility. The ability to edit speech with high fidelity and speaker consistency has significant implications for media production and communication. However, as with all speech editing technologies, there are potential risks regarding misinformation and deepfakes, necessitating responsible use and detection methods. The paper presents SIEDD, a novel discrete diffusion framework that leverages the hierarchical structure of neural codecs for superior text-guided speech inpainting and editing. By introducing HiCoDD, which enforces coarse-to-fine generation and leakage-free conditioning, the authors address key limitations of existing autoregressive and naive diffusion approaches, achieving state-of-the-art performance on the RealEdit benchmark.
Reinforcement learning for flow-matching text-to-speech is complicated by deterministic ODE sampling: trajectory-level policy-gradient methods typically convert the ODE into an SDE and track per-step likelihood ratios, introducing stochastic perturbations and substantial overhead. We propose GROW, a group-relative advantage-weighted on-policy RL method that acts directly on the standard flow-matching objective. For each prompt, GROW samples a group of on-policy utterances, separately standardizes intelligibility and speaker-similarity rewards within the group, and combines them to reweight flow-matching regression. A Wasserstein-2 velocity penalty anchors the updated model to a frozen pretrained reference. A group-mean reward baseline is introduced to convert reward weighting into advantage weighting. For strong pretrained TTS models with concentrated rewards, positive exponential weighting is dominated by reward-agnostic self-imitation, whereas a zero-mean signed advantage preserves effective within-group credit assignment. Instantiated on DiTAR and evaluated on LibriSpeech and Seed-TTS EN/ZH, GROW reduces average WER from 2.016 to 1.558 and raises speaker similarity from 0.676 to 0.715 while keeping UTMOS. With 10-NFE training rollouts and 32-NFE evaluation, GROW retains comparable performance while training 2.9x faster than 32-NFE DiTAR-GRPO. We will open-source complete GROW codes, faithful DiTAR reproduction, and all model checkpoints.
Primary: Shanghai Innovation Institute
All Institutions: Shanghai Innovation Institute, Shanghai Jiao Tong University
GROW introduces a novel, efficient, and theoretically motivated advantage-weighted regression framework for flow-matching TTS, demonstrating superior performance and training efficiency over trajectory-based RL methods. The paper makes a significant technical contribution by bridging the gap between discrete-token RL alignment techniques and continuous flow-matching generators, offering a scalable solution for post-training autoregressive-diffusion models.
The paper proposes GROW, a reinforcement learning method tailored for autoregressive-diffusion Text-to-Speech (TTS) models, specifically instantiated on DiTAR. The core innovation lies in addressing the inefficiency and instability of converting deterministic ODE sampling (used in flow-matching) into Stochastic Differential Equations (SDEs) for policy gradient methods. Instead, GROW applies advantage-weighted regression directly to the flow-matching objective. It samples a group of on-policy rollouts, computes group-relative advantages for intelligibility and speaker similarity rewards, and uses these signed, normalized advantages to weight the standard flow-matching loss. A Wasserstein-2 velocity penalty is used to anchor the model to the pretrained reference, preventing divergence. The methodology is theoretically grounded, drawing on Taylor expansions to justify why linear advantage weighting outperforms exponential weighting in the high-reward, concentrated regime typical of strong pretrained TTS models. The approach is elegant in its simplicity, avoiding the overhead of trajectory storage and per-step likelihood ratio calculations required by methods like GRPO or PPO in continuous spaces.
The experimental evaluation is comprehensive and robust. The authors compare GROW against a strong baseline (DiTAR-GRPO) and the pretrained model across three benchmarks: LibriSpeech-PC, Seed-TTS EN, and Seed-TTS ZH. Results show consistent improvements in Word Error Rate (WER) and Speaker Similarity (SIM) while maintaining or slightly improving UTMOS. Notably, GROW achieves these gains with significantly higher training efficiency, being 2.9x faster than the 32-NFE DiTAR-GRPO baseline when using 10-NFE rollouts. The ablation studies are particularly strong, providing detailed analysis on the impact of rollout NFE, CFG settings, learning rates, W2 anchor strength, and reward design. The decomposition of self-imitation vs. reward-contrast provides valuable insight into why the proposed weighting scheme works better than exponential alternatives.
The paper provides extensive implementation details, including model architecture (Qwen3-0.6B backbone, LocDiT), hyperparameters (learning rates, group size, NFEs), and dataset descriptions. The authors explicitly state they will open-source the code, reproduction, and checkpoints, which significantly enhances reproducibility. The algorithm is clearly defined with pseudocode. The use of standard metrics (WER, SIM, UTMOS) and public datasets further supports reproducibility.
The method is currently evaluated primarily on zero-shot TTS tasks using specific reward models (ASR for WER, Speaker Encoder for SIM). The generalizability to other TTS objectives (e.g., emotion, prosody) or other flow-matching architectures (e.g., F5-TTS) is suggested but not extensively demonstrated. The reliance on frozen reward models introduces potential biases inherent in those models (e.g., ASR errors affecting WER rewards). Additionally, the performance gain, while consistent, is modest in absolute terms (e.g., WER reduction of ~0.45), though significant in the context of already high-performing models. The method's effectiveness on very short or very long utterances is not explicitly analyzed.
This work contributes to the efficient alignment of generative audio models, potentially lowering the computational cost of post-training for high-quality TTS systems. This could democratize access to advanced speech synthesis technologies. However, as with any TTS advancement, there are risks associated with improved voice cloning capabilities, including potential misuse for deepfakes or misinformation. The authors' open-source commitment promotes transparency and further research in this area. GROW introduces a novel, efficient, and theoretically motivated advantage-weighted regression framework for flow-matching TTS, demonstrating superior performance and training efficiency over trajectory-based RL methods. The paper makes a significant technical contribution by bridging the gap between discrete-token RL alignment techniques and continuous flow-matching generators, offering a scalable solution for post-training autoregressive-diffusion models.
What is music style? Though often described using text labels such as "swing," "classical," or "emotional," the real style remains implicit and hidden in concrete music examples. In this paper, we introduce a cross-modal framework that learns implicit music styles from raw audio and applies them to symbolic music generation. Inspired by BLIP-2, our model leverages a Querying Transformer (Q-Former) to extract style representations from a large, pre-trained audio language model (LM), and further applies them to condition a symbolic LM for generating piano arrangements. We adopt a two-stage training strategy: contrastive learning to align auditory style with symbolic expression, followed by generative modeling for music arrangement. Our model generates piano performances jointly conditioned on a lead sheet (content) and a reference audio example (style), enabling controllable and stylistically faithful arrangement. Experiments demonstrate the effectiveness of our approach in piano cover generation, style transfer, and audio-to-MIDI retrieval, achieving substantial improvements in style-aware alignment and music quality.
Primary: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
All Institutions: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), National University of Singapore, School of Computing
The paper presents a novel and technically sound approach to cross-modal music generation, effectively applying vision-language bootstrapping techniques to the audio-symbolic domain for piano arrangement.
The paper proposes a cross-modal framework for piano arrangement by leveraging a Querying Transformer (Q-Former) to bridge audio and symbolic domains. The core innovation lies in using the Q-Former to extract implicit style representations from a frozen audio language model (MusicGen) and conditioning a symbolic language model (MuseCoco) with these representations, alongside a lead sheet for content. The two-stage training process—contrastive/matching learning for style alignment followed by generative modeling—is a logical extension of vision-language bootstrapping methods (like BLIP-2) to the audio-symbolic domain. The approach effectively disentangles content (lead sheet) from style (audio reference), addressing a significant gap in current music generation systems that often rely on text prompts or explicit symbolic controls. The use of LoRA adapters to condition the frozen symbolic LM is a practical and efficient design choice.
The experimental evaluation is comprehensive, covering objective metrics (MCA, CA, GPC, VCC, TA) and subjective listening tests. The inclusion of out-of-distribution datasets (Ballroom, GTZAN) strengthens the claim of generalization. The results show competitive performance against baselines like PiCoGen2 and A2M, particularly in style coherence metrics. The audio-to-MIDI retrieval task further validates the quality of the learned cross-modal representations. However, the lower content preservation scores on the in-distribution POP909 set compared to PCG2 are a notable weakness, attributed to errors in the upstream Sheetsage transcription. The subjective evaluation, while well-designed with ANOVA analysis, relies on a small sample size (21 participants), which limits the statistical power of the findings.
The paper provides detailed model configurations, including parameter counts, training epochs, batch sizes, and optimizer settings. The use of open-source or widely available backbones (MusicGen, MuseCoco, Sheetsage) enhances reproducibility. The data pairing strategy and specific metric definitions are clearly described. The lack of a public code repository is a minor drawback, but the detailed appendix and demo page mitigate this to some extent.
The authors acknowledge that the style representation is segment-level (4-bar), which may not capture long-term stylistic evolution or transitions. The reliance on Sheetsage for lead sheet extraction introduces potential error propagation. The method is currently limited to piano arrangement, and generalizing to multi-track or more complex instrumental arrangements is identified as future work. The subjective evaluation sample size is small.
This work contributes to the field of AI-assisted music creation by providing a tool for expressive piano arrangement. It enables musicians to reinterpret existing audio with specific stylistic nuances while maintaining harmonic and melodic integrity. The cross-modal representation learning technique has broader applications in audio-symbolic retrieval and style transfer. The potential for misuse in generating copyrighted music arrangements without permission is a consideration, though the focus on arrangement rather than direct cloning mitigates some risks. The paper presents a novel and technically sound approach to cross-modal music generation, effectively applying vision-language bootstrapping techniques to the audio-symbolic domain for piano arrangement.
Humans recognize a musical passage even when it is shifted in time or transposed in pitch, indicating a notion of equivariance in the representation space. Our analysis, however, shows that standard music transformers map such time-shifted or pitch-transposed inputs onto uncorrelated representations: these models become progressively less equivariant as they scale in size or train longer. This suggests that in standard music transformers, additional model capacity is allocated to memorizing absolute patterns rather than capturing shared musical structures. In this paper, we propose the Equivariant Music Transformer (EMT), which enforces equivariance through self-distillation by jointly optimizing a next-token-prediction and an auxiliary equivariance regularization loss. We find that the additional equivariance loss acts as a beneficial regularizer, simultaneously improving next-token prediction and producing equivariant latent representations. Through both objective and subjective evaluations, EMT demonstrates superior equivariance and generative capability compared to data augmentation, feature engineering, and state-of-the-art (SOTA) baselines. More broadly, our findings reveal that standard language modeling methods alone do not capture music's translational symmetries, and dedicated inductive biases are required to produce better music representations. The code, weights and demos are available online.
Primary: Queen Mary University of London
All Institutions: Queen Mary University of London, UKRI Centre for Doctoral Training (CDT) in Artificial Intelligence and Music (AIM)
The paper presents a solid technical contribution by introducing an equivariance regularization loss for music transformers, demonstrating that explicit inductive biases improve both equivariance and generative performance, and revealing that equivariance does not emerge naturally with scale.
The paper proposes the Equivariant Music Transformer (EMT), which introduces an auxiliary equivariance regularization loss to a standard next-token prediction transformer. The method involves a self-distillation setup where a shifted version of the input is processed, and the output distribution is aligned with the shifted original distribution using KL divergence. The authors argue that this enforces latent space equivariance. The approach is conceptually sound and addresses a specific gap in symbolic music generation: the lack of explicit inductive biases for pitch and time translational symmetry. However, the novelty is somewhat incremental, as equivariance has been explored in other domains (e.g., vision, NLP) and feature-level equivariance (FME/MRA) is already established in music transformers. The contribution lies primarily in the specific application of this regularization to generative music transformers and the empirical analysis of how equivariance degrades with scale.
The experimental evaluation is comprehensive. The authors conduct ablation studies comparing their method against data augmentation, feature-level equivariance, and a combination of both. They also benchmark against strong external baselines (Anticipatory Music Transformer, MIDI-LLM). The key finding—that standard transformers become *less* equivariant as they scale—is an interesting empirical observation that challenges the assumption that equivariance emerges naturally. The objective metrics (cross-entropy loss, equivariance loss, Top-1/Top-5 accuracy) show consistent improvements. The subjective listening test (MOS) adds valuable human-centric validation, demonstrating that EMT maintains quality under shifted prompts better than baselines. The use of a paired Wilcoxon signed-rank test for statistical significance is appropriate.
The paper provides detailed implementation details, including dataset preprocessing (LMD clean), model architecture (12 layers, 768 hidden size), training hyperparameters (AdamW, learning rate schedule, loss weights), and evaluation protocols. The code and demos are available online, which significantly aids reproducibility. The description of the self-distillation mechanism and the specific transformations (pitch/time shifts) is clear.
The paper acknowledges that the equivariance loss acts as a regularizer but does not deeply analyze the potential trade-offs or failure modes. For instance, if the shift distribution in the loss does not match the shift distribution in the test data, performance might degrade. The reliance on LakhMIDI, which has known quality issues (noise, transcription errors), is a limitation, although the authors filter the data. The claim that "standard language modeling methods alone do not capture music's translational symmetries" is supported, but the extent to which this applies to non-music domains or other symbolic domains is not discussed. The computational cost of the auxiliary branch (doubling forward passes) is noted but not extensively analyzed in terms of efficiency gains vs. performance.
This work contributes to the broader field of AI for Music (AIM) by providing a robust method for generating symbolic music that respects musical structure (transposition/tempo). This can improve the usability of AI music assistants, allowing users to transpose or shift prompts without losing generation quality. It also highlights the importance of inductive biases in deep learning for structured data. The ethical statement correctly addresses copyright concerns associated with training on LakhMIDI. The paper presents a solid technical contribution by introducing an equivariance regularization loss for music transformers, demonstrating that explicit inductive biases improve both equivariance and generative performance, and revealing that equivariance does not emerge naturally with scale.
Text-to-music language models begin with a choice usually made by default: how to tokenize music. Normally entangled with backbone, data, and recipe, its effect has never been measured in isolation. We fix pretrained Qwen3.5 (0.8B-27B), data, budget, and decoding, and swap only the representation across seven tokenizations, anchoring texture metrics to each representation's model-free ceiling. The ordering is clean and surprising: representation, not model size, is the binding variable for distributional fidelity. Scaling the backbone 34x barely moves Frechet Music Distance (FMD), whereas switching representation halves it. PMT, a performance-resolution stream we release (10 ms timing, per-note velocity, multi-track texture; 609 symbols), reaches FMD 159 at 0.8B against 272-286 for beat grids (1.7-1.8x lower, up to 2.8x elsewhere; non-overlapping bootstrap CIs), so a 0.8B performance-resolution model beats a 27B beat grid. It reappears on a 26M from-scratch backbone and a second performance-resolution tokenizer: a property of the class, not one lucky vocabulary. Nor is it a finer-lattice artifact: snapping PMT's onsets to the beat grids' resolution still leaves it 67-129 FMD ahead of both (n=500). The effect is distributional; whether it is audible is a separate question, left open by our probe, with a human study pre-registered. Native caption adherence is weak but separable: a lightweight decode-time constraint doubles instrument-F1 (.28 to .60) and Correct-Key (.16 to .35) at no distributional cost. We release the harness, 25+ checkpoints, two corpora (86.6k aligned across caption/MIDI/ABC/audio; 6.25M captioned, the largest for music), and an imprinting diagnostic: published text-to-MIDI systems reproduce their training distribution near-invariant to the caption (72% vs. 71% chord-time on disjoint domains). The field's next representation claim can now be measured, not asserted.
Primary: Unknown (Affiliations not explicitly named in text, only numbered)
All Institutions: Unknown
The paper provides a rigorous, controlled evaluation demonstrating that token representation design is more critical than model scale for distributional fidelity in text-to-symbolic-music generation, supported by the release of new datasets and benchmarks.
The paper proposes "PMT" (Performance-timed Music Tokens), a tokenization scheme for symbolic music generation that preserves 10ms micro-timing and per-note velocity, contrasting with traditional beat-grid tokenizations (like REMI) that quantize timing to metrical positions. The core methodological contribution is not just the tokenizer itself, but the rigorous "controlled swap" evaluation protocol. The authors fix the backbone (Qwen3.5, 0.8B-27B), data, training budget, and decoding strategy, varying only the tokenization scheme across seven different representations. This allows for an isolation of the representation's effect on distributional fidelity. The integration recipe involves vocabulary extension and caption-masked supervised fine-tuning. The approach is technically sound and addresses a significant confound in current music LLM research where representation, data, and model size are often entangled.
The experimental design is the strongest aspect of the paper. By controlling for model scale and data, the authors demonstrate that representation quality dominates model size for distributional fidelity (measured by Fréchet Music Distance). A 0.8B model using PMT outperforms a 27B model using beat grids. They further validate this on a 26M parameter from-scratch backbone and a second performance-resolution tokenizer (PerTok), suggesting the result is a property of the representation class. They also provide an "imprinting diagnostic" showing that existing systems (MIDI-LLM, text2midi) reproduce training distributions rather than following captions, highlighting a limitation in current text-conditioned symbolic generation. The release of two large datasets (86.6k aligned, 6.25M captioned) and a benchmark harness adds substantial empirical value.
High. The authors release the PMT tokenizer, the evaluation harness, 25+ checkpoints, and the datasets. The controlled swap protocol is clearly defined, allowing other researchers to replicate the comparison. The use of standard backbones (Qwen3.5) and open metrics (FMD) further aids reproducibility.
The paper explicitly acknowledges that the distributional fidelity gains do not necessarily translate to perceptual/audible quality, which is left for a pre-registered human study. Additionally, caption adherence (e.g., key, instrument) is weak for PMT compared to dedicated systems, although the authors show a lightweight decode-time constraint can mitigate this. The "imprinting" finding suggests a broader issue with text-conditioned symbolic generation that may require architectural changes beyond tokenization to fully resolve. The lack of explicit institution names in the provided text makes institutional assessment difficult, though the technical content is strong.
This work has significant implications for the field of symbolic music generation and potentially other structured artifact generation domains (e.g., code, 3D assets). It challenges the assumption that scaling models alone will solve quality issues, emphasizing the critical role of representation design. The release of large, high-quality datasets and benchmarks will facilitate future research. The finding that representations can dominate scale is a crucial lesson for efficient model development. The paper provides a rigorous, controlled evaluation demonstrating that token representation design is more critical than model scale for distributional fidelity in text-to-symbolic-music generation, supported by the release of new datasets and benchmarks.
Reference-conditioned melody-preserving lyric editing replaces words while retaining a performance's timing, singer identity, and naturalness. Continuous-latent autoregression avoids finite codebooks and offers stepwise generation with learned stopping. Editing creates a conflict absent from ordinary reconstruction: training pairs reference cues with original lyrics, whereas inference asks revised lyrics to override source-lyric-correlated cues; one source-following patch can propagate through AR history. We introduce CLASVS. Its State-Control-Transition (SCT) routing keeps target-lyric and reference-melody controls persistent, returns semantic feedback on phonetic progress to the causal planner, and confines the previous latent patch to the local Transition. Progressive State-Control Grounding (PSCG) learns this contract through paired-edit-free, content-consistent Mandarin reconstruction. On two Mandarin benchmarks, CLASVS improves all four operations over discrete-AR Vevo2 and reduces macro-PER by 46.2%, while maintaining melody, singer similarity, and perceptual quality. Together, these results establish a strong continuous-AR operating point for score-annotation-free lyric edits and a basis for broader stepwise control. Audio demonstrations are available on our project page: https://piedpiperg.github.io/Liyric-SVS/.
Primary: Tencent AI Lab
All Institutions: Tencent AI Lab, Tsinghua University, Peking University
CLASVS introduces a novel continuous-latent autoregressive framework with a State-Control-Transition routing mechanism that effectively addresses the conflict between melody preservation and lyric editing in singing voice synthesis, achieving state-of-the-art performance in Mandarin lyric editing tasks.
The paper proposes CLASVS, a continuous-latent autoregressive model for melody-preserving lyric editing in singing voice synthesis. The core methodological contribution is the State-Control-Transition (SCT) routing mechanism, which attempts to resolve the conflict between preserving reference melody/timing and generating new lyrics. It separates the control signals (target lyrics, reference melody) from the generative state (semantic feedback, previous latent patch). The training strategy, Progressive State-Control Grounding (PSCG), relies on content-consistent reconstruction (training on the same audio/transcript) rather than paired counterfactual edits, which is a significant theoretical challenge. The use of a frozen causal semantic encoder for phonetic progress feedback is a novel architectural choice for maintaining lyric alignment in an autoregressive setting. However, the reliance on existing components (AudioVAE, Qwen3-0.6B, Flow-DiT) limits the novelty of the underlying generative primitives, focusing innovation primarily on the routing and training protocol.
The evaluation is comprehensive, comparing CLASVS against discrete-AR (Vevo2) and continuous-NAR (YingMusic-Singer-Plus) baselines. The paper reports significant improvements in macro-Phone Error Rate (PER) and perceptual metrics (MOS) over Vevo2. The inclusion of a "target-preference margin" metric to audit whether the model actually follows target lyrics rather than reverting to source lyrics is a strong methodological addition to the evaluation suite. The ablation studies effectively isolate the contributions of the SCT components. However, the evaluation is restricted to Mandarin, and the comparison with YingMusic+ shows mixed results (CLASVS is better at deletion/insertion, but YingMusic+ is better at substitution), suggesting that the autoregressive approach may have limitations in handling large lexical changes compared to parallel generation.
The paper provides detailed descriptions of the architecture, training stages, and hyperparameters. It mentions the release of code and checkpoints upon publication, which supports reproducibility. The use of standard datasets (Emilia, curated singing) and public baselines facilitates comparison. The specific "CLA-LyricEdit-320" benchmark is described in detail, allowing for independent verification.
The primary limitation is the lack of paired counterfactual training data, which forces the model to generalize from reconstruction tasks. This may limit its ability to handle complex edits or non-Mandarin languages. The autoregressive nature of the model results in slower inference speeds compared to NAR baselines like YingMusic+. The evaluation is limited to Mandarin, and cross-lingual generalization is not tested. Additionally, the reliance on automatic melody extraction (chroma-derived tokens) may introduce errors that propagate through the generation process.
This work contributes to the field of controllable singing voice synthesis, enabling more natural and flexible lyric editing without the need for manual score annotations. This has applications in music production, karaoke, and accessibility. However, the ability to edit lyrics while preserving singer identity raises ethical concerns regarding voice impersonation and copyright, which the authors acknowledge and address with proposed safety measures. CLASVS introduces a novel continuous-latent autoregressive framework with a State-Control-Transition routing mechanism that effectively addresses the conflict between melody preservation and lyric editing in singing voice synthesis, achieving state-of-the-art performance in Mandarin lyric editing tasks.
Speech and audio generation is often needed in animation dubbing, audio drama, movies, advertising, games, podcasts, and short-video production. In these scenarios, creators may need to design voices without reference recordings, control speaker styles with natural language, support acoustic scenes with environments and audio effects, and later reuse the designed voices. Therefore, it is important to support multi-speaker speech and audio generation for both instruct and zero-shot tasks. The instruct task requires a caption of the environment, speaker styles, and fine-grained content, while the zero-shot task uses reference audio together with the same fine-grained content. We address these tasks from both the data and model sides. First, we propose SwanData-Caption, which cleans raw speech and audio data, adds targeted synthetic coverage, and annotates diverse and accurate multi-level captions. Then, we propose SwanTale, a multi-speaker expressive speech and audio generation model that supports both zero-shot and instruct tasks. We introduce SwanVAE to support high-quality multi-audio-modality generation. Then, we adopt reward-conditioned quality control and Engram conditioning, along with Unified MoE for multi-task and multi-audio-modality modeling. In addition, we use curriculum learning and GRPO post-training to let the model progressively learn and strengthen its capabilities. Experimental results show that SwanTale leads on multiple key zero-shot and instruct metrics, achieves the best expressiveness scores in both tasks, and supports complex instruct generation involving multi-speaker speech and audio. Demos can be found at https://swanaigc.github.io/#swantale.
Primary: ByteDance
All Institutions: ByteDance, Zhejiang University
SwanTale presents a robust and unified framework for multi-speaker speech and audio generation, effectively bridging the gap between zero-shot cloning and instruct-based creative control through a sophisticated data pipeline and a novel VAE-MoE architecture, significantly advancing the state-of-the-art in expressive audio synthesis.
The paper proposes SwanTale, a unified model for multi-speaker speech and audio generation supporting both zero-shot (reference-based) and instruct (caption-based) tasks. The methodology is comprehensive, addressing data, model architecture, and training. Key technical contributions include: 1) SwanData-Caption, a pipeline for generating fine-grained multi-level captions using an LLM (Seed2.0 Lite) with a persona library to guide annotation. 2) SwanVAE, a custom VAE with a Gaussian bottleneck and Transformer-based decoder for 48kHz audio, designed to balance reconstruction quality and latent learnability. 3) A flow-based Transformer prior with Unified MoE (Mixture of Experts) to handle multiple tasks and audio modalities (speech, environment, effects, singing). 4) Reward-conditioned quality control and GRPO (Group Relative Policy Optimization) post-training to improve pronunciation, stability, and caption adherence. The approach to unify zero-shot and instruct generation in a single model with complex audio scene modeling is technically sound and represents a significant engineering effort. The use of Engram conditioning and curriculum learning is also noted.
The evaluation covers zero-shot TTS, instruct TTS, and acoustic quality. Benchmarks used include SwanBench-Speech, InstructTTSEval, SwanBench-Scene, and SwanBench-Caption. The paper claims leadership on multiple key metrics. The data pipeline includes rigorous filtering (DNSMOS, SQUIM) and human auditing (best-worst comparison for expressiveness). The inclusion of synthetic data for underrepresented demographics (elderly) and challenging pronunciations is a strong point for robustness. The evaluation of multi-speaker dialogue and complex instruct generation adds depth beyond standard single-speaker TTS.
The paper provides detailed descriptions of the data pipeline, model architecture (SwanVAE, Unified MoE), and training procedures (curriculum, GRPO). However, the code is not publicly available (Project URL is none), and some components (e.g., Seed2.0 Lite, SwanAligner, SwanVerifier) are referenced as internal or proprietary tools. While the methodology is described well, full reproducibility might be challenging without access to the specific internal models and the full dataset (70M records).
The reliance on internal data and proprietary models (Seed2.0, SwanAligner) limits immediate reproducibility. The paper mentions "SwanBench" which appears to be a new benchmark, potentially lacking the long-term community validation of established benchmarks like LibriSpeech or VCTK. The complexity of the model (Unified MoE, GRPO) might make inference and fine-tuning resource-intensive. The "Engram conditioning" is mentioned but not fully detailed in the provided text snippet, which might obscure its specific novelty or implementation details.
This work has significant potential for creative industries (animation, gaming, film) by enabling more expressive and controllable audio generation. The ability to generate complex audio scenes with speech and effects from natural language instructions is a powerful tool for content creation. However, the ease of voice cloning and impersonation raises ethical concerns regarding deepfakes and misinformation, which the authors should address in terms of safety guardrails. SwanTale presents a robust and unified framework for multi-speaker speech and audio generation, effectively bridging the gap between zero-shot cloning and instruct-based creative control through a sophisticated data pipeline and a novel VAE-MoE architecture, significantly advancing the state-of-the-art in expressive audio synthesis.
Audio reasoning is essential for machine understanding of the acoustic world. Reinforcement learning with verifiable rewards can elicit such reasoning, yet existing reward designs are complementary in their limitations: outcome-based rewards supervise only the final answer and let the model reach it without attending to the audio, whereas process-based rewards score the reasoning itself but rely on coarse, hand-crafted, and fixed criteria that neither adapt to each question nor stay grounded in the acoustic evidence. Moreover, questions differ in what they demand, with some hinging on perception and others on multi-step reasoning, and any static criterion weakens as the policy improves. Supervising the reasoning process with fine-grained, audio-grounded, and adaptive rewards is therefore crucial, yet challenging since such rewards are impractical to design by hand for every sample. To this end, we introduce AudioRubrics, a reinforcement learning framework that supervises audio reasoning with self-evolving, audio-grounded rubric rewards. AudioRubrics synthesizes per-sample rubrics from the raw waveform and, conditioned on the model's own rollouts, regenerates and reweights criteria per group, supplying a continuous learning signal that keeps targeting the current policy's weaknesses as static criteria saturate. Comprehensive evaluations across three audio reasoning benchmarks reveal that AudioRubrics substantially outperforms a wide range of open-source and training-based baselines. Furthermore, our analysis shows that the gains scale with the capability of the rubric generator and judge, and AudioRubrics converges to a stable reasoning length that avoids both degenerate collapse and unbounded growth. The improvement in audio perception further demonstrates the effectiveness of anchoring supervision in the acoustic evidence. Our project page is available at https://audiorubrics.github.io.
Primary: Microsoft Research
All Institutions: University of Maryland, College Park, University of Illinois Urbana-Champaign, University of Illinois Chicago, Microsoft Research, MBZUAI
AudioRubrics introduces a novel reinforcement learning framework that utilizes self-evolving, audio-grounded rubrics to provide dense, adaptive process-level rewards for audio reasoning, significantly outperforming static reward baselines on multiple benchmarks.
The paper proposes "AudioRubrics," a reinforcement learning framework designed to enhance the reasoning capabilities of Large Audio-Language Models (LALMs). The core innovation lies in replacing static, hand-crafted process rewards with self-evolving, audio-grounded rubrics. The methodology involves three key steps: (1) Initializing rubrics from raw audio waveforms using an audio-capable model to ensure grounding; (2) Evolving these rubrics during training by eliciting new criteria from the model's own rollouts, pruning non-discriminative ones (those with zero variance in judgments across a group), and re-weighting the survivors; (3) Combining these evolving rubric rewards with outcome accuracy and an overthinking penalty (to prevent verbose, hallucinated reasoning) within a Group Relative Policy Optimization (GRPO) framework. The approach addresses the saturation of static rewards and the lack of audio grounding in text-based process rewards. The technical design is sound, leveraging the heterogeneity of rollouts to generate adaptive evaluation criteria.
The authors evaluate AudioRubrics on three benchmarks: MMAU, MMAR, and MMSU. They compare against a wide range of proprietary (GPT-4o-Audio, Gemini) and open-source baselines (Qwen2-Audio, Phi-4-Multimodal, etc.), as well as other RL-based reasoning methods. The results indicate substantial improvements over baselines, particularly in multi-step reasoning tasks. The paper includes analysis showing that the rubrics evolve to become more sophisticated as training progresses, and that the gains scale with the capability of the rubric generator. The inclusion of an overthinking penalty is shown to stabilize reasoning length. The evaluation is comprehensive and covers diverse audio domains (speech, sound, music).
The paper provides a GitHub repository link and a Hugging Face collection for model weights and datasets. The methodology is described in detail, including the GRPO objective and the specific steps for rubric elicitation and filtering. However, the exact hyperparameters for the rubric generation model, the specific taxonomy of initial dimensions, and the precise implementation details of the "negative criteria" elicitation might require careful reading of the appendix (not fully provided here, but implied). The code availability significantly aids reproducibility.
The approach relies heavily on the quality of the rubric generator/judge model. If this model is biased or lacks audio understanding, the evolving rubrics may propagate errors. The method introduces additional computational overhead during training due to the continuous generation and evaluation of rubrics. The paper mentions that static criteria saturate, but does not extensively discuss the potential for "rubric collapse" where the model optimizes for a narrow set of evolving criteria that do not generalize. The reliance on an external judge model for rubric generation and judging introduces a dependency that might not be available in all settings.
This work contributes to the development of more robust and interpretable audio reasoning systems. By grounding reasoning in audio evidence and providing dense process-level feedback, it reduces hallucinations and improves the reliability of AI systems in critical applications like healthcare and multimedia analysis. The self-evolving nature of the rewards could inspire similar adaptive reward mechanisms in other multimodal reasoning tasks. AudioRubrics introduces a novel reinforcement learning framework that utilizes self-evolving, audio-grounded rubrics to provide dense, adaptive process-level rewards for audio reasoning, significantly outperforming static reward baselines on multiple benchmarks.
Token compression in OmniLLMs is typically posed as a single saliency-ranking problem: score each multimodal token, keep the top-K. We argue this abstraction is mis-specified. The same attention score simultaneously decides two things: how much retained capacity each modality receives, and which tokens within a modality are kept. A shared top-K rule therefore inherits this audio-favoring allocation prior, spending retained capacity on audio before video tokens have a chance to compete. We propose Macer, a training-free compressor that first assigns explicit audio and video budgets, then performs allocation-normalized ranking within each modality at modality-specific shallow layers. Macer significantly reduces token cost while preserving accuracy across audio-grounded, audio--video joint, visual-dominant, and video-centric benchmarks. At 25 % retention, Macer preserves 98.7 % of full-token performance on Qwen2.5-Omni-7B and 97.3 % on Qwen2.5-Omni-3B. On Qwen2.5-Omni-7B, this 25 % setting reaches OmniZip-level performance at 45 % retention while using lower FLOPs. On OmniVinci-9B, the same allocation-before-ranking principle improves over shared top-K ranking by up to 12.9 points.
Primary: University of Houston
All Institutions: University of Houston, The University of Texas at Arlington
The paper makes a significant contribution to multimodal efficiency by identifying and correcting a fundamental mis-specification in how shared-attention mechanisms handle token compression, proposing a theoretically grounded, training-free solution that significantly improves the efficiency-accuracy trade-off for OmniLLMs.
The paper proposes "Macer," a training-free token compression method for OmniLLMs (Omnimodal Large Language Models). The core theoretical contribution is the identification of a structural flaw in existing shared-decoder compression methods: the coupling of cross-modal capacity allocation and within-modality token selection via a single shared attention score. The authors argue that raw shared attention inherently favors audio tokens due to architectural priors, leading to suboptimal video retention. Macer addresses this by decoupling these two decisions. It first explicitly allocates a budget for audio and video tokens based on a user-defined ratio. Then, it performs modality-local ranking using "Allocation-Normalized Modality Scoring" (ANMS), which normalizes attention scores within each modality to remove the cross-modal mass bias. Furthermore, it leverages the observation that audio and video saliency become readable at different shallow layers, applying pruning at modality-specific depths ($L_a$ for audio, $L_v$ for video). A lightweight temporal coverage term is added for video to prevent temporal clustering. The approach is inference-time only, requiring no training or model modification.
The evaluation is conducted on three OmniLLM backbones: Qwen2.5-Omni-7B, Qwen2.5-Omni-3B, and OmniVinci-9B. Benchmarks cover audio-grounded, audio-video joint, visual-dominant, and video-centric tasks. The results claim significant performance preservation at low retention ratios (e.g., 98.7% of full performance at 25% retention on Qwen2.5-Omni-7B). The paper compares Macer against ToMe-style merging and random dropping, demonstrating that explicit budget allocation outperforms implicit budgeting induced by shared ranking. The ablation studies support the three key observations: (1) shared attention is not neutral, (2) budgeting is the primary driver of cross-modal balance, and (3) modality-specific readout layers are beneficial. The results are strong and directly validate the proposed mechanism.
The paper provides detailed mathematical formulations for the ANMS scores and the budget allocation logic. It specifies the hyperparameters used (readout layers, budget ratios). Since the method is training-free and applied to open-source models (Qwen2.5-Omni, OmniVinci), reproduction is feasible for researchers with access to the necessary compute. The paper mentions an appendix with additional details, which is standard for reproducibility.
The method relies on specific architectural assumptions of shared-decoder OmniLLMs where audio and video tokens are interleaved and processed by the same attention mechanism. Its applicability to encoder-decoder architectures or models with separate modality-specific encoders that feed into a unified decoder might require adaptation. The "training-free" nature means it cannot learn task-specific compression patterns that a trained compressor might, potentially capping its maximum efficiency compared to learned methods. The paper notes that the temporal coverage term is lightweight but does not fully address complex long-range video dependencies if the budget is extremely tight.
This work addresses a critical bottleneck in deploying OmniLLMs: inference cost and latency. By enabling efficient compression without retraining, it lowers the barrier to using these models in resource-constrained environments. The insight into modality bias in shared attention mechanisms is broadly applicable to the design and analysis of multimodal models, potentially influencing future model architectures to be more modality-agnostic or to handle allocation explicitly. The paper makes a significant contribution to multimodal efficiency by identifying and correcting a fundamental mis-specification in how shared-attention mechanisms handle token compression, proposing a theoretically grounded, training-free solution that significantly improves the efficiency-accuracy trade-off for OmniLLMs.
We present JoyAI-Talker, a full-duplex speech dialogue system that delivers robust foundation model capabilities while empowering empathetic interaction and voice agent intelligence. JoyAI-Talker adopts a modular Thinker-Talker architecture and further implements a unified speech-text joint training pipeline to mitigate the common "cognitive degradation" bottleneck, thereby largely preserving the model's core textual reasoning, STEM, and logical capabilities while extending them to speech-based interaction. For expressive speech synthesis, the Talker module employs a text-controllable generation paradigm that enables natural-language instructions to flexibly control vocal attributes and localized paralinguistic events, such as laughter and sighs, supporting more expressive and fine-grained speech responses. To enhance conversational empathy, we introduce the Persona-Adaptive Empathetic Response (PAER) framework. PAER employs a hierarchical cognitive pipeline to extract non-verbal speaker cues, such as gender, age, and emotional state, from raw input audio, incorporate them into the Thinker's CoT reasoning, and generate context-adaptive responses that align semantically appropriate text with fine-grained control over utterance-level expressiveness and localized paralinguistic events, including sighs, speaking rate, and volume. We further integrate Joy-Duplex, a state-driven, plug-and-play full-duplex framework that functions as an efficient gating engine for real-time turn control. Extensive evaluations show that JoyAI-Talker achieves highly competitive performance on foundational T2T and S2T benchmarks. In full-duplex evaluation, the system reaches a high response rate of 0.88 under user interruptions while maintaining an extremely low false-trigger rate under background speech, demonstrating its readiness for fluid and natural speech dialogue.
Primary: JD.com
All Institutions: JD.com
[One sentence main contribution]. [The paper presents JoyAI-Talker, a modular full-duplex speech dialogue system that integrates a large MoE language model with an expressive speech generator, utilizing a novel state-driven gating mechanism and a persona-adaptive empathy framework to achieve robust, low-latency, and emotionally intelligent voice interactions while preserving the reasoning capabilities of the underlying LLM.]
The paper proposes JoyAI-Talker, a full-duplex speech dialogue system built on a decoupled "Thinker-Talker" architecture. The Thinker is a 48.9B parameter MoE LLM (JoyAI-LLM Flash) adapted for speech via a projector, while the Talker is a speech generation model based on JoyVoice. Key methodological contributions include: 1) A unified speech-text joint training pipeline starting from mid-training to mitigate "cognitive degradation" in LLMs; 2) The Persona-Adaptive Empathetic Response (PAER) framework, which uses Chain-of-Thought reasoning to extract speaker attributes (gender, age, emotion) from audio and inject them into the response generation; 3) A state-driven full-duplex framework (Joy-Duplex) using a lightweight 1.7B decoder to predict interleaved text and state tokens (e.g., `<|accept|>`, `<|reject|>`) for turn-taking control, rather than relying on energy-based VAD. The approach is technically sound and addresses significant pain points in current LSLM (Large Speech Language Model) systems, particularly the trade-off between reasoning capability and acoustic alignment, and the fragility of full-duplex interaction. However, the core components (MoE LLMs, DiT-based TTS, CoT for empathy) are incremental combinations of existing state-of-the-art techniques rather than fundamentally new algorithmic breakthroughs.
The evaluation is extensive and covers multiple dimensions: T2T benchmarks (to verify reasoning retention), S2T benchmarks (ASR, translation, QA), empathy perception (AIR-Bench, MER2025), empathetic dialogue generation (EchoMind), and full-duplex interaction (Full-Duplex-Bench v1.5). The results show competitive performance on T2T and S2T tasks, demonstrating that the joint training did not catastrophically forget textual capabilities. The full-duplex evaluation highlights a strong balance between interruption response (0.88) and low false-trigger rates, which is a practical and significant achievement. The inclusion of a commercial baseline (Gemini 3.1 Live) adds credibility. However, the paper relies heavily on LLM-as-a-Judge for empathy metrics, which can be biased. The specific architectural novelty of the "Thinker-Talker" interface is standard in many recent systems (e.g., Qwen2-Audio, Moshi), though the specific implementation of the state-token gating is a notable engineering contribution.
The paper provides detailed descriptions of the training stages (Mid-training, Context Extension, SFT, DPO), the model architecture (MoE, MLA), and the data construction strategies (e.g., voice conversion for Talker adaptation). The use of standard frameworks (Megatron-Core) and public benchmarks enhances reproducibility. However, the specific "JoyAI-LLM Flash" backbone and the internal data mixes are proprietary to JD.com, making exact replication impossible without access to their internal models and data. The description of the Joy-Duplex state machine is clear enough for implementation.
The decoupled architecture introduces latency overhead compared to unified end-to-end models, as noted by the authors. The reliance on a separate 1.7B model for duplex control adds computational complexity. The empathy framework relies on the Thinker's ability to correctly infer attributes via CoT, which can fail if the audio is noisy or ambiguous, potentially leading to hallucinated persona traits. The "cognitive degradation" mitigation is empirical; while results are good, the theoretical guarantee is absent. Furthermore, the system's performance is heavily dependent on the quality of the ASR transcription passed to the Thinker, creating a potential error propagation bottleneck in the decoupled design.
This work contributes to the development of more natural and robust voice assistants, which has significant implications for human-computer interaction, accessibility, and companion AI. The focus on empathy and full-duplex interaction moves the field closer to seamless human-like dialogue. However, the deployment of such systems raises concerns about privacy (continuous audio processing), bias in empathetic responses (stereotyping based on inferred gender/age), and the potential for deceptive anthropomorphism. The open-weight nature of the underlying components (if released) could accelerate research in this area. [One sentence main contribution]. [The paper presents JoyAI-Talker, a modular full-duplex speech dialogue system that integrates a large MoE language model with an expressive speech generator, utilizing a novel state-driven gating mechanism and a persona-adaptive empathy framework to achieve robust, low-latency, and emotionally intelligent voice interactions while preserving the reasoning capabilities of the underlying LLM.]
Automatic Drum Transcription (ADT) is commonly formulated as a direct mapping from a music mixture to symbolic drum events. While effective for transcription, this formulation discards the acoustic stems that are useful for editing, remixing, and production. We revisit an alternative separate-and-detect formulation, where a drum source separation front end first produces five editable drum stems, and a fixed onset detector then converts each stem into symbolic events. The separator is built on a five-stem latent diffusion model that jointly generates kick, snare, toms, hi-hats, and cymbals in a compact VAE latent space. We further study two training-only auxiliary branches--an onset branch (OB) and a timbre branch (TB)--which shape the separator during learning but are discarded at inference. Trained on synthetic drum multitracks and evaluated on MDB Drums and ENST-Drums, the proposed pipeline consistently improves over a strong U-Net-based drum separation baseline in overall transcription F1. It also outperforms a representative end-to-end ADT system on kick and snare F1 under our evaluation protocol, while additionally providing separated audio stems. The ablation results show that OB gives the most stable transcription gains, whereas TB changes the trade-off between reconstruction, acoustic stem quality, and onset detection. These results suggest that generative drum demixing can serve not only as a source separation model, but also as a practical front end for interpretable drum transcription.
Primary: National Taiwan University
All Institutions: National Taiwan University, AI Center of Research Excellence (AI-CoRE), Data Science Degree Program, Institute of Information Science, Academia Sinica, Rhythm Culture Corporation
The paper presents a novel "separate-and-detect" framework for automatic drum transcription that leverages a latent diffusion model with training-only auxiliary branches to optimize for both audio separation quality and transcription accuracy, demonstrating significant improvements over existing baselines on standard datasets.
The paper proposes a "separate-and-detect" pipeline for Automatic Drum Transcription (ADT). Instead of direct event prediction, it uses a five-stem latent diffusion model (based on MSG-LD) to separate drum stems (kick, snare, toms, hi-hats, cymbals) from a drum-dominant mixture, followed by a fixed onset detector. The core methodological contribution is the introduction of two training-only auxiliary branches: an Onset Branch (OB) that encourages sparse percussive structure, and a Timbre Branch (TB) that regularizes acoustic features using DrumGAN descriptors. This design aims to optimize the generative model not just for audio reconstruction, but for downstream transcription utility. The approach is technically sound, leveraging the strengths of diffusion models for high-fidelity separation while addressing the specific needs of rhythmic transcription through auxiliary supervision.
The evaluation is conducted on MDB Drums and ENST-Drums datasets. The authors compare their method against a strong U-Net baseline (LarsNet) and an end-to-end ADT system (ADTOF). Results show that the proposed method, particularly with the +OB auxiliary branch, significantly improves overall transcription F1 over LarsNet and outperforms ADTOF on kick and snare classes. The ablation study effectively isolates the impact of the auxiliary branches, showing that OB provides stable transcription gains, while TB influences the trade-off between reconstruction quality and timbral accuracy. The use of Fréchet Audio Distance (FAD) and mel-MSE provides a comprehensive view of separation quality, complementing the transcription metrics.
The paper provides clear descriptions of the dataset preprocessing, model architecture modifications, and training details. The code and demo are publicly available on GitHub and a dedicated website, which significantly enhances reproducibility. The use of standard datasets (MDB, ENST) and established baselines (ADTOF, LarsNet) allows for fair comparison. The specific hyperparameters for the onset detector tuning are mentioned, though the exact random seeds are not explicitly detailed in the text, the public code likely contains them.
The authors acknowledge several limitations. The system operates at 16 kHz, which limits high-frequency content recovery, affecting hi-hats and cymbals. The iterative nature of diffusion sampling makes inference slower (approx. 25 seconds for 10 seconds of audio) compared to single-pass models. The method struggles with dense, broadband textures like cymbals due to inter-class confusion and VAE latent compression. Additionally, the reliance on a fixed upstream source separation model (Demucs) means errors in the initial drum extraction cannot be corrected by the downstream components.
This work contributes to the field of audio signal processing and music information retrieval by demonstrating that generative models can be optimized for symbolic transcription tasks. The ability to provide both editable stems and symbolic events is valuable for music production, education, and analysis. The "separate-and-detect" paradigm offers a new perspective on ADT, potentially inspiring similar hybrid approaches in other domains where both audio fidelity and symbolic interpretation are required. The paper presents a novel "separate-and-detect" framework for automatic drum transcription that leverages a latent diffusion model with training-only auxiliary branches to optimize for both audio separation quality and transcription accuracy, demonstrating significant improvements over existing baselines on standard datasets.
In this paper, we introduce noise-aware self-supervised learning (NA-SSL) models for noise-aware anomalous sound detection (NA-ASD). NA-ASD is an ASD task with two-channel audio recordings, where one microphone is located close to the target machine and the other is located farther away to capture noise. For this task, we simulate two-channel recordings using diverse audio datasets and train NA-SSL models to extract clean SSL representations of the close-microphone signal by using the far-microphone recording dominated by background noise as auxiliary information. The NA-SSL models are then used as frontends in the standard ASD framework. Our experimental evaluation on the DCASE 2026 Challenge Task 2 development dataset demonstrates the effectiveness of the NA-SSL framework across three base SSL models (BEATs, EAT, and Dasheng), both with and without discriminative fine-tuning. Furthermore, the challenge results proved the effectiveness of the proposed approach, where the NA-BEATs system won the challenge by a large margin, achieving an official score of 70.24%, while the second-place system achieved 65.46%.
Primary: Mitsubishi Electric Research Laboratories (MERL)
All Institutions: Mitsubishi Electric Research Laboratories (MERL), Nagoya University, Information Technology R&D Center, Mitsubishi Electric Corporation
The paper presents a robust and effective Noise-Aware Self-Supervised Learning framework that successfully leverages auxiliary noise information to improve Anomalous Sound Detection, achieving first place in the DCASE 2026 Challenge.
The paper proposes a Noise-Aware Self-Supervised Learning (NA-SSL) framework tailored for the Noise-Aware Anomalous Sound Detection (NA-ASD) task. The core methodology involves adapting existing Self-Supervised Learning (SSL) models (BEATs, EAT, Dasheng) by inserting trainable Noise-Aware (NA) layers that utilize cross-attention to condition the representation of the close-microphone signal on the far-microphone noise signal. This approach leverages the physical setup of the NA-ASD task, where a distant microphone captures ambient noise. The authors simulate two-channel recordings using diverse datasets (FSD50K, WHAM!, DEMAND, QUT-NOISE) to pre-train these NA-SSL models. The method is then integrated into a standard ASD pipeline with discriminative fine-tuning and various backend aggregation techniques (BEAM, RDP). While the application of cross-attention for noise conditioning is not entirely new in speech processing, its systematic adaptation and evaluation across multiple modern, large-scale audio SSL backbones for the specific NA-ASD challenge context represents a solid engineering contribution. The novelty lies in the specific integration strategy and the demonstration of its efficacy across different architectural families (masked prediction, masked bootstrapping, masked autoencoder).
The experimental evaluation is comprehensive and rigorous, centered on the DCASE 2026 Challenge Task 2. The authors evaluate their method on the development set with detailed ablation studies across different base models, fine-tuning strategies (with/without discriminative fine-tuning), and backend techniques. The results show consistent improvements over the original SSL models. Crucially, the paper reports the official challenge results, where the proposed NA-BEATs system won the challenge with a significant margin (70.24% vs 65.46% for second place). This real-world validation on a blind test set provides strong evidence of the method's effectiveness and robustness. The inclusion of variance analysis and ensemble results further strengthens the empirical claims. The comparison against other top-performing systems highlights the efficiency of the proposed approach, which achieves state-of-the-art performance using a single model rather than large ensembles.
The paper provides substantial implementation details, including the architecture of the NA layers (number of heads, hidden size), training hyperparameters (optimizer, learning rate, batch size, EMA decay), data simulation parameters (room impulse response settings, SNR ranges), and backend configurations. The use of standard libraries (Pyroomacoustics) and well-known base models facilitates reproduction. However, the specific random seeds for the "three trials" are mentioned but not explicitly listed in the text, which might introduce slight variability in exact replication. The code for the NA layers is not explicitly linked, though the architecture is described in sufficient detail for implementation.
The primary limitation is the reliance on simulated data for pre-training the NA-SSL models. While the simulation uses diverse datasets, the gap between simulated room acoustics and real-world factory environments may still exist. The method is specifically designed for the NA-ASD task with two-microphone setups; its applicability to single-microphone scenarios or other noise-aware tasks is not explored. Additionally, the performance gain, while significant, comes with the computational overhead of the additional NA layers and the need for the far-microphone signal, which may not always be available or feasible to deploy in all industrial settings. The paper does not extensively discuss the failure cases or scenarios where the far-microphone signal might be corrupted or uninformative.
This work contributes to the field of industrial anomaly detection, which has significant implications for predictive maintenance, safety, and efficiency in manufacturing. By improving the robustness of ASD systems in noisy environments, it enables more reliable deployment of automated monitoring systems. The noise-aware SSL framework could potentially be adapted to other audio processing tasks requiring robustness to environmental noise, such as speech recognition or audio event detection in challenging acoustic conditions. The paper presents a robust and effective Noise-Aware Self-Supervised Learning framework that successfully leverages auxiliary noise information to improve Anomalous Sound Detection, achieving first place in the DCASE 2026 Challenge.
Bandwidth extension (BWE) aims to recover missing high-frequency content from band-limited speech. Existing methods often formulate BWE as a fixed or predefined bandwidth conversion problem, potentially requiring cutoff-specific models or retraining when the input bandwidth changes. This assumption limits their applicability to practical scenarios where speech may arrive with diverse cutoff frequencies. We propose AnyBand, a unified BWE framework that recasts bandwidth extension as in-context spectral infilling. Motivated by prompt-based zero-shot speech generation, AnyBand conditions high-frequency generation on the observed low-frequency spectrum, using the available band as a frequency-domain prompt that conveys content, speaker, prosodic, and spectral-envelope cues. This formulation enables a single model to perform cutoff-conditioned generation over a continuous range of input bandwidths. AnyBand is trained with missing-band conditional flow matching and an Easy-to-Balanced cutoff curriculum over continuously sampled cutoff frequencies. To better exploit the spectral prompt, we introduce a frequency-aware Diffusion Transformer that models cross-frequency interactions and long-range temporal dependencies, followed by a physically motivated multi-view adversarial refinement stage to enhance spectral realism, envelope coherence, and harmonic consistency. Experiments on multiple datasets and bandwidth settings show that AnyBand consistently improves spectral reconstruction over existing baselines while achieving competitive perceptual quality across both standard and irregular input cutoffs. Audio samples are available.
Primary: National University of Singapore
All Institutions: National University of Singapore, Unknown
[One sentence main contribution]. AnyBand introduces a unified, in-context spectral infilling framework for speech bandwidth extension using a frequency-aware Diffusion Transformer and a novel cutoff curriculum. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a well-motivated and technically sophisticated approach to a long-standing problem in audio processing. By reframing BWE as a conditional generation task conditioned on a spectral prompt, it elegantly handles continuous bandwidth variations. The integration of frequency-aware modeling within a DiT architecture is a strong technical contribution that addresses the specific inductive biases required for spectral data. While the novelty is incremental in the context of the rapid evolution of diffusion models, the specific application to unified BWE with the proposed curriculum and adversarial refinement is valuable. The work is significant for its practical utility in creating flexible, single-model solutions for diverse bandwidth scenarios, potentially simplifying deployment in real-world applications.
The paper proposes AnyBand, a unified framework for speech bandwidth extension (BWE) that treats the task as in-context spectral infilling. The core innovation lies in conditioning a Diffusion Transformer (DiT) on the observed low-frequency spectrum as a "frequency-domain prompt" to generate the missing high-frequency content. The architecture augments a temporal DiT backbone with explicit frequency encoders/decoders to model cross-frequency interactions. The training strategy employs a novel "Easy-to-Balanced" cutoff curriculum to handle continuously varying cutoff frequencies, avoiding the need for separate models per bandwidth. Additionally, a multi-view adversarial refinement stage is introduced to enhance spectral realism, envelope coherence, and harmonic consistency. The approach is technically sound, leveraging recent advances in flow matching and diffusion transformers, and applies the concept of in-context learning (typically seen in LLMs) to the spectral domain.
The authors evaluate AnyBand on VCTK and EARS datasets, comparing against baselines like NU-Wave 2, AudioSR, FLowHigh, and Fre-Painter. Metrics include LSD, LF-LSD, HF-LSD, NISQA, COL, and STOI. The results show consistent improvements in spectral reconstruction (lower LSD) and competitive perceptual quality (NISQA) across various input cutoffs (2kHz, 4kHz, etc.). The experiments are comprehensive, covering both in-domain and out-of-domain settings. However, the reliance on objective metrics like NISQA and LSD, while standard, lacks extensive subjective human evaluation (MOS/MUSHRA) which is crucial for perceptual quality claims in audio restoration. The comparison is fair, but the marginal gains in some perceptual metrics over strong baselines like AudioSR warrant caution in claiming a transformative leap.
The paper provides detailed architectural descriptions, loss functions, and training procedures. The inclusion of a demo page suggests code or at least pre-trained models might be available, though no GitHub URL is explicitly provided in the text. The "Easy-to-Balanced" curriculum and specific discriminator designs are described with sufficient mathematical detail for reproduction. The use of standard solvers (Heun) and common datasets enhances reproducibility.
The method assumes the input low-frequency spectrum is clean and correctly aligned; it does not address blind BWE where the cutoff frequency is unknown or the input is degraded by noise/compression beyond simple band-limiting (though BABE is cited as related work, AnyBand itself requires the mask/cutoff extent). The computational cost of the DiT backbone and adversarial refinement may be higher than lightweight CNN-based BWE systems. The paper does not provide extensive ablation studies on the contribution of each discriminator component or the specific impact of the frequency encoder/decoder versus a standard DiT.
AnyBand has significant potential for improving speech quality in legacy communication systems, audio restoration, and voice conversion where bandwidth constraints are dynamic. By unifying multiple bandwidth settings into a single model, it reduces the engineering overhead of deploying multiple specialized models. However, like all generative audio tools, it carries risks of misuse in deepfake generation or misrepresentation, although BWE is generally considered a benign enhancement task. [One sentence main contribution]. AnyBand introduces a unified, in-context spectral infilling framework for speech bandwidth extension using a frequency-aware Diffusion Transformer and a novel cutoff curriculum. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a well-motivated and technically sophisticated approach to a long-standing problem in audio processing. By reframing BWE as a conditional generation task conditioned on a spectral prompt, it elegantly handles continuous bandwidth variations. The integration of frequency-aware modeling within a DiT architecture is a strong technical contribution that addresses the specific inductive biases required for spectral data. While the novelty is incremental in the context of the rapid evolution of diffusion models, the specific application to unified BWE with the proposed curriculum and adversarial refinement is valuable. The work is significant for its practical utility in creating flexible, single-model solutions for diverse bandwidth scenarios, potentially simplifying deployment in real-world applications.