In this report, we introduce Qwen-Music, a powerful music generation model capable of producing highly musical and high-fidelity songs with complete vocal singing. Qwen-Music supports two core tasks: Text to Music Generation, which create entirely new songs from text descriptions, lyrics, and musical attributes, and Cover Song Generation, which reinterprets existing songs with different styles and vocal characteristics. Architecturally, Qwen-Music integrates three core components: Qwen-Music-Tokenizer, Qwen-Music-LLM, and Qwen-Music-Render. Qwen-Music-Tokenizer compresses audio into a 25 Hz single-codebook stream of Music Semantic Tokens that preserve semantic and melodic information for LLM prediction. Based on these tokens, Qwen-Music-LLM performs autoregressive music semantic modeling, with a key novelty being a melody-token-based chain-of-thought (Melody-CoT) mechanism that plans melodies before full-song generation, improving creativity, musicality, structural coherence, and reference-audio-based melody cloning. To overcome the fidelity limitations of discrete semantic tokens, Qwen-Music-Render performs generative stereo rendering, enriching acoustic details and producing high-fidelity stereo waveforms. Finally, we train Qwen-Music-LLM on more than 5 million hours of multilingual music data covering hundreds of languages. We first apply quality-aware pre-training curriculum, then use progressive post-training, comprising supervised initialization, offline DPO, and online GSPO, to further improve musicality and instruction-following ability. Across 600 Chinese and English prompts, Qwen-Music achieves state-of-the-art results in 13 of 16 objective musicality and audio-quality metrics. Professional evaluators also prefer Qwen-Music over leading proprietary systems. For cover song generation, Qwen-Music preserves reference melodies more accurately than leading proprietary systems.
Primary: Qwen Team
All Institutions: Qwen Team
This paper introduces Qwen-Music, a state-of-the-art music generation system that effectively bridges semantic composition and acoustic rendering through a novel Melody-CoT mechanism and a high-fidelity generative renderer, achieving superior performance in both objective metrics and human preference tests.
The paper presents a comprehensive three-stage architecture for music generation: a semantic tokenizer, an autoregressive language model (LLM), and a generative renderer. The core methodological contributions lie in the specific design choices for handling the complexity of full-song generation. First, the **Qwen-Music-Tokenizer** employs a four-stage training recipe (BestRQ pretraining, causal adaptation, multi-task SFT, and VQ insertion) to create a 25 Hz single-codebook stream. This is a significant engineering contribution, as it balances semantic richness with computational tractability for LLMs. Second, the **Melody-CoT** mechanism is a notable innovation. By explicitly planning melody tokens before generating full-mixture semantic tokens, the model addresses the "composition vs. rendering" mismatch. This chain-of-thought approach allows for better structural coherence and enables cover song generation by conditioning on reference melody tokens while allowing style/timbre changes. The use of relative MIDI representations for melody tokens is a smart design to prevent leakage of absolute pitch/key information that might conflict with style tags. Third, the **Qwen-Music-Render** utilizes a Diffusion Transformer (DiT) conditioned on both semantic tokens and textual descriptions, followed by a novel **Spec-VAE** and **Band-Mode Refiner**. The refiner’s frequency-adaptive correction (phase-only for low, joint for mid, magnitude-only for high) is a sophisticated touch to address specific spectral artifacts. The training pipeline is also rigorous, featuring a quality-graded pre-training curriculum (Q1-Q6 buckets) and a progressive post-training alignment (SFT -> Offline DPO -> Online GSPO), which is state-of-the-art practice for aligning generative models with human preference.
The evaluation is extensive and robust. The authors conduct blind A/B preference tests with 50 professional human raters, showing Qwen-Music outperforms leading proprietary systems (Suno V5.5, MiniMax, Mureka) with a 59.1% win rate. They also report performance on the Artificial Analysis leaderboard. Objective evaluations cover 16 metrics across SongBench, SongEval, and AudioBox-Aesthetic, where Qwen-Music leads in 13/16 metrics. The cover song generation evaluation is particularly strong, demonstrating superior melody preservation compared to baselines. The inclusion of a detailed acoustic data quality pipeline for training data filtering (detecting fake stereo, noise floors, cutoffs) adds credibility to the high-fidelity claims. The ablation studies on the renderer and the effectiveness of the Melody-CoT mechanism are well-supported by the results.
The paper provides detailed architectural specifications, including layer counts, parameter sizes, and training objectives. The tokenizer training stages are clearly defined. The post-training pipeline (SFT, DPO, GSPO) is described with sufficient detail for replication by other research groups. However, the specific "quality-aware" reward models and the exact composition of the 5 million-hour dataset are proprietary and not fully disclosed, which limits exact reproducibility of the pre-training phase. The code for the tokenizer and renderer is not explicitly linked in the provided text, though the Qwen team typically releases code for their models.
The paper relies heavily on proprietary baselines (Suno, MiniMax) for comparison, which makes independent verification difficult. The "quality-graded" pre-training assumes the availability of a reliable internal reward model, which may not be generalizable to all settings. The Melody-CoT mechanism, while effective, adds complexity and inference time. The model's performance on non-vocal music or highly abstract musical forms (e.g., free jazz, classical symphonies without lyrics) is less emphasized compared to vocal pop/rock genres, which dominate the training data. The reliance on a single codebook for the tokenizer might limit the expressiveness compared to multi-codebook approaches (like Encodec) in capturing fine-grained timbral details, although the renderer is designed to mitigate this.
Qwen-Music represents a significant step forward in accessible, high-fidelity music generation. By open-sourcing the technical report and potentially the model (implied by the Qwen lineage), it democratizes access to state-of-the-art music AI. The ability to generate cover songs and control musical attributes empowers creators. However, the potential for misuse in generating copyrighted material or deepfake vocals remains a concern, necessitating robust safety filters and watermarking, which are not detailed in this technical report. The high computational requirements for training such large models also contribute to the centralization of AI capabilities. This paper introduces Qwen-Music, a state-of-the-art music generation system that effectively bridges semantic composition and acoustic rendering through a novel Melody-CoT mechanism and a high-fidelity generative renderer, achieving superior performance in both objective metrics and human preference tests.
Current voice AI benchmarks typically evaluate isolated capabilities such as speech intelligibility, word error rate, or text-based dialogue quality, but they rarely test whether systems harness the acoustic information that distinguishes spoken language from its textual representation. To this end, we introduce the Real World Voice EQ Bench, a multidimensional benchmark for evaluating voice AI across text-to-speech (TTS), speech-to-speech (STS), speech understanding (SU), and automatic speech recognition (ASR). Our evaluations indicate that performance is highly dimension-specific. For TTS, naturalness, expressiveness, identity stability, and reliability are largely independent evaluation dimensions. For STS, access to audio does not guarantee use of vocal affect, and some agents remain largely transcript-driven. For SU, models perform unevenly across paralinguistic tasks. For ASR, real world accent, emotion, noise, and conversational conditions expose failures that are not captured by established clean-speech benchmarks. Together, these results show that voice AI should be evaluated as a profile of acoustic, expressive, interactional, and robustness capabilities rather than by a single aggregate score.
Primary: Hume AI Research
All Institutions: Hume AI Research
The paper presents RW-Voice-EQ Bench, a rigorous, multidimensional benchmark that reveals the dimension-specific nature of voice AI performance, exposing critical gaps in current evaluation practices such as benchmark optimization in ASR and the underutilization of paralinguistic cues in STS systems.
The paper introduces a comprehensive, multidimensional benchmarking framework called RW-Voice-EQ Bench, designed to evaluate voice AI systems across four distinct domains: Text-to-Speech (TTS), Speech-to-Speech (STS), Speech Understanding (SU), and Automatic Speech Recognition (ASR). The methodology is rigorous, separating linguistic content from paralinguistic acoustic cues. For perceptual tasks (TTS, STS), it employs large-scale human evaluation (over 1 million ratings) using specific rubrics and mixed-effects models to control for rater bias. For objective tasks (SU, ASR), it uses label-based scoring. A significant methodological contribution is the diagnostic suite for "benchmark optimization" (or "benchmaxxing") in ASR, which tests for memorization via consensus disagreement, audio masking, orthographic switching, and synthetic voice probes. The paper also validates Speech-Language Models (SLMs) as judges against human raters, providing a nuanced analysis of where automated judges succeed (verification-dependent tasks) and fail (open-ended perceptual tasks).
The experimental evaluation is extensive, covering 31 TTS configurations, 15 STS configurations, and analyzing top ASR models on the Hugging Face leaderboard. The results clearly demonstrate that performance is highly dimension-specific; no single model dominates all categories. For instance, in TTS, models strong in expressiveness (e.g., Gemini 3.1 Flash) differ from those strong in reliability or long-form stability (e.g., Kokoro-82M). In STS, the paper highlights that many systems remain largely transcript-driven, failing to leverage vocal affect even when present. The ASR analysis provides compelling evidence that high WER scores on clean benchmarks do not correlate with robustness to accents, noise, or emotional speech, and that many top models exhibit signs of overfitting to benchmark artifacts. The statistical analysis using mixed-effects models and Spearman correlations adds robustness to the findings.
The paper provides significant detail on the evaluation setup, including the number of raters, rating scales, and the specific dimensions evaluated. The benchmark and judge leaderboards are hosted on Hugging Face Spaces, ensuring accessibility. However, the full dataset of prompts and the specific code for the mixed-effects modeling pipeline are not explicitly linked in the provided text, though the Hugging Face links suggest open resources. The detailed description of the "benchmark optimization" diagnostics allows for replication of those specific tests.
The primary limitation is the reliance on human evaluation for TTS and STS, which is resource-intensive and potentially subject to demographic biases, although the paper attempts to mitigate this with large sample sizes and rater filtering. The STS evaluation, while innovative, is limited to specific scripted scenarios (hostile, urgent, ambiguous tone) and may not capture the full breadth of spontaneous conversational dynamics. The ASR benchmark optimization analysis is preliminary, focusing on VoxPopuli, and the authors acknowledge the need for future work to expand this to other datasets. Additionally, the SLM judge validation is currently limited to TTS, leaving STS judge reliability less explored.
This paper has substantial impact on the voice AI community by challenging the reliance on single aggregate scores and clean-speech benchmarks. It provides a roadmap for more realistic and robust evaluation of voice systems, which is critical for deployment in real-world applications. By exposing benchmark optimization in ASR, it encourages the development of more generalizable models and the use of withheld test sets. The multidimensional profile approach helps developers select models based on specific use-case requirements (e.g., customer service vs. entertainment), driving more targeted research and development. The paper presents RW-Voice-EQ Bench, a rigorous, multidimensional benchmark that reveals the dimension-specific nature of voice AI performance, exposing critical gaps in current evaluation practices such as benchmark optimization in ASR and the underutilization of paralinguistic cues in STS systems.
We introduce the REAL-TSE Challenge, an IEEE SLT 2026 satellite challenge on target speaker extraction~(TSE) from real conversational recordings. Given a multi-speaker mixture and one or more enrollment utterances from a target speaker, participating systems must recover only the target speech. Unlike simulated read-speech benchmarks, REAL-TSE evaluates Mandarin and English recordings that contain natural overlap, reverberation, noise, channel mismatch, and conversational dynamics. The challenge defines two complementary tracks: an Online track for low-latency streaming extraction and an Offline track for full-context processing. Systems are evaluated with Token Error Rate (TER), Speaker Similarity (SpkSim), DNSMOS, and target-speaker activity F1. This overview paper describes the task definition, datasets, baselines, evaluation protocol, submitted systems, condition-wise findings, and lessons for future real-world TSE benchmarks.
Primary: Nanjing University
All Institutions: Nanjing University, Chinese University of Hong Kong (Shenzhen), Brno University of Technology, Northwestern Polytechnical University, Shanghai Jiao Tong University, NTT, Inc.
The REAL-TSE Challenge establishes a critical new benchmark for real-world Target Speaker Extraction, revealing significant gaps between simulated and real-world performance and highlighting the dangers of metric over-optimization in neural audio evaluation.
This paper introduces the REAL-TSE Challenge, a benchmarking framework rather than a novel algorithmic architecture. The methodological contribution lies in the rigorous definition of a real-world Target Speaker Extraction (TSE) task. It distinguishes itself from prior simulated benchmarks (like LibriMix) by utilizing real conversational recordings with natural overlap, reverberation, noise, and channel mismatch. The challenge is structured into two tracks: Online (low-latency, <100ms) and Offline (full-context). The evaluation methodology is multi-dimensional, employing Token Error Rate (TER), Speaker Similarity (SpkSim), DNSMOS-P808, and Target-Speaker Activity F1. A critical methodological insight presented is the detection of metric over-optimization, specifically regarding DNSMOS-OVRL, leading to the adoption of DNSMOS-P808 for ranking. The paper details the data simulation strategies used by participants (pseudo-labeling, real-data adaptation) and analyzes the impact of channel mismatch and scenario types on performance.
The experimental section provides a comprehensive analysis of 24 systems from 18 teams. It includes baseline results (BSRNN variants) and aggregate performance metrics. The paper conducts a detailed condition-wise analysis on the EVAL-2 set, examining performance across different recording devices (H1, H2, Phone), target-speaker activity ratios, and recording scenarios (meeting, cafe, home, vehicle). Key findings include the significant impact of mixture recording device quality, the lack of consistent correlation between nominal scenario labels and difficulty, and the trade-offs between different metrics (e.g., systems optimizing for SpkSim may suffer in TER). The analysis of DNSMOS reliability is a significant experimental contribution, demonstrating how neural metrics can be gamed without perceptual improvement.
The challenge promotes reproducibility and transparency. The authors release the evaluation toolkit, baseline code (WeSep-based), and the dataset splits (Development and Evaluation). The training policy is open but restricted regarding specific test-set leakage (forbidden to use dev/test splits of source corpora for training). The paper provides detailed descriptions of the latency verification process (perturbation-based) and metric computation. However, as a challenge overview, it does not provide full training code for the winning systems, which is standard for such venues, but it does release the infrastructure for future replication.
The primary limitation is the inherent difficulty of real-world evaluation: the lack of ground-truth clean targets for most real recordings makes absolute performance measurement challenging, relying instead on proxies like ASR and speaker similarity. The paper acknowledges that metric-aware over-optimization is a significant issue, suggesting that current automatic metrics may not fully align with human perception or true extraction quality. Additionally, the "unseen" nature of EVAL-2 means that generalization to entirely new, unrecorded environments remains an open question, though the challenge design attempts to mitigate this by using diverse, newly recorded data.
This work has a substantial impact on the speech processing community by shifting the focus from simulated to real-world TSE evaluation. It highlights the gap between lab-performance and real-world deployment, emphasizing the need for robustness to channel mismatch, noise, and conversational dynamics. The findings on metric reliability (DNSMOS) have broader implications for the entire field of audio quality assessment and neural metric usage. By providing a standardized benchmark and revealing the limitations of current approaches, it guides future research towards more practical, multi-objective optimization and better data simulation techniques. The REAL-TSE Challenge establishes a critical new benchmark for real-world Target Speaker Extraction, revealing significant gaps between simulated and real-world performance and highlighting the dangers of metric over-optimization in neural audio evaluation.
Audio mixing style encompasses the artistic and technical decisions a mix engineer makes, including level balancing, spatialization, and the choice, ordering, and parameterization of audio effects (FX) on each stem. FX chains are a key determinant of this style, yet existing approaches to modeling them remain limited. Some operate on stereo mixtures without explicit per-stem FX chain modeling, others fix the number or type of effects per track, and many require differentiable effect implementations or scarce multitrack datasets. We present StemFX, a framework that learns mixing style representations by autoregressively predicting variable-length FX chains on source-separated stems. A Transformer decoder predicts tokenized FX chains autoregressively, while a band-split multi-band CNN encoder with FiLM conditioning captures per-stem spectral structure. To enable large-scale paired training, we extract pseudo-stems from about 105K songs via source separation and augment them using MultiAFx, a toolkit unifying 85 audio effects from 7 Python libraries. Evaluated on mixing style retrieval, StemFX outperforms all baseline models across all tested chain lengths. On paired mixing style transfer, StemFX achieves the best spectral fidelity and the highest listener preference, over 4000 times faster than iterative optimization.
Primary: National Taiwan University
All Institutions: National Taiwan University
This paper presents a significant advancement in automatic mixing and audio representation learning by introducing a scalable, autoregressive framework for FX chain prediction that overcomes data scarcity through a novel pseudo-stem generation pipeline, achieving state-of-the-art performance in both style retrieval and transfer tasks.
The paper proposes StemFX, a novel framework for learning mixing style representations by treating FX chain prediction as an autoregressive sequence generation problem. The architecture combines a band-split multi-band CNN encoder with FiLM conditioning (incorporating hand-crafted mixing features) and a Transformer decoder. A key methodological innovation is the "StemFX Pipeline," which leverages source separation on large-scale unpaired datasets (FMA) to create pseudo-stems, which are then augmented with random FX chains using a unified toolkit (MultiAFx) to generate paired training data at scale. This approach circumvents the scarcity of real multitrack datasets with ground-truth FX metadata. The tokenization scheme for variable-length FX chains (effect names, parameter names, and quantized values) is well-designed for autoregressive modeling. The use of FiLM conditioning to inject explicit mixing statistics into the CNN layers is a strong technical choice that bridges the gap between low-level spectral features and high-level mixing decisions.
The experimental evaluation is comprehensive and robust. The authors evaluate on two tasks: mixing style retrieval and paired mixing style transfer. For retrieval, StemFX significantly outperforms baselines (AFx-Rep, Fx-Encoder++, CLAP, and a controlled contrastive variant BSFiLM-CL) across various chain lengths, demonstrating that the learned embeddings capture mixing style distinct from musical content. The ablation studies effectively isolate the contributions of the encoder architecture, FiLM conditioning, and the data pipeline. The style transfer experiments are particularly compelling, showing superior spectral fidelity (MRSTFT) and human preference (MUSHRA scores) compared to fixed-chain methods and iterative optimization techniques (ST-ITO). The claim of being 4000x faster than iterative optimization is a significant practical advantage. The use of out-of-domain effects (pedalboard) in evaluation tests generalization well.
The paper provides detailed implementation details, including hyperparameters, dataset sizes, and architecture specifications. The release of both the StemFX codebase and the MultiAFx toolkit on GitHub greatly enhances reproducibility. The description of the data generation pipeline, including source separation models and augmentation strategies, is clear enough for other researchers to replicate the training data generation process. The use of standard metrics (MRSTFT, MUSHRA) also aids in reproducibility of evaluation.
The authors acknowledge several limitations. The model is restricted to the 85 effects included in the MultiAFx toolkit; adding new effects requires retraining. The reliance on source separation introduces potential artifacts from the separation model (e.g., SCNet), which may propagate into the learned representations. The use of pseudo-stems from different songs (cross-song mixing) may introduce musical incoherence, although the authors argue this decorrelates content from style. The model currently operates on four stems (vocals, bass, drums, other), which is a coarser decomposition than professional multitracks. Finally, the training data uses random effect chains, which may not reflect the structured, genre-dependent processing choices of human engineers.
StemFX has significant potential impact on music production technology. By enabling efficient and accurate mixing style transfer and retrieval, it can assist producers in learning mixing techniques, automate parts of the mixing workflow, and enhance music information retrieval systems. The open-source release of the toolkit and models promotes further research in audio effect modeling and automatic mixing. The ability to generate interpretable FX chains also provides insights into the decision-making process of mix engineers, potentially serving as an educational tool. This paper presents a significant advancement in automatic mixing and audio representation learning by introducing a scalable, autoregressive framework for FX chain prediction that overcomes data scarcity through a novel pseudo-stem generation pipeline, achieving state-of-the-art performance in both style retrieval and transfer tasks.
Current voice AI benchmarks typically evaluate isolated capabilities such as speech intelligibility, word error rate, or text-based dialogue quality, but they rarely test whether systems harness the acoustic information that distinguishes spoken language from its textual representation. To this end, we introduce the Real World Voice EQ Bench, a multidimensional benchmark for evaluating voice AI across text-to-speech (TTS), speech-to-speech (STS), speech understanding (SU), and automatic speech recognition (ASR). Our evaluations indicate that performance is highly dimension-specific. For TTS, naturalness, expressiveness, identity stability, and reliability are largely independent evaluation dimensions. For STS, access to audio does not guarantee use of vocal affect, and some agents remain largely transcript-driven. For SU, models perform unevenly across paralinguistic tasks. For ASR, real world accent, emotion, noise, and conversational conditions expose failures that are not captured by established clean-speech benchmarks. Together, these results show that voice AI should be evaluated as a profile of acoustic, expressive, interactional, and robustness capabilities rather than by a single aggregate score.
Primary: Hume AI Research
All Institutions: Hume AI Research
The paper presents RW-Voice-EQ Bench, a rigorous, multidimensional benchmark that reveals the dimension-specific nature of voice AI performance, exposing critical gaps in current evaluation practices such as benchmark optimization in ASR and the underutilization of paralinguistic cues in STS systems.
The paper introduces a comprehensive, multidimensional benchmarking framework called RW-Voice-EQ Bench, designed to evaluate voice AI systems across four distinct domains: Text-to-Speech (TTS), Speech-to-Speech (STS), Speech Understanding (SU), and Automatic Speech Recognition (ASR). The methodology is rigorous, separating linguistic content from paralinguistic acoustic cues. For perceptual tasks (TTS, STS), it employs large-scale human evaluation (over 1 million ratings) using specific rubrics and mixed-effects models to control for rater bias. For objective tasks (SU, ASR), it uses label-based scoring. A significant methodological contribution is the diagnostic suite for "benchmark optimization" (or "benchmaxxing") in ASR, which tests for memorization via consensus disagreement, audio masking, orthographic switching, and synthetic voice probes. The paper also validates Speech-Language Models (SLMs) as judges against human raters, providing a nuanced analysis of where automated judges succeed (verification-dependent tasks) and fail (open-ended perceptual tasks).
The experimental evaluation is extensive, covering 31 TTS configurations, 15 STS configurations, and analyzing top ASR models on the Hugging Face leaderboard. The results clearly demonstrate that performance is highly dimension-specific; no single model dominates all categories. For instance, in TTS, models strong in expressiveness (e.g., Gemini 3.1 Flash) differ from those strong in reliability or long-form stability (e.g., Kokoro-82M). In STS, the paper highlights that many systems remain largely transcript-driven, failing to leverage vocal affect even when present. The ASR analysis provides compelling evidence that high WER scores on clean benchmarks do not correlate with robustness to accents, noise, or emotional speech, and that many top models exhibit signs of overfitting to benchmark artifacts. The statistical analysis using mixed-effects models and Spearman correlations adds robustness to the findings.
The paper provides significant detail on the evaluation setup, including the number of raters, rating scales, and the specific dimensions evaluated. The benchmark and judge leaderboards are hosted on Hugging Face Spaces, ensuring accessibility. However, the full dataset of prompts and the specific code for the mixed-effects modeling pipeline are not explicitly linked in the provided text, though the Hugging Face links suggest open resources. The detailed description of the "benchmark optimization" diagnostics allows for replication of those specific tests.
The primary limitation is the reliance on human evaluation for TTS and STS, which is resource-intensive and potentially subject to demographic biases, although the paper attempts to mitigate this with large sample sizes and rater filtering. The STS evaluation, while innovative, is limited to specific scripted scenarios (hostile, urgent, ambiguous tone) and may not capture the full breadth of spontaneous conversational dynamics. The ASR benchmark optimization analysis is preliminary, focusing on VoxPopuli, and the authors acknowledge the need for future work to expand this to other datasets. Additionally, the SLM judge validation is currently limited to TTS, leaving STS judge reliability less explored.
This paper has substantial impact on the voice AI community by challenging the reliance on single aggregate scores and clean-speech benchmarks. It provides a roadmap for more realistic and robust evaluation of voice systems, which is critical for deployment in real-world applications. By exposing benchmark optimization in ASR, it encourages the development of more generalizable models and the use of withheld test sets. The multidimensional profile approach helps developers select models based on specific use-case requirements (e.g., customer service vs. entertainment), driving more targeted research and development. The paper presents RW-Voice-EQ Bench, a rigorous, multidimensional benchmark that reveals the dimension-specific nature of voice AI performance, exposing critical gaps in current evaluation practices such as benchmark optimization in ASR and the underutilization of paralinguistic cues in STS systems.
We introduce the REAL-TSE Challenge, an IEEE SLT 2026 satellite challenge on target speaker extraction~(TSE) from real conversational recordings. Given a multi-speaker mixture and one or more enrollment utterances from a target speaker, participating systems must recover only the target speech. Unlike simulated read-speech benchmarks, REAL-TSE evaluates Mandarin and English recordings that contain natural overlap, reverberation, noise, channel mismatch, and conversational dynamics. The challenge defines two complementary tracks: an Online track for low-latency streaming extraction and an Offline track for full-context processing. Systems are evaluated with Token Error Rate (TER), Speaker Similarity (SpkSim), DNSMOS, and target-speaker activity F1. This overview paper describes the task definition, datasets, baselines, evaluation protocol, submitted systems, condition-wise findings, and lessons for future real-world TSE benchmarks.
Primary: Nanjing University
All Institutions: Nanjing University, Chinese University of Hong Kong (Shenzhen), Brno University of Technology, Northwestern Polytechnical University, Shanghai Jiao Tong University, NTT, Inc.
The REAL-TSE Challenge establishes a critical new benchmark for real-world Target Speaker Extraction, revealing significant gaps between simulated and real-world performance and highlighting the dangers of metric over-optimization in neural audio evaluation.
This paper introduces the REAL-TSE Challenge, a benchmarking framework rather than a novel algorithmic architecture. The methodological contribution lies in the rigorous definition of a real-world Target Speaker Extraction (TSE) task. It distinguishes itself from prior simulated benchmarks (like LibriMix) by utilizing real conversational recordings with natural overlap, reverberation, noise, and channel mismatch. The challenge is structured into two tracks: Online (low-latency, <100ms) and Offline (full-context). The evaluation methodology is multi-dimensional, employing Token Error Rate (TER), Speaker Similarity (SpkSim), DNSMOS-P808, and Target-Speaker Activity F1. A critical methodological insight presented is the detection of metric over-optimization, specifically regarding DNSMOS-OVRL, leading to the adoption of DNSMOS-P808 for ranking. The paper details the data simulation strategies used by participants (pseudo-labeling, real-data adaptation) and analyzes the impact of channel mismatch and scenario types on performance.
The experimental section provides a comprehensive analysis of 24 systems from 18 teams. It includes baseline results (BSRNN variants) and aggregate performance metrics. The paper conducts a detailed condition-wise analysis on the EVAL-2 set, examining performance across different recording devices (H1, H2, Phone), target-speaker activity ratios, and recording scenarios (meeting, cafe, home, vehicle). Key findings include the significant impact of mixture recording device quality, the lack of consistent correlation between nominal scenario labels and difficulty, and the trade-offs between different metrics (e.g., systems optimizing for SpkSim may suffer in TER). The analysis of DNSMOS reliability is a significant experimental contribution, demonstrating how neural metrics can be gamed without perceptual improvement.
The challenge promotes reproducibility and transparency. The authors release the evaluation toolkit, baseline code (WeSep-based), and the dataset splits (Development and Evaluation). The training policy is open but restricted regarding specific test-set leakage (forbidden to use dev/test splits of source corpora for training). The paper provides detailed descriptions of the latency verification process (perturbation-based) and metric computation. However, as a challenge overview, it does not provide full training code for the winning systems, which is standard for such venues, but it does release the infrastructure for future replication.
The primary limitation is the inherent difficulty of real-world evaluation: the lack of ground-truth clean targets for most real recordings makes absolute performance measurement challenging, relying instead on proxies like ASR and speaker similarity. The paper acknowledges that metric-aware over-optimization is a significant issue, suggesting that current automatic metrics may not fully align with human perception or true extraction quality. Additionally, the "unseen" nature of EVAL-2 means that generalization to entirely new, unrecorded environments remains an open question, though the challenge design attempts to mitigate this by using diverse, newly recorded data.
This work has a substantial impact on the speech processing community by shifting the focus from simulated to real-world TSE evaluation. It highlights the gap between lab-performance and real-world deployment, emphasizing the need for robustness to channel mismatch, noise, and conversational dynamics. The findings on metric reliability (DNSMOS) have broader implications for the entire field of audio quality assessment and neural metric usage. By providing a standardized benchmark and revealing the limitations of current approaches, it guides future research towards more practical, multi-objective optimization and better data simulation techniques. The REAL-TSE Challenge establishes a critical new benchmark for real-world Target Speaker Extraction, revealing significant gaps between simulated and real-world performance and highlighting the dangers of metric over-optimization in neural audio evaluation.
Machine-learnt models are increasingly used to predict ISO 3382-1 room acoustic parameters from sparse measurements, with reported coefficients of determination frequently above 0.85. This paper shows that such figures are often determined by the evaluation protocol rather than by the model. Using a multi-condition measurement campaign in a 264-seat conference hall and a 180-seat concert hall, three model families were evaluated under a factorial protocol ablation: validation splits either row-based or grouped by receiver position, and input features either including measured-at-test quantities or restricted to source-receiver geometry and environmental state. Row-based splits with measured-at-test inputs reproduce the high reported accuracies (mean $R^2$ 0.81 for the core parameters); grouping the splits by position and restricting inputs to information available at an unmeasured position reduces these to 0.09-0.57, reordering the apparent difficulty of parameter classes. A hybrid CNN evaluated with the target's own impulse response as input is shown to exploit it as a position fingerprint rather than as transferable acoustic information; training-only signal access yields no gain for any parameter tested, including reverberation time. Under the deployment-consistent protocol, the spread between Random Forest, the hybrid CNN, and inverse-distance weighting is an order of magnitude smaller than the spread between protocols for a fixed model; the learnt models retain a genuine advantage for sound strength and reverberation time, and the high accuracy of the original pipelines re-emerges as condition interpolation at measured positions (band means 0.80-0.88), a distinct and operationally useful task.
Primary: Alanya Alaaddin Keykubat University
All Institutions: Alanya Alaaddin Keykubat University
This paper provides a critical methodological audit of machine learning in room acoustics, demonstrating that reported high accuracies are often artifacts of evaluation protocols rather than model capability, and establishes a rigorous framework for distinguishing between condition interpolation and spatial prediction.
The paper employs a rigorous factorial ablation study to dissect the evaluation protocols of machine learning models in room acoustics. It introduces a taxonomy distinguishing between "condition interpolation" (predicting parameters at measured positions under varying environmental states) and "spatial prediction" (predicting parameters at unmeasured positions). The methodology involves re-evaluating existing pipelines (Random Forest, Hybrid CNN) under strict "deployment-consistent" protocols where validation folds are grouped by receiver position and inputs are restricted to geometry and environmental state. A key methodological insight is the demonstration that neural networks can exploit the target position's own impulse response as a "position fingerprint" rather than learning transferable acoustic features, particularly in venues with few measurement positions. This approach effectively isolates the source of high reported accuracy, revealing it to be largely an artifact of data leakage via correlated rows and privileged inputs.
The experimental design is robust, utilizing a multi-condition measurement campaign in two distinct halls (a concert hall and a conference hall). The authors systematically vary split designs (row-based vs. grouped) and input features (full vs. geometry-only). The results are compelling: while row-based splits with full inputs yield high $R^2$ scores (0.81), the deployment-consistent protocol reduces these significantly (0.09-0.57), with simple baselines like inverse-distance weighting often matching or exceeding complex neural networks for certain parameters (e.g., clarity). The paper provides a clear reordering of parameter difficulty, showing that sound strength and reverberation time are more predictable spatially than clarity parameters, which depend on fine-grained early reflection structures not captured by the available geometric features. The inclusion of a "privileged information" experiment further strengthens the argument by showing that training-time signal access does not confer a significant advantage for spatial prediction.
The paper demonstrates high commitment to reproducibility. The author explicitly states that evaluation scripts, split definitions, random seeds, and per-fold results will be deposited in a public repository. The underlying measurement data is available upon request. The detailed description of the model architectures, feature sets, and experimental protocols allows for independent verification. The clear distinction between the two evaluation modes and the provision of specific $R^2$ values for each configuration enhance the transparency of the findings.
The primary limitation is the scope of the data: the study is based on a single multi-condition campaign in only two halls. While the findings are likely generalizable to similar indoor environments, they may not hold for outdoor acoustics or highly complex, non-diffuse fields. The author acknowledges this and suggests cross-venue replication is needed. Additionally, the study focuses on specific ISO 3382-1 parameters; the generalizability to other acoustic metrics or perceptual qualities is not addressed. The network architecture, while effective for the task, is not novel in itself, and the paper's contribution lies in the evaluation rather than the model design.
This paper has significant implications for the field of data-driven acoustics and spatial audio. By exposing the fragility of high accuracy claims in sparse measurement scenarios, it urges the community to adopt more rigorous evaluation standards. The proposed reporting checklist provides a practical framework for future research, ensuring that claims about spatial prediction are substantiated by appropriate protocols. This work helps prevent the dissemination of misleading performance metrics and guides the development of more robust and truly generalizable models for room acoustic prediction. It also highlights the enduring value of simple baselines and the importance of understanding what models are actually learning. This paper provides a critical methodological audit of machine learning in room acoustics, demonstrating that reported high accuracies are often artifacts of evaluation protocols rather than model capability, and establishes a rigorous framework for distinguishing between condition interpolation and spatial prediction.
Recent advances in text-to-speech and voice cloning make high-quality spoofing inexpensive and scalable, threatening voice authentication systems, especially automatic speaker verification (ASV). Existing defenses mainly address this threat through binary countermeasures (CMs) for deepfake detection or spoofing-aware speaker verification (SASV), where current systems are dominated by modular ASV-CM fusion and cascaded pipelines. Although large audio language models (LALMs) have shown promise on related audio tasks, including CM and ASV, their use for SASV remains unexplored, despite their capacity to produce natural-language rationales for auditing and robustness beyond discriminative predictions. This work systematically evaluates LALMs for SASV against conventional pipelines under zero-shot prompting, supervised adaptation, reasoning-oriented training, and reinforcement-learning-based optimization. Our results show that pretrained LALMs are near chance in the zero-shot setting, confirming that they are not natively suited to SASV, but that task-specific adaptation closes this gap. We further find that competitive SASV performance can be achieved through several distinct routes. These findings position LALMs as a promising and auditable foundation for unified SASV, while clarifying where conventional cascade systems still lead.
Primary: Applied AI Institute
All Institutions: Applied AI Institute, MIRAI, HSE, MTUCI
This paper provides the first systematic evaluation of Large Audio Language Models for Spoofing-Aware Speaker Verification, demonstrating that while they can be adapted to compete with conventional pipelines, they currently lag in threshold-based metrics and require significant task-specific adaptation to overcome their lack of native security awareness.
The paper proposes a novel adaptation of Large Audio Language Models (LALMs) for Spoofing-Aware Speaker Verification (SASV), a task that traditionally relies on cascaded or fused discriminative pipelines (ASV + CM). The authors formulate SASV as a three-way classification problem (target, non-target, spoof) using LALMs like SALMONN and Qwen-Audio. Key methodological contributions include: 1) A composite loss function combining cross-entropy, Additive Angular Margin (AAM) for speaker discrimination, and binary cross-entropy for spoof detection to balance the competing objectives. 2) Hard-sample mining to focus training on difficult pairs. 3) An exploration of Chain-of-Thought (CoT) reasoning and Group Relative Policy Optimization (GRPO) to generate interpretable rationales for decisions. The approach is technically sound, leveraging parameter-efficient fine-tuning (LoRA) and addressing the specific challenge of balancing speaker identity verification with anti-spoofing detection within a generative framework.
The evaluation is conducted on the ASVspoof 5 dataset, a challenging benchmark for SASV. The results demonstrate that while zero-shot LALMs perform near chance, supervised fine-tuning (SFT) significantly improves performance. The proposed method with composite losses and hard mining achieves competitive results against strong baselines (ECAPA-TDNN + WavLM/W2V2-AASIST fusion), particularly in terms of overall accuracy and min a-DCF. However, the conventional baselines still outperform the LALM in EER, which the authors attribute to the lack of calibrated continuous scores in the token-based LALM output. The ablation studies effectively isolate the contribution of each component (AAM head, spoof head, hard mining). The reasoning component, while interesting, did not yield performance gains over hard-label SFT, which is a significant finding but limits the technical impact of that specific branch.
The paper provides sufficient detail regarding model architectures (SALMONN-7B, Qwen2-Audio-7B), training hyperparameters (LoRA rank, learning rate, batch size), and data processing (resampling, augmentation). The dataset (ASVspoof 5 + VoxCeleb) is standard. However, the code and pre-trained checkpoints are not mentioned as available, and the specific cold-start reasoning generation process using Step-Audio R1 is described but the model weights are not provided, making full reproduction of the CoT component difficult. The evaluation on a stratified sample of 20K pairs rather than the full set is noted, which aids speed but requires careful interpretation of metrics.
The primary limitation is the lack of performance superiority over established discriminative baselines in key metrics like EER. The LALM approach is computationally more expensive and slower than conventional pipelines. The reasoning traces, while interpretable, are not validated for correctness or utility in a human-evaluation setting, and the authors admit they are prone to hallucination. The "reasoning" capability does not translate to better hard-label accuracy, questioning the practical value of the CoT component for the core task. The reliance on external models (Step-Audio R1) for data generation introduces a dependency that is not fully analyzed.
This work opens a new direction for using LALMs in security-critical audio applications, offering the potential for interpretable decision-making which is crucial for auditing voice authentication systems. It highlights the gap between general audio understanding and specific security tasks. The findings help guide future research on whether reasoning capabilities can be effectively integrated into security systems or if discriminative models remain superior for raw performance. It contributes to the broader understanding of LALM capabilities and limitations in specialized domains. This paper provides the first systematic evaluation of Large Audio Language Models for Spoofing-Aware Speaker Verification, demonstrating that while they can be adapted to compete with conventional pipelines, they currently lag in threshold-based metrics and require significant task-specific adaptation to overcome their lack of native security awareness.
Automatic symbolic music analysis has made substantial progress, yet existing systems are typically designed for a single mode of use, such as full-score prediction, and therefore do not match the broader range of operations that arise in analysis workflows, including partial completion, local correction, and iterative refinement. As a result, there remains a gap between strong benchmark models and systems that can support interactive analytical use. We present a unified framework for symbolic Roman-numeral (RN) analysis that narrows this gap by combining strong predictive performance with direct support for constrained completion and revision. The method is designed to provide a practical trade-off between accuracy and interactive responsiveness by computing expensive pretrained representations once and reusing them during iterative refinement, making powerful pretrained models more amenable to interactive settings. It supports complete score analysis, targeted revision of existing labels, and inference of missing annotations from partial context through a shared modeling framework. Experiments on Dilemmadata, the largest and most heterogeneous benchmark of its kind, show that the proposed approach is a strong RN-analysis baseline while also supporting masked completion from partial labels. Together with a prototype interface for multi-level candidate inspection and editing, these results position automatic RN analysis not only as a prediction problem, but also as a foundation for future interactive tools for music analysis.
Primary: Johannes Kepler University Linz
All Institutions: Johannes Kepler University Linz
This paper presents a unified framework for interactive Roman Numeral analysis that effectively bridges the gap between high-accuracy predictive models and practical, analyst-in-the-loop workflows through a hybrid MusicBERT-GNN architecture and edit-conditioned masked prediction.
The paper proposes a hybrid architecture combining a pretrained MusicBERT sequence encoder with a Graph Neural Network (GNN) for symbolic Roman Numeral (RN) analysis. The core methodological contribution is the "prediction-to-analysis continuum," which unifies blind full-score prediction, masked completion, and interactive revision within a single framework. The authors introduce an edit-conditioned masked-prediction model that uses label-conditioning modules to inject known labels as biases, allowing for constrained completion. They also employ conflict-aware optimizers (PCGrad, CAGrad) to handle gradient conflicts in the multi-task learning setup (20 tasks). The approach is technically sound, leveraging the complementary strengths of sequence modeling (context) and graph modeling (structural coherence). However, the architectural novelty is incremental, primarily consisting of combining existing strong components (RNBert + AnalysisGNN) and adding a specific conditioning mechanism for interactivity.
The evaluation is conducted on "Dilemmadata," a new benchmark combining AugmentedNet and Distant Listening Corpus. The experiments demonstrate that the hybrid model achieves state-of-the-art or competitive results on blind full-score inference compared to baselines like RNBert and AnalysisGNN. Crucially, the paper provides empirical evidence for the interactive use case: masked completion accuracy improves monotonically as more contextual labels are provided, validating the model's ability to leverage partial information. The inclusion of a prototype interface and the analysis of post-hoc decoding strategies (voters, beam search) adds practical value. The results are robust, though the paper notes that post-hoc decoding modules do not improve blind inference, highlighting the specific design for the interactive regime.
The paper provides a GitHub repository and a demo URL. The methodology is described in sufficient detail, including the tokenization scheme (REMI), graph construction, and training objectives (multi-task cross-entropy, KL distillation, L2-SP). The use of standard optimizers and augmentation techniques aids reproducibility. The definition of the Dilemmadata benchmark and the specific masking protocols are clearly outlined, facilitating future comparisons.
The paper acknowledges that the evaluation of masked completion uses randomly sampled masks rather than realistic analyst correction patterns, which may overestimate performance in real-world interactive settings. Additionally, the post-hoc decoding modules (voters, beam search) did not improve blind inference accuracy, suggesting they are specialized for the interactive regime and their utility is context-dependent. The study is primarily technical and lacks user studies with musicologists to quantify the actual benefit of the interactive workflow.
This work significantly advances the field of symbolic music analysis by shifting the focus from static prediction to interactive, analyst-in-the-loop tools. By providing a unified framework and a prototype interface, it lowers the barrier for developing practical music analysis software. The emphasis on interpretability (multi-level candidates, contradiction highlighting) and the handling of ambiguous musical passages aligns with the needs of musicologists, potentially bridging the gap between computational models and human analytical practices. This paper presents a unified framework for interactive Roman Numeral analysis that effectively bridges the gap between high-accuracy predictive models and practical, analyst-in-the-loop workflows through a hybrid MusicBERT-GNN architecture and edit-conditioned masked prediction.
Supervised fine-tuning (SFT) is widely used to adapt self-supervised speech representations to downstream classification tasks. Small gains observed under a single pretrained checkpoint are often interpreted as method-level improvements, i.e., a higher attainable performance ceiling. We show that such conclusions are not always reliable because SFT outcomes depend strongly on the specific pretrained instance. We conduct a systematic study on 3 SUPERB classification tasks, evaluating 8 SFT variants across 9 pretrained checkpoints from wav2vec~2.0, HuBERT, and WavLM, with multi-seed repetitions on representative base-scale models. We find that the identity of the statistically indistinguishable top-group SFT recipe is often checkpoint-dependent, with limited transferability across pretrained instances. These findings suggest that many reported downstream gains reflect instance and seed dependent elicitation match, rather than universally improving the attainable performance ceiling.
Primary: Kyoto University
All Institutions: Graduate School of Informatics, Kyoto University
This paper provides a crucial methodological critique of speech foundation model fine-tuning, demonstrating that reported SFT gains are often instance-dependent artifacts rather than universal improvements, thereby advocating for more rigorous and stable evaluation standards in the field.
The paper proposes a critical re-evaluation of standard practices in Supervised Fine-Tuning (SFT) for speech foundation models. Rather than proposing a new architecture, it introduces a rigorous empirical framework to test the stability of SFT performance across different pretrained instances (checkpoints) and random seeds. The methodology involves a systematic sweep of 8 SFT configurations (varying feature extraction modes and freezing strategies) across 9 checkpoints from three major families (wav2vec 2.0, HuBERT, WavLM) on 3 SUPERB tasks. The core theoretical contribution is the "elicitation match" hypothesis, which posits that performance gains are often due to better alignment with the specific latent structure of a checkpoint rather than a universal increase in capacity. This reframes the interpretation of SFT results from "method superiority" to "activation reliability."
The experimental design is robust and comprehensive. The authors utilize a large-scale evaluation matrix (9 checkpoints x 8 configs x 3 tasks) and crucially include multi-seed repetitions for base-scale models to isolate stochastic variability. The use of paired exact McNemar tests to define "top groups" provides a statistically sound basis for comparing configurations, moving beyond simple mean accuracy comparisons. The results clearly demonstrate that the "best" SFT configuration is not consistent across checkpoints, even within the same family and size, and that seed changes can lead to significant performance flips (under-activation vs. full activation). This effectively debunks the assumption that a single SFT recipe is universally optimal.
The paper provides high reproducibility standards. It specifies the exact checkpoints, model families, and pretraining data sources. The training setup follows official SUPERB pipelines, ensuring that hyperparameters are standardized. The inclusion of specific random seeds (1337, 2048, 7395) and the detailed description of the SFT configuration space (Feature Mode, Freeze Mode) allow other researchers to replicate the findings. The use of standard evaluation metrics and scripts further enhances reproducibility.
The study is limited to base and large scale models; it does not explore the behavior of smaller or significantly larger models, which might exhibit different stability characteristics. The SFT configurations tested are representative but not exhaustive (e.g., it does not test LoRA or other parameter-efficient fine-tuning methods, which are increasingly common). Additionally, the "elicitation match" hypothesis is interpretive; while supported by the data, it is not formally proven through mechanistic analysis of the representations. The focus on classification tasks also means the findings may not directly generalize to regression or generation tasks without further investigation.
This work has significant implications for the speech community. It cautions against over-interpreting small performance gains in single-checkpoint ablation studies and advocates for more rigorous evaluation protocols that account for instance and seed variability. By highlighting the fragility of SFT performance, it encourages researchers to adopt more robust evaluation practices, potentially reducing the publication of marginal improvements that do not generalize. This promotes a more mature and reliable scientific discourse in speech representation learning. This paper provides a crucial methodological critique of speech foundation model fine-tuning, demonstrating that reported SFT gains are often instance-dependent artifacts rather than universal improvements, thereby advocating for more rigorous and stable evaluation standards in the field.
Recent text-to-audio models generate high-quality audio, but often fail to follow instructions involving multiple sound events and temporal order. This gap arises because existing evaluation and training signals mainly emphasize global similarity or perceptual quality, with limited supervision on instruction-level correctness. We propose an instruction-level framework that uses audio-aware large language models (ALLMs) as fine-grained judges to verify target event presence and temporal relations in generated audio. After validating ALLM judgments on benchmarks and through human verification, we use their feedback to construct preference pairs for direct preference optimization. We further introduce S3Bench, a narrative benchmark for evaluating multi-event temporal instruction following. Experiments show that our method improves event completeness, temporal ordering, and joint instruction-following accuracy across existing benchmarks and S3Bench, while maintaining audio quality.
Primary: National Taiwan University
All Institutions: National Taiwan University, Amazon
The paper presents a significant advancement in Text-to-Audio generation by introducing a novel preference optimization framework that leverages Audio-Aware LLMs for fine-grained instruction verification, effectively bridging the gap between global audio quality and precise temporal event control, while contributing a valuable new benchmark to the community.
The paper proposes a novel framework, ALLM-Judged Preference Optimization (AJPO), to address the lack of fine-grained instruction following in Text-to-Audio (TTA) generation. The core innovation lies in leveraging Audio-Aware Large Language Models (ALLMs) as structured judges to verify sound event existence and temporal ordering, rather than relying on global similarity metrics like CLAPScore. These structured judgments are used to construct preference pairs for Direct Preference Optimization (DPO). The methodology is sound and addresses a specific, well-identified gap in the field: the misalignment between global perceptual quality and fine-grained instruction adherence. The use of ALLMs as reward models/judges is a timely approach, capitalizing on the emergent reasoning capabilities of recent multimodal models. However, the novelty is somewhat incremental as the concept of using LLMs/ALLMs for preference optimization is becoming more common across modalities; the specific application to temporal event ordering in audio is the key differentiator.
The experimental evaluation is comprehensive and robust. The authors first validate the capability of ALLMs (Qwen2.5-Omni, Audio Flamingo) as judges on existing benchmarks and synthetic datasets, showing strong correlation with human judgments. They then introduce S3Bench, a new benchmark for narrative multi-event audio, which adds significant value to the community. The ablation studies effectively demonstrate the superiority of fine-grained ALLM feedback over CLAP-based ranking and caption-based matching. The comparison with existing preference datasets (Audio-Alpaca, Baton) highlights the benefits of dynamic, instruction-aligned preference construction. The results show consistent improvements in event completeness and temporal ordering without degrading audio quality metrics (FAD, IS, CLAPScore). The inclusion of online DPO experiments further strengthens the technical contribution by showing the importance of dynamic preference reconstruction.
The paper provides sufficient detail for reproduction, including model architectures (TangoFlux-base), training hyperparameters (learning rate, batch size, warm-up steps), and the construction of the S3Bench dataset. The use of open-source models (Qwen2.5-Omni, Audio Flamingo) and standard TTA baselines enhances reproducibility. The project page link suggests availability of qualitative examples and potentially code, which would further aid reproducibility.
The approach relies heavily on the accuracy of the ALLM judge. While validated, ALLMs can still hallucinate or make errors, particularly in complex, noisy audio scenarios (as noted in the evaluation of AudioCaps-test vs. AudioTime). The performance of the TTA model is thus bounded by the judge's reliability. Additionally, the synthetic MultiEvent-Temporal benchmarks, while useful for controlled evaluation, may not fully capture the complexity of real-world audio scenes with overlapping sounds and background noise, although S3Bench attempts to mitigate this. The computational cost of running ALLM judgments for preference construction, especially in online DPO, is non-trivial.
This work significantly advances the controllability of TTA systems, making them more suitable for complex applications like film scoring, game audio design, and assistive technologies where precise instruction following is critical. The introduction of S3Bench provides a standardized benchmark for future research in this area. The methodological framework of using multimodal LLMs for fine-grained preference optimization could be generalized to other generative tasks. The paper presents a significant advancement in Text-to-Audio generation by introducing a novel preference optimization framework that leverages Audio-Aware LLMs for fine-grained instruction verification, effectively bridging the gap between global audio quality and precise temporal event control, while contributing a valuable new benchmark to the community.
Automatic speech recognition is dominated by autoregressive decoders that emit one token at a time. We ask whether a discrete diffusion language model can transcribe speech instead, refining a whole transcript in parallel over a small number of denoising steps. We train an audio-native interface for DiffusionGemma, a 26B mixture-of-experts model that generates text by uniform, random-token discrete diffusion rather than the absorbing-mask scheme common to recent diffusion language models. A frozen Whisper encoder supplies acoustic features, a lightweight projector maps them into the model embedding space, and low-rank adapters let the frozen backbone attend to the new modality. About 42M parameters are trained, which is 0.16 percent of the backbone. We find that the natural training objectives fail to ground the audio because their gradient reaches the projector only through attention that has already dismissed it. A connectionist temporal classification loss applied through the frozen output head breaks this deadlock. The resulting model reaches 6.6 percent word error rate on LibriSpeech test-clean, transcribes in roughly eight parallel steps regardless of utterance length, and uses a single adapter trained on six languages, which we evaluate here on English, Hindi, and Mandarin.
Primary: Interfaze AI
All Institutions: Interfaze AI
This paper presents a compelling proof-of-concept for audio-native discrete diffusion language models, effectively solving the gradient flow problem in frozen multimodal adapters through a novel CTC-based grounding strategy, thereby enabling efficient, parallel speech recognition with minimal trainable parameters.
The paper proposes a novel architecture for Automatic Speech Recognition (ASR) by adapting a frozen 26B-parameter discrete-diffusion language model (DiffusionGemma) to audio inputs. The core methodological contribution is the "audio-native" pathway, which uses a frozen Whisper encoder, a lightweight trainable projector, and Low-Rank Adapters (LoRA) to inject acoustic features into the LLM's embedding space. A significant technical insight is the diagnosis of the "grounding deadlock," where standard diffusion and autoregressive losses fail to propagate gradients to the audio projector because the frozen attention mechanism ignores the ungrounded audio tokens. The authors resolve this by introducing a Connectionist Temporal Classification (CTC) loss applied directly through the frozen output head, which provides a direct gradient signal to the projector independent of attention. This allows the model to learn to attend to audio features in subsequent steps. The use of uniform random-token diffusion (rather than absorbing masks) is also noted as a specific characteristic of the backbone used.
The evaluation is conducted on LibriSpeech (English) and FLEURS (Multilingual: English, Hindi, Mandarin). The model achieves a Word Error Rate (WER) of 6.6% on LibriSpeech test-clean, which is competitive given the extremely low number of trainable parameters (42M, or 0.16% of the backbone). The experiments demonstrate that the model converges in approximately eight parallel denoising steps, decoupling inference cost from transcript length. Ablation studies effectively rule out encoder quality and token resolution as the primary bottlenecks, pointing instead to data scale as the main limitation compared to fully trained autoregressive systems like Whisper. The multilingual results show the adapter's ability to generalize across languages, though performance drops on non-English scripts compared to English.
The paper provides detailed architectural specifications, including the Whisper encoder size, projector dimensions, LoRA ranks, and training hyperparameters (AdamW, learning rate, batch size). The description of the diffusion process, corruption kernel, and sampling algorithm is sufficiently detailed for reproduction. However, the code is not explicitly linked in the text provided, and the specific random seeds or exact data splits (beyond standard LibriSpeech/FLEURS) are not fully enumerated, though standard practices are implied.
The primary limitation is the reliance on a frozen Whisper encoder, which restricts the input window to 30 seconds and prevents true streaming recognition without complex segmentation. The model trails state-of-the-art autoregressive systems (like Whisper-large) in accuracy, particularly on multilingual and noisy data, which the authors attribute to the limited training data (219 hours) compared to the millions of hours used for Whisper. Additionally, the diffusion decoder exhibits repetition artifacts on short utterances, requiring post-processing. The evaluation lacks a controlled head-to-head comparison with other diffusion-based ASR systems on identical hardware and data splits.
This work demonstrates that large language models can be efficiently adapted for speech recognition without retraining the backbone, potentially lowering the barrier to entry for multimodal ASR systems. It challenges the dominance of autoregressive decoding in ASR by showing that parallel diffusion decoding is viable and efficient. The multilingual adapter suggests potential for low-resource language support through parameter-efficient fine-tuning. However, the reliance on pre-trained encoders and large models still poses environmental and accessibility challenges. This paper presents a compelling proof-of-concept for audio-native discrete diffusion language models, effectively solving the gradient flow problem in frozen multimodal adapters through a novel CTC-based grounding strategy, thereby enabling efficient, parallel speech recognition with minimal trainable parameters.
Recent diffusion-based generative models have achieved strong results in domain-specific audio generation tasks such as speech, singing, and instrumental music synthesis. However, these models are typically specialized and do not generalize well to mixed or intermediate audio types. In this work, we adapt a diffusion-based model originally designed for multi-instrument music synthesis to voice conversion, covering both speech and singing within a unified framework. Specifically, we extend musical note-based conditioning to include phonetic posteriorgrams (PPGs) and pitch contours, and reinterpret timbre conditioning as speaker or singer identity via feature-wise linear modulation. Experiments show that the adapted model matches or surpasses a dedicated voice conversion system in terms of naturalness and performer similarity, while maintaining accurate pitch control across speech and singing. At the same time, we observe limitations in phonetic fidelity and a degradation in vocal quality when incorporating instrumental training data. Furthermore, we demonstrate that off-the-shelf feature extractors provide effective conditioning signals, enabling large-scale self-supervised training without manual annotations. These results highlight the potential of cross-domain model transfer towards unified audio generation systems capable of handling speech, singing, and music. Qualitative samples can be found on our project page: https://benadar293.github.io/voice-conversion
Primary: International Audio Laboratories Erlangen
All Institutions: International Audio Laboratories Erlangen, Fraunhofer IIS
This paper effectively demonstrates that a diffusion model designed for instrumental music synthesis can be adapted to high-quality speech and singing voice conversion by extending its conditioning mechanisms to include phonetic and pitch information, achieving performance comparable to specialized systems while highlighting the inherent trade-offs in unified audio generation frameworks.
The paper proposes a method to adapt a diffusion-based model originally designed for multi-instrument music synthesis (T5-based) to the task of speech and singing voice conversion (VC). The core technical contribution lies in the conditioning framework: it extends musical note-based conditioning (piano rolls) to include phonetic posteriorgrams (PPGs) and pitch contours ($f_0$) for vocal content, and reinterprets timbre conditioning via Feature-wise Linear Modulation (FiLM) as speaker/singer identity. The authors also implement pitch range adaptation to handle the disparity between source and target vocal ranges. The methodology is sound and leverages existing, robust components (CREPE for pitch, wav2vec 2.0 for PPGs, TRILL for embeddings), focusing on the architectural adaptation and conditioning strategy rather than proposing a new neural architecture from scratch. The comparison with a specialized VC model (FlowMAC) provides a strong baseline for evaluating the efficacy of the cross-domain transfer.
The experimental setup is comprehensive, covering both speech and singing domains. The authors employ a rigorous evaluation protocol including MUSHRA-style listening tests for naturalness and similarity, as well as objective metrics like Fréchet Audio Distance (FAD), Raw Pitch Accuracy (RPA), and phonetic fidelity metrics (Jensen-Shannon and Wasserstein distances on PPGs). The results indicate that the adapted music synthesis model (T5-Voc) matches or slightly surpasses the dedicated VC model (MAC-Voc) in naturalness and performer similarity, while maintaining accurate pitch control. However, the paper honestly reports limitations: phonetic fidelity is lower in the T5 model compared to the specialized model, and incorporating instrumental data (T5-All) degrades vocal quality. The inclusion of a "Vocals with Instrumentals" experiment demonstrates the potential of the unified framework, showing improvements in FAD when combining vocal and instrumental conditioning. The statistical analysis (Wilcoxon signed-rank test) adds credibility to the subjective findings.
The paper provides significant detail regarding the model architecture, conditioning mechanisms, and training procedures. It specifies the use of off-the-shelf feature extractors (CREPE, wav2vec 2.0, TRILL) and mentions the use of publicly available implementations for the baseline models. The dataset description is somewhat vague regarding the proprietary speech data ("proprietary and public sources"), which limits exact replication of the training set. However, the authors state that the feature extractors are public, allowing for the creation of pseudo-annotated datasets from equivalent audio. The project page link provides qualitative samples, aiding in verification. The code is not explicitly linked in the text provided, but the reliance on public components enhances reproducibility potential.
The paper identifies several key limitations. First, there is a trade-off between generative quality/naturalness and phonetic fidelity; the T5 model, while sounding more natural, is less precise in preserving specific phonetic content compared to the specialized FlowMAC model. Second, the degradation in vocal quality when training on mixed vocal-instrumental data (T5-All) highlights the difficulty of unified modeling without careful architectural or training interventions. Third, the reliance on source separation for mixed audio introduces potential errors, although the authors claim high-quality separation. Finally, the evaluation of phonetic fidelity relies on automatic metrics which may not fully correlate with human perception of intelligibility, a gap the authors acknowledge.
This work contributes to the growing field of unified audio generation models, demonstrating that domain-specific architectures can be effectively repurposed for new tasks through careful conditioning design. It highlights the potential for cross-domain transfer in audio synthesis, which could lead to more versatile and efficient generative systems. The findings also have implications for the development of foundation models for audio, suggesting that while unified models are promising, they currently face challenges in balancing performance across diverse domains (speech vs. singing vs. instrumental). The open release of qualitative samples and the discussion of self-supervised training with off-the-shelf features support the broader ML community's interest in scalable, annotation-efficient audio generation methods. This paper effectively demonstrates that a diffusion model designed for instrumental music synthesis can be adapted to high-quality speech and singing voice conversion by extending its conditioning mechanisms to include phonetic and pitch information, achieving performance comparable to specialized systems while highlighting the inherent trade-offs in unified audio generation frameworks.
In this report, we introduce Qwen-Music, a powerful music generation model capable of producing highly musical and high-fidelity songs with complete vocal singing. Qwen-Music supports two core tasks: Text to Music Generation, which create entirely new songs from text descriptions, lyrics, and musical attributes, and Cover Song Generation, which reinterprets existing songs with different styles and vocal characteristics. Architecturally, Qwen-Music integrates three core components: Qwen-Music-Tokenizer, Qwen-Music-LLM, and Qwen-Music-Render. Qwen-Music-Tokenizer compresses audio into a 25 Hz single-codebook stream of Music Semantic Tokens that preserve semantic and melodic information for LLM prediction. Based on these tokens, Qwen-Music-LLM performs autoregressive music semantic modeling, with a key novelty being a melody-token-based chain-of-thought (Melody-CoT) mechanism that plans melodies before full-song generation, improving creativity, musicality, structural coherence, and reference-audio-based melody cloning. To overcome the fidelity limitations of discrete semantic tokens, Qwen-Music-Render performs generative stereo rendering, enriching acoustic details and producing high-fidelity stereo waveforms. Finally, we train Qwen-Music-LLM on more than 5 million hours of multilingual music data covering hundreds of languages. We first apply quality-aware pre-training curriculum, then use progressive post-training, comprising supervised initialization, offline DPO, and online GSPO, to further improve musicality and instruction-following ability. Across 600 Chinese and English prompts, Qwen-Music achieves state-of-the-art results in 13 of 16 objective musicality and audio-quality metrics. Professional evaluators also prefer Qwen-Music over leading proprietary systems. For cover song generation, Qwen-Music preserves reference melodies more accurately than leading proprietary systems.
Primary: Qwen Team
All Institutions: Qwen Team
This paper introduces Qwen-Music, a state-of-the-art music generation system that effectively bridges semantic composition and acoustic rendering through a novel Melody-CoT mechanism and a high-fidelity generative renderer, achieving superior performance in both objective metrics and human preference tests.
The paper presents a comprehensive three-stage architecture for music generation: a semantic tokenizer, an autoregressive language model (LLM), and a generative renderer. The core methodological contributions lie in the specific design choices for handling the complexity of full-song generation. First, the **Qwen-Music-Tokenizer** employs a four-stage training recipe (BestRQ pretraining, causal adaptation, multi-task SFT, and VQ insertion) to create a 25 Hz single-codebook stream. This is a significant engineering contribution, as it balances semantic richness with computational tractability for LLMs. Second, the **Melody-CoT** mechanism is a notable innovation. By explicitly planning melody tokens before generating full-mixture semantic tokens, the model addresses the "composition vs. rendering" mismatch. This chain-of-thought approach allows for better structural coherence and enables cover song generation by conditioning on reference melody tokens while allowing style/timbre changes. The use of relative MIDI representations for melody tokens is a smart design to prevent leakage of absolute pitch/key information that might conflict with style tags. Third, the **Qwen-Music-Render** utilizes a Diffusion Transformer (DiT) conditioned on both semantic tokens and textual descriptions, followed by a novel **Spec-VAE** and **Band-Mode Refiner**. The refiner’s frequency-adaptive correction (phase-only for low, joint for mid, magnitude-only for high) is a sophisticated touch to address specific spectral artifacts. The training pipeline is also rigorous, featuring a quality-graded pre-training curriculum (Q1-Q6 buckets) and a progressive post-training alignment (SFT -> Offline DPO -> Online GSPO), which is state-of-the-art practice for aligning generative models with human preference.
The evaluation is extensive and robust. The authors conduct blind A/B preference tests with 50 professional human raters, showing Qwen-Music outperforms leading proprietary systems (Suno V5.5, MiniMax, Mureka) with a 59.1% win rate. They also report performance on the Artificial Analysis leaderboard. Objective evaluations cover 16 metrics across SongBench, SongEval, and AudioBox-Aesthetic, where Qwen-Music leads in 13/16 metrics. The cover song generation evaluation is particularly strong, demonstrating superior melody preservation compared to baselines. The inclusion of a detailed acoustic data quality pipeline for training data filtering (detecting fake stereo, noise floors, cutoffs) adds credibility to the high-fidelity claims. The ablation studies on the renderer and the effectiveness of the Melody-CoT mechanism are well-supported by the results.
The paper provides detailed architectural specifications, including layer counts, parameter sizes, and training objectives. The tokenizer training stages are clearly defined. The post-training pipeline (SFT, DPO, GSPO) is described with sufficient detail for replication by other research groups. However, the specific "quality-aware" reward models and the exact composition of the 5 million-hour dataset are proprietary and not fully disclosed, which limits exact reproducibility of the pre-training phase. The code for the tokenizer and renderer is not explicitly linked in the provided text, though the Qwen team typically releases code for their models.
The paper relies heavily on proprietary baselines (Suno, MiniMax) for comparison, which makes independent verification difficult. The "quality-graded" pre-training assumes the availability of a reliable internal reward model, which may not be generalizable to all settings. The Melody-CoT mechanism, while effective, adds complexity and inference time. The model's performance on non-vocal music or highly abstract musical forms (e.g., free jazz, classical symphonies without lyrics) is less emphasized compared to vocal pop/rock genres, which dominate the training data. The reliance on a single codebook for the tokenizer might limit the expressiveness compared to multi-codebook approaches (like Encodec) in capturing fine-grained timbral details, although the renderer is designed to mitigate this.
Qwen-Music represents a significant step forward in accessible, high-fidelity music generation. By open-sourcing the technical report and potentially the model (implied by the Qwen lineage), it democratizes access to state-of-the-art music AI. The ability to generate cover songs and control musical attributes empowers creators. However, the potential for misuse in generating copyrighted material or deepfake vocals remains a concern, necessitating robust safety filters and watermarking, which are not detailed in this technical report. The high computational requirements for training such large models also contribute to the centralization of AI capabilities. This paper introduces Qwen-Music, a state-of-the-art music generation system that effectively bridges semantic composition and acoustic rendering through a novel Melody-CoT mechanism and a high-fidelity generative renderer, achieving superior performance in both objective metrics and human preference tests.
Large audio-language models (LALMs) often underperform on fine-grained, non-semantic attributes of speech, such as a speaker's emotion, despite strong performance on speech content. Improving this without the cost of retraining calls for an effective inference-time intervention, yet most existing methods intervene only after the audio encoder and operate at a relatively coarse granularity. The encoder itself, where acoustic information is first extracted from the waveform, remains largely unexplored, especially at the level of individual neurons. We introduce IAAN, Identifying and Amplifying Acoustic Neurons, a training-free and label-free method that scores each feed-forward neuron in the audio encoder by contrasting its activation on the real waveform with that on a noise reference lacking the real audio's acoustic information. IAAN then amplifies a small set of the highest-scoring neurons at inference. Across ten non-semantic speech attributes, IAAN improves average accuracy by 25.7 points on Audio-Flamingo-3, 21.4 on Qwen2.5-Omni, and 9.7 on Kimi-Audio. It also improves a model already explicitly fine-tuned to prioritize acoustic evidence. In controlled comparisons, both the encoder locus and neuron-level selectivity prove necessary for this gain. Intervening after the encoder, at the decoding side or inside the language model, yields little to no improvement, or even deteriorates accuracy. The improvement also depends on which specific neurons are amplified, not merely on their number, confirming that IAAN's acoustic score succeeds in identifying the neurons that matter. These results show that a small, precisely targeted intervention inside the audio encoder is an effective and largely untapped way to strengthen the acoustic understanding of LALMs, opening a new direction for inference-time methods that improve acoustic perception through neuron-level access to the encoder.
Primary: National Taiwan University
All Institutions: National Taiwan University, NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), ASUS Open Cloud Infrastructure Software Center
[One sentence main contribution]. The paper introduces IAAN, a training-free inference-time intervention that identifies and amplifies specific feed-forward neurons in the audio encoder of LALMs by contrasting their activations on real audio versus noise, significantly improving non-semantic acoustic perception. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. This paper makes a valuable contribution to the field of multimodal machine learning by shifting the focus of inference-time interventions from the language model backbone to the audio encoder. By proposing a simple yet effective contrastive scoring mechanism at the individual neuron level, IAAN addresses a critical gap in how LALMs process fine-grained acoustic information. The rigorous experimental evaluation, including comprehensive ablations against other intervention loci and granularities, strongly validates the necessity of encoder-side, neuron-level targeting. The significant performance gains across multiple state-of-the-art LALMs on a challenging benchmark (VoxParadox) demonstrate the practical utility of this approach. While the computational cost of dual encoder passes is a minor drawback, the method's label-free and training-free nature makes it a highly attractive tool for enhancing the acoustic grounding of existing models.
The paper proposes IAAN (Identifying and Amplifying Acoustic Neurons), a training-free, label-free inference-time intervention method. The core methodology involves contrasting the activation of feed-forward neurons in the audio encoder of Large Audio-Language Models (LALMs) between the real audio waveform and a noise reference (Gaussian noise with matched length/variance). Neurons that activate more strongly on real audio are identified as "acoustic neurons" and their activations are amplified by a gain factor $g$ during inference. The approach is conceptually simple and elegant, leveraging the intuition that acoustic-specific neurons should be sensitive to the presence of structured audio versus unstructured noise. The method operates at the individual neuron level within the audio encoder, which is a finer granularity than previous interventions that typically target the language model backbone or output logits. The technical execution involves standard transformer operations (FFN activations) and does not require backpropagation or external datasets for calibration, relying instead on an internal contrastive signal.
The authors evaluate IAAN on three open-source LALMs (Audio-Flamingo-3, Qwen2.5-Omni, Kimi-Audio) and one specialized model (AF3-PD) using the VoxParadox benchmark, which tests ten non-semantic speech attributes (e.g., emotion, gender, pitch) where semantic content may contradict acoustic cues. The results are significant: IAAN improves average accuracy by 25.7 points on Audio-Flamingo-3, 21.4 on Qwen2.5-Omni, and 9.7 on Kimi-Audio. Crucially, the paper includes rigorous ablation studies. It compares IAAN against other inference-time interventions (Audio-aware decoding, LLM-amplify, Hidden-state steering) and demonstrates that intervening *inside* the encoder is necessary, as post-encoder interventions yield little to no gain. It also shows that the specific selection of neurons (via the acoustic score) is critical, as random selection or whole-layer amplification fails to reproduce the gains. The method also improves a model (AF3-PD) that was already fine-tuned for acoustic tasks, suggesting broad applicability. The use of a held-out development set (LISTEN) for hyperparameter tuning adds credibility, although the transfer of optimal hyperparameters across models is noted to be variable.
The paper provides sufficient detail for reproduction. It defines the neuron definition (FFN hidden activations), the reference signal generation (Gaussian noise with fixed seed), the scoring equation, and the amplification hook. The models and benchmark (VoxParadox) are open-source and widely available. The hyperparameter selection process is clearly described, though the reliance on an external development set (LISTEN) for tuning $(K, g)$ is a minor constraint for fully zero-shot application. The code is not explicitly linked in the text provided, but the method is algorithmic and straightforward to implement given the open-source nature of the base models.
The primary limitation is the computational overhead of running two forward passes through the audio encoder (real vs. noise reference) for every inference step. While the encoder is smaller than the LLM, this still doubles the encoder compute time. Second, the method's effectiveness is somewhat model-dependent; Kimi-Audio showed smaller gains and higher sensitivity to hyperparameters compared to Audio-Flamingo-3. The noise reference, while effective, is a heuristic; it assumes that neurons responding to "non-noise" are the acoustic ones, which might not capture all nuanced acoustic features equally well. Finally, the method is evaluated only on multiple-choice and free-text description tasks; it does not address potential negative impacts on semantic understanding or other modalities, though the authors claim no deterioration in semantic tasks.
This work opens a new direction for improving multimodal models by focusing on the encoder side, which has been largely ignored in favor of LLM-centric interventions. It demonstrates that fine-grained, neuron-level steering within encoders can be highly effective for specific perceptual tasks. This could lead to more efficient and targeted methods for enhancing multimodal capabilities without the cost of retraining. However, the ability to easily steer model behavior via activation amplification also raises questions about robustness and potential adversarial vulnerabilities, although these are not explored in depth here. [One sentence main contribution]. The paper introduces IAAN, a training-free inference-time intervention that identifies and amplifies specific feed-forward neurons in the audio encoder of LALMs by contrasting their activations on real audio versus noise, significantly improving non-semantic acoustic perception. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. This paper makes a valuable contribution to the field of multimodal machine learning by shifting the focus of inference-time interventions from the language model backbone to the audio encoder. By proposing a simple yet effective contrastive scoring mechanism at the individual neuron level, IAAN addresses a critical gap in how LALMs process fine-grained acoustic information. The rigorous experimental evaluation, including comprehensive ablations against other intervention loci and granularities, strongly validates the necessity of encoder-side, neuron-level targeting. The significant performance gains across multiple state-of-the-art LALMs on a challenging benchmark (VoxParadox) demonstrate the practical utility of this approach. While the computational cost of dual encoder passes is a minor drawback, the method's label-free and training-free nature makes it a highly attractive tool for enhancing the acoustic grounding of existing models.
Multimodal information is pivotal for e-commerce search ranking. Existing works leverage multimodal data typically by fine-tuning general Multimodal Large Language Models (MLLMs) via collaborative signals, subsequently integrating the derived representations into ranking models as item features. Despite their efficacy, these methods face two primary limitations: (1) they rely on a single collaborative signal for MLLM fine-tuning, failing to exploit the heterogeneous signals essential for multitask ranking; and (2) they treat multimodal representations as regular item features in ranking models, underutilizing their latent potential for user behavior modeling. To address these challenges, we propose the Multiplex Multimodal Representation Model (MMRM), a unified framework that aligns MLLMs with diverse collaborative signals. By employing a shared backbone with task-specific tokens and projection layers, MMRM simultaneously learns from multiple signals and generates comprehensive multiplex item representations in a single inference pass. Furthermore, we introduce a multiplex user representation strategy in ranking models, which derives task-specific user representations via search-based behavior sequence modeling leveraging multiplex item representations. Extensive experiments demonstrate MMRM's superior efficiency and effectiveness. Notably, MMRM has been successfully deployed in the JD e-commerce search engine, yielding significant performance gains for millions of daily users.
Primary: JD.COM
All Institutions: JD.COM
[One sentence main contribution]. The paper introduces MMRM, a unified framework that aligns MLLMs with diverse collaborative signals using task-specific tokens, enabling efficient generation of multiplex item representations and significantly improving e-commerce search ranking performance. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. MMRM addresses critical limitations in existing multimodal ranking approaches by effectively integrating heterogeneous collaborative signals and disentangling task-specific representations, leading to superior offline and online performance in a major e-commerce platform.
The paper proposes MMRM, a framework for aligning Multimodal Large Language Models (MLLMs) with multiple heterogeneous collaborative signals (query-to-item clicks, and item-to-item clicks, carts, orders) in e-commerce search. The core technical innovation lies in using a shared MLLM backbone with task-specific special tokens ([SEARCH], [CLICK], [CART], [ORDER]) to generate disentangled item representations in a single forward pass. This addresses the limitation of previous methods that either use single signals or require separate models/forward passes for different signals. The method also introduces a multiplex user representation strategy in the downstream ranking model, leveraging these task-specific item embeddings for behavior sequence modeling. The approach is technically sound, leveraging established techniques like contrastive learning, GradCache for memory efficiency, and MMoE for multitask learning, but applies them in a specific, effective configuration for e-commerce.
The evaluation is extensive and robust, covering both offline representation learning metrics (F1@5, NDCG@5) and online A/B testing metrics (UCTR, UACR, UCVR). The offline results demonstrate that MMRM outperforms single-task baselines and a vanilla multitask baseline (Vanilla-Multi), validating the effectiveness of the task-specific token design. The online A/B test on JD's search engine provides strong evidence of real-world utility, showing significant improvements in key business metrics. The ablation studies on batch size and the detailed description of dataset construction (including hard negative sampling strategies) add credibility to the experimental claims.
The paper provides sufficient implementation details for reproduction, including the base model (Qwen3-VL-4B-Instruct), training hyperparameters (learning rate, batch size, temperature), and optimization techniques (GradCache, FlashAttention-2). The dataset construction process is described in detail, although the exact proprietary data distributions are not fully open. The code is not publicly released, which is common for industry papers, but the methodological description is clear enough for a competent researcher to implement.
The paper does not explicitly discuss the computational overhead of the MMRM approach compared to single-task models, although it claims efficiency gains by avoiding multiple forward passes. The generalizability of the findings to other e-commerce platforms or non-e-commerce domains is not explored. Additionally, the reliance on proprietary JD data limits the ability of external researchers to fully validate the scale and diversity of the collaborative signals used. The paper also does not discuss potential biases introduced by the graph-based sampling methods.
This work has significant implications for the field of information retrieval, particularly in e-commerce search and recommendation systems. By demonstrating how to effectively leverage multimodal data and multiple collaborative signals within a unified framework, it sets a new standard for industrial-scale ranking models. The successful deployment and positive impact on millions of users highlight the practical value of the research. It also contributes to the broader understanding of how to adapt large foundation models for specific, complex downstream tasks with heterogeneous data sources. [One sentence main contribution]. The paper introduces MMRM, a unified framework that aligns MLLMs with diverse collaborative signals using task-specific tokens, enabling efficient generation of multiplex item representations and significantly improving e-commerce search ranking performance. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. MMRM addresses critical limitations in existing multimodal ranking approaches by effectively integrating heterogeneous collaborative signals and disentangling task-specific representations, leading to superior offline and online performance in a major e-commerce platform.
Speech enhancement (SE) can substantially improve perceptual quality, yet enhanced speech does not necessarily improve automatic speech recognition (ASR). Existing remedies, such as retraining the enhancer jointly with recognizer or interpolating enhanced speech with the noisy input, can mitigate this mismatch, but common explanations such as artifacts and over-suppression remain qualitative and do not localize which enhancement component harms recognition. We propose inference time polar projection, a diagnosis for STFT domain enhancement. Given a mask $M=Ae^{jφ}$, polar projection forms $M_{α,γ}=A^αe^{jγφ}$, where $α$ controls magnitude strength and $γ$ controls phase correction. Sweeping these controls on frozen SE and ASR models turns ASR degradation into measurable magnitude and phase effects. Our projection analysis shows that magnitude strength is the operative axis, while estimated phase correction provides no recognition benefit. The optimal magnitude strength is recognizer dependent: waveform-input wav2vec2.0 favors strong correction, whereas log-Mel-input, noise-robust Whisper prefers weaker correction. Finally, the projection provides a simple mitigation for any SE front end in the STFT mask domain, without retraining either the enhancer or the recognizer, making it directly useful for voice assistants and agents that rely on enhanced speech.
Primary: University of Illinois Urbana-Champaign
All Institutions: University of Illinois Urbana-Champaign, Wuhan University
The paper presents a rigorous diagnostic framework for understanding the mismatch between speech enhancement quality and automatic speech recognition performance, demonstrating that magnitude strength is the critical factor and that phase correction is largely superfluous for recognition tasks.
The paper introduces "Inference Time Polar Projection," a diagnostic framework that factorizes STFT-domain complex masks into independent magnitude and phase controls. By parameterizing the mask as $M_{\alpha,\gamma} = A^\alpha e^{j\gamma\phi}$, the authors decouple the effects of magnitude scaling ($\alpha$) and phase correction ($\gamma$) on Automatic Speech Recognition (ASR) performance. This approach is methodologically sound and offers a rigorous way to dissect the black-box nature of Speech Enhancement (SE) models. The derivation of the effective mask for non-multiplicative models (like MossFormerGAN-SE) is a clever technical addition that extends the applicability of the diagnosis. However, the core concept of sweeping mask parameters is not entirely new; it builds on existing work in mask exponentiation and observation adding, though the specific disentanglement of magnitude vs. phase for ASR diagnosis is a distinct contribution.
The experimental evaluation is comprehensive and well-structured. The authors test on the VoiceBank+DEMAND benchmark, which is standard for this domain, using six different SE models (both discriminative and generative) and two distinct ASR backbones (Whisper and wav2vec 2.0). The results clearly demonstrate the SE-ASR mismatch and validate the polar projection hypothesis: magnitude strength is the primary driver of ASR performance, while phase correction provides negligible benefit. The analysis of recognizer dependence (wav2vec 2.0 favoring strong suppression vs. Whisper preferring mild correction) is insightful and aligns with the known robustness differences between waveform-based and feature-based models. The statistical significance testing (paired bootstrap) adds rigor to the claims.
The paper provides sufficient detail for reproduction. It specifies the datasets (VoiceBank+DEMAND), the specific SE models used (with references to ClearerVoice-Studio implementations), and the ASR models. The mathematical formulation of the polar projection is clear. However, the code for the "effective mask" calculation for MossFormerGAN-SE is described conceptually but not provided in the text, which might require some inference to implement exactly. The use of standard benchmarks and publicly available models enhances reproducibility.
The primary limitation is the scope of the evaluation. The experiments are conducted on a single dataset (VoiceBank+DEMAND), which, while standard, may not generalize to more complex real-world conditions with heavy reverberation or diverse noise types not present in the test set. The authors acknowledge this in the limitations section. Additionally, the diagnosis is limited to STFT-domain enhancements; while they extend it to effective masks for other architectures, it may not apply to pure time-domain models that do not operate in the spectral domain. The phase result, while robust for the tested models, is specific to the estimated phase of those enhancers and does not rule out the potential utility of explicitly modeled phase in other contexts.
This work has significant implications for the deployment of SE systems in voice assistants and agents. By showing that phase correction is largely unnecessary for ASR and that magnitude strength is recognizer-dependent, it suggests that SE systems can be simplified or calibrated specifically for machine listening, potentially reducing computational cost and improving robustness. It shifts the paradigm from optimizing for perceptual quality (PESQ, SI-SDR) to optimizing for recognition utility, which is crucial for the next generation of speech-centric AI applications. The paper presents a rigorous diagnostic framework for understanding the mismatch between speech enhancement quality and automatic speech recognition performance, demonstrating that magnitude strength is the critical factor and that phase correction is largely superfluous for recognition tasks.
Zero-shot instrument cloning aims to render an arbitrary [Target MIDI] sequence with the acoustic identity of an unseen instrument given only a short [Reference Audio, Reference MIDI] pair. Existing methods rely on pre-trained embeddings (e.g., CLAP) that compress the reference audio into a fixed-length vector, discarding fine-grained acoustic cues essential for faithful timbre reconstruction. We present Anysynth, an embedding-free neural synthesizer based on in-context flow matching. By conditioning a Diffusion Transformer (DiT) directly on the uncompressed reference audio and target MIDI, our model allows self-attention to dynamically retrieve acoustic details at generation time. Experiments show that \tool outperforms embedding-based and auto-regressive baselines in audio quality, timbre similarity, and melody adherence. Notably, the model exhibits prompt-length scaling: longer reference prompts yield steadily better timbre fidelity, a property absent in embedding-based systems. To optimize controllability, we further propose Asymmetric Hierarchical CFG, which structurally decouples MIDI and reference-timbre guidance based on their natural semantic-acoustic dependency. This asymmetric formulation avoids gradient conflicts and improves both note accuracy and timbre fidelity, pushing the boundary of expressive, zero-shot instrument cloning. Demo audios are available at https://anysynth-demo.github.io/
Primary: Central Media Technology Institute
All Institutions: Central Media Technology Institute, Chinese University of Hong Kong
The paper presents a significant advancement in zero-shot instrument cloning by replacing information-bottleneck embeddings with in-context learning in a DiT architecture, supported by a novel Asymmetric CFG strategy that effectively decouples semantic and acoustic guidance.
The paper proposes "Anysynth," an embedding-free neural synthesizer for zero-shot instrument cloning. The core methodological innovation lies in reformulating the task as an In-Context Learning (ICL) problem using a Diffusion Transformer (DiT) with Flow Matching. Instead of compressing reference audio into a fixed-length embedding (like CLAP), the model conditions directly on the uncompressed reference mel spectrogram and MIDI as a prefix. This allows the self-attention mechanism to dynamically attend to fine-grained acoustic details (transients, room reverb) from the reference. A key technical contribution is the "Asymmetric Hierarchical Classifier-Free Guidance" (CFG). The authors argue that standard symmetric CFG causes gradient conflicts between semantic (MIDI) and acoustic (timbre) conditions. By factorizing the probability $p(c_m, c_r | x_t)$ as $p(c_r | c_m, x_t)p(c_m | x_t)$, they decouple the guidance, applying strong structural guidance first and then refining with timbre. This is a theoretically sound and practically effective modification to standard diffusion/flow matching conditioning strategies. The MIDI encoding is also enhanced with a binary onset channel to better capture note boundaries, addressing a common failure mode in piano roll representations.
The experimental setup is robust, utilizing a mix of synthetic datasets (NSynth, Slakh) and real-world recordings (MAESTRO, GuitarSet). The evaluation metrics are comprehensive, including PANNs for timbre similarity, Onset F1 for melody adherence, MERT for perceptual quality, and Audiobox Aesthetics for overall quality. The results demonstrate that Anysynth outperforms embedding-based baselines (CLAP-conditioned DiT) and autoregressive/token-based baselines (TokenSynth, CTD). A significant finding is the "prompt-length scaling" property: unlike embedding-based models which saturate quickly, Anysynth's performance improves with longer reference prompts, validating the ICL approach. The ablation studies effectively isolate the benefits of in-context conditioning over CLAP embeddings and validate the superiority of Asymmetric CFG over Combined and Symmetric CFG variants. The UMAP visualizations provide qualitative support for the clustering of timbres.
The paper provides detailed implementation specifics, including the DiT architecture (1024 hidden dim, 25 layers, 16 heads), training hyperparameters (batch size, learning rate, warmup), and dataset construction details (filtering rules for NSynth/Slakh). The use of standard components (Vocos vocoder, RoPE, AdaLN) aids reproducibility. However, the code repository is not explicitly linked in the text (only a demo page), which is a minor hurdle for immediate reproduction, though the architectural details are sufficient for a competent researcher to implement.
The paper acknowledges that performance gains from prompt length diminish after 8 seconds, suggesting a limit to the utility of very long prompts for short generation tasks. The model is evaluated only on keyboard, guitar, and bass instruments; generalization to more complex orchestral instruments or vocals is not demonstrated. The reliance on a pre-trained vocoder (Vocos) means that the quality of the final waveform is partially dependent on the vocoder's capability, although this is standard practice. The computational cost of attending to long reference sequences in a DiT might be higher than embedding-based methods during inference, although the authors claim no major overhead due to parallel computation.
This work advances the field of controllable audio generation by providing a more faithful and flexible method for instrument cloning. It addresses a critical bottleneck in current systems (information loss in embeddings) and offers a scalable solution (ICL) that improves with more data/context. This has implications for music production tools, virtual instrument synthesis, and audio restoration. The Asymmetric CFG technique is generalizable to other multimodal generation tasks where semantic and stylistic/acoustic conditions are entangled. The paper presents a significant advancement in zero-shot instrument cloning by replacing information-bottleneck embeddings with in-context learning in a DiT architecture, supported by a novel Asymmetric CFG strategy that effectively decouples semantic and acoustic guidance.
Automated assessment of second language (L2) speaking proficiency relies on large-scale annotated speech data, which remains scarce compared to widely available written learner corpora. A promising direction for addressing this imbalance is to use text-to-speech (TTS) and voice cloning to convert written L2 production into synthetic speech. However, written and spoken L2 differ fundamentally: spontaneous speech includes disfluencies and discourse markers, while writing is more planned and complex. This raises the question of what is required to generate synthetic L2 speech suitable for assessment. We address this through a systematic analysis of speaker-text relationships using COREFL, a publicly available corpus containing paired spoken and written responses from the same L2 learners to the same questions across modalities. In our proposed framework, we first address the structural differences between written and spoken language by transforming written responses into spoken-style transcripts ("speechification") using a large language model. These transcripts are then converted into speech using a TTS/voice-cloning model. To assign a voice to each synthetic response, we investigate different speaker-text pairing strategies based on shared learner attributes (proficiency level, first language, both, or neither). We evaluate our data augmentation techniques on the language assessment task, with improvements shown in both wav2vec2 (audio-based) and ModernBERT (text-based) scoring systems. Results show that matching speakers and texts by proficiency level yields the most robust synthetic speech. Moreover, raw written text leads to a strong mismatch with spoken language, while speechification substantially reduces this gap and improves grading performance.
Primary: University of Cambridge
All Institutions: University of Cambridge
The paper presents a systematic framework for augmenting L2 speaking assessment data by speechifying written texts and synthesizing speech, demonstrating that matching proficiency levels and modeling spoken linguistic structures significantly improves downstream grading performance.
The paper proposes a data augmentation framework for L2 speaking assessment by converting written learner essays into synthetic speech. The core methodological contribution is the "speechification" step, where an LLM (GPT-4.1) rewrites written text into spoken-style transcripts incorporating disfluencies, hesitations, and learner errors, conditioned on CEFR proficiency levels. This is followed by zero-shot voice cloning using OmniVoice. The study systematically evaluates speaker-text pairing strategies (matching by CEFR, L1, both, or neither) to determine the optimal conditions for generating synthetic data that mimics real L2 speech characteristics. The approach is well-motivated by the scarcity of annotated L2 speech data and the structural differences between written and spoken language.
The authors utilize the COREFL corpus, which provides paired written and spoken responses, allowing for a controlled analysis of modality gaps. They evaluate the synthetic data using two downstream grading models: wav2vec2 (audio-based) and ModernBERT (text-based). The experiments demonstrate that speechification significantly improves performance, particularly for the text-based grader, by reducing the linguistic mismatch. Matching speakers and texts by proficiency level was found to be the most robust strategy. The results show consistent improvements in PCC, SRC, and RMSE metrics when using augmented data. However, the evaluation relies heavily on automatic metrics (WER, PCC, SRC) and speaker verification cosine similarity, lacking subjective human evaluation (e.g., MOS) for speech naturalness or assessment accuracy, which is a notable gap for TTS research.
The paper provides detailed descriptions of the pipeline, including the LLM prompt for speechification, the TTS model (OmniVoice), and the grading architectures. The COREFL dataset is publicly available. The code for the grading systems is not explicitly linked, but the model architectures (wav2vec2, ModernBERT) are standard. The specific hyperparameters are mentioned to be in a table (referenced but not fully detailed in the snippet provided, though standard practices are implied). The cost of LLM usage is noted, adding transparency. Overall, the methodology is clear enough for reproduction, assuming access to the proprietary OmniVoice model or a suitable open-source alternative.
The reliance on GPT-4.1 for speechification introduces a dependency on a specific, potentially costly, and non-deterministic component. The synthetic speech, while linguistically improved via speechification, may still lack the acoustic variability and prosodic nuances of real spontaneous speech, as evidenced by the performance gap between real and synthetic data. The study focuses on English L2 learners, limiting generalizability to other languages without further validation. The lack of human evaluation for the synthetic speech quality and the grading performance limits the confidence in the "realism" of the data from a pedagogical perspective. The use of zero-shot TTS means the synthetic voices are clones but may not capture the full idiosyncratic speech patterns of the original speakers.
This work has significant potential to address the data scarcity problem in automated language assessment, particularly for L2 speaking. By enabling the generation of large-scale, annotated synthetic speech data from abundant written corpora, it could accelerate the development of more robust and fair assessment tools. However, ethical considerations regarding the use of synthetic data in high-stakes testing environments must be addressed, including issues of bias, transparency, and the potential for over-reliance on synthetic data that may not fully represent human speech variability. The paper presents a systematic framework for augmenting L2 speaking assessment data by speechifying written texts and synthesizing speech, demonstrating that matching proficiency levels and modeling spoken linguistic structures significantly improves downstream grading performance.
Dance-to-music generation is a promising task for applications such as choreography support and automatic accompaniment, where temporal coordination between body movement and sound is essential. In particular, using human joint positions as the motion representation is attractive because they explicitly capture body dynamics while being lightweight, privacy-preserving, and easy to integrate with motion capture and pose-estimation pipelines. A central challenge in this setting, however, is the scarcity of high-quality paired dance-music data, since collecting accurately synchronized pairs is costly and often constrained by copyright and performance rights. This makes it difficult to train end-to-end models solely from paired data. To address this issue, we propose a dance-conditioned music generation framework that efficiently exploits both unpaired and paired data. Our method combines pretrained unimodal encoders for motion and music, beat-guided contrastive pretraining to align their feature spaces, and a ControlNet-style conditioning module on top of a pretrained text-to-audio diffusion model. Experiments on AIST++ demonstrate that the proposed techniques improve both dance-music alignment and audio quality, as confirmed by quantitative and qualitative evaluations. Compared to a state-of-the-art method, our approach achieves superior dance alignment performance and competitive audio quality. Code is available at https://github.com/kmraven/AudioLDM-ControlNet .
Primary: Sony Computer Science Laboratories
All Institutions: Sony Computer Science Laboratories, Keio University, Georgia Institute of Technology
The paper presents a competent application of contrastive alignment and ControlNet-style conditioning to the dance-to-music generation task, achieving state-of-the-art alignment performance on AIST++. While the methodological novelty is incremental, the technical execution is solid, and the results demonstrate the effectiveness of combining unimodal pretraining with cross-modal alignment for this specific, data-scarce problem.
The paper proposes a two-stage framework for dance-to-music generation. Stage 1 involves beat-guided contrastive pretraining to align features from a pretrained MotionBERT encoder (for dance) and MERT encoder (for music). Stage 2 uses these aligned motion features to condition a frozen AudioLDM backbone via a ControlNet-style adapter. The methodology is technically sound and follows established patterns in multimodal generation (contrastive alignment + diffusion conditioning). The use of MotionBERT for 2D keypoints is a strong choice for capturing kinematic structure. However, the novelty is moderate; the core idea of using contrastive learning to align modalities (like CLIP) and then conditioning a diffusion model (like ControlNet) is a well-known paradigm in computer vision and has been applied to audio (e.g., AudioLDM). The specific application to dance-to-music with beat-guided alignment is a valid contribution but represents an incremental adaptation of existing techniques rather than a fundamental methodological breakthrough. The "beat-guided" aspect is a specific engineering detail rather than a new theoretical framework.
The experiments are conducted on the AIST++ dataset, which is the standard benchmark for this task. The evaluation includes both objective metrics (BHS, BCS, F1, TD, FAD, CLAP, BAS) and subjective user studies (MOS). The results show that the proposed method outperforms the baseline (Li et al.) in dance-alignment metrics (BAS and MOS), which is the primary goal. However, the subjective sound quality MOS is slightly lower than the baseline, although objective audio quality metrics (FAD, CLAP) are better. The ablation studies effectively demonstrate the contribution of the contrastive pretraining and the MotionBERT encoder. The comparison with Li et al. is fair as they use the same dataset split. One limitation is the small test set (36 pairs), which can lead to high variance in metrics, though the authors acknowledge this by using both objective and subjective measures. The results are consistent with the claims.
The paper provides sufficient detail for reproduction. It specifies the datasets (AIST++), preprocessing steps (resampling, normalization), model architectures (MotionBERT, MERT, AudioLDM, ControlNet), and training hyperparameters (learning rates, batch sizes, epochs). The code is available on GitHub. The use of standard pretrained models (AudioLDM, MotionBERT, MERT) facilitates reproducibility. The specific beat extraction method (Librosa) and contrastive loss details are clearly described.
The paper acknowledges the small scale of paired dance-music data. The subjective sound quality is slightly inferior to the baseline, which the authors attribute to the backbone (AudioLDM vs MusicGen). The reliance on 2D keypoints may lose some 3D motion information, although MotionBERT is designed to handle this. The beat extraction is heuristic-based (velocity smoothing), which might not be robust to all dance styles. The evaluation is limited to AIST++, and generalizability to other datasets or styles is not tested. The subjective study has a small number of participants (35) and samples (10 per participant), which may not fully capture perceptual nuances.
This work contributes to the field of multimodal generation, specifically the intersection of dance and music. It has potential applications in choreography support, automatic accompaniment, and entertainment. The use of joint positions is privacy-preserving compared to video-based methods. The framework could be extended to other motion-to-audio tasks. The open-source code promotes further research in this area. The paper presents a competent application of contrastive alignment and ControlNet-style conditioning to the dance-to-music generation task, achieving state-of-the-art alignment performance on AIST++. While the methodological novelty is incremental, the technical execution is solid, and the results demonstrate the effectiveness of combining unimodal pretraining with cross-modal alignment for this specific, data-scarce problem.
Generative streaming models for Target Speaker Extraction (TSE) commonly exhibit a quality--intelligibility trade-off, wherein naive optimization for perceptual audio quality tends to degrade speech intelligibility, and conversely. We reveal that this trade-off arises not from the constraints of streaming architectures, but from an inappropriate choice of optimization anchor. Directly optimizing against audio quality metrics induces catastrophic reward hacking, where content critical to pronunciation and intelligibility is systematically erased to maximize a proxy score. To break this bottleneck, we propose two complementary improvements: an enlarged Conformer convolution kernel for richer local spectro-temporal modeling, and WavLM-anchored Direct Preference Optimization (DPO) fine-tuning strategy. DPO preference pairs are ranked by WavLM cosine similarity, a deep acoustic feature encoding both phonetic structure and speaker identity, providing an optimization anchor that resists hacking. Under a 560 ms streaming chunk size, the proposed method achieves a 10.9% relative intelligibility improvement (word error rate: 0.138 to 0.123), with marginal simultaneous gains in audio quality and speaker similarity.
Primary: Tsinghua University
All Institutions: Tsinghua University, The Chinese University of Hong Kong, SenseTime
The paper makes a significant contribution to streaming TSE by identifying and resolving the reward hacking problem in DPO through deep-feature anchoring, achieving a Pareto improvement in quality and intelligibility.
The paper proposes a novel approach to the quality-intelligibility trade-off in streaming Target Speaker Extraction (TSE). The core methodological contribution is the identification and mitigation of "reward hacking" in Direct Preference Optimization (DPO) when applied to generative speech models. The authors correctly diagnose that optimizing for perceptual metrics (DNSMOS) leads to intelligibility collapse, while optimizing for Word Error Rate (WER) fails to improve quality. Their solution involves two steps: architectural (enlarging Conformer kernel size to $k=15$ for better local context) and algorithmic (using WavLM cosine similarity as the preference anchor for DPO). The separation of the semantic pathway (optimized via DPO) and acoustic pathway (frozen) is a sound engineering decision to prevent semantic drift during acoustic refinement. The use of WavLM, which encodes both phonetic and speaker information, as a proxy for "good" speech is a clever heuristic that effectively balances the competing objectives.
The experimental setup is rigorous, utilizing the Libri2Mix dataset and standard metrics (DNSMOS, WER, Speaker Similarity). The ablation studies are particularly strong, clearly demonstrating the failure modes of $DPO_{DNSMOS}$ (intelligibility collapse) and $DPO_{WER}$ (no gain, quality drop), thereby validating the necessity of the $DPO_{WavLM}$ approach. The results show a statistically significant improvement in WER (10.9% relative) with simultaneous gains in quality and speaker similarity. The inclusion of Inference Success Rate (ISR) is a valuable addition for streaming contexts. The evaluation covers multiple KL penalty coefficients and checkpoints, providing a robust analysis of hyperparameter sensitivity.
The paper provides sufficient implementation details, including model architecture (Conformer kernel size, SELM/ARLM layers), training hyperparameters (learning rate, optimizer), and dataset specifications. The use of standard libraries (FunCodec, Whisper, WavLM) aids reproducibility. However, the exact code for the "Historical Context Refinement" mechanism and the specific stochastic sampling procedure for generating preference pairs could be more detailed. The claim of "catastrophic reward hacking" is well-supported by the provided analysis (spectrograms and metric trends), which enhances trust in the reproducibility of the findings.
The primary limitation is the reliance on WavLM as a proxy for human preference. While effective, WavLM is a self-supervised model trained on large-scale speech data, and its alignment with human perceptual quality is not perfect. The method is evaluated only on Libri2Mix, a relatively clean, two-speaker mixture dataset. Real-world scenarios with heavy noise, reverberation, and more than two speakers are not addressed. Furthermore, the streaming latency is 560ms, which is acceptable for many applications but not for strict real-time (<200ms) interaction, as noted by the authors. The computational cost of generating 16 candidates for DPO training is also non-trivial.
This work has significant implications for the deployment of generative AI in real-time communication systems, such as teleconferencing and voice assistants. By breaking the quality-intelligibility trade-off, it enables systems that are both natural-sounding and easy to understand. The insight regarding reward hacking in generative speech models is broadly applicable to other text-to-speech and audio generation tasks. It highlights the danger of using naive proxy metrics for optimization and advocates for more semantically aware anchors. The paper makes a significant contribution to streaming TSE by identifying and resolving the reward hacking problem in DPO through deep-feature anchoring, achieving a Pareto improvement in quality and intelligibility.
Melody skeleton extraction aims to derive a shorter melody that preserves structural notes while removing ornaments. Prior methods rely on hand-crafted reduction rules or note-wise salience classifiers trained with heuristically or procedurally generated pseudo-labels. Such supervision can inherit generator bias and does not explicitly optimize a coherent reduced melody. We introduce MeloBottleneck, a self-supervised framework that represents a skeleton as a length-controlled, order-preserving latent subsequence. A hard-bottleneck extractor selects note events, a rhythmic-closure operator produces a self-consistent skeleton, and a re-ornamentation decoder reconstructs the input melody. Training combines reconstruction, a frozen autoregressive melody prior, ornament-invariant consistency across procedurally ornamented views, and ornament exclusion. We evaluate three regimes: synthetic out-of-distribution ornament-to-skeleton, TAVERN variation-to-theme, and Jiugong ornamented-to-gongche. A matched pseudo-label classifier excels on the synthetic benchmark, while MeloBottleneck transfers better, achieving competitive selection quality on TAVERN and Jiugong. Skeletonized melodies also improve BM25-based fragment retrieval, boosting Recall@K and MRR while reducing query time. Overall, the results suggest that learning skeletons as latent subsequences yields more robust transfer than pseudo-label imitation.
Primary: Beijing University of Posts and Telecommunications
All Institutions: Beijing University of Posts and Telecommunications, Central Conservatory of Music
MeloBottleneck introduces a novel self-supervised framework for melody skeleton extraction that models skeletons as latent subsequences, demonstrating superior cross-domain transfer and practical utility in music retrieval compared to prior pseudo-label or heuristic methods.
The paper proposes MeloBottleneck, a self-supervised framework for melody skeleton extraction. The core innovation lies in formulating the skeleton not as a set of independent note-wise labels, but as a length-controlled, order-preserving latent subsequence. This is a significant methodological shift from prior work that relies on heuristic rules or pseudo-label classifiers. The architecture employs a Transformer encoder to select top-K notes, followed by a deterministic "rhythmic closure" operator that converts the sparse subsequence into a standalone melody by absorbing deleted time into preceding notes. Training is multi-objective: it includes reconstruction loss (re-ornamentation), a frozen autoregressive melody prior for musicality, and ornament-invariant consistency losses to ensure stability under procedural ornamentation. The approach effectively decouples note selection from rhythmic coherence, addressing a key limitation in previous discrete latent compression methods for music.
The evaluation is comprehensive, covering three distinct regimes: synthetic out-of-distribution (OOD) ornament-to-skeleton, cross-domain variation-to-theme (TAVERN), and in-domain ornamented-to-gongche (Jiugong). The results demonstrate a clear trade-off: while a pseudo-label classifier excels on synthetic data where it directly imitates the generator, MeloBottleneck shows superior transferability to zero-shot and cross-domain settings. This supports the central claim that learning structural representations is more robust than imitation. Additionally, the paper provides a practical downstream application, showing that skeletonized melodies improve BM25-based fragment retrieval by reducing noise and query time. The ablation studies rigorously validate the contribution of each loss component, particularly the ornament exclusion and consistency losses.
The paper provides detailed implementation specifics, including the SimpleMono event representation, hyperparameters for the Transformer backbone (6 encoder/3 decoder layers), and specific loss coefficients. The authors also provide a link to supplementary materials including code and a demo web page, which significantly enhances reproducibility. The use of standard symbolic music representations and clear algorithmic descriptions of the rhythmic closure and selection mechanisms makes the method straightforward to implement.
The authors acknowledge several limitations. First, the method relies on a procedural ornamenter for training views; if this ornamenter does not match the target style, the invariance signal may be suboptimal. Second, the evaluation of selection quality uses an oracle compression ratio, meaning free-length prediction is only indirectly tested. Third, the rhythmic closure operator is simple (absorbing time into the preceding note), which may not preserve musical phrasing as well as more complex closure strategies. Finally, the focus is strictly on monophonic melodies, limiting immediate applicability to polyphonic music without significant extension.
This work has significant implications for symbolic music analysis, retrieval, and compression. By providing a robust method to extract structural skeletons, it facilitates better theme tracing, variation recognition, and efficient indexing of large music corpora. The self-supervised nature of the approach reduces reliance on expensive human annotations, making it scalable to diverse musical traditions. The improved retrieval performance suggests practical utility in digital library systems and music information retrieval tools. MeloBottleneck introduces a novel self-supervised framework for melody skeleton extraction that models skeletons as latent subsequences, demonstrating superior cross-domain transfer and practical utility in music retrieval compared to prior pseudo-label or heuristic methods.
While recent few-step sampling text-to-audio generation models like MeanAudio substantially accelerate generation by modeling average velocities, their strict one-step generation quality still lags significantly behind multi-step counterparts. We propose FdAudio to bridge this gap. Unlike MeanAudio, which relies solely on regression against target velocity fields, our post-training approach optimizes the final one-step distribution directly across pre-trained embedding spaces via a multi-representation Fréchet-distance (FD) loss. Crucially, to prevent the multi-step degradation that naive post-training with FD-loss causes, we introduce a MeanFlow consistency objective as a structural anchor. Results demonstrate that FdAudio establishes state-of-the-art one-step T2A generation quality among few-step systems, yielding an 11.4% reduction in FD score and a 28.8% improvement in FAD score relative to the baseline MeanAudio framework. Notably, we solve FD post-training's naive multi-step degradation issue by proposing the MeanFlow anchor, enabling a 25-step sampling path to maintain high-fidelity audio synthesis that matches or surpasses strong multi-step models at a fraction of their computational latency.
Primary: National Taiwan University
All Institutions: National Taiwan University, Amazon
FdAudio presents a practical and effective post-training strategy for one-step text-to-audio generation, successfully bridging the quality gap with multi-step models while preserving sampling flexibility through MeanFlow anchoring. The work is technically sound, well-evaluated, and addresses a relevant problem in the field of generative audio.
The paper addresses a critical issue in flow-matching based text-to-audio generation: the trade-off between one-step generation quality and multi-step sampling flexibility. The authors propose FdAudio, a post-training framework that combines Fréchet Distance (FD) loss with a MeanFlow consistency anchor. The core technical contribution is the identification that naive FD post-training collapses the velocity field, destroying multi-step sampling capability. By introducing a MeanFlow regularization term, they preserve the trajectory while optimizing the one-step output distribution. The methodology is sound and logically derived, leveraging existing concepts (FD loss, MeanFlow) in a novel combination. However, the novelty is incremental; it is an effective engineering solution rather than a fundamental theoretical breakthrough. The use of a queue-based estimator for FD loss is also a known technique in GAN/diffusion post-training.
The experimental evaluation is comprehensive and convincing. The authors demonstrate state-of-the-art one-step performance on AudioCaps, significantly outperforming MeanAudio and other few-step baselines. Crucially, they show that their method retains high-quality multi-step sampling, a feat that naive FD post-training fails to achieve. The ablation studies are thorough, analyzing the impact of the MeanFlow anchor weight, the choice of encoders for the multi-representation FD loss, and the comparison against various baselines. The inclusion of subjective human evaluation (MOS) alongside objective metrics (FAD, CLAP, IS) strengthens the claims. The latency measurements further validate the practical utility of the approach.
The paper provides sufficient detail for reproduction. It specifies the base model (MeanAudio-S-Full), the training data (AudioCaps + WavCaps subset), the hyperparameters (learning rate, batch size, queue size), and the specific encoders used. The code and demo links are provided, which greatly enhances reproducibility. The training budget is mentioned as limited (1 epoch on 80k samples), which might affect generalizability, but the setup is clear.
The primary limitation is the reliance on a limited training budget (1 epoch on a subset of data). While the results are strong, it is unclear if the method scales linearly with more data or longer training. Additionally, the method is specific to flow-matching/consistency models and may not generalize to other generative paradigms without adaptation. The improvement in multi-step quality is present but not revolutionary; the main gain is in one-step quality while maintaining multi-step capability.
This work has significant implications for real-time audio generation applications, such as interactive games, virtual assistants, and content creation tools, where low latency is critical. By enabling high-quality one-step generation without sacrificing the flexibility of multi-step sampling, it provides a more robust solution for diverse deployment scenarios. It also contributes to the broader understanding of post-training techniques for generative models, highlighting the importance of trajectory preservation. FdAudio presents a practical and effective post-training strategy for one-step text-to-audio generation, successfully bridging the quality gap with multi-step models while preserving sampling flexibility through MeanFlow anchoring. The work is technically sound, well-evaluated, and addresses a relevant problem in the field of generative audio.