In this report, we introduce Qwen-Music, a powerful music generation model capable of producing highly musical and high-fidelity songs with complete vocal singing. Qwen-Music supports two core tasks: Text to Music Generation, which create entirely new songs from text descriptions, lyrics, and musical attributes, and Cover Song Generation, which reinterprets existing songs with different styles and vocal characteristics. Architecturally, Qwen-Music integrates three core components: Qwen-Music-Tokenizer, Qwen-Music-LLM, and Qwen-Music-Render. Qwen-Music-Tokenizer compresses audio into a 25 Hz single-codebook stream of Music Semantic Tokens that preserve semantic and melodic information for LLM prediction. Based on these tokens, Qwen-Music-LLM performs autoregressive music semantic modeling, with a key novelty being a melody-token-based chain-of-thought (Melody-CoT) mechanism that plans melodies before full-song generation, improving creativity, musicality, structural coherence, and reference-audio-based melody cloning. To overcome the fidelity limitations of discrete semantic tokens, Qwen-Music-Render performs generative stereo rendering, enriching acoustic details and producing high-fidelity stereo waveforms. Finally, we train Qwen-Music-LLM on more than 5 million hours of multilingual music data covering hundreds of languages. We first apply quality-aware pre-training curriculum, then use progressive post-training, comprising supervised initialization, offline DPO, and online GSPO, to further improve musicality and instruction-following ability. Across 600 Chinese and English prompts, Qwen-Music achieves state-of-the-art results in 13 of 16 objective musicality and audio-quality metrics. Professional evaluators also prefer Qwen-Music over leading proprietary systems. For cover song generation, Qwen-Music preserves reference melodies more accurately than leading proprietary systems.
Primary: Qwen Team
All Institutions: Qwen Team
This paper introduces Qwen-Music, a state-of-the-art music generation system that effectively bridges semantic composition and acoustic rendering through a novel Melody-CoT mechanism and a high-fidelity generative renderer, achieving superior performance in both objective metrics and human preference tests.
The paper presents a comprehensive three-stage architecture for music generation: a semantic tokenizer, an autoregressive language model (LLM), and a generative renderer. The core methodological contributions lie in the specific design choices for handling the complexity of full-song generation. First, the **Qwen-Music-Tokenizer** employs a four-stage training recipe (BestRQ pretraining, causal adaptation, multi-task SFT, and VQ insertion) to create a 25 Hz single-codebook stream. This is a significant engineering contribution, as it balances semantic richness with computational tractability for LLMs. Second, the **Melody-CoT** mechanism is a notable innovation. By explicitly planning melody tokens before generating full-mixture semantic tokens, the model addresses the "composition vs. rendering" mismatch. This chain-of-thought approach allows for better structural coherence and enables cover song generation by conditioning on reference melody tokens while allowing style/timbre changes. The use of relative MIDI representations for melody tokens is a smart design to prevent leakage of absolute pitch/key information that might conflict with style tags. Third, the **Qwen-Music-Render** utilizes a Diffusion Transformer (DiT) conditioned on both semantic tokens and textual descriptions, followed by a novel **Spec-VAE** and **Band-Mode Refiner**. The refiner’s frequency-adaptive correction (phase-only for low, joint for mid, magnitude-only for high) is a sophisticated touch to address specific spectral artifacts. The training pipeline is also rigorous, featuring a quality-graded pre-training curriculum (Q1-Q6 buckets) and a progressive post-training alignment (SFT -> Offline DPO -> Online GSPO), which is state-of-the-art practice for aligning generative models with human preference.
The evaluation is extensive and robust. The authors conduct blind A/B preference tests with 50 professional human raters, showing Qwen-Music outperforms leading proprietary systems (Suno V5.5, MiniMax, Mureka) with a 59.1% win rate. They also report performance on the Artificial Analysis leaderboard. Objective evaluations cover 16 metrics across SongBench, SongEval, and AudioBox-Aesthetic, where Qwen-Music leads in 13/16 metrics. The cover song generation evaluation is particularly strong, demonstrating superior melody preservation compared to baselines. The inclusion of a detailed acoustic data quality pipeline for training data filtering (detecting fake stereo, noise floors, cutoffs) adds credibility to the high-fidelity claims. The ablation studies on the renderer and the effectiveness of the Melody-CoT mechanism are well-supported by the results.
The paper provides detailed architectural specifications, including layer counts, parameter sizes, and training objectives. The tokenizer training stages are clearly defined. The post-training pipeline (SFT, DPO, GSPO) is described with sufficient detail for replication by other research groups. However, the specific "quality-aware" reward models and the exact composition of the 5 million-hour dataset are proprietary and not fully disclosed, which limits exact reproducibility of the pre-training phase. The code for the tokenizer and renderer is not explicitly linked in the provided text, though the Qwen team typically releases code for their models.
The paper relies heavily on proprietary baselines (Suno, MiniMax) for comparison, which makes independent verification difficult. The "quality-graded" pre-training assumes the availability of a reliable internal reward model, which may not be generalizable to all settings. The Melody-CoT mechanism, while effective, adds complexity and inference time. The model's performance on non-vocal music or highly abstract musical forms (e.g., free jazz, classical symphonies without lyrics) is less emphasized compared to vocal pop/rock genres, which dominate the training data. The reliance on a single codebook for the tokenizer might limit the expressiveness compared to multi-codebook approaches (like Encodec) in capturing fine-grained timbral details, although the renderer is designed to mitigate this.
Qwen-Music represents a significant step forward in accessible, high-fidelity music generation. By open-sourcing the technical report and potentially the model (implied by the Qwen lineage), it democratizes access to state-of-the-art music AI. The ability to generate cover songs and control musical attributes empowers creators. However, the potential for misuse in generating copyrighted material or deepfake vocals remains a concern, necessitating robust safety filters and watermarking, which are not detailed in this technical report. The high computational requirements for training such large models also contribute to the centralization of AI capabilities. This paper introduces Qwen-Music, a state-of-the-art music generation system that effectively bridges semantic composition and acoustic rendering through a novel Melody-CoT mechanism and a high-fidelity generative renderer, achieving superior performance in both objective metrics and human preference tests.
Current voice AI benchmarks typically evaluate isolated capabilities such as speech intelligibility, word error rate, or text-based dialogue quality, but they rarely test whether systems harness the acoustic information that distinguishes spoken language from its textual representation. To this end, we introduce the Real World Voice EQ Bench, a multidimensional benchmark for evaluating voice AI across text-to-speech (TTS), speech-to-speech (STS), speech understanding (SU), and automatic speech recognition (ASR). Our evaluations indicate that performance is highly dimension-specific. For TTS, naturalness, expressiveness, identity stability, and reliability are largely independent evaluation dimensions. For STS, access to audio does not guarantee use of vocal affect, and some agents remain largely transcript-driven. For SU, models perform unevenly across paralinguistic tasks. For ASR, real world accent, emotion, noise, and conversational conditions expose failures that are not captured by established clean-speech benchmarks. Together, these results show that voice AI should be evaluated as a profile of acoustic, expressive, interactional, and robustness capabilities rather than by a single aggregate score.
Primary: Hume AI Research
All Institutions: Hume AI Research
The paper presents RW-Voice-EQ Bench, a rigorous, multidimensional benchmark that reveals the dimension-specific nature of voice AI performance, exposing critical gaps in current evaluation practices such as benchmark optimization in ASR and the underutilization of paralinguistic cues in STS systems.
The paper introduces a comprehensive, multidimensional benchmarking framework called RW-Voice-EQ Bench, designed to evaluate voice AI systems across four distinct domains: Text-to-Speech (TTS), Speech-to-Speech (STS), Speech Understanding (SU), and Automatic Speech Recognition (ASR). The methodology is rigorous, separating linguistic content from paralinguistic acoustic cues. For perceptual tasks (TTS, STS), it employs large-scale human evaluation (over 1 million ratings) using specific rubrics and mixed-effects models to control for rater bias. For objective tasks (SU, ASR), it uses label-based scoring. A significant methodological contribution is the diagnostic suite for "benchmark optimization" (or "benchmaxxing") in ASR, which tests for memorization via consensus disagreement, audio masking, orthographic switching, and synthetic voice probes. The paper also validates Speech-Language Models (SLMs) as judges against human raters, providing a nuanced analysis of where automated judges succeed (verification-dependent tasks) and fail (open-ended perceptual tasks).
The experimental evaluation is extensive, covering 31 TTS configurations, 15 STS configurations, and analyzing top ASR models on the Hugging Face leaderboard. The results clearly demonstrate that performance is highly dimension-specific; no single model dominates all categories. For instance, in TTS, models strong in expressiveness (e.g., Gemini 3.1 Flash) differ from those strong in reliability or long-form stability (e.g., Kokoro-82M). In STS, the paper highlights that many systems remain largely transcript-driven, failing to leverage vocal affect even when present. The ASR analysis provides compelling evidence that high WER scores on clean benchmarks do not correlate with robustness to accents, noise, or emotional speech, and that many top models exhibit signs of overfitting to benchmark artifacts. The statistical analysis using mixed-effects models and Spearman correlations adds robustness to the findings.
The paper provides significant detail on the evaluation setup, including the number of raters, rating scales, and the specific dimensions evaluated. The benchmark and judge leaderboards are hosted on Hugging Face Spaces, ensuring accessibility. However, the full dataset of prompts and the specific code for the mixed-effects modeling pipeline are not explicitly linked in the provided text, though the Hugging Face links suggest open resources. The detailed description of the "benchmark optimization" diagnostics allows for replication of those specific tests.
The primary limitation is the reliance on human evaluation for TTS and STS, which is resource-intensive and potentially subject to demographic biases, although the paper attempts to mitigate this with large sample sizes and rater filtering. The STS evaluation, while innovative, is limited to specific scripted scenarios (hostile, urgent, ambiguous tone) and may not capture the full breadth of spontaneous conversational dynamics. The ASR benchmark optimization analysis is preliminary, focusing on VoxPopuli, and the authors acknowledge the need for future work to expand this to other datasets. Additionally, the SLM judge validation is currently limited to TTS, leaving STS judge reliability less explored.
This paper has substantial impact on the voice AI community by challenging the reliance on single aggregate scores and clean-speech benchmarks. It provides a roadmap for more realistic and robust evaluation of voice systems, which is critical for deployment in real-world applications. By exposing benchmark optimization in ASR, it encourages the development of more generalizable models and the use of withheld test sets. The multidimensional profile approach helps developers select models based on specific use-case requirements (e.g., customer service vs. entertainment), driving more targeted research and development. The paper presents RW-Voice-EQ Bench, a rigorous, multidimensional benchmark that reveals the dimension-specific nature of voice AI performance, exposing critical gaps in current evaluation practices such as benchmark optimization in ASR and the underutilization of paralinguistic cues in STS systems.
Neural audio codecs with residual vector quantization (RVQ) normally treat all frequencies uniformly, so their codebooks become spectrally entangled. Truncating stages then removes an unpredictable mix of frequencies. Parallel band decomposition addresses this by splitting audio into independent bands, but fragments the latent space and loses cross-frequency coherence. We introduce HARP (Harmonic-Aware Residual Partitioning), a training strategy that partitions RVQ stages into frequency-ordered groups where each group refines its target band while the decoder retains access to all lower frequencies. Overtones are reconstructed in the context of their fundamentals, preserving coherence that parallel methods lose. HARP requires no architectural changes; it only modifies the training loss, leaving inference identical to standard RVQ. On speech, music, and general audio, HARP outperforms both standard RVQ and parallel decomposition. MUSHRA listening tests also show perceptual improvements.
Primary: Georgia Institute of Technology
All Institutions: Georgia Institute of Technology, The Chinese University of Hong Kong, Tencent Music Entertainment
HARP introduces a training-only strategy for neural audio codecs that partitions RVQ stages into frequency-ordered groups with cumulative decoding and soft subband supervision, effectively addressing spectral entanglement and improving both objective metrics and perceptual quality, particularly at low bitrates, without architectural changes.
The paper proposes HARP, a training-time strategy for Residual Vector Quantization (RVQ) in neural audio codecs. The core innovation lies in partitioning RVQ stages into frequency-ordered groups and enforcing spectral specialization through a novel loss function. Key components include: 1) Cumulative Decoding: Ensuring higher-frequency stages are reconstructed in the context of lower-frequency content to preserve harmonic coherence. 2) Subband Contribution Supervision: Supervising the isolated contribution of each group to its target band using stop-gradient operators to prevent gradient interference. 3) Soft Band Weighting: Using learnable Gaussian weights over mel bins to guide specialization without hard spectral cutoffs. The approach is architecturally agnostic, requiring only modifications to the training objective. This is a clever and relatively simple modification that addresses the "spectral entanglement" problem inherent in standard RVQ, where early and late stages capture similar frequency content.
The authors evaluate HARP against standard DAC (same architecture, no band supervision) and BSCodec (parallel band decomposition). They use a diverse dataset including MUSDB18-HQ, LibriTTS, and AudioSet. Metrics include SI-SDR, KAD, PESQ, STOI, and MUSHRA listening tests. Results show consistent improvements in SI-SDR and KAD over both baselines, with particularly notable gains at low bitrates (2.6 kbps). The ablation studies effectively isolate the contributions of cumulative decoding, soft boundaries, and stop-gradient mechanisms. The harmonic coherence analysis using synthetic tones provides strong evidence for the preservation of phase/amplitude relationships. The MUSHRA tests confirm perceptual improvements, especially in bass fullness and timbral naturalness.
The paper provides detailed descriptions of the architecture, training objectives, and hyperparameters. The code is made publicly available on GitHub. The use of standard components (DAC backbone, EMA updates) and clear algorithmic descriptions enhances reproducibility. The training cost increase (2-3x) is noted, which is a practical consideration but does not hinder reproducibility.
The primary limitation is that the performance gains, while consistent, are modest in absolute terms (e.g., +0.25 to +0.96 dB SI-SDR). The method relies on the assumption that natural audio has a specific energy distribution (low-to-high), which might not hold for all types of audio (e.g., highly inharmonic or noise-heavy content, though the paper argues it still applies). The method is training-only, so it does not offer architectural efficiency gains. The "soft" boundaries mean that some spectral leakage or imperfect separation might still occur, potentially limiting the maximum potential of the specialization compared to hard architectural splits (though the paper argues this is offset by coherence).
This work contributes to the field of neural audio compression by providing a principled way to impose spectral structure on RVQ without architectural complexity. This can lead to more robust and interpretable audio codecs, which are foundational for speech synthesis, music generation, and audio understanding tasks. The improved low-bitrate performance is particularly relevant for streaming applications. The open-source release facilitates further research in this area. HARP introduces a training-only strategy for neural audio codecs that partitions RVQ stages into frequency-ordered groups with cumulative decoding and soft subband supervision, effectively addressing spectral entanglement and improving both objective metrics and perceptual quality, particularly at low bitrates, without architectural changes.
Neural audio codecs with residual vector quantization (RVQ) normally treat all frequencies uniformly, so their codebooks become spectrally entangled. Truncating stages then removes an unpredictable mix of frequencies. Parallel band decomposition addresses this by splitting audio into independent bands, but fragments the latent space and loses cross-frequency coherence. We introduce HARP (Harmonic-Aware Residual Partitioning), a training strategy that partitions RVQ stages into frequency-ordered groups where each group refines its target band while the decoder retains access to all lower frequencies. Overtones are reconstructed in the context of their fundamentals, preserving coherence that parallel methods lose. HARP requires no architectural changes; it only modifies the training loss, leaving inference identical to standard RVQ. On speech, music, and general audio, HARP outperforms both standard RVQ and parallel decomposition. MUSHRA listening tests also show perceptual improvements.
Primary: Georgia Institute of Technology
All Institutions: Georgia Institute of Technology, The Chinese University of Hong Kong, Tencent Music Entertainment
HARP introduces a training-only strategy for neural audio codecs that partitions RVQ stages into frequency-ordered groups with cumulative decoding and soft subband supervision, effectively addressing spectral entanglement and improving both objective metrics and perceptual quality, particularly at low bitrates, without architectural changes.
The paper proposes HARP, a training-time strategy for Residual Vector Quantization (RVQ) in neural audio codecs. The core innovation lies in partitioning RVQ stages into frequency-ordered groups and enforcing spectral specialization through a novel loss function. Key components include: 1) Cumulative Decoding: Ensuring higher-frequency stages are reconstructed in the context of lower-frequency content to preserve harmonic coherence. 2) Subband Contribution Supervision: Supervising the isolated contribution of each group to its target band using stop-gradient operators to prevent gradient interference. 3) Soft Band Weighting: Using learnable Gaussian weights over mel bins to guide specialization without hard spectral cutoffs. The approach is architecturally agnostic, requiring only modifications to the training objective. This is a clever and relatively simple modification that addresses the "spectral entanglement" problem inherent in standard RVQ, where early and late stages capture similar frequency content.
The authors evaluate HARP against standard DAC (same architecture, no band supervision) and BSCodec (parallel band decomposition). They use a diverse dataset including MUSDB18-HQ, LibriTTS, and AudioSet. Metrics include SI-SDR, KAD, PESQ, STOI, and MUSHRA listening tests. Results show consistent improvements in SI-SDR and KAD over both baselines, with particularly notable gains at low bitrates (2.6 kbps). The ablation studies effectively isolate the contributions of cumulative decoding, soft boundaries, and stop-gradient mechanisms. The harmonic coherence analysis using synthetic tones provides strong evidence for the preservation of phase/amplitude relationships. The MUSHRA tests confirm perceptual improvements, especially in bass fullness and timbral naturalness.
The paper provides detailed descriptions of the architecture, training objectives, and hyperparameters. The code is made publicly available on GitHub. The use of standard components (DAC backbone, EMA updates) and clear algorithmic descriptions enhances reproducibility. The training cost increase (2-3x) is noted, which is a practical consideration but does not hinder reproducibility.
The primary limitation is that the performance gains, while consistent, are modest in absolute terms (e.g., +0.25 to +0.96 dB SI-SDR). The method relies on the assumption that natural audio has a specific energy distribution (low-to-high), which might not hold for all types of audio (e.g., highly inharmonic or noise-heavy content, though the paper argues it still applies). The method is training-only, so it does not offer architectural efficiency gains. The "soft" boundaries mean that some spectral leakage or imperfect separation might still occur, potentially limiting the maximum potential of the specialization compared to hard architectural splits (though the paper argues this is offset by coherence).
This work contributes to the field of neural audio compression by providing a principled way to impose spectral structure on RVQ without architectural complexity. This can lead to more robust and interpretable audio codecs, which are foundational for speech synthesis, music generation, and audio understanding tasks. The improved low-bitrate performance is particularly relevant for streaming applications. The open-source release facilitates further research in this area. HARP introduces a training-only strategy for neural audio codecs that partitions RVQ stages into frequency-ordered groups with cumulative decoding and soft subband supervision, effectively addressing spectral entanglement and improving both objective metrics and perceptual quality, particularly at low bitrates, without architectural changes.
Singing Quality Assessment (SQA) has become increasingly important for practical multimedia applications and Music AI systems, yet existing studies predominantly focus on short singing clips and remain insufficient for full-length songs. Unlike clip-level assessment, full-length song SQA requires modeling how singing quality varies across different audio segments and how these local variations influence the overall evaluation of vocal performance. Moreover, the scarcity of segment-level annotations makes effective supervision challenging, as directly assigning a single overall score label to every segment tends to treat different segment qualities as equivalent. To address these challenges, we propose SongSQA, a two-stage framework for full-length song SQA. In the first stage, a Segment Score Predictor is trained with pseudo labels generated by a pre-trained teacher model, enabling segment-level singing quality prediction without requiring manual segment annotations. In the second stage, a Song Quality Aggregator integrates segment features and predicted segment scores into unified segment embeddings, and employs a learnable song embedding together with self-attention to capture the connection between segment-level vocal performance and overall song quality. In this way, SongSQA dynamically aggregates critical quality cues across the song to produce a holistic quality prediction, while also generating a temporal segment-level quality curve. Experimental results demonstrate the effectiveness of SongSQA for full-length song SQA, achieving up to a 13.95% relative improvement in KTAU over the strongest baseline, while consistently improving other evaluation metrics across all datasets.
Primary: The Chinese University of Hong Kong, Shenzhen
All Institutions: Xi'an Jiaotong University, The Chinese University of Hong Kong, Shenzhen
[One sentence main contribution]. SongSQA introduces a two-stage framework for full-length song singing quality assessment that utilizes pseudo-labels from a teacher model for segment-level supervision and a Transformer-based aggregator to capture non-linear segment interactions. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a solid technical contribution to the field of audio quality assessment by shifting the paradigm from clip-level to full-song evaluation. The proposed methodology is innovative in its handling of the annotation scarcity problem through a teacher-student pseudo-labeling strategy, which is a clever adaptation of semi-supervised learning principles to a specific perceptual task. The integration of segment-level predictions into a global context via self-attention allows the model to mimic human perceptual weighting of vocal highlights and flaws. The experimental results are strong and well-supported by ablation studies and subjective evaluations. While the novelty is incremental in terms of architectural components (using Transformers and SSL features is common), the specific application to full-song SQA with temporal curves and the pseudo-labeling strategy for segment supervision is a valuable addition to the literature. The work is technically sound and addresses a clear gap in current SQA systems.
The paper proposes SongSQA, a two-stage framework for full-length song singing quality assessment (SQA). The core methodological innovation lies in addressing the lack of segment-level annotations by using a pre-trained teacher model to generate pseudo-labels for a Segment Score Predictor. This predictor extracts segment-level features and scores, which are then fused and aggregated by a Song Quality Aggregator using a Transformer with a learnable [CLS]-like token. The approach effectively bridges the gap between clip-level SQA and full-song assessment. The use of pseudo-labels from a specialized teacher (SongEval) rather than simple label sharing is a pragmatic and technically sound solution to the data scarcity problem. The aggregation mechanism using self-attention to model the non-linear influence of segments on overall quality is well-motivated by human perception studies.
The experimental evaluation is robust, utilizing two datasets: the public Lyra-SA dataset and a proprietary Internal Dataset annotated by experts. The paper compares SongSQA against strong baselines including clip-level SOTA models (UTMOSv2, PS-SQA, RAMP+) and SSL-based full-song adaptations. SongSQA achieves significant relative improvements (up to 13.95% in KTAU) over the strongest baseline. The ablation studies are thorough, validating the contributions of the teacher model choice, feature fusion strategy, and aggregation design. The case study and subjective evaluation further support the perceptual validity of the temporal score curves generated by the model. The results on the Internal Dataset show even higher performance, suggesting the model benefits from high-quality expert annotations.
The paper provides detailed implementation details, including dataset splits, segment duration (10s with 5s overlap), model architecture parameters (number of layers, loss weights), and optimizer settings. The use of publicly available datasets (Lyra-SA) and pre-trained models (MuQ, SongEval) enhances reproducibility. However, the Internal Dataset is proprietary, which limits the ability to fully replicate the results on that specific benchmark. The code is not explicitly linked in the text provided, though a demo page is available.
The primary limitation is the reliance on a pre-trained teacher model for pseudo-label generation. The quality of the final model is thus bounded by the teacher's ability to discern segment-level singing quality. If the teacher model has biases or errors, these will propagate to the student model. Additionally, the performance on the proprietary Internal Dataset, while impressive, cannot be directly compared to other works due to the lack of public access. The model's performance on non-Mandarin/Cantonese singing (only a small proportion in the internal dataset) is not thoroughly evaluated, limiting generalizability claims.
This work has significant implications for Music AI, particularly in the evaluation of Singing Voice Synthesis (SVS) and Singing Voice Conversion (SVC) systems. By providing a tool for fine-grained, full-song assessment, it enables more precise feedback for model improvement and content moderation on multimedia platforms. The generated temporal score curves can also be used for music education and analysis. The methodology of using pseudo-labels for segment-level supervision in audio quality assessment could be adapted to other domains lacking granular annotations. [One sentence main contribution]. SongSQA introduces a two-stage framework for full-length song singing quality assessment that utilizes pseudo-labels from a teacher model for segment-level supervision and a Transformer-based aggregator to capture non-linear segment interactions. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a solid technical contribution to the field of audio quality assessment by shifting the paradigm from clip-level to full-song evaluation. The proposed methodology is innovative in its handling of the annotation scarcity problem through a teacher-student pseudo-labeling strategy, which is a clever adaptation of semi-supervised learning principles to a specific perceptual task. The integration of segment-level predictions into a global context via self-attention allows the model to mimic human perceptual weighting of vocal highlights and flaws. The experimental results are strong and well-supported by ablation studies and subjective evaluations. While the novelty is incremental in terms of architectural components (using Transformers and SSL features is common), the specific application to full-song SQA with temporal curves and the pseudo-labeling strategy for segment supervision is a valuable addition to the literature. The work is technically sound and addresses a clear gap in current SQA systems.
End-to-end speech language models increasingly represent user speech with speech tokens rather than relying exclusively on cascaded ASR--LLM--TTS pipelines. Although these tokens support expressive and low-latency spoken interaction, they may also preserve sensitive speaker characteristics. We investigate whether exposed speech tokens leak voiceprints and formulate this risk as a speaker inversion attack. We introduce Audio BERT (AuB), a trainable model that constructs token embeddings from discrete codebooks and aggregates them into speaker-sensitive representations, and propose SpInv, a two-stage inversion method built on AuB to recover embeddings in the space of an attacker-specified speaker encoder. We evaluate Moshi, Higgs3, Kimi-Audio, and Qwen3-Omni using speaker-disjoint protocols on the VoxCeleb dataset. Extensive experiments show that, with only three seconds of frontend output, SpInv achieves cosine similarities above 0.70 in the attacker-specified speaker-encoder space.
Primary: Harbin Institute of Technology
All Institutions: Harbin Institute of Technology
The paper presents a significant and timely contribution to the intersection of speech processing and machine learning security. By formulating and executing a speaker inversion attack against modern end-to-end speech language models, it exposes a critical privacy vulnerability in emerging AI architectures. The proposed SpInv method is technically robust, leveraging advanced distillation and contrastive learning techniques to recover speaker embeddings from discrete tokens with high fidelity. The comprehensive evaluation across multiple state-of-the-art models and rigorous experimental protocols strengthens the validity of the findings. This work is essential reading for researchers and practitioners in speech AI, highlighting the urgent need for privacy-preserving design patterns in next-generation speech interfaces.
The paper proposes a novel threat model called "Speaker Inversion Attack" (SIA) against end-to-end speech language models (SLMs). The core methodological contribution is SpInv, a two-stage attack framework. Stage A uses a distillation pretraining approach with an auxiliary model called Audio BERT (AuB) to map discrete speech tokens (or continuous features) to a target speaker embedding space defined by a frozen encoder (e.g., ECAPA-TDNN). This involves alignment loss, relational loss, and self-distillation (DINO-style) to prevent collapse and preserve geometry. Stage B fine-tunes the model with discriminative supervision (ArcFace) to enhance speaker discrimination. The approach is technically sound, leveraging established contrastive and distillation techniques (DINO, VICReg) in a new context. The mapping from discrete tokens to continuous embeddings via a learned lookup table is a clever adaptation to handle the black-box nature of the tokenizers.
The evaluation is comprehensive and rigorous. The authors test against four state-of-the-art SLM frontends (Moshi, Higgs3, Kimi-Audio, Qwen3-Omni), covering different tokenization strategies (residual codebooks, hybrid, continuous). They use speaker-disjoint protocols on VoxCeleb1 and VoxCeleb2, which is crucial for validating generalization. The results are strong: cosine similarities >0.70 are achieved with only 3 seconds of audio. The ablation studies on input duration and target speaker encoder spaces add depth. The fact that they test on "Qwen3-Omni" (which appears to be a very recent or potentially misnamed model in the context of current public releases, possibly referring to Qwen-Audio or a specific internal variant, but treated as a valid target here) suggests the work is timely. The metrics (CosSim, EER, Acc) are appropriate for the task.
The paper provides detailed descriptions of the AuB architecture, loss functions, and training hyperparameters (learning rates, batch sizes, loss weights). The datasets (VoxCeleb1/2) and target encoders (ECAPA-TDNN, etc.) are publicly available. The threat model assumes query access to the frontend, which is a standard and realistic assumption for API-based services. The code is not explicitly linked, but the methodological description is sufficient for reproduction by a competent researcher.
The authors acknowledge two main limitations: focusing on embedding recovery rather than waveform reconstruction, and using only VoxCeleb datasets. The reliance on VoxCeleb, while standard, may not fully capture the diversity of "in-the-wild" speech (accents, noise, channel variations) that might be present in real-world SLM deployments. Additionally, the attack requires a training phase where the adversary collects data from the target frontend; while feasible, this is a resource-intensive prerequisite. The paper does not explore defense mechanisms, only the attack.
This work has significant implications for the privacy of users interacting with AI voice assistants and SLMs. It demonstrates that even when raw audio is not transmitted, the intermediate representations (tokens) can leak sensitive biometric information (voiceprints). This necessitates the development of privacy-preserving tokenization techniques or anonymization methods for speech tokens. It serves as a wake-up call for the SLM community to consider privacy in the design of frontend tokenizers. The paper presents a significant and timely contribution to the intersection of speech processing and machine learning security. By formulating and executing a speaker inversion attack against modern end-to-end speech language models, it exposes a critical privacy vulnerability in emerging AI architectures. The proposed SpInv method is technically robust, leveraging advanced distillation and contrastive learning techniques to recover speaker embeddings from discrete tokens with high fidelity. The comprehensive evaluation across multiple state-of-the-art models and rigorous experimental protocols strengthens the validity of the findings. This work is essential reading for researchers and practitioners in speech AI, highlighting the urgent need for privacy-preserving design patterns in next-generation speech interfaces.
Interpretability is critical in clinical decision support. Concept bottleneck frameworks improve it by representing inputs as human-understandable concepts and restricting predictions solely on them. However, research on their use for voice-based health assessment remains limited. In this study, we propose a voice concept bottleneck framework for interpretable health assessment using an audio language model (ALM). The ALM is fine-tuned on a voice quality assessment dataset to enhance its understanding of voice concepts and serves as an independent concept extractor, producing discrete, interpretable scores for a lightweight downstream classifier. The discrete concept scores provide intuitive interpretation, while the lightweight classifier facilitates post-hoc interpretability analyses. Results on depression and dysarthria assessment tasks demonstrate that the proposed framework can flexibly adapt voice concepts to different health conditions and consistently outperforms openSMILE-based and self-supervised speech model-based baselines.
Primary: Columbia University
All Institutions: Columbia University
This paper presents a compelling application of Audio Language Models as interpretable concept bottlenecks for health assessment, demonstrating that decoupling concept extraction from classification can yield both high performance and clinical interpretability, although validation on larger, more diverse datasets is needed to confirm robustness.
The paper proposes a novel framework that leverages Audio Language Models (ALMs), specifically Audio Flamingo Next, as independent concept extractors for interpretable health assessment. The core innovation lies in decoupling the concept extraction (via fine-tuned ALM producing discrete 1-5 ordinal scores) from the downstream classification (via XGBoost). This approach aims to mitigate shortcut learning and enhance interpretability by restricting predictions to human-understandable concepts (Clinical Voice Quality and Speaking Behavior). The methodology is sound in its intent, utilizing LoRA for efficient fine-tuning and aggregating speaker-level statistics for the classifier. However, the novelty is somewhat incremental, as concept bottleneck models (CBMs) are an established paradigm, and applying them to audio via LLMs is a logical but not radically new extension. The specific adaptation of ALMs to produce discrete ordinal concepts rather than continuous embeddings or text descriptions is a notable technical choice.
The experimental evaluation covers two distinct health conditions: depression (Android Corpus) and dysarthria (TORGO corpus). The results demonstrate that the proposed ALM-based framework outperforms baselines using openSMILE and self-supervised models (Vox-Profile). The paper provides SHAP analysis to validate interpretability, showing that features like pitch variability and speech tempo align with clinical expectations for depression. For dysarthria, the separation of CVQ scores across severity levels is shown. While the performance gains are consistent, the baselines are relatively standard for the domain. The comparison with Vox-Profile is fair, but the lack of comparison with other recent SSL-based CBMs or end-to-end interpretable models limits the comprehensiveness of the benchmark. The results are promising but rely on small datasets, particularly for dysarthria (only 15 speakers total), which raises questions about generalizability.
The paper provides sufficient detail regarding the model architecture (Audio Flamingo Next), fine-tuning parameters (LoRA rank, alpha, dropout), and downstream classifier settings (XGBoost hyperparameters). The datasets are publicly available, and the voice activity detection tool is cited. However, the code is not explicitly linked in the provided text, and specific random seeds or exact data splitting procedures beyond the standard protocols are not detailed. The reliance on a specific Hugging Face model checkpoint aids reproducibility, but the fine-tuning process on the PVQD dataset requires careful handling of the discrete label generation which is described but could be more precise.
The primary limitation is the small sample size of the evaluation datasets, especially the dysarthria corpus, which limits the statistical power of the results. The ALM's pretraining bias is acknowledged and partially corrected via fine-tuning, but the paper notes that the model still struggles with certain concepts like abnormal loudness and pitch due to definition mismatches. The discrete 1-5 scale, while interpretable, may lose nuance compared to continuous ratings. Furthermore, the "independent" nature of the concept extractor assumes that the fine-tuned ALM captures the true underlying concepts without leakage from the downstream task, which is hard to guarantee without rigorous ablation studies on concept fidelity.
This work has significant potential impact in clinical decision support systems by providing interpretable voice biomarkers. By making AI predictions transparent through human-understandable concepts, it can help clinicians trust and validate AI outputs. The framework could be extended to other health conditions affecting voice, such as Parkinson's disease or Alzheimer's. However, ethical considerations regarding privacy, bias in health assessment, and the risk of over-reliance on automated diagnostics must be addressed. The use of ALMs also raises questions about the environmental cost and accessibility of such models in resource-constrained clinical settings. This paper presents a compelling application of Audio Language Models as interpretable concept bottlenecks for health assessment, demonstrating that decoupling concept extraction from classification can yield both high performance and clinical interpretability, although validation on larger, more diverse datasets is needed to confirm robustness.
We propose the concept of noise-aware audio self-supervised learning (SSL), whose goal is to encode audio mixtures while suppressing undesired noise, and present Noise-Aware BEATs (NABEATs) as a BEATs-based realization of this framework. Audio SSL models are designed to handle a wide range of audio signals. Consequently, under noisy conditions, they cannot effectively focus on the target sounds relevant to a downstream task, resulting in degraded performance. To address this issue, NABEATs is trained to estimate clean BEATs representations from a noisy audio signal with an auxiliary reference noise input. This reference noise enables the model to account for specific noise characteristics at inference time, thereby achieving better generalization across operating environments. Our experimental evaluations demonstrate that NABEATs significantly improves performance of various downstream tasks under noisy conditions and also generalizes well to unseen noise types.
Primary: Mitsubishi Electric Research Laboratories (MERL)
All Institutions: Mitsubishi Electric Research Laboratories (MERL), Nagoya University
This paper presents a practical and effective method for enhancing the noise robustness of audio self-supervised learning models by conditioning representation learning on reference noise, demonstrating significant improvements in downstream task performance under both seen and unseen noisy conditions.
The paper proposes NABEATs, a noise-aware adaptation of the BEATs self-supervised learning (SSL) model. The core innovation lies in conditioning the representation learning process on a reference noise signal. The authors introduce two mechanisms to integrate this reference noise: Cross-Attention (CA) and Feature-wise Linear Modulation (FiLM). The model is trained via distillation from a frozen, pre-trained BEATs model, aiming to estimate clean representations from noisy inputs. This approach is theoretically sound and addresses a specific gap in general-purpose audio SSL, which typically lacks robustness to environmental noise. The distinction between using a reference noise (NABEATs) versus unconditional denoising (DBEATs) is clearly defined. However, the method is essentially an architectural modification to an existing SSL framework rather than a fundamentally new learning paradigm. The use of reference noise is analogous to techniques in speech enhancement and speaker verification, making the novelty incremental rather than transformative.
The experimental setup is rigorous and well-structured. The authors evaluate NABEATs on six diverse downstream tasks (environmental sound, speech, music) under various noisy conditions using multiple noise datasets (WHAM!, CHiME-3, MUSDB18). The inclusion of an "unseen noise" generalization test (MUSDB18) is a strong point, demonstrating the practical utility of the reference noise mechanism. The results show consistent improvements over the baseline BEATs and the unconditional denoising baseline (DBEATs), particularly in unseen noise conditions. The use of t-SNE visualizations and SNR analysis of representations provides good qualitative and quantitative support for the claims. The evaluation on the DCASE 2025 Task 2 benchmark adds real-world relevance, although the performance gain is modest. The ablation between CA and FiLM variants provides useful insight into how reference information is utilized.
The paper provides sufficient detail for reproduction. It specifies the base model (BEATs_iter3.pt), the additional layer architectures (Transformer layers with MHSA/MHCA, RMSNorm, SwiGLU), training hyperparameters (AdamW, learning rate, batch size, epochs), and dataset sources (FSD50K, WHAM!, DEMAND, QUT-NOISE). The training procedure for distillation is clearly described. However, the code is not publicly linked in the provided text, and some details regarding the exact mixing ratios and SNR distributions for the downstream evaluation splits could be more precise. The use of standard datasets and a widely available base model enhances reproducibility.
The primary limitation is the dependency on a reference noise signal. While the authors argue this is practical in many scenarios (e.g., pre-recorded noise), it is not universally available. The method does not address the case where no reference noise is present. Furthermore, the performance gains, while statistically significant, are not massive, and the model introduces additional complexity and computational overhead compared to the base BEATs. The reliance on MSE loss for distillation may not perfectly align with downstream classification objectives, as noted in the discussion of the CRM-D task. The evaluation is limited to 16kHz audio, which may not reflect high-fidelity audio applications.
This work contributes to the development of robust audio AI systems that can operate effectively in real-world, noisy environments. By improving the generalization of SSL models, it facilitates the deployment of audio understanding systems in diverse settings, from smart homes to industrial monitoring. The concept of noise-aware SSL could inspire similar approaches in other modalities where background interference is a major challenge. However, the potential for misuse in creating robust surveillance or eavesdropping systems is a minor concern, though mitigated by the requirement for reference noise. This paper presents a practical and effective method for enhancing the noise robustness of audio self-supervised learning models by conditioning representation learning on reference noise, demonstrating significant improvements in downstream task performance under both seen and unseen noisy conditions.
Audio mixing style encompasses the artistic and technical decisions a mix engineer makes, including level balancing, spatialization, and the choice, ordering, and parameterization of audio effects (FX) on each stem. FX chains are a key determinant of this style, yet existing approaches to modeling them remain limited. Some operate on stereo mixtures without explicit per-stem FX chain modeling, others fix the number or type of effects per track, and many require differentiable effect implementations or scarce multitrack datasets. We present StemFX, a framework that learns mixing style representations by autoregressively predicting variable-length FX chains on source-separated stems. A Transformer decoder predicts tokenized FX chains autoregressively, while a band-split multi-band CNN encoder with FiLM conditioning captures per-stem spectral structure. To enable large-scale paired training, we extract pseudo-stems from about 105K songs via source separation and augment them using MultiAFx, a toolkit unifying 85 audio effects from 7 Python libraries. Evaluated on mixing style retrieval, StemFX outperforms all baseline models across all tested chain lengths. On paired mixing style transfer, StemFX achieves the best spectral fidelity and the highest listener preference, over 4000 times faster than iterative optimization.
Primary: National Taiwan University
All Institutions: National Taiwan University
This paper presents a significant advancement in automatic mixing and audio representation learning by introducing a scalable, autoregressive framework for FX chain prediction that overcomes data scarcity through a novel pseudo-stem generation pipeline, achieving state-of-the-art performance in both style retrieval and transfer tasks.
The paper proposes StemFX, a novel framework for learning mixing style representations by treating FX chain prediction as an autoregressive sequence generation problem. The architecture combines a band-split multi-band CNN encoder with FiLM conditioning (incorporating hand-crafted mixing features) and a Transformer decoder. A key methodological innovation is the "StemFX Pipeline," which leverages source separation on large-scale unpaired datasets (FMA) to create pseudo-stems, which are then augmented with random FX chains using a unified toolkit (MultiAFx) to generate paired training data at scale. This approach circumvents the scarcity of real multitrack datasets with ground-truth FX metadata. The tokenization scheme for variable-length FX chains (effect names, parameter names, and quantized values) is well-designed for autoregressive modeling. The use of FiLM conditioning to inject explicit mixing statistics into the CNN layers is a strong technical choice that bridges the gap between low-level spectral features and high-level mixing decisions.
The experimental evaluation is comprehensive and robust. The authors evaluate on two tasks: mixing style retrieval and paired mixing style transfer. For retrieval, StemFX significantly outperforms baselines (AFx-Rep, Fx-Encoder++, CLAP, and a controlled contrastive variant BSFiLM-CL) across various chain lengths, demonstrating that the learned embeddings capture mixing style distinct from musical content. The ablation studies effectively isolate the contributions of the encoder architecture, FiLM conditioning, and the data pipeline. The style transfer experiments are particularly compelling, showing superior spectral fidelity (MRSTFT) and human preference (MUSHRA scores) compared to fixed-chain methods and iterative optimization techniques (ST-ITO). The claim of being 4000x faster than iterative optimization is a significant practical advantage. The use of out-of-domain effects (pedalboard) in evaluation tests generalization well.
The paper provides detailed implementation details, including hyperparameters, dataset sizes, and architecture specifications. The release of both the StemFX codebase and the MultiAFx toolkit on GitHub greatly enhances reproducibility. The description of the data generation pipeline, including source separation models and augmentation strategies, is clear enough for other researchers to replicate the training data generation process. The use of standard metrics (MRSTFT, MUSHRA) also aids in reproducibility of evaluation.
The authors acknowledge several limitations. The model is restricted to the 85 effects included in the MultiAFx toolkit; adding new effects requires retraining. The reliance on source separation introduces potential artifacts from the separation model (e.g., SCNet), which may propagate into the learned representations. The use of pseudo-stems from different songs (cross-song mixing) may introduce musical incoherence, although the authors argue this decorrelates content from style. The model currently operates on four stems (vocals, bass, drums, other), which is a coarser decomposition than professional multitracks. Finally, the training data uses random effect chains, which may not reflect the structured, genre-dependent processing choices of human engineers.
StemFX has significant potential impact on music production technology. By enabling efficient and accurate mixing style transfer and retrieval, it can assist producers in learning mixing techniques, automate parts of the mixing workflow, and enhance music information retrieval systems. The open-source release of the toolkit and models promotes further research in audio effect modeling and automatic mixing. The ability to generate interpretable FX chains also provides insights into the decision-making process of mix engineers, potentially serving as an educational tool. This paper presents a significant advancement in automatic mixing and audio representation learning by introducing a scalable, autoregressive framework for FX chain prediction that overcomes data scarcity through a novel pseudo-stem generation pipeline, achieving state-of-the-art performance in both style retrieval and transfer tasks.
In X-lingual automatic speaker verification (ASV), fixed front-end scores vary in reliability with language match, duration, and score source. We propose AMECxSV, an adaptive metadata-driven embedding-fusion calibration backend for metadata-available settings. AMECxSV fuses trial scores with metadata to produce calibrated target posteriors, with optional posterior-confidence abstention; metadata serve as calibration context, not speaker evidence. On a development-derived speaker-disjoint held-out split, score+metadata heads reduce equal error rate (EER) from 3.15% to 2.42% for the official TidyVoice score source and from 0.64% to 0.43% for LI-MSV; the dual-score head reaches 0.43% full-coverage EER. At 0.79 coverage, abstention yields 0.03% accepted-trial EER, not a full-coverage metric. Matched score-only, metadata-permutation, and metadata-only controls support a calibration-context interpretation and limit claims to metadata-available scoring.
Primary: University of Southern California
All Institutions: University of Southern California, Signal Analysis and Interpretation Laboratory (SAIL), University of Southern California
AMECxSV presents a practical and effective metadata-driven calibration framework that improves the reliability of X-lingual speaker verification systems by adaptively adjusting scores based on language and duration context, demonstrating significant gains in calibration quality and error rates through rigorous ablation and control studies.
The paper proposes AMECxSV, a metadata-driven calibration backend for X-lingual Automatic Speaker Verification (ASV). The core methodology involves fusing trial scores from fixed embedding front-ends with metadata (language match, duration) using a lightweight Multi-Layer Perceptron (MLP). The approach is framed as "embedding-fusion calibration," where metadata serves as context for score reliability rather than direct speaker evidence. The method includes a full-coverage variant (AMEC-FC) and a confidence-based abstention variant (AMEC-ABS). The technical contribution is relatively incremental, building upon established logistic/isotonic calibration and multi-system fusion techniques by adding a structured metadata input to an MLP. While the specific application to X-lingual reliability is well-motivated, the architectural novelty is limited to the feature engineering (score summaries, order statistics, interactions) and the specific conditioning on language/duration.
The evaluation is conducted on the TidyVoiceX-ASV development protocol, using a speaker-disjoint held-out split. The authors compare AMECxSV against single-score baselines, global calibration methods, and a multi-score fusion baseline (MultiScore-FC). Results show consistent improvements in Equal Error Rate (EER) and $C_llr$ when metadata is added. The paper includes rigorous controls, such as metadata permutation and matched score-only architectures, to validate that gains are due to the metadata content and not just model capacity. External evaluations on TidyVoice and LI-MSV score sources further support the method's generalizability. The statistical significance is addressed via bootstrap replicates. However, the results are primarily on development splits, and the magnitude of improvement, while consistent, is modest in absolute terms (e.g., 3.15% to 2.42% EER on one source).
The paper provides detailed implementation details, including the architecture of the MLP, training hyperparameters (AdamW, learning rates, batch sizes), and the specific metadata features used. The codebase is linked via an anonymous repository. The dataset split is clearly defined as development-derived, which aids reproducibility but limits direct comparison to official challenge results. The use of fixed, publicly available embedding systems (SpeechBrain, WeSpeaker, etc.) ensures that the front-end components are reproducible.
The primary limitation is the dependency on metadata. The system requires language match and duration information, which may not be available or reliable in all deployment scenarios (e.g., language-blind protocols). The authors acknowledge that language metadata might need to be predicted by a separate LID module, introducing potential error propagation. Additionally, the improvements are source-dependent, and the method does not address the underlying representation learning issues of the front-ends. The abstention mechanism, while effective for reducing risk, reduces coverage, which may not be desirable for all applications.
This work contributes to the robustness and reliability of ASV systems, particularly in multilingual contexts. By providing a framework for adaptive calibration, it helps mitigate the performance degradation caused by language mismatch and short utterances. This can lead to more fair and accurate speaker verification in real-world applications where trial conditions are heterogeneous. The emphasis on calibration and abstention also promotes safer deployment by allowing the system to defer uncertain decisions. AMECxSV presents a practical and effective metadata-driven calibration framework that improves the reliability of X-lingual speaker verification systems by adaptively adjusting scores based on language and duration context, demonstrating significant gains in calibration quality and error rates through rigorous ablation and control studies.
Current voice AI benchmarks typically evaluate isolated capabilities such as speech intelligibility, word error rate, or text-based dialogue quality, but they rarely test whether systems harness the acoustic information that distinguishes spoken language from its textual representation. To this end, we introduce the Real World Voice EQ Bench, a multidimensional benchmark for evaluating voice AI across text-to-speech (TTS), speech-to-speech (STS), speech understanding (SU), and automatic speech recognition (ASR). Our evaluations indicate that performance is highly dimension-specific. For TTS, naturalness, expressiveness, identity stability, and reliability are largely independent evaluation dimensions. For STS, access to audio does not guarantee use of vocal affect, and some agents remain largely transcript-driven. For SU, models perform unevenly across paralinguistic tasks. For ASR, real world accent, emotion, noise, and conversational conditions expose failures that are not captured by established clean-speech benchmarks. Together, these results show that voice AI should be evaluated as a profile of acoustic, expressive, interactional, and robustness capabilities rather than by a single aggregate score.
Primary: Hume AI Research
All Institutions: Hume AI Research
The paper presents RW-Voice-EQ Bench, a rigorous, multidimensional benchmark that reveals the dimension-specific nature of voice AI performance, exposing critical gaps in current evaluation practices such as benchmark optimization in ASR and the underutilization of paralinguistic cues in STS systems.
The paper introduces a comprehensive, multidimensional benchmarking framework called RW-Voice-EQ Bench, designed to evaluate voice AI systems across four distinct domains: Text-to-Speech (TTS), Speech-to-Speech (STS), Speech Understanding (SU), and Automatic Speech Recognition (ASR). The methodology is rigorous, separating linguistic content from paralinguistic acoustic cues. For perceptual tasks (TTS, STS), it employs large-scale human evaluation (over 1 million ratings) using specific rubrics and mixed-effects models to control for rater bias. For objective tasks (SU, ASR), it uses label-based scoring. A significant methodological contribution is the diagnostic suite for "benchmark optimization" (or "benchmaxxing") in ASR, which tests for memorization via consensus disagreement, audio masking, orthographic switching, and synthetic voice probes. The paper also validates Speech-Language Models (SLMs) as judges against human raters, providing a nuanced analysis of where automated judges succeed (verification-dependent tasks) and fail (open-ended perceptual tasks).
The experimental evaluation is extensive, covering 31 TTS configurations, 15 STS configurations, and analyzing top ASR models on the Hugging Face leaderboard. The results clearly demonstrate that performance is highly dimension-specific; no single model dominates all categories. For instance, in TTS, models strong in expressiveness (e.g., Gemini 3.1 Flash) differ from those strong in reliability or long-form stability (e.g., Kokoro-82M). In STS, the paper highlights that many systems remain largely transcript-driven, failing to leverage vocal affect even when present. The ASR analysis provides compelling evidence that high WER scores on clean benchmarks do not correlate with robustness to accents, noise, or emotional speech, and that many top models exhibit signs of overfitting to benchmark artifacts. The statistical analysis using mixed-effects models and Spearman correlations adds robustness to the findings.
The paper provides significant detail on the evaluation setup, including the number of raters, rating scales, and the specific dimensions evaluated. The benchmark and judge leaderboards are hosted on Hugging Face Spaces, ensuring accessibility. However, the full dataset of prompts and the specific code for the mixed-effects modeling pipeline are not explicitly linked in the provided text, though the Hugging Face links suggest open resources. The detailed description of the "benchmark optimization" diagnostics allows for replication of those specific tests.
The primary limitation is the reliance on human evaluation for TTS and STS, which is resource-intensive and potentially subject to demographic biases, although the paper attempts to mitigate this with large sample sizes and rater filtering. The STS evaluation, while innovative, is limited to specific scripted scenarios (hostile, urgent, ambiguous tone) and may not capture the full breadth of spontaneous conversational dynamics. The ASR benchmark optimization analysis is preliminary, focusing on VoxPopuli, and the authors acknowledge the need for future work to expand this to other datasets. Additionally, the SLM judge validation is currently limited to TTS, leaving STS judge reliability less explored.
This paper has substantial impact on the voice AI community by challenging the reliance on single aggregate scores and clean-speech benchmarks. It provides a roadmap for more realistic and robust evaluation of voice systems, which is critical for deployment in real-world applications. By exposing benchmark optimization in ASR, it encourages the development of more generalizable models and the use of withheld test sets. The multidimensional profile approach helps developers select models based on specific use-case requirements (e.g., customer service vs. entertainment), driving more targeted research and development. The paper presents RW-Voice-EQ Bench, a rigorous, multidimensional benchmark that reveals the dimension-specific nature of voice AI performance, exposing critical gaps in current evaluation practices such as benchmark optimization in ASR and the underutilization of paralinguistic cues in STS systems.
We introduce the REAL-TSE Challenge, an IEEE SLT 2026 satellite challenge on target speaker extraction~(TSE) from real conversational recordings. Given a multi-speaker mixture and one or more enrollment utterances from a target speaker, participating systems must recover only the target speech. Unlike simulated read-speech benchmarks, REAL-TSE evaluates Mandarin and English recordings that contain natural overlap, reverberation, noise, channel mismatch, and conversational dynamics. The challenge defines two complementary tracks: an Online track for low-latency streaming extraction and an Offline track for full-context processing. Systems are evaluated with Token Error Rate (TER), Speaker Similarity (SpkSim), DNSMOS, and target-speaker activity F1. This overview paper describes the task definition, datasets, baselines, evaluation protocol, submitted systems, condition-wise findings, and lessons for future real-world TSE benchmarks.
Primary: Nanjing University
All Institutions: Nanjing University, Chinese University of Hong Kong (Shenzhen), Brno University of Technology, Northwestern Polytechnical University, Shanghai Jiao Tong University, NTT, Inc.
The REAL-TSE Challenge establishes a critical new benchmark for real-world Target Speaker Extraction, revealing significant gaps between simulated and real-world performance and highlighting the dangers of metric over-optimization in neural audio evaluation.
This paper introduces the REAL-TSE Challenge, a benchmarking framework rather than a novel algorithmic architecture. The methodological contribution lies in the rigorous definition of a real-world Target Speaker Extraction (TSE) task. It distinguishes itself from prior simulated benchmarks (like LibriMix) by utilizing real conversational recordings with natural overlap, reverberation, noise, and channel mismatch. The challenge is structured into two tracks: Online (low-latency, <100ms) and Offline (full-context). The evaluation methodology is multi-dimensional, employing Token Error Rate (TER), Speaker Similarity (SpkSim), DNSMOS-P808, and Target-Speaker Activity F1. A critical methodological insight presented is the detection of metric over-optimization, specifically regarding DNSMOS-OVRL, leading to the adoption of DNSMOS-P808 for ranking. The paper details the data simulation strategies used by participants (pseudo-labeling, real-data adaptation) and analyzes the impact of channel mismatch and scenario types on performance.
The experimental section provides a comprehensive analysis of 24 systems from 18 teams. It includes baseline results (BSRNN variants) and aggregate performance metrics. The paper conducts a detailed condition-wise analysis on the EVAL-2 set, examining performance across different recording devices (H1, H2, Phone), target-speaker activity ratios, and recording scenarios (meeting, cafe, home, vehicle). Key findings include the significant impact of mixture recording device quality, the lack of consistent correlation between nominal scenario labels and difficulty, and the trade-offs between different metrics (e.g., systems optimizing for SpkSim may suffer in TER). The analysis of DNSMOS reliability is a significant experimental contribution, demonstrating how neural metrics can be gamed without perceptual improvement.
The challenge promotes reproducibility and transparency. The authors release the evaluation toolkit, baseline code (WeSep-based), and the dataset splits (Development and Evaluation). The training policy is open but restricted regarding specific test-set leakage (forbidden to use dev/test splits of source corpora for training). The paper provides detailed descriptions of the latency verification process (perturbation-based) and metric computation. However, as a challenge overview, it does not provide full training code for the winning systems, which is standard for such venues, but it does release the infrastructure for future replication.
The primary limitation is the inherent difficulty of real-world evaluation: the lack of ground-truth clean targets for most real recordings makes absolute performance measurement challenging, relying instead on proxies like ASR and speaker similarity. The paper acknowledges that metric-aware over-optimization is a significant issue, suggesting that current automatic metrics may not fully align with human perception or true extraction quality. Additionally, the "unseen" nature of EVAL-2 means that generalization to entirely new, unrecorded environments remains an open question, though the challenge design attempts to mitigate this by using diverse, newly recorded data.
This work has a substantial impact on the speech processing community by shifting the focus from simulated to real-world TSE evaluation. It highlights the gap between lab-performance and real-world deployment, emphasizing the need for robustness to channel mismatch, noise, and conversational dynamics. The findings on metric reliability (DNSMOS) have broader implications for the entire field of audio quality assessment and neural metric usage. By providing a standardized benchmark and revealing the limitations of current approaches, it guides future research towards more practical, multi-objective optimization and better data simulation techniques. The REAL-TSE Challenge establishes a critical new benchmark for real-world Target Speaker Extraction, revealing significant gaps between simulated and real-world performance and highlighting the dangers of metric over-optimization in neural audio evaluation.
Machine-learnt models are increasingly used to predict ISO 3382-1 room acoustic parameters from sparse measurements, with reported coefficients of determination frequently above 0.85. This paper shows that such figures are often determined by the evaluation protocol rather than by the model. Using a multi-condition measurement campaign in a 264-seat conference hall and a 180-seat concert hall, three model families were evaluated under a factorial protocol ablation: validation splits either row-based or grouped by receiver position, and input features either including measured-at-test quantities or restricted to source-receiver geometry and environmental state. Row-based splits with measured-at-test inputs reproduce the high reported accuracies (mean $R^2$ 0.81 for the core parameters); grouping the splits by position and restricting inputs to information available at an unmeasured position reduces these to 0.09-0.57, reordering the apparent difficulty of parameter classes. A hybrid CNN evaluated with the target's own impulse response as input is shown to exploit it as a position fingerprint rather than as transferable acoustic information; training-only signal access yields no gain for any parameter tested, including reverberation time. Under the deployment-consistent protocol, the spread between Random Forest, the hybrid CNN, and inverse-distance weighting is an order of magnitude smaller than the spread between protocols for a fixed model; the learnt models retain a genuine advantage for sound strength and reverberation time, and the high accuracy of the original pipelines re-emerges as condition interpolation at measured positions (band means 0.80-0.88), a distinct and operationally useful task.
Primary: Alanya Alaaddin Keykubat University
All Institutions: Alanya Alaaddin Keykubat University
This paper provides a critical methodological audit of machine learning in room acoustics, demonstrating that reported high accuracies are often artifacts of evaluation protocols rather than model capability, and establishes a rigorous framework for distinguishing between condition interpolation and spatial prediction.
The paper employs a rigorous factorial ablation study to dissect the evaluation protocols of machine learning models in room acoustics. It introduces a taxonomy distinguishing between "condition interpolation" (predicting parameters at measured positions under varying environmental states) and "spatial prediction" (predicting parameters at unmeasured positions). The methodology involves re-evaluating existing pipelines (Random Forest, Hybrid CNN) under strict "deployment-consistent" protocols where validation folds are grouped by receiver position and inputs are restricted to geometry and environmental state. A key methodological insight is the demonstration that neural networks can exploit the target position's own impulse response as a "position fingerprint" rather than learning transferable acoustic features, particularly in venues with few measurement positions. This approach effectively isolates the source of high reported accuracy, revealing it to be largely an artifact of data leakage via correlated rows and privileged inputs.
The experimental design is robust, utilizing a multi-condition measurement campaign in two distinct halls (a concert hall and a conference hall). The authors systematically vary split designs (row-based vs. grouped) and input features (full vs. geometry-only). The results are compelling: while row-based splits with full inputs yield high $R^2$ scores (0.81), the deployment-consistent protocol reduces these significantly (0.09-0.57), with simple baselines like inverse-distance weighting often matching or exceeding complex neural networks for certain parameters (e.g., clarity). The paper provides a clear reordering of parameter difficulty, showing that sound strength and reverberation time are more predictable spatially than clarity parameters, which depend on fine-grained early reflection structures not captured by the available geometric features. The inclusion of a "privileged information" experiment further strengthens the argument by showing that training-time signal access does not confer a significant advantage for spatial prediction.
The paper demonstrates high commitment to reproducibility. The author explicitly states that evaluation scripts, split definitions, random seeds, and per-fold results will be deposited in a public repository. The underlying measurement data is available upon request. The detailed description of the model architectures, feature sets, and experimental protocols allows for independent verification. The clear distinction between the two evaluation modes and the provision of specific $R^2$ values for each configuration enhance the transparency of the findings.
The primary limitation is the scope of the data: the study is based on a single multi-condition campaign in only two halls. While the findings are likely generalizable to similar indoor environments, they may not hold for outdoor acoustics or highly complex, non-diffuse fields. The author acknowledges this and suggests cross-venue replication is needed. Additionally, the study focuses on specific ISO 3382-1 parameters; the generalizability to other acoustic metrics or perceptual qualities is not addressed. The network architecture, while effective for the task, is not novel in itself, and the paper's contribution lies in the evaluation rather than the model design.
This paper has significant implications for the field of data-driven acoustics and spatial audio. By exposing the fragility of high accuracy claims in sparse measurement scenarios, it urges the community to adopt more rigorous evaluation standards. The proposed reporting checklist provides a practical framework for future research, ensuring that claims about spatial prediction are substantiated by appropriate protocols. This work helps prevent the dissemination of misleading performance metrics and guides the development of more robust and truly generalizable models for room acoustic prediction. It also highlights the enduring value of simple baselines and the importance of understanding what models are actually learning. This paper provides a critical methodological audit of machine learning in room acoustics, demonstrating that reported high accuracies are often artifacts of evaluation protocols rather than model capability, and establishes a rigorous framework for distinguishing between condition interpolation and spatial prediction.
Recent advances in text-to-speech and voice cloning make high-quality spoofing inexpensive and scalable, threatening voice authentication systems, especially automatic speaker verification (ASV). Existing defenses mainly address this threat through binary countermeasures (CMs) for deepfake detection or spoofing-aware speaker verification (SASV), where current systems are dominated by modular ASV-CM fusion and cascaded pipelines. Although large audio language models (LALMs) have shown promise on related audio tasks, including CM and ASV, their use for SASV remains unexplored, despite their capacity to produce natural-language rationales for auditing and robustness beyond discriminative predictions. This work systematically evaluates LALMs for SASV against conventional pipelines under zero-shot prompting, supervised adaptation, reasoning-oriented training, and reinforcement-learning-based optimization. Our results show that pretrained LALMs are near chance in the zero-shot setting, confirming that they are not natively suited to SASV, but that task-specific adaptation closes this gap. We further find that competitive SASV performance can be achieved through several distinct routes. These findings position LALMs as a promising and auditable foundation for unified SASV, while clarifying where conventional cascade systems still lead.
Primary: Applied AI Institute
All Institutions: Applied AI Institute, MIRAI, HSE, MTUCI
This paper provides the first systematic evaluation of Large Audio Language Models for Spoofing-Aware Speaker Verification, demonstrating that while they can be adapted to compete with conventional pipelines, they currently lag in threshold-based metrics and require significant task-specific adaptation to overcome their lack of native security awareness.
The paper proposes a novel adaptation of Large Audio Language Models (LALMs) for Spoofing-Aware Speaker Verification (SASV), a task that traditionally relies on cascaded or fused discriminative pipelines (ASV + CM). The authors formulate SASV as a three-way classification problem (target, non-target, spoof) using LALMs like SALMONN and Qwen-Audio. Key methodological contributions include: 1) A composite loss function combining cross-entropy, Additive Angular Margin (AAM) for speaker discrimination, and binary cross-entropy for spoof detection to balance the competing objectives. 2) Hard-sample mining to focus training on difficult pairs. 3) An exploration of Chain-of-Thought (CoT) reasoning and Group Relative Policy Optimization (GRPO) to generate interpretable rationales for decisions. The approach is technically sound, leveraging parameter-efficient fine-tuning (LoRA) and addressing the specific challenge of balancing speaker identity verification with anti-spoofing detection within a generative framework.
The evaluation is conducted on the ASVspoof 5 dataset, a challenging benchmark for SASV. The results demonstrate that while zero-shot LALMs perform near chance, supervised fine-tuning (SFT) significantly improves performance. The proposed method with composite losses and hard mining achieves competitive results against strong baselines (ECAPA-TDNN + WavLM/W2V2-AASIST fusion), particularly in terms of overall accuracy and min a-DCF. However, the conventional baselines still outperform the LALM in EER, which the authors attribute to the lack of calibrated continuous scores in the token-based LALM output. The ablation studies effectively isolate the contribution of each component (AAM head, spoof head, hard mining). The reasoning component, while interesting, did not yield performance gains over hard-label SFT, which is a significant finding but limits the technical impact of that specific branch.
The paper provides sufficient detail regarding model architectures (SALMONN-7B, Qwen2-Audio-7B), training hyperparameters (LoRA rank, learning rate, batch size), and data processing (resampling, augmentation). The dataset (ASVspoof 5 + VoxCeleb) is standard. However, the code and pre-trained checkpoints are not mentioned as available, and the specific cold-start reasoning generation process using Step-Audio R1 is described but the model weights are not provided, making full reproduction of the CoT component difficult. The evaluation on a stratified sample of 20K pairs rather than the full set is noted, which aids speed but requires careful interpretation of metrics.
The primary limitation is the lack of performance superiority over established discriminative baselines in key metrics like EER. The LALM approach is computationally more expensive and slower than conventional pipelines. The reasoning traces, while interpretable, are not validated for correctness or utility in a human-evaluation setting, and the authors admit they are prone to hallucination. The "reasoning" capability does not translate to better hard-label accuracy, questioning the practical value of the CoT component for the core task. The reliance on external models (Step-Audio R1) for data generation introduces a dependency that is not fully analyzed.
This work opens a new direction for using LALMs in security-critical audio applications, offering the potential for interpretable decision-making which is crucial for auditing voice authentication systems. It highlights the gap between general audio understanding and specific security tasks. The findings help guide future research on whether reasoning capabilities can be effectively integrated into security systems or if discriminative models remain superior for raw performance. It contributes to the broader understanding of LALM capabilities and limitations in specialized domains. This paper provides the first systematic evaluation of Large Audio Language Models for Spoofing-Aware Speaker Verification, demonstrating that while they can be adapted to compete with conventional pipelines, they currently lag in threshold-based metrics and require significant task-specific adaptation to overcome their lack of native security awareness.
Automatic symbolic music analysis has made substantial progress, yet existing systems are typically designed for a single mode of use, such as full-score prediction, and therefore do not match the broader range of operations that arise in analysis workflows, including partial completion, local correction, and iterative refinement. As a result, there remains a gap between strong benchmark models and systems that can support interactive analytical use. We present a unified framework for symbolic Roman-numeral (RN) analysis that narrows this gap by combining strong predictive performance with direct support for constrained completion and revision. The method is designed to provide a practical trade-off between accuracy and interactive responsiveness by computing expensive pretrained representations once and reusing them during iterative refinement, making powerful pretrained models more amenable to interactive settings. It supports complete score analysis, targeted revision of existing labels, and inference of missing annotations from partial context through a shared modeling framework. Experiments on Dilemmadata, the largest and most heterogeneous benchmark of its kind, show that the proposed approach is a strong RN-analysis baseline while also supporting masked completion from partial labels. Together with a prototype interface for multi-level candidate inspection and editing, these results position automatic RN analysis not only as a prediction problem, but also as a foundation for future interactive tools for music analysis.
Primary: Johannes Kepler University Linz
All Institutions: Johannes Kepler University Linz
This paper presents a unified framework for interactive Roman Numeral analysis that effectively bridges the gap between high-accuracy predictive models and practical, analyst-in-the-loop workflows through a hybrid MusicBERT-GNN architecture and edit-conditioned masked prediction.
The paper proposes a hybrid architecture combining a pretrained MusicBERT sequence encoder with a Graph Neural Network (GNN) for symbolic Roman Numeral (RN) analysis. The core methodological contribution is the "prediction-to-analysis continuum," which unifies blind full-score prediction, masked completion, and interactive revision within a single framework. The authors introduce an edit-conditioned masked-prediction model that uses label-conditioning modules to inject known labels as biases, allowing for constrained completion. They also employ conflict-aware optimizers (PCGrad, CAGrad) to handle gradient conflicts in the multi-task learning setup (20 tasks). The approach is technically sound, leveraging the complementary strengths of sequence modeling (context) and graph modeling (structural coherence). However, the architectural novelty is incremental, primarily consisting of combining existing strong components (RNBert + AnalysisGNN) and adding a specific conditioning mechanism for interactivity.
The evaluation is conducted on "Dilemmadata," a new benchmark combining AugmentedNet and Distant Listening Corpus. The experiments demonstrate that the hybrid model achieves state-of-the-art or competitive results on blind full-score inference compared to baselines like RNBert and AnalysisGNN. Crucially, the paper provides empirical evidence for the interactive use case: masked completion accuracy improves monotonically as more contextual labels are provided, validating the model's ability to leverage partial information. The inclusion of a prototype interface and the analysis of post-hoc decoding strategies (voters, beam search) adds practical value. The results are robust, though the paper notes that post-hoc decoding modules do not improve blind inference, highlighting the specific design for the interactive regime.
The paper provides a GitHub repository and a demo URL. The methodology is described in sufficient detail, including the tokenization scheme (REMI), graph construction, and training objectives (multi-task cross-entropy, KL distillation, L2-SP). The use of standard optimizers and augmentation techniques aids reproducibility. The definition of the Dilemmadata benchmark and the specific masking protocols are clearly outlined, facilitating future comparisons.
The paper acknowledges that the evaluation of masked completion uses randomly sampled masks rather than realistic analyst correction patterns, which may overestimate performance in real-world interactive settings. Additionally, the post-hoc decoding modules (voters, beam search) did not improve blind inference accuracy, suggesting they are specialized for the interactive regime and their utility is context-dependent. The study is primarily technical and lacks user studies with musicologists to quantify the actual benefit of the interactive workflow.
This work significantly advances the field of symbolic music analysis by shifting the focus from static prediction to interactive, analyst-in-the-loop tools. By providing a unified framework and a prototype interface, it lowers the barrier for developing practical music analysis software. The emphasis on interpretability (multi-level candidates, contradiction highlighting) and the handling of ambiguous musical passages aligns with the needs of musicologists, potentially bridging the gap between computational models and human analytical practices. This paper presents a unified framework for interactive Roman Numeral analysis that effectively bridges the gap between high-accuracy predictive models and practical, analyst-in-the-loop workflows through a hybrid MusicBERT-GNN architecture and edit-conditioned masked prediction.
Supervised fine-tuning (SFT) is widely used to adapt self-supervised speech representations to downstream classification tasks. Small gains observed under a single pretrained checkpoint are often interpreted as method-level improvements, i.e., a higher attainable performance ceiling. We show that such conclusions are not always reliable because SFT outcomes depend strongly on the specific pretrained instance. We conduct a systematic study on 3 SUPERB classification tasks, evaluating 8 SFT variants across 9 pretrained checkpoints from wav2vec~2.0, HuBERT, and WavLM, with multi-seed repetitions on representative base-scale models. We find that the identity of the statistically indistinguishable top-group SFT recipe is often checkpoint-dependent, with limited transferability across pretrained instances. These findings suggest that many reported downstream gains reflect instance and seed dependent elicitation match, rather than universally improving the attainable performance ceiling.
Primary: Kyoto University
All Institutions: Graduate School of Informatics, Kyoto University
This paper provides a crucial methodological critique of speech foundation model fine-tuning, demonstrating that reported SFT gains are often instance-dependent artifacts rather than universal improvements, thereby advocating for more rigorous and stable evaluation standards in the field.
The paper proposes a critical re-evaluation of standard practices in Supervised Fine-Tuning (SFT) for speech foundation models. Rather than proposing a new architecture, it introduces a rigorous empirical framework to test the stability of SFT performance across different pretrained instances (checkpoints) and random seeds. The methodology involves a systematic sweep of 8 SFT configurations (varying feature extraction modes and freezing strategies) across 9 checkpoints from three major families (wav2vec 2.0, HuBERT, WavLM) on 3 SUPERB tasks. The core theoretical contribution is the "elicitation match" hypothesis, which posits that performance gains are often due to better alignment with the specific latent structure of a checkpoint rather than a universal increase in capacity. This reframes the interpretation of SFT results from "method superiority" to "activation reliability."
The experimental design is robust and comprehensive. The authors utilize a large-scale evaluation matrix (9 checkpoints x 8 configs x 3 tasks) and crucially include multi-seed repetitions for base-scale models to isolate stochastic variability. The use of paired exact McNemar tests to define "top groups" provides a statistically sound basis for comparing configurations, moving beyond simple mean accuracy comparisons. The results clearly demonstrate that the "best" SFT configuration is not consistent across checkpoints, even within the same family and size, and that seed changes can lead to significant performance flips (under-activation vs. full activation). This effectively debunks the assumption that a single SFT recipe is universally optimal.
The paper provides high reproducibility standards. It specifies the exact checkpoints, model families, and pretraining data sources. The training setup follows official SUPERB pipelines, ensuring that hyperparameters are standardized. The inclusion of specific random seeds (1337, 2048, 7395) and the detailed description of the SFT configuration space (Feature Mode, Freeze Mode) allow other researchers to replicate the findings. The use of standard evaluation metrics and scripts further enhances reproducibility.
The study is limited to base and large scale models; it does not explore the behavior of smaller or significantly larger models, which might exhibit different stability characteristics. The SFT configurations tested are representative but not exhaustive (e.g., it does not test LoRA or other parameter-efficient fine-tuning methods, which are increasingly common). Additionally, the "elicitation match" hypothesis is interpretive; while supported by the data, it is not formally proven through mechanistic analysis of the representations. The focus on classification tasks also means the findings may not directly generalize to regression or generation tasks without further investigation.
This work has significant implications for the speech community. It cautions against over-interpreting small performance gains in single-checkpoint ablation studies and advocates for more rigorous evaluation protocols that account for instance and seed variability. By highlighting the fragility of SFT performance, it encourages researchers to adopt more robust evaluation practices, potentially reducing the publication of marginal improvements that do not generalize. This promotes a more mature and reliable scientific discourse in speech representation learning. This paper provides a crucial methodological critique of speech foundation model fine-tuning, demonstrating that reported SFT gains are often instance-dependent artifacts rather than universal improvements, thereby advocating for more rigorous and stable evaluation standards in the field.
Recent text-to-audio models generate high-quality audio, but often fail to follow instructions involving multiple sound events and temporal order. This gap arises because existing evaluation and training signals mainly emphasize global similarity or perceptual quality, with limited supervision on instruction-level correctness. We propose an instruction-level framework that uses audio-aware large language models (ALLMs) as fine-grained judges to verify target event presence and temporal relations in generated audio. After validating ALLM judgments on benchmarks and through human verification, we use their feedback to construct preference pairs for direct preference optimization. We further introduce S3Bench, a narrative benchmark for evaluating multi-event temporal instruction following. Experiments show that our method improves event completeness, temporal ordering, and joint instruction-following accuracy across existing benchmarks and S3Bench, while maintaining audio quality.
Primary: National Taiwan University
All Institutions: National Taiwan University, Amazon
The paper presents a significant advancement in Text-to-Audio generation by introducing a novel preference optimization framework that leverages Audio-Aware LLMs for fine-grained instruction verification, effectively bridging the gap between global audio quality and precise temporal event control, while contributing a valuable new benchmark to the community.
The paper proposes a novel framework, ALLM-Judged Preference Optimization (AJPO), to address the lack of fine-grained instruction following in Text-to-Audio (TTA) generation. The core innovation lies in leveraging Audio-Aware Large Language Models (ALLMs) as structured judges to verify sound event existence and temporal ordering, rather than relying on global similarity metrics like CLAPScore. These structured judgments are used to construct preference pairs for Direct Preference Optimization (DPO). The methodology is sound and addresses a specific, well-identified gap in the field: the misalignment between global perceptual quality and fine-grained instruction adherence. The use of ALLMs as reward models/judges is a timely approach, capitalizing on the emergent reasoning capabilities of recent multimodal models. However, the novelty is somewhat incremental as the concept of using LLMs/ALLMs for preference optimization is becoming more common across modalities; the specific application to temporal event ordering in audio is the key differentiator.
The experimental evaluation is comprehensive and robust. The authors first validate the capability of ALLMs (Qwen2.5-Omni, Audio Flamingo) as judges on existing benchmarks and synthetic datasets, showing strong correlation with human judgments. They then introduce S3Bench, a new benchmark for narrative multi-event audio, which adds significant value to the community. The ablation studies effectively demonstrate the superiority of fine-grained ALLM feedback over CLAP-based ranking and caption-based matching. The comparison with existing preference datasets (Audio-Alpaca, Baton) highlights the benefits of dynamic, instruction-aligned preference construction. The results show consistent improvements in event completeness and temporal ordering without degrading audio quality metrics (FAD, IS, CLAPScore). The inclusion of online DPO experiments further strengthens the technical contribution by showing the importance of dynamic preference reconstruction.
The paper provides sufficient detail for reproduction, including model architectures (TangoFlux-base), training hyperparameters (learning rate, batch size, warm-up steps), and the construction of the S3Bench dataset. The use of open-source models (Qwen2.5-Omni, Audio Flamingo) and standard TTA baselines enhances reproducibility. The project page link suggests availability of qualitative examples and potentially code, which would further aid reproducibility.
The approach relies heavily on the accuracy of the ALLM judge. While validated, ALLMs can still hallucinate or make errors, particularly in complex, noisy audio scenarios (as noted in the evaluation of AudioCaps-test vs. AudioTime). The performance of the TTA model is thus bounded by the judge's reliability. Additionally, the synthetic MultiEvent-Temporal benchmarks, while useful for controlled evaluation, may not fully capture the complexity of real-world audio scenes with overlapping sounds and background noise, although S3Bench attempts to mitigate this. The computational cost of running ALLM judgments for preference construction, especially in online DPO, is non-trivial.
This work significantly advances the controllability of TTA systems, making them more suitable for complex applications like film scoring, game audio design, and assistive technologies where precise instruction following is critical. The introduction of S3Bench provides a standardized benchmark for future research in this area. The methodological framework of using multimodal LLMs for fine-grained preference optimization could be generalized to other generative tasks. The paper presents a significant advancement in Text-to-Audio generation by introducing a novel preference optimization framework that leverages Audio-Aware LLMs for fine-grained instruction verification, effectively bridging the gap between global audio quality and precise temporal event control, while contributing a valuable new benchmark to the community.
Automatic speech recognition is dominated by autoregressive decoders that emit one token at a time. We ask whether a discrete diffusion language model can transcribe speech instead, refining a whole transcript in parallel over a small number of denoising steps. We train an audio-native interface for DiffusionGemma, a 26B mixture-of-experts model that generates text by uniform, random-token discrete diffusion rather than the absorbing-mask scheme common to recent diffusion language models. A frozen Whisper encoder supplies acoustic features, a lightweight projector maps them into the model embedding space, and low-rank adapters let the frozen backbone attend to the new modality. About 42M parameters are trained, which is 0.16 percent of the backbone. We find that the natural training objectives fail to ground the audio because their gradient reaches the projector only through attention that has already dismissed it. A connectionist temporal classification loss applied through the frozen output head breaks this deadlock. The resulting model reaches 6.6 percent word error rate on LibriSpeech test-clean, transcribes in roughly eight parallel steps regardless of utterance length, and uses a single adapter trained on six languages, which we evaluate here on English, Hindi, and Mandarin.
Primary: Interfaze AI
All Institutions: Interfaze AI
This paper presents a compelling proof-of-concept for audio-native discrete diffusion language models, effectively solving the gradient flow problem in frozen multimodal adapters through a novel CTC-based grounding strategy, thereby enabling efficient, parallel speech recognition with minimal trainable parameters.
The paper proposes a novel architecture for Automatic Speech Recognition (ASR) by adapting a frozen 26B-parameter discrete-diffusion language model (DiffusionGemma) to audio inputs. The core methodological contribution is the "audio-native" pathway, which uses a frozen Whisper encoder, a lightweight trainable projector, and Low-Rank Adapters (LoRA) to inject acoustic features into the LLM's embedding space. A significant technical insight is the diagnosis of the "grounding deadlock," where standard diffusion and autoregressive losses fail to propagate gradients to the audio projector because the frozen attention mechanism ignores the ungrounded audio tokens. The authors resolve this by introducing a Connectionist Temporal Classification (CTC) loss applied directly through the frozen output head, which provides a direct gradient signal to the projector independent of attention. This allows the model to learn to attend to audio features in subsequent steps. The use of uniform random-token diffusion (rather than absorbing masks) is also noted as a specific characteristic of the backbone used.
The evaluation is conducted on LibriSpeech (English) and FLEURS (Multilingual: English, Hindi, Mandarin). The model achieves a Word Error Rate (WER) of 6.6% on LibriSpeech test-clean, which is competitive given the extremely low number of trainable parameters (42M, or 0.16% of the backbone). The experiments demonstrate that the model converges in approximately eight parallel denoising steps, decoupling inference cost from transcript length. Ablation studies effectively rule out encoder quality and token resolution as the primary bottlenecks, pointing instead to data scale as the main limitation compared to fully trained autoregressive systems like Whisper. The multilingual results show the adapter's ability to generalize across languages, though performance drops on non-English scripts compared to English.
The paper provides detailed architectural specifications, including the Whisper encoder size, projector dimensions, LoRA ranks, and training hyperparameters (AdamW, learning rate, batch size). The description of the diffusion process, corruption kernel, and sampling algorithm is sufficiently detailed for reproduction. However, the code is not explicitly linked in the text provided, and the specific random seeds or exact data splits (beyond standard LibriSpeech/FLEURS) are not fully enumerated, though standard practices are implied.
The primary limitation is the reliance on a frozen Whisper encoder, which restricts the input window to 30 seconds and prevents true streaming recognition without complex segmentation. The model trails state-of-the-art autoregressive systems (like Whisper-large) in accuracy, particularly on multilingual and noisy data, which the authors attribute to the limited training data (219 hours) compared to the millions of hours used for Whisper. Additionally, the diffusion decoder exhibits repetition artifacts on short utterances, requiring post-processing. The evaluation lacks a controlled head-to-head comparison with other diffusion-based ASR systems on identical hardware and data splits.
This work demonstrates that large language models can be efficiently adapted for speech recognition without retraining the backbone, potentially lowering the barrier to entry for multimodal ASR systems. It challenges the dominance of autoregressive decoding in ASR by showing that parallel diffusion decoding is viable and efficient. The multilingual adapter suggests potential for low-resource language support through parameter-efficient fine-tuning. However, the reliance on pre-trained encoders and large models still poses environmental and accessibility challenges. This paper presents a compelling proof-of-concept for audio-native discrete diffusion language models, effectively solving the gradient flow problem in frozen multimodal adapters through a novel CTC-based grounding strategy, thereby enabling efficient, parallel speech recognition with minimal trainable parameters.
Recent diffusion-based generative models have achieved strong results in domain-specific audio generation tasks such as speech, singing, and instrumental music synthesis. However, these models are typically specialized and do not generalize well to mixed or intermediate audio types. In this work, we adapt a diffusion-based model originally designed for multi-instrument music synthesis to voice conversion, covering both speech and singing within a unified framework. Specifically, we extend musical note-based conditioning to include phonetic posteriorgrams (PPGs) and pitch contours, and reinterpret timbre conditioning as speaker or singer identity via feature-wise linear modulation. Experiments show that the adapted model matches or surpasses a dedicated voice conversion system in terms of naturalness and performer similarity, while maintaining accurate pitch control across speech and singing. At the same time, we observe limitations in phonetic fidelity and a degradation in vocal quality when incorporating instrumental training data. Furthermore, we demonstrate that off-the-shelf feature extractors provide effective conditioning signals, enabling large-scale self-supervised training without manual annotations. These results highlight the potential of cross-domain model transfer towards unified audio generation systems capable of handling speech, singing, and music. Qualitative samples can be found on our project page: https://benadar293.github.io/voice-conversion
Primary: International Audio Laboratories Erlangen
All Institutions: International Audio Laboratories Erlangen, Fraunhofer IIS
This paper effectively demonstrates that a diffusion model designed for instrumental music synthesis can be adapted to high-quality speech and singing voice conversion by extending its conditioning mechanisms to include phonetic and pitch information, achieving performance comparable to specialized systems while highlighting the inherent trade-offs in unified audio generation frameworks.
The paper proposes a method to adapt a diffusion-based model originally designed for multi-instrument music synthesis (T5-based) to the task of speech and singing voice conversion (VC). The core technical contribution lies in the conditioning framework: it extends musical note-based conditioning (piano rolls) to include phonetic posteriorgrams (PPGs) and pitch contours ($f_0$) for vocal content, and reinterprets timbre conditioning via Feature-wise Linear Modulation (FiLM) as speaker/singer identity. The authors also implement pitch range adaptation to handle the disparity between source and target vocal ranges. The methodology is sound and leverages existing, robust components (CREPE for pitch, wav2vec 2.0 for PPGs, TRILL for embeddings), focusing on the architectural adaptation and conditioning strategy rather than proposing a new neural architecture from scratch. The comparison with a specialized VC model (FlowMAC) provides a strong baseline for evaluating the efficacy of the cross-domain transfer.
The experimental setup is comprehensive, covering both speech and singing domains. The authors employ a rigorous evaluation protocol including MUSHRA-style listening tests for naturalness and similarity, as well as objective metrics like Fréchet Audio Distance (FAD), Raw Pitch Accuracy (RPA), and phonetic fidelity metrics (Jensen-Shannon and Wasserstein distances on PPGs). The results indicate that the adapted music synthesis model (T5-Voc) matches or slightly surpasses the dedicated VC model (MAC-Voc) in naturalness and performer similarity, while maintaining accurate pitch control. However, the paper honestly reports limitations: phonetic fidelity is lower in the T5 model compared to the specialized model, and incorporating instrumental data (T5-All) degrades vocal quality. The inclusion of a "Vocals with Instrumentals" experiment demonstrates the potential of the unified framework, showing improvements in FAD when combining vocal and instrumental conditioning. The statistical analysis (Wilcoxon signed-rank test) adds credibility to the subjective findings.
The paper provides significant detail regarding the model architecture, conditioning mechanisms, and training procedures. It specifies the use of off-the-shelf feature extractors (CREPE, wav2vec 2.0, TRILL) and mentions the use of publicly available implementations for the baseline models. The dataset description is somewhat vague regarding the proprietary speech data ("proprietary and public sources"), which limits exact replication of the training set. However, the authors state that the feature extractors are public, allowing for the creation of pseudo-annotated datasets from equivalent audio. The project page link provides qualitative samples, aiding in verification. The code is not explicitly linked in the text provided, but the reliance on public components enhances reproducibility potential.
The paper identifies several key limitations. First, there is a trade-off between generative quality/naturalness and phonetic fidelity; the T5 model, while sounding more natural, is less precise in preserving specific phonetic content compared to the specialized FlowMAC model. Second, the degradation in vocal quality when training on mixed vocal-instrumental data (T5-All) highlights the difficulty of unified modeling without careful architectural or training interventions. Third, the reliance on source separation for mixed audio introduces potential errors, although the authors claim high-quality separation. Finally, the evaluation of phonetic fidelity relies on automatic metrics which may not fully correlate with human perception of intelligibility, a gap the authors acknowledge.
This work contributes to the growing field of unified audio generation models, demonstrating that domain-specific architectures can be effectively repurposed for new tasks through careful conditioning design. It highlights the potential for cross-domain transfer in audio synthesis, which could lead to more versatile and efficient generative systems. The findings also have implications for the development of foundation models for audio, suggesting that while unified models are promising, they currently face challenges in balancing performance across diverse domains (speech vs. singing vs. instrumental). The open release of qualitative samples and the discussion of self-supervised training with off-the-shelf features support the broader ML community's interest in scalable, annotation-efficient audio generation methods. This paper effectively demonstrates that a diffusion model designed for instrumental music synthesis can be adapted to high-quality speech and singing voice conversion by extending its conditioning mechanisms to include phonetic and pitch information, achieving performance comparable to specialized systems while highlighting the inherent trade-offs in unified audio generation frameworks.
In this report, we introduce Qwen-Music, a powerful music generation model capable of producing highly musical and high-fidelity songs with complete vocal singing. Qwen-Music supports two core tasks: Text to Music Generation, which create entirely new songs from text descriptions, lyrics, and musical attributes, and Cover Song Generation, which reinterprets existing songs with different styles and vocal characteristics. Architecturally, Qwen-Music integrates three core components: Qwen-Music-Tokenizer, Qwen-Music-LLM, and Qwen-Music-Render. Qwen-Music-Tokenizer compresses audio into a 25 Hz single-codebook stream of Music Semantic Tokens that preserve semantic and melodic information for LLM prediction. Based on these tokens, Qwen-Music-LLM performs autoregressive music semantic modeling, with a key novelty being a melody-token-based chain-of-thought (Melody-CoT) mechanism that plans melodies before full-song generation, improving creativity, musicality, structural coherence, and reference-audio-based melody cloning. To overcome the fidelity limitations of discrete semantic tokens, Qwen-Music-Render performs generative stereo rendering, enriching acoustic details and producing high-fidelity stereo waveforms. Finally, we train Qwen-Music-LLM on more than 5 million hours of multilingual music data covering hundreds of languages. We first apply quality-aware pre-training curriculum, then use progressive post-training, comprising supervised initialization, offline DPO, and online GSPO, to further improve musicality and instruction-following ability. Across 600 Chinese and English prompts, Qwen-Music achieves state-of-the-art results in 13 of 16 objective musicality and audio-quality metrics. Professional evaluators also prefer Qwen-Music over leading proprietary systems. For cover song generation, Qwen-Music preserves reference melodies more accurately than leading proprietary systems.
Primary: Qwen Team
All Institutions: Qwen Team
This paper introduces Qwen-Music, a state-of-the-art music generation system that effectively bridges semantic composition and acoustic rendering through a novel Melody-CoT mechanism and a high-fidelity generative renderer, achieving superior performance in both objective metrics and human preference tests.
The paper presents a comprehensive three-stage architecture for music generation: a semantic tokenizer, an autoregressive language model (LLM), and a generative renderer. The core methodological contributions lie in the specific design choices for handling the complexity of full-song generation. First, the **Qwen-Music-Tokenizer** employs a four-stage training recipe (BestRQ pretraining, causal adaptation, multi-task SFT, and VQ insertion) to create a 25 Hz single-codebook stream. This is a significant engineering contribution, as it balances semantic richness with computational tractability for LLMs. Second, the **Melody-CoT** mechanism is a notable innovation. By explicitly planning melody tokens before generating full-mixture semantic tokens, the model addresses the "composition vs. rendering" mismatch. This chain-of-thought approach allows for better structural coherence and enables cover song generation by conditioning on reference melody tokens while allowing style/timbre changes. The use of relative MIDI representations for melody tokens is a smart design to prevent leakage of absolute pitch/key information that might conflict with style tags. Third, the **Qwen-Music-Render** utilizes a Diffusion Transformer (DiT) conditioned on both semantic tokens and textual descriptions, followed by a novel **Spec-VAE** and **Band-Mode Refiner**. The refiner’s frequency-adaptive correction (phase-only for low, joint for mid, magnitude-only for high) is a sophisticated touch to address specific spectral artifacts. The training pipeline is also rigorous, featuring a quality-graded pre-training curriculum (Q1-Q6 buckets) and a progressive post-training alignment (SFT -> Offline DPO -> Online GSPO), which is state-of-the-art practice for aligning generative models with human preference.
The evaluation is extensive and robust. The authors conduct blind A/B preference tests with 50 professional human raters, showing Qwen-Music outperforms leading proprietary systems (Suno V5.5, MiniMax, Mureka) with a 59.1% win rate. They also report performance on the Artificial Analysis leaderboard. Objective evaluations cover 16 metrics across SongBench, SongEval, and AudioBox-Aesthetic, where Qwen-Music leads in 13/16 metrics. The cover song generation evaluation is particularly strong, demonstrating superior melody preservation compared to baselines. The inclusion of a detailed acoustic data quality pipeline for training data filtering (detecting fake stereo, noise floors, cutoffs) adds credibility to the high-fidelity claims. The ablation studies on the renderer and the effectiveness of the Melody-CoT mechanism are well-supported by the results.
The paper provides detailed architectural specifications, including layer counts, parameter sizes, and training objectives. The tokenizer training stages are clearly defined. The post-training pipeline (SFT, DPO, GSPO) is described with sufficient detail for replication by other research groups. However, the specific "quality-aware" reward models and the exact composition of the 5 million-hour dataset are proprietary and not fully disclosed, which limits exact reproducibility of the pre-training phase. The code for the tokenizer and renderer is not explicitly linked in the provided text, though the Qwen team typically releases code for their models.
The paper relies heavily on proprietary baselines (Suno, MiniMax) for comparison, which makes independent verification difficult. The "quality-graded" pre-training assumes the availability of a reliable internal reward model, which may not be generalizable to all settings. The Melody-CoT mechanism, while effective, adds complexity and inference time. The model's performance on non-vocal music or highly abstract musical forms (e.g., free jazz, classical symphonies without lyrics) is less emphasized compared to vocal pop/rock genres, which dominate the training data. The reliance on a single codebook for the tokenizer might limit the expressiveness compared to multi-codebook approaches (like Encodec) in capturing fine-grained timbral details, although the renderer is designed to mitigate this.
Qwen-Music represents a significant step forward in accessible, high-fidelity music generation. By open-sourcing the technical report and potentially the model (implied by the Qwen lineage), it democratizes access to state-of-the-art music AI. The ability to generate cover songs and control musical attributes empowers creators. However, the potential for misuse in generating copyrighted material or deepfake vocals remains a concern, necessitating robust safety filters and watermarking, which are not detailed in this technical report. The high computational requirements for training such large models also contribute to the centralization of AI capabilities. This paper introduces Qwen-Music, a state-of-the-art music generation system that effectively bridges semantic composition and acoustic rendering through a novel Melody-CoT mechanism and a high-fidelity generative renderer, achieving superior performance in both objective metrics and human preference tests.
Large audio-language models (LALMs) often underperform on fine-grained, non-semantic attributes of speech, such as a speaker's emotion, despite strong performance on speech content. Improving this without the cost of retraining calls for an effective inference-time intervention, yet most existing methods intervene only after the audio encoder and operate at a relatively coarse granularity. The encoder itself, where acoustic information is first extracted from the waveform, remains largely unexplored, especially at the level of individual neurons. We introduce IAAN, Identifying and Amplifying Acoustic Neurons, a training-free and label-free method that scores each feed-forward neuron in the audio encoder by contrasting its activation on the real waveform with that on a noise reference lacking the real audio's acoustic information. IAAN then amplifies a small set of the highest-scoring neurons at inference. Across ten non-semantic speech attributes, IAAN improves average accuracy by 25.7 points on Audio-Flamingo-3, 21.4 on Qwen2.5-Omni, and 9.7 on Kimi-Audio. It also improves a model already explicitly fine-tuned to prioritize acoustic evidence. In controlled comparisons, both the encoder locus and neuron-level selectivity prove necessary for this gain. Intervening after the encoder, at the decoding side or inside the language model, yields little to no improvement, or even deteriorates accuracy. The improvement also depends on which specific neurons are amplified, not merely on their number, confirming that IAAN's acoustic score succeeds in identifying the neurons that matter. These results show that a small, precisely targeted intervention inside the audio encoder is an effective and largely untapped way to strengthen the acoustic understanding of LALMs, opening a new direction for inference-time methods that improve acoustic perception through neuron-level access to the encoder.
Primary: National Taiwan University
All Institutions: National Taiwan University, NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), ASUS Open Cloud Infrastructure Software Center
[One sentence main contribution]. The paper introduces IAAN, a training-free inference-time intervention that identifies and amplifies specific feed-forward neurons in the audio encoder of LALMs by contrasting their activations on real audio versus noise, significantly improving non-semantic acoustic perception. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. This paper makes a valuable contribution to the field of multimodal machine learning by shifting the focus of inference-time interventions from the language model backbone to the audio encoder. By proposing a simple yet effective contrastive scoring mechanism at the individual neuron level, IAAN addresses a critical gap in how LALMs process fine-grained acoustic information. The rigorous experimental evaluation, including comprehensive ablations against other intervention loci and granularities, strongly validates the necessity of encoder-side, neuron-level targeting. The significant performance gains across multiple state-of-the-art LALMs on a challenging benchmark (VoxParadox) demonstrate the practical utility of this approach. While the computational cost of dual encoder passes is a minor drawback, the method's label-free and training-free nature makes it a highly attractive tool for enhancing the acoustic grounding of existing models.
The paper proposes IAAN (Identifying and Amplifying Acoustic Neurons), a training-free, label-free inference-time intervention method. The core methodology involves contrasting the activation of feed-forward neurons in the audio encoder of Large Audio-Language Models (LALMs) between the real audio waveform and a noise reference (Gaussian noise with matched length/variance). Neurons that activate more strongly on real audio are identified as "acoustic neurons" and their activations are amplified by a gain factor $g$ during inference. The approach is conceptually simple and elegant, leveraging the intuition that acoustic-specific neurons should be sensitive to the presence of structured audio versus unstructured noise. The method operates at the individual neuron level within the audio encoder, which is a finer granularity than previous interventions that typically target the language model backbone or output logits. The technical execution involves standard transformer operations (FFN activations) and does not require backpropagation or external datasets for calibration, relying instead on an internal contrastive signal.
The authors evaluate IAAN on three open-source LALMs (Audio-Flamingo-3, Qwen2.5-Omni, Kimi-Audio) and one specialized model (AF3-PD) using the VoxParadox benchmark, which tests ten non-semantic speech attributes (e.g., emotion, gender, pitch) where semantic content may contradict acoustic cues. The results are significant: IAAN improves average accuracy by 25.7 points on Audio-Flamingo-3, 21.4 on Qwen2.5-Omni, and 9.7 on Kimi-Audio. Crucially, the paper includes rigorous ablation studies. It compares IAAN against other inference-time interventions (Audio-aware decoding, LLM-amplify, Hidden-state steering) and demonstrates that intervening *inside* the encoder is necessary, as post-encoder interventions yield little to no gain. It also shows that the specific selection of neurons (via the acoustic score) is critical, as random selection or whole-layer amplification fails to reproduce the gains. The method also improves a model (AF3-PD) that was already fine-tuned for acoustic tasks, suggesting broad applicability. The use of a held-out development set (LISTEN) for hyperparameter tuning adds credibility, although the transfer of optimal hyperparameters across models is noted to be variable.
The paper provides sufficient detail for reproduction. It defines the neuron definition (FFN hidden activations), the reference signal generation (Gaussian noise with fixed seed), the scoring equation, and the amplification hook. The models and benchmark (VoxParadox) are open-source and widely available. The hyperparameter selection process is clearly described, though the reliance on an external development set (LISTEN) for tuning $(K, g)$ is a minor constraint for fully zero-shot application. The code is not explicitly linked in the text provided, but the method is algorithmic and straightforward to implement given the open-source nature of the base models.
The primary limitation is the computational overhead of running two forward passes through the audio encoder (real vs. noise reference) for every inference step. While the encoder is smaller than the LLM, this still doubles the encoder compute time. Second, the method's effectiveness is somewhat model-dependent; Kimi-Audio showed smaller gains and higher sensitivity to hyperparameters compared to Audio-Flamingo-3. The noise reference, while effective, is a heuristic; it assumes that neurons responding to "non-noise" are the acoustic ones, which might not capture all nuanced acoustic features equally well. Finally, the method is evaluated only on multiple-choice and free-text description tasks; it does not address potential negative impacts on semantic understanding or other modalities, though the authors claim no deterioration in semantic tasks.
This work opens a new direction for improving multimodal models by focusing on the encoder side, which has been largely ignored in favor of LLM-centric interventions. It demonstrates that fine-grained, neuron-level steering within encoders can be highly effective for specific perceptual tasks. This could lead to more efficient and targeted methods for enhancing multimodal capabilities without the cost of retraining. However, the ability to easily steer model behavior via activation amplification also raises questions about robustness and potential adversarial vulnerabilities, although these are not explored in depth here. [One sentence main contribution]. The paper introduces IAAN, a training-free inference-time intervention that identifies and amplifies specific feed-forward neurons in the audio encoder of LALMs by contrasting their activations on real audio versus noise, significantly improving non-semantic acoustic perception. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. This paper makes a valuable contribution to the field of multimodal machine learning by shifting the focus of inference-time interventions from the language model backbone to the audio encoder. By proposing a simple yet effective contrastive scoring mechanism at the individual neuron level, IAAN addresses a critical gap in how LALMs process fine-grained acoustic information. The rigorous experimental evaluation, including comprehensive ablations against other intervention loci and granularities, strongly validates the necessity of encoder-side, neuron-level targeting. The significant performance gains across multiple state-of-the-art LALMs on a challenging benchmark (VoxParadox) demonstrate the practical utility of this approach. While the computational cost of dual encoder passes is a minor drawback, the method's label-free and training-free nature makes it a highly attractive tool for enhancing the acoustic grounding of existing models.
Multimodal information is pivotal for e-commerce search ranking. Existing works leverage multimodal data typically by fine-tuning general Multimodal Large Language Models (MLLMs) via collaborative signals, subsequently integrating the derived representations into ranking models as item features. Despite their efficacy, these methods face two primary limitations: (1) they rely on a single collaborative signal for MLLM fine-tuning, failing to exploit the heterogeneous signals essential for multitask ranking; and (2) they treat multimodal representations as regular item features in ranking models, underutilizing their latent potential for user behavior modeling. To address these challenges, we propose the Multiplex Multimodal Representation Model (MMRM), a unified framework that aligns MLLMs with diverse collaborative signals. By employing a shared backbone with task-specific tokens and projection layers, MMRM simultaneously learns from multiple signals and generates comprehensive multiplex item representations in a single inference pass. Furthermore, we introduce a multiplex user representation strategy in ranking models, which derives task-specific user representations via search-based behavior sequence modeling leveraging multiplex item representations. Extensive experiments demonstrate MMRM's superior efficiency and effectiveness. Notably, MMRM has been successfully deployed in the JD e-commerce search engine, yielding significant performance gains for millions of daily users.
Primary: JD.COM
All Institutions: JD.COM
[One sentence main contribution]. The paper introduces MMRM, a unified framework that aligns MLLMs with diverse collaborative signals using task-specific tokens, enabling efficient generation of multiplex item representations and significantly improving e-commerce search ranking performance. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. MMRM addresses critical limitations in existing multimodal ranking approaches by effectively integrating heterogeneous collaborative signals and disentangling task-specific representations, leading to superior offline and online performance in a major e-commerce platform.
The paper proposes MMRM, a framework for aligning Multimodal Large Language Models (MLLMs) with multiple heterogeneous collaborative signals (query-to-item clicks, and item-to-item clicks, carts, orders) in e-commerce search. The core technical innovation lies in using a shared MLLM backbone with task-specific special tokens ([SEARCH], [CLICK], [CART], [ORDER]) to generate disentangled item representations in a single forward pass. This addresses the limitation of previous methods that either use single signals or require separate models/forward passes for different signals. The method also introduces a multiplex user representation strategy in the downstream ranking model, leveraging these task-specific item embeddings for behavior sequence modeling. The approach is technically sound, leveraging established techniques like contrastive learning, GradCache for memory efficiency, and MMoE for multitask learning, but applies them in a specific, effective configuration for e-commerce.
The evaluation is extensive and robust, covering both offline representation learning metrics (F1@5, NDCG@5) and online A/B testing metrics (UCTR, UACR, UCVR). The offline results demonstrate that MMRM outperforms single-task baselines and a vanilla multitask baseline (Vanilla-Multi), validating the effectiveness of the task-specific token design. The online A/B test on JD's search engine provides strong evidence of real-world utility, showing significant improvements in key business metrics. The ablation studies on batch size and the detailed description of dataset construction (including hard negative sampling strategies) add credibility to the experimental claims.
The paper provides sufficient implementation details for reproduction, including the base model (Qwen3-VL-4B-Instruct), training hyperparameters (learning rate, batch size, temperature), and optimization techniques (GradCache, FlashAttention-2). The dataset construction process is described in detail, although the exact proprietary data distributions are not fully open. The code is not publicly released, which is common for industry papers, but the methodological description is clear enough for a competent researcher to implement.
The paper does not explicitly discuss the computational overhead of the MMRM approach compared to single-task models, although it claims efficiency gains by avoiding multiple forward passes. The generalizability of the findings to other e-commerce platforms or non-e-commerce domains is not explored. Additionally, the reliance on proprietary JD data limits the ability of external researchers to fully validate the scale and diversity of the collaborative signals used. The paper also does not discuss potential biases introduced by the graph-based sampling methods.
This work has significant implications for the field of information retrieval, particularly in e-commerce search and recommendation systems. By demonstrating how to effectively leverage multimodal data and multiple collaborative signals within a unified framework, it sets a new standard for industrial-scale ranking models. The successful deployment and positive impact on millions of users highlight the practical value of the research. It also contributes to the broader understanding of how to adapt large foundation models for specific, complex downstream tasks with heterogeneous data sources. [One sentence main contribution]. The paper introduces MMRM, a unified framework that aligns MLLMs with diverse collaborative signals using task-specific tokens, enabling efficient generation of multiplex item representations and significantly improving e-commerce search ranking performance. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. MMRM addresses critical limitations in existing multimodal ranking approaches by effectively integrating heterogeneous collaborative signals and disentangling task-specific representations, leading to superior offline and online performance in a major e-commerce platform.
Speech enhancement (SE) can substantially improve perceptual quality, yet enhanced speech does not necessarily improve automatic speech recognition (ASR). Existing remedies, such as retraining the enhancer jointly with recognizer or interpolating enhanced speech with the noisy input, can mitigate this mismatch, but common explanations such as artifacts and over-suppression remain qualitative and do not localize which enhancement component harms recognition. We propose inference time polar projection, a diagnosis for STFT domain enhancement. Given a mask $M=Ae^{jφ}$, polar projection forms $M_{α,γ}=A^αe^{jγφ}$, where $α$ controls magnitude strength and $γ$ controls phase correction. Sweeping these controls on frozen SE and ASR models turns ASR degradation into measurable magnitude and phase effects. Our projection analysis shows that magnitude strength is the operative axis, while estimated phase correction provides no recognition benefit. The optimal magnitude strength is recognizer dependent: waveform-input wav2vec2.0 favors strong correction, whereas log-Mel-input, noise-robust Whisper prefers weaker correction. Finally, the projection provides a simple mitigation for any SE front end in the STFT mask domain, without retraining either the enhancer or the recognizer, making it directly useful for voice assistants and agents that rely on enhanced speech.
Primary: University of Illinois Urbana-Champaign
All Institutions: University of Illinois Urbana-Champaign, Wuhan University
The paper presents a rigorous diagnostic framework for understanding the mismatch between speech enhancement quality and automatic speech recognition performance, demonstrating that magnitude strength is the critical factor and that phase correction is largely superfluous for recognition tasks.
The paper introduces "Inference Time Polar Projection," a diagnostic framework that factorizes STFT-domain complex masks into independent magnitude and phase controls. By parameterizing the mask as $M_{\alpha,\gamma} = A^\alpha e^{j\gamma\phi}$, the authors decouple the effects of magnitude scaling ($\alpha$) and phase correction ($\gamma$) on Automatic Speech Recognition (ASR) performance. This approach is methodologically sound and offers a rigorous way to dissect the black-box nature of Speech Enhancement (SE) models. The derivation of the effective mask for non-multiplicative models (like MossFormerGAN-SE) is a clever technical addition that extends the applicability of the diagnosis. However, the core concept of sweeping mask parameters is not entirely new; it builds on existing work in mask exponentiation and observation adding, though the specific disentanglement of magnitude vs. phase for ASR diagnosis is a distinct contribution.
The experimental evaluation is comprehensive and well-structured. The authors test on the VoiceBank+DEMAND benchmark, which is standard for this domain, using six different SE models (both discriminative and generative) and two distinct ASR backbones (Whisper and wav2vec 2.0). The results clearly demonstrate the SE-ASR mismatch and validate the polar projection hypothesis: magnitude strength is the primary driver of ASR performance, while phase correction provides negligible benefit. The analysis of recognizer dependence (wav2vec 2.0 favoring strong suppression vs. Whisper preferring mild correction) is insightful and aligns with the known robustness differences between waveform-based and feature-based models. The statistical significance testing (paired bootstrap) adds rigor to the claims.
The paper provides sufficient detail for reproduction. It specifies the datasets (VoiceBank+DEMAND), the specific SE models used (with references to ClearerVoice-Studio implementations), and the ASR models. The mathematical formulation of the polar projection is clear. However, the code for the "effective mask" calculation for MossFormerGAN-SE is described conceptually but not provided in the text, which might require some inference to implement exactly. The use of standard benchmarks and publicly available models enhances reproducibility.
The primary limitation is the scope of the evaluation. The experiments are conducted on a single dataset (VoiceBank+DEMAND), which, while standard, may not generalize to more complex real-world conditions with heavy reverberation or diverse noise types not present in the test set. The authors acknowledge this in the limitations section. Additionally, the diagnosis is limited to STFT-domain enhancements; while they extend it to effective masks for other architectures, it may not apply to pure time-domain models that do not operate in the spectral domain. The phase result, while robust for the tested models, is specific to the estimated phase of those enhancers and does not rule out the potential utility of explicitly modeled phase in other contexts.
This work has significant implications for the deployment of SE systems in voice assistants and agents. By showing that phase correction is largely unnecessary for ASR and that magnitude strength is recognizer-dependent, it suggests that SE systems can be simplified or calibrated specifically for machine listening, potentially reducing computational cost and improving robustness. It shifts the paradigm from optimizing for perceptual quality (PESQ, SI-SDR) to optimizing for recognition utility, which is crucial for the next generation of speech-centric AI applications. The paper presents a rigorous diagnostic framework for understanding the mismatch between speech enhancement quality and automatic speech recognition performance, demonstrating that magnitude strength is the critical factor and that phase correction is largely superfluous for recognition tasks.
Zero-shot instrument cloning aims to render an arbitrary [Target MIDI] sequence with the acoustic identity of an unseen instrument given only a short [Reference Audio, Reference MIDI] pair. Existing methods rely on pre-trained embeddings (e.g., CLAP) that compress the reference audio into a fixed-length vector, discarding fine-grained acoustic cues essential for faithful timbre reconstruction. We present Anysynth, an embedding-free neural synthesizer based on in-context flow matching. By conditioning a Diffusion Transformer (DiT) directly on the uncompressed reference audio and target MIDI, our model allows self-attention to dynamically retrieve acoustic details at generation time. Experiments show that \tool outperforms embedding-based and auto-regressive baselines in audio quality, timbre similarity, and melody adherence. Notably, the model exhibits prompt-length scaling: longer reference prompts yield steadily better timbre fidelity, a property absent in embedding-based systems. To optimize controllability, we further propose Asymmetric Hierarchical CFG, which structurally decouples MIDI and reference-timbre guidance based on their natural semantic-acoustic dependency. This asymmetric formulation avoids gradient conflicts and improves both note accuracy and timbre fidelity, pushing the boundary of expressive, zero-shot instrument cloning. Demo audios are available at https://anysynth-demo.github.io/
Primary: Central Media Technology Institute
All Institutions: Central Media Technology Institute, Chinese University of Hong Kong
The paper presents a significant advancement in zero-shot instrument cloning by replacing information-bottleneck embeddings with in-context learning in a DiT architecture, supported by a novel Asymmetric CFG strategy that effectively decouples semantic and acoustic guidance.
The paper proposes "Anysynth," an embedding-free neural synthesizer for zero-shot instrument cloning. The core methodological innovation lies in reformulating the task as an In-Context Learning (ICL) problem using a Diffusion Transformer (DiT) with Flow Matching. Instead of compressing reference audio into a fixed-length embedding (like CLAP), the model conditions directly on the uncompressed reference mel spectrogram and MIDI as a prefix. This allows the self-attention mechanism to dynamically attend to fine-grained acoustic details (transients, room reverb) from the reference. A key technical contribution is the "Asymmetric Hierarchical Classifier-Free Guidance" (CFG). The authors argue that standard symmetric CFG causes gradient conflicts between semantic (MIDI) and acoustic (timbre) conditions. By factorizing the probability $p(c_m, c_r | x_t)$ as $p(c_r | c_m, x_t)p(c_m | x_t)$, they decouple the guidance, applying strong structural guidance first and then refining with timbre. This is a theoretically sound and practically effective modification to standard diffusion/flow matching conditioning strategies. The MIDI encoding is also enhanced with a binary onset channel to better capture note boundaries, addressing a common failure mode in piano roll representations.
The experimental setup is robust, utilizing a mix of synthetic datasets (NSynth, Slakh) and real-world recordings (MAESTRO, GuitarSet). The evaluation metrics are comprehensive, including PANNs for timbre similarity, Onset F1 for melody adherence, MERT for perceptual quality, and Audiobox Aesthetics for overall quality. The results demonstrate that Anysynth outperforms embedding-based baselines (CLAP-conditioned DiT) and autoregressive/token-based baselines (TokenSynth, CTD). A significant finding is the "prompt-length scaling" property: unlike embedding-based models which saturate quickly, Anysynth's performance improves with longer reference prompts, validating the ICL approach. The ablation studies effectively isolate the benefits of in-context conditioning over CLAP embeddings and validate the superiority of Asymmetric CFG over Combined and Symmetric CFG variants. The UMAP visualizations provide qualitative support for the clustering of timbres.
The paper provides detailed implementation specifics, including the DiT architecture (1024 hidden dim, 25 layers, 16 heads), training hyperparameters (batch size, learning rate, warmup), and dataset construction details (filtering rules for NSynth/Slakh). The use of standard components (Vocos vocoder, RoPE, AdaLN) aids reproducibility. However, the code repository is not explicitly linked in the text (only a demo page), which is a minor hurdle for immediate reproduction, though the architectural details are sufficient for a competent researcher to implement.
The paper acknowledges that performance gains from prompt length diminish after 8 seconds, suggesting a limit to the utility of very long prompts for short generation tasks. The model is evaluated only on keyboard, guitar, and bass instruments; generalization to more complex orchestral instruments or vocals is not demonstrated. The reliance on a pre-trained vocoder (Vocos) means that the quality of the final waveform is partially dependent on the vocoder's capability, although this is standard practice. The computational cost of attending to long reference sequences in a DiT might be higher than embedding-based methods during inference, although the authors claim no major overhead due to parallel computation.
This work advances the field of controllable audio generation by providing a more faithful and flexible method for instrument cloning. It addresses a critical bottleneck in current systems (information loss in embeddings) and offers a scalable solution (ICL) that improves with more data/context. This has implications for music production tools, virtual instrument synthesis, and audio restoration. The Asymmetric CFG technique is generalizable to other multimodal generation tasks where semantic and stylistic/acoustic conditions are entangled. The paper presents a significant advancement in zero-shot instrument cloning by replacing information-bottleneck embeddings with in-context learning in a DiT architecture, supported by a novel Asymmetric CFG strategy that effectively decouples semantic and acoustic guidance.
Automated assessment of second language (L2) speaking proficiency relies on large-scale annotated speech data, which remains scarce compared to widely available written learner corpora. A promising direction for addressing this imbalance is to use text-to-speech (TTS) and voice cloning to convert written L2 production into synthetic speech. However, written and spoken L2 differ fundamentally: spontaneous speech includes disfluencies and discourse markers, while writing is more planned and complex. This raises the question of what is required to generate synthetic L2 speech suitable for assessment. We address this through a systematic analysis of speaker-text relationships using COREFL, a publicly available corpus containing paired spoken and written responses from the same L2 learners to the same questions across modalities. In our proposed framework, we first address the structural differences between written and spoken language by transforming written responses into spoken-style transcripts ("speechification") using a large language model. These transcripts are then converted into speech using a TTS/voice-cloning model. To assign a voice to each synthetic response, we investigate different speaker-text pairing strategies based on shared learner attributes (proficiency level, first language, both, or neither). We evaluate our data augmentation techniques on the language assessment task, with improvements shown in both wav2vec2 (audio-based) and ModernBERT (text-based) scoring systems. Results show that matching speakers and texts by proficiency level yields the most robust synthetic speech. Moreover, raw written text leads to a strong mismatch with spoken language, while speechification substantially reduces this gap and improves grading performance.
Primary: University of Cambridge
All Institutions: University of Cambridge
The paper presents a systematic framework for augmenting L2 speaking assessment data by speechifying written texts and synthesizing speech, demonstrating that matching proficiency levels and modeling spoken linguistic structures significantly improves downstream grading performance.
The paper proposes a data augmentation framework for L2 speaking assessment by converting written learner essays into synthetic speech. The core methodological contribution is the "speechification" step, where an LLM (GPT-4.1) rewrites written text into spoken-style transcripts incorporating disfluencies, hesitations, and learner errors, conditioned on CEFR proficiency levels. This is followed by zero-shot voice cloning using OmniVoice. The study systematically evaluates speaker-text pairing strategies (matching by CEFR, L1, both, or neither) to determine the optimal conditions for generating synthetic data that mimics real L2 speech characteristics. The approach is well-motivated by the scarcity of annotated L2 speech data and the structural differences between written and spoken language.
The authors utilize the COREFL corpus, which provides paired written and spoken responses, allowing for a controlled analysis of modality gaps. They evaluate the synthetic data using two downstream grading models: wav2vec2 (audio-based) and ModernBERT (text-based). The experiments demonstrate that speechification significantly improves performance, particularly for the text-based grader, by reducing the linguistic mismatch. Matching speakers and texts by proficiency level was found to be the most robust strategy. The results show consistent improvements in PCC, SRC, and RMSE metrics when using augmented data. However, the evaluation relies heavily on automatic metrics (WER, PCC, SRC) and speaker verification cosine similarity, lacking subjective human evaluation (e.g., MOS) for speech naturalness or assessment accuracy, which is a notable gap for TTS research.
The paper provides detailed descriptions of the pipeline, including the LLM prompt for speechification, the TTS model (OmniVoice), and the grading architectures. The COREFL dataset is publicly available. The code for the grading systems is not explicitly linked, but the model architectures (wav2vec2, ModernBERT) are standard. The specific hyperparameters are mentioned to be in a table (referenced but not fully detailed in the snippet provided, though standard practices are implied). The cost of LLM usage is noted, adding transparency. Overall, the methodology is clear enough for reproduction, assuming access to the proprietary OmniVoice model or a suitable open-source alternative.
The reliance on GPT-4.1 for speechification introduces a dependency on a specific, potentially costly, and non-deterministic component. The synthetic speech, while linguistically improved via speechification, may still lack the acoustic variability and prosodic nuances of real spontaneous speech, as evidenced by the performance gap between real and synthetic data. The study focuses on English L2 learners, limiting generalizability to other languages without further validation. The lack of human evaluation for the synthetic speech quality and the grading performance limits the confidence in the "realism" of the data from a pedagogical perspective. The use of zero-shot TTS means the synthetic voices are clones but may not capture the full idiosyncratic speech patterns of the original speakers.
This work has significant potential to address the data scarcity problem in automated language assessment, particularly for L2 speaking. By enabling the generation of large-scale, annotated synthetic speech data from abundant written corpora, it could accelerate the development of more robust and fair assessment tools. However, ethical considerations regarding the use of synthetic data in high-stakes testing environments must be addressed, including issues of bias, transparency, and the potential for over-reliance on synthetic data that may not fully represent human speech variability. The paper presents a systematic framework for augmenting L2 speaking assessment data by speechifying written texts and synthesizing speech, demonstrating that matching proficiency levels and modeling spoken linguistic structures significantly improves downstream grading performance.
Dance-to-music generation is a promising task for applications such as choreography support and automatic accompaniment, where temporal coordination between body movement and sound is essential. In particular, using human joint positions as the motion representation is attractive because they explicitly capture body dynamics while being lightweight, privacy-preserving, and easy to integrate with motion capture and pose-estimation pipelines. A central challenge in this setting, however, is the scarcity of high-quality paired dance-music data, since collecting accurately synchronized pairs is costly and often constrained by copyright and performance rights. This makes it difficult to train end-to-end models solely from paired data. To address this issue, we propose a dance-conditioned music generation framework that efficiently exploits both unpaired and paired data. Our method combines pretrained unimodal encoders for motion and music, beat-guided contrastive pretraining to align their feature spaces, and a ControlNet-style conditioning module on top of a pretrained text-to-audio diffusion model. Experiments on AIST++ demonstrate that the proposed techniques improve both dance-music alignment and audio quality, as confirmed by quantitative and qualitative evaluations. Compared to a state-of-the-art method, our approach achieves superior dance alignment performance and competitive audio quality. Code is available at https://github.com/kmraven/AudioLDM-ControlNet .
Primary: Sony Computer Science Laboratories
All Institutions: Sony Computer Science Laboratories, Keio University, Georgia Institute of Technology
The paper presents a competent application of contrastive alignment and ControlNet-style conditioning to the dance-to-music generation task, achieving state-of-the-art alignment performance on AIST++. While the methodological novelty is incremental, the technical execution is solid, and the results demonstrate the effectiveness of combining unimodal pretraining with cross-modal alignment for this specific, data-scarce problem.
The paper proposes a two-stage framework for dance-to-music generation. Stage 1 involves beat-guided contrastive pretraining to align features from a pretrained MotionBERT encoder (for dance) and MERT encoder (for music). Stage 2 uses these aligned motion features to condition a frozen AudioLDM backbone via a ControlNet-style adapter. The methodology is technically sound and follows established patterns in multimodal generation (contrastive alignment + diffusion conditioning). The use of MotionBERT for 2D keypoints is a strong choice for capturing kinematic structure. However, the novelty is moderate; the core idea of using contrastive learning to align modalities (like CLIP) and then conditioning a diffusion model (like ControlNet) is a well-known paradigm in computer vision and has been applied to audio (e.g., AudioLDM). The specific application to dance-to-music with beat-guided alignment is a valid contribution but represents an incremental adaptation of existing techniques rather than a fundamental methodological breakthrough. The "beat-guided" aspect is a specific engineering detail rather than a new theoretical framework.
The experiments are conducted on the AIST++ dataset, which is the standard benchmark for this task. The evaluation includes both objective metrics (BHS, BCS, F1, TD, FAD, CLAP, BAS) and subjective user studies (MOS). The results show that the proposed method outperforms the baseline (Li et al.) in dance-alignment metrics (BAS and MOS), which is the primary goal. However, the subjective sound quality MOS is slightly lower than the baseline, although objective audio quality metrics (FAD, CLAP) are better. The ablation studies effectively demonstrate the contribution of the contrastive pretraining and the MotionBERT encoder. The comparison with Li et al. is fair as they use the same dataset split. One limitation is the small test set (36 pairs), which can lead to high variance in metrics, though the authors acknowledge this by using both objective and subjective measures. The results are consistent with the claims.
The paper provides sufficient detail for reproduction. It specifies the datasets (AIST++), preprocessing steps (resampling, normalization), model architectures (MotionBERT, MERT, AudioLDM, ControlNet), and training hyperparameters (learning rates, batch sizes, epochs). The code is available on GitHub. The use of standard pretrained models (AudioLDM, MotionBERT, MERT) facilitates reproducibility. The specific beat extraction method (Librosa) and contrastive loss details are clearly described.
The paper acknowledges the small scale of paired dance-music data. The subjective sound quality is slightly inferior to the baseline, which the authors attribute to the backbone (AudioLDM vs MusicGen). The reliance on 2D keypoints may lose some 3D motion information, although MotionBERT is designed to handle this. The beat extraction is heuristic-based (velocity smoothing), which might not be robust to all dance styles. The evaluation is limited to AIST++, and generalizability to other datasets or styles is not tested. The subjective study has a small number of participants (35) and samples (10 per participant), which may not fully capture perceptual nuances.
This work contributes to the field of multimodal generation, specifically the intersection of dance and music. It has potential applications in choreography support, automatic accompaniment, and entertainment. The use of joint positions is privacy-preserving compared to video-based methods. The framework could be extended to other motion-to-audio tasks. The open-source code promotes further research in this area. The paper presents a competent application of contrastive alignment and ControlNet-style conditioning to the dance-to-music generation task, achieving state-of-the-art alignment performance on AIST++. While the methodological novelty is incremental, the technical execution is solid, and the results demonstrate the effectiveness of combining unimodal pretraining with cross-modal alignment for this specific, data-scarce problem.