Unifying speech, sound, and music generation in one model is hindered by tradeoffs between fidelity, end-to-end training, in-context conditioning, and variable-length synthesis that no current paradigm fully resolves. To address this challenge, we present AudioCALM, a universal audio generation framework that extends autoregressive (AR) next-token prediction from discrete tokens to continuous audio latents: a thin flow-matching head replaces the softmax to predict rectified-flow velocities at each position, and a block-causal AR-Flow attention pattern produces arbitrary-length output. Joint training of multiple audio generation tasks faces an asymmetric text--audio mismatch: speech transcripts align to specific time spans and demand tight, time-aligned attention, whereas sound and music captions describe only overall semantics and rely on diffuse, holistic attention; mixing the two disproportionately degrades sound and music generation. We address this asymmetry at two levels: a data reformulation strategy that unifies all three tasks under a single description-style conditioning interface, and a novel architecture Asymmetric Mixture-of-Modality-Experts (A-MoME), which adds a dedicated residual expert for speech while sound and music share the backbone, incurring no inference overhead on non-speech inputs. Experimental results demonstrate that AudioCALM matches modality-specific state-of-the-art and outperforms prior unified baselines on speech, sound, and music generation benchmarks.
Primary: Hong Kong University of Science and Technology (HKUST)
All Institutions: Hong Kong University of Science and Technology (HKUST), Alibaba Group
AudioCALM presents a compelling unified audio generation framework that effectively bridges the gap between discrete autoregressive modeling and continuous flow matching, achieving state-of-the-art performance across speech, sound, and music domains while introducing novel architectural and data-level solutions to cross-modal interference.
The paper proposes AudioCALM, a unified framework for text-to-speech, text-to-sound, and text-to-music generation. The core methodological innovation is "Continuous Autoregressive Language Modeling" (CALM), which replaces the discrete softmax output of standard autoregressive language models with a continuous flow-matching head that predicts rectified-flow velocities over VAE latents. This allows the model to leverage the streaming and in-context capabilities of AR models while avoiding the information bottleneck of discrete tokenization. Key technical components include: 1) AR-Flow Attention: A block-causal attention pattern that allows bidirectional flow matching within a block of latents while maintaining autoregressive commitment across blocks, enabling variable-length generation. 2) Asymmetric Mixture-of-Modality-Experts (A-MoME): A novel architectural design that adds a dedicated residual expert for speech (which requires tight local alignment) while sharing the backbone for sound and music (which rely on global semantics), addressing the identified "asymmetric mismatch" in joint training. 3) Description-Style Conditioning: A data reformulation strategy using an MLLM to generate long-form, modality-specific descriptions from short captions/transcripts, unifying the conditioning interface across modalities. The approach is theoretically sound and addresses specific pain points in unified audio generation (fidelity vs. flexibility, cross-modal interference).
The evaluation is comprehensive, covering three distinct audio modalities on standard benchmarks (LibriTTS, SeedTTS for speech; AudioCaps for sound; Song-Describer for music). The paper compares AudioCALM against both modality-specific state-of-the-art systems (e.g., CosyVoice 3.0, Stable Audio Open) and prior unified models (UniAudio, UniFlow-Audio). Results show that AudioCALM matches or exceeds SOTA on most metrics, particularly in sound and music generation (FAD, CLAP score) and speech intelligibility (WER). The ablation studies are particularly strong, effectively isolating the contributions of the continuous head, the description-style conditioning, and the A-MoME architecture. The finding that adding speech data disproportionately degrades non-speech generation (and vice versa) is a significant empirical insight that justifies the asymmetric design. The use of both objective metrics (FAD, WER, CLAP) and subjective evaluations (MOS) provides a robust assessment.
The paper provides detailed implementation details, including the VAE architecture (CNN-GAN with iSTFT head), training hyperparameters (AdamW, batch size, learning rate), and the specific prompts used for the MLLM captioning pipeline. The authors release code and weights, and provide the cached annotations for public datasets, which significantly aids reproducibility. The use of open-source datasets (LibriTTS, VGGSound, FMA, etc.) ensures that the training data is accessible. The only potential hurdle is the reliance on Gemini 3 Pro for the offline captioning step, but the authors mitigate this by releasing the prompts and the resulting captions.
The authors acknowledge several limitations: 1) The training data is restricted to English speech and public sound/music corpora, limiting generalization to non-English speech, singing voice, and rare audio events. 2) The backbone scale is limited to 1.7B-4B parameters, leaving open questions about scaling behavior. 3) Long-form generation coherence and termination are not deeply investigated, with the current system relying on a simple stop head. 4) The use of a closed-source MLLM for data preparation introduces a dependency that may not be fully reproducible by all researchers without access to similar models.
AudioCALM represents a significant step towards universal audio generation, which has broad applications in accessibility (TTS), creative industries (music/sound design), and research (data augmentation). However, the power of unified models to clone voices and generate realistic sound effects raises serious concerns about misuse, including impersonation, fraud, and disinformation. The authors address this by implementing safeguards in the license (prohibiting non-consensual cloning) and discussing the need for synthetic audio detection. The release of such a powerful model requires careful consideration of these risks. AudioCALM presents a compelling unified audio generation framework that effectively bridges the gap between discrete autoregressive modeling and continuous flow matching, achieving state-of-the-art performance across speech, sound, and music domains while introducing novel architectural and data-level solutions to cross-modal interference.
While large language model (LLM)-based text-to-speech (TTS) systems have achieved high-quality speech synthesis, most existing systems focus on English and Chinese. Japanese, however, remains under-explored, and its unique linguistic challenges, such as widespread context-dependent kanji polyphony, have yet to be adequately tackled. Here we introduce Sarashina2.2-TTS (https://github.com/sbintuitions/sarashina2.2-tts), a Japanese-centric LLM-TTS system that tackles these challenges through a dual approach: data strategy and evaluation methodology. First, we scale training to approximately 361k hours of speech, incorporating a balanced mix of Japanese and English data. Furthermore, we design a targeted data augmentation pipeline covering all 2,136 Joyo (regular-use) kanji designated by Japan's Agency for Cultural Affairs to efficiently address kanji polyphony disambiguation. Second, we introduce the Joyo Kanji Yomi Benchmark (https://github.com/sbintuitions/JoyoKanji-Yomi-Benchmark), covering all 2,136 Joyo kanji and their 4,378 readings. Alongside this benchmark, we propose Kana-CER, a metric that compares synthesized speech against reference readings in the kana space, eliminating orthographic variations to directly measure pronunciation correctness. Experiments demonstrate that our targeted data augmentation significantly improves reading accuracy. Overall, Sarashina2.2-TTS achieves state-of-the-art kanji-level reading accuracy and matches top baselines on general sentence-level pronunciation, while delivering the highest speaker similarity in zero-shot Japanese speech synthesis. Furthermore, cross-lingual evaluation reveals that Sarashina2.2-TTS is the only system that maintains stable Japanese pronunciation regardless of the prompt language, confirming that our balanced training approach improves cross-lingual robustness.
Primary: SB Intuitions
All Institutions: SB Intuitions
Sarashina2.2-TTS makes a significant contribution to Japanese speech synthesis by introducing a targeted synthetic data augmentation pipeline for kanji polyphony and a novel kana-based evaluation metric, achieving state-of-the-art reading accuracy and cross-lingual robustness.
The paper proposes a comprehensive data-centric strategy to address the specific linguistic challenge of kanji polyphony in Japanese TTS. The core methodological contribution is the construction of a massive 361k-hour multilingual dataset with a balanced Japanese-English ratio, which is unusually large for open-source Japanese TTS. The most novel technical component is the "Pronunciation Steering" (PronSteering) mechanism, which uses special tokens to inject explicit kana readings and pitch-accent tags into the LLM context. This is leveraged in a targeted synthetic data generation pipeline to cover rare kanji readings. The authors also introduce a novel evaluation metric, Kana-CER, which operates in the phonological (kana) space rather than the orthographic (kanji) space to eliminate errors caused by Japanese orthographic variation. The architecture itself (S3Tokenizer + LLM + Flow Matching) is derivative of existing LLM-TTS systems (like CosyVoice), but the data engineering and evaluation framework are distinct and highly relevant to the subfield. EXPERIMENTAL_EVALUTION: The experimental evaluation is rigorous and well-designed for the specific problem. The authors introduce the "Joyo Kanji Yomi Benchmark," a human-verified dataset covering all 2,136 Joyo kanji and 4,378 readings, which fills a critical gap in the field. Results show state-of-the-art performance on this benchmark, significantly outperforming baselines like Qwen3-TTS and FishAudio S1-mini in kanji-level accuracy. The cross-lingual robustness experiments are particularly compelling, demonstrating that the balanced training data prevents the degradation of Japanese pronunciation when prompted with non-Japanese speech, a common failure mode in multilingual models. The use of standard CER vs. Kana-CER effectively highlights the limitations of existing evaluation metrics for Japanese.
The paper provides code and model weights for the TTS system and the benchmark. The detailed description of the PronSteering tokenization and the synthetic data generation pipeline enhances reproducibility. However, the exact sources of the 361k hours of data are not fully enumerated (only domains are described), which is typical for large-scale proprietary data curation but limits full reproducibility of the data distribution. The Kana-ASR model is also released, aiding in the reproducibility of the evaluation metric.
The PronSteering capability is explicitly stated as *not* included in the open-source release; users only get the model trained with synthetic data generated by it. This limits the immediate utility of the method for users who want to control pronunciation dynamically. The reliance on LLM-generated sentences for the synthetic data and benchmark introduces potential biases or unnatural phrasing, although human verification mitigates this. The Kana-ASR model, while effective, may struggle with highly expressive or colloquial speech, as noted by the authors.
This work significantly advances the state of Japanese TTS, a language often underserved compared to English and Chinese. By providing a standardized benchmark and evaluation metric, it facilitates fairer comparisons and drives progress in handling complex orthographic-to-phonological mappings. The balanced training strategy offers insights for improving cross-lingual robustness in multilingual models. The open-source release of the benchmark and tools will likely spur further research into low-resource language handling and polyphony disambiguation. Sarashina2.2-TTS makes a significant contribution to Japanese speech synthesis by introducing a targeted synthetic data augmentation pipeline for kanji polyphony and a novel kana-based evaluation metric, achieving state-of-the-art reading accuracy and cross-lingual robustness.
Speech-to-speech translation (S2ST) should preserve not only lexical meaning, but also expressive attributes: emotion, scenario style (e.g., news reporting vs. dramatic dialogue), and nonverbal vocalizations (NVs). Moreover, collecting cross-lingual target speech that is both translation-faithful and expressively aligned with the source is difficult at scale, making reference-based evaluation impractical. We introduce STEB (Speech-to-Speech Translation Expressiveness Benchmark), a 32.6-hour Chinese--English benchmark that evaluates both standard dimensions (translation fidelity, speaker similarity, duration alignment) and expressiveness dimensions (emotion, scenario style, NV preservation). For expressiveness evaluation, STEB uses a caption-then-summarize framework that converts speech into structured expressive attributes and compares source and hypothesis attributes with an LLM judge. Human validation shows statistically significant correlations with listener judgments across all expressive dimensions. We evaluate six S2ST systems covering cascaded systems, end-to-end models, and speech large language models. Many systems, especially cascaded ones, achieve strong translation fidelity, but they still struggle with emotion preservation (best: 3.82/5) and NV preservation (best: 2.31/5). These results reveal a gap between semantic transfer and expressive transfer, identifying expressiveness preservation as an open challenge for S2ST. Audio samples are available at https://cmots.github.io/steb.github.io/.
Primary: Hong Kong University of Science and Technology
All Institutions: Hong Kong University of Science and Technology, Tencent Youtu Lab, Shenzhen International Graduate School, Tsinghua University
This paper presents a significant contribution to the field of Speech-to-Speech Translation by introducing STEB, a comprehensive benchmark that evaluates not just translation accuracy but also the preservation of expressive attributes such as emotion, scenario style, and nonverbal vocalizations. The proposed "caption-then-summarize" LLM-based evaluation framework provides a scalable and reference-free solution to a previously intractable problem, validated by strong human correlation. The empirical results reveal a critical gap between semantic transfer and expressive transfer in current S2ST systems, offering valuable insights for future model development and establishing a new standard for evaluating expressive speech technologies.
The paper introduces a novel evaluation framework for Speech-to-Speech Translation (S2ST) that moves beyond semantic fidelity to assess expressive attributes (emotion, scenario style, nonverbal vocalizations). The core methodological contribution is the "caption-then-summarize" pipeline, which leverages multimodal LLMs to convert audio into structured textual descriptions of expressiveness, enabling reference-free comparison via an LLM-as-a-judge. This approach addresses the critical lack of parallel expressive S2ST references. The data curation pipeline is rigorous, involving source separation, speaker diarization, and multi-stage quality filtering using both automatic metrics (DNSMOS, BEATs) and human validation. The methodology is sound and addresses a significant gap in the field, although it relies heavily on the capabilities of current multimodal LLMs for annotation and judging.
The experimental setup is comprehensive, evaluating six diverse S2ST systems (cascaded, end-to-end, and speech LLMs) on a 32.6-hour Chinese-English benchmark. The results clearly demonstrate the decoupling of translation fidelity and expressiveness preservation, with cascaded systems excelling in BLEU but failing in emotion/NV preservation, while end-to-end models show better expressiveness but lower translation accuracy. The human correlation study provides strong validation for the LLM judge, showing statistically significant agreement with human raters, particularly for emotion and NVs. The analysis of why explicit NV markers help cascaded systems but not end-to-end models offers valuable insights into system design.
The paper provides detailed descriptions of the data curation pipeline, including specific models used (BS-Roformer, Silero VAM, pyannote, Qwen3 variants) and hyperparameters. The inclusion of prompts for the LLM judges and the annotation pipeline enhances reproducibility. The release of code and audio samples (or metadata for copyrighted audio) further supports reproducibility. The strict quality control steps are well-documented, allowing other researchers to replicate the benchmark construction.
The benchmark is currently limited to Chinese-English pairs, restricting its generalizability to other language pairs. The reliance on LLM-based evaluation introduces potential biases inherent in the judge models, although human correlation mitigates this to some extent. The scenario style dimension remains subjective and shows lower correlation with human judgments compared to emotion and NVs. The benchmark size, while substantial, may not cover the full diversity of real-world speech scenarios, particularly rare or highly specialized contexts.
This work significantly advances the field of S2ST by highlighting the importance of expressiveness preservation, which is crucial for applications like dubbing, virtual assistants, and cross-lingual communication. By providing a standardized benchmark and evaluation metric, it enables fair comparison of future S2ST systems and drives research towards more human-like and expressive translation. The findings suggest that current systems are not yet ready for high-quality expressive dubbing, setting a clear direction for future improvements. This paper presents a significant contribution to the field of Speech-to-Speech Translation by introducing STEB, a comprehensive benchmark that evaluates not just translation accuracy but also the preservation of expressive attributes such as emotion, scenario style, and nonverbal vocalizations. The proposed "caption-then-summarize" LLM-based evaluation framework provides a scalable and reference-free solution to a previously intractable problem, validated by strong human correlation. The empirical results reveal a critical gap between semantic transfer and expressive transfer in current S2ST systems, offering valuable insights for future model development and establishing a new standard for evaluating expressive speech technologies.
Learning discrete speech representations that preserve similarity across variable-length utterances is central to query-by-example spoken term detection (QbE-STD). While wav2tok introduced CTC-based sequence alignment to enforce token consistency, its tightly coupled clustering and alignment training recipe limits scalability. We propose wav2tok 2.0, a scalable alignment-aware speech tokenizer built on the BEST-STD backbone. wav2tok 2.0 employs staged training, first learning discriminative, speaker-invariant representations via contrastive learning and vector quantization, and then enforcing pairwise token consistency using a CTC alignment loss and a novel DTW-aligned framewise prediction objective with adaptive weighting. Experiments show that wav2tok 2.0 consistently outperforms BEST-STD and general-purpose tokenizers on QbE-STD while remaining efficient and scalable.
Primary: Indian Institute of Technology Kanpur
All Institutions: Indian Institute of Technology Kanpur, KU Leuven
wav2tok 2.0 introduces a scalable, alignment-aware speech tokenizer that combines contrastive learning with explicit CTC and DTW-aligned framewise alignment objectives, achieving state-of-the-art performance in QbE-STD tasks while maintaining computational efficiency.
The paper proposes wav2tok 2.0, a scalable speech tokenizer for Query-by-Example Spoken Term Detection (QbE-STD). It builds upon the BEST-STD architecture by introducing a two-stage training process. Stage I uses contrastive learning and vector quantization to learn discriminative, speaker-invariant representations. Stage II enforces pairwise token consistency using a CTC-based alignment loss and a novel DTW-aligned framewise token prediction objective with adaptive weighting. The methodology addresses the scalability issues of the original wav2tok by decoupling representation learning from alignment constraints. The introduction of the DTW-aligned framewise prediction loss is a specific technical contribution aimed at fine-grained alignment, though it relies on existing DTW and CTC mechanisms.
The authors evaluate wav2tok 2.0 on LibriSpeech and TIMIT datasets using standard QbE-STD metrics (MAP, MRR, MTWV). They compare against general-purpose tokenizers (HuBERT, WavLM, SpeechTokenizer, EnCodec), conventional STD baselines (MFCC, BNF), and prior speech-specific tokenizers (BEST-STD, wav2tok). Results indicate that wav2tok 2.0 consistently outperforms these baselines across various codebook sizes and query types (IV/OOV). The ablation studies demonstrate the contribution of both the CTC alignment and the novel framewise prediction loss. The experiments are well-structured and provide a clear comparison, although the dataset scope is limited to English speech corpora.
The paper provides detailed implementation details, including encoder architecture (Mamba-based), codebook sizes, loss weights, and training epochs. A GitHub repository link is provided. The use of standard libraries for CTC and DTW suggests high reproducibility. The staged training approach is clearly defined, facilitating replication.
The primary limitation is the reliance on English-only datasets (LibriSpeech, TIMIT), which limits the assessment of multilingual generalization. The paper acknowledges this and suggests future work on multilingual settings. Additionally, while the method is more scalable than the original wav2tok, it still requires paired utterances for Stage II training, which may be a constraint for some retrieval scenarios. The performance gain, while consistent, is marginal in some metrics compared to the strong BEST-STD baseline, suggesting diminishing returns from the added complexity.
This work contributes to the field of efficient audio retrieval and spoken term detection. By improving the scalability and accuracy of discrete speech tokenizers, it facilitates more robust audio indexing and search applications. The techniques for explicit pairwise alignment could be relevant to other sequence modeling tasks in speech processing. However, the impact is somewhat niche, primarily benefiting researchers and practitioners in the specific domain of QbE-STD. wav2tok 2.0 introduces a scalable, alignment-aware speech tokenizer that combines contrastive learning with explicit CTC and DTW-aligned framewise alignment objectives, achieving state-of-the-art performance in QbE-STD tasks while maintaining computational efficiency.
We introduce DNSMOS-C, a compact end-to-end speech quality assessment model that extends the DNSMOS Pro framework by integrating a MOS-guided triplet-based contrastive loss. Applied directly to the intermediate embeddings, this contrastive supervision encourages the latent space to be better organized with respect to perceptual quality while preserving the simplicity and efficiency of DNSMOS Pro. Unlike prior methods that depend on large pre-trained self-supervised learning (SSL) encoders and multi-stage training, DNSMOS-C jointly learns speech representations and MOS regression within a single, unified framework. Experiments on multiple datasets show that DNSMOS-C consistently improves correlation metrics over DNSMOS Pro and achieves better generalization on challenging out-of-domain test sets. Furthermore, latent space analyses indicate that our approach learns representations that exhibit an emergent low-dimensional quality ordering, which enhances interpretability and improves training stability. These findings demonstrate that MOS-guided contrastive learning enables more robust and accurate quality predictions without incurring additional computational overhead.
Primary: KTH Royal Institute of Technology
All Institutions: KTH Royal Institute of Technology, Google LLC
DNSMOS-C improves the robustness and generalization of lightweight speech quality models by integrating MOS-guided contrastive learning into the DNSMOS Pro framework, offering a practical balance between performance, efficiency, and training stability for real-world deployment.
The paper proposes DNSMOS-C, a modification of the existing DNSMOS Pro architecture. The core methodological contribution is the integration of a MOS-guided triplet-based contrastive loss (adapted from SCOREQ) into the training objective of a compact, end-to-end convolutional model. The authors argue that this encourages the latent space to be organized by perceptual quality rather than specific distortion types. While the application of contrastive learning to speech quality is not entirely new (SCOREQ did this for SSL features), applying it directly to the intermediate embeddings of a lightweight, end-to-end CNN without pre-trained SSL encoders is a valid and pragmatic engineering contribution. The approach is technically sound but relies heavily on adapting existing loss functions rather than proposing a novel architectural primitive or theoretical framework. The integration is straightforward: adding a weighted contrastive term to the Gaussian Negative Log-Likelihood (GNLL) loss.
The experimental evaluation is comprehensive in terms of dataset variety, covering synthetic (BVCC), simulated (NISQA, Tencent), and real-world (TCD-VoIP, ESC50) data. The results show consistent improvements in correlation metrics (LCC, SRCC) over the DNSMOS Pro baseline, particularly in out-of-domain generalization scenarios. The latent space analysis using PCA and clustering provides qualitative support for the claim that the model learns a "quality manifold." The inclusion of standard deviation over 10 runs adds credibility to the stability claims. However, the performance gains, while consistent, are modest in absolute terms (e.g., LCC improvements of ~0.01-0.02 on some splits). The trade-off analysis regarding distortion clustering vs. quality ordering is insightful but highlights a limitation in interpretability for specific artifact types.
The paper provides significant detail on the methodology, including hyperparameters (learning rate, epochs, margin), data preprocessing steps (16kHz, 10s padding, log-magnitude spectrograms), and the specific loss formulations. The authors explicitly state that code and checkpoints will be available on GitHub, which significantly enhances reproducibility. The use of standard datasets and clear evaluation metrics allows for direct comparison with prior work.
The primary limitation is the incremental nature of the novelty; it adapts a known technique (contrastive regression) to a known architecture (DNSMOS Pro). The performance gains, while statistically significant in correlation, may not be transformative for all applications. The latent space analysis shows a degradation in the ability to separate specific distortion types, which might be a drawback for diagnostic applications where identifying the *cause* of poor quality is as important as the *score*. Furthermore, the model is still limited by the capacity of a small CNN compared to larger SSL-based models, though this is a trade-off for efficiency.
This work contributes to the field of automatic speech quality assessment, a critical component for VoIP, streaming services, and generative speech models. By providing a more robust, efficient, and generalizable model, it facilitates the deployment of high-quality monitoring tools in resource-constrained environments. The emphasis on generalization to unseen domains addresses a key pain point in the industry. DNSMOS-C improves the robustness and generalization of lightweight speech quality models by integrating MOS-guided contrastive learning into the DNSMOS Pro framework, offering a practical balance between performance, efficiency, and training stability for real-world deployment.
Recently, zero-shot text-to-speech (TTS) has enabled high-fidelity and expressive speech synthesis, but it often fails to imitate unseen speaking styles from uncommon scenarios (e.g., crosstalk, dialects). Moreover, fine-tuning pretrained models requires large, high-quality datasets, limiting rapid personalization. We propose VoiceTTA, a reinforcement learning-based test-time adaptation (TTA) method that improves voice imitation of pretrained zero-shot TTS models. VoiceTTA introduces two style rewards based on coefficient-of-variation differences of F0 and energy, combined with speaker similarity and intelligibility (WER from a pretrained Whisper model), and optimizes learnable prefixes via group relative preference optimization (GRPO) in a flow matching-based model at inference time. Extensive experiments demonstrate substantial improvements on uncommon speech prompts, outperforming state-of-the-art baselines. Audio samples are available at https://voicetta.pages.dev/
Primary: The Hong Kong University of Science and Technology (Guangzhou)
All Institutions: The Hong Kong University of Science and Technology (Guangzhou), Tencent
[One sentence main contribution]. This paper introduces VoiceTTA, a reinforcement learning-based test-time adaptation method that optimizes learnable prefixes in flow-matching TTS models using a composite reward of prosodic variation and speaker similarity. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The work represents a novel application of LLM-centric RL algorithms (GRPO) to continuous audio generation, offering a parameter-efficient way to adapt zero-shot TTS models to unseen, low-resource styles. While the technical approach is innovative and the results show clear improvements in objective style similarity metrics, the reliance on an internal dataset and the modest gains in perceptual naturalness limit its immediate impact on the broader TTS community. It serves as a proof-of-concept for RL-based TTA in audio, paving the way for more sophisticated reward designs and public benchmarking.
The paper proposes VoiceTTA, a test-time adaptation (TTA) framework for zero-shot Text-to-Speech (TTS) models. The core innovation lies in applying Group Relative Policy Optimization (GRPO)—an algorithm typically associated with Large Language Model (LLM) alignment—to optimize learnable prefixes in a flow-matching-based TTS model during inference. The method introduces a composite reward function consisting of style rewards (Coefficient of Variation differences for F0 and energy) and a speaker similarity reward, balanced with an intelligibility reward (Word Error Rate from Whisper). The approach is technically sound in its adaptation of RL techniques to continuous generation tasks, although the use of CV differences as a proxy for prosodic style is a simplification that may not capture complex temporal dynamics. The derivation of the probability ratio using flow-matching loss as a proxy is a clever workaround for the lack of discrete token probabilities in diffusion/flow models.
The experiments are conducted on a custom internal dataset of "uncommon" speech styles (accented, children, slurred, dialects) and the KeSpeech dialect dataset. The baseline comparisons include F5-TTS, CosyVoice, MaskGCT, and Vevo. The results show improvements in Speaker Similarity (S-SIM) and Word Error Rate (WER) compared to the base F5-TTS model. However, the subjective evaluation (MOS) shows only marginal improvements in style similarity (3.27 vs 3.07 for F5-TTS) and a slight drop in naturalness compared to CosyVoice. The use of an internal, undisclosed dataset for the primary "uncommon" evaluation is a significant limitation for reproducibility and fair comparison. The ablation studies provide some insight into the reward weights and number of prefixes, but the overall performance gains, while statistically significant in objective metrics, appear modest in perceptual quality.
The paper provides hyperparameters for the GRPO optimization (learning rate, number of prefixes, candidate sampling temperature range). However, the primary evaluation dataset is internal and not publicly available, which severely hinders reproducibility. The code is not explicitly linked in the text (only a demo page is provided), and the specific versions of the backbone models (F5-TTS, Whisper, speaker embedding models) are not fully detailed. The reliance on a "pretrained Whisper model" for WER calculation is standard, but the exact configuration is needed for exact replication.
The primary limitation is the lack of public data for the main experimental claims. The use of Coefficient of Variation for F0 and Energy is a coarse metric for prosody and may fail to capture nuanced stylistic elements like rhythm or phrasing. The GRPO adaptation is performed at inference time, which adds computational overhead per utterance, potentially limiting real-time applicability despite the "lightweight" parameter claim. The subjective MOS scores are low across the board (around 3.0-3.5), suggesting that while the method improves similarity metrics, the overall audio quality remains mediocre compared to state-of-the-art systems trained on massive datasets.
This work contributes to the field of efficient model adaptation, demonstrating that RL-based TTA can be effective for audio generation tasks. It highlights the potential for personalizing large generative models without full fine-tuning. However, the reliance on proprietary internal data limits the broader scientific impact. The method could be valuable for niche applications where data collection is difficult, but the marginal gains in naturalness may limit widespread adoption over existing fine-tuned or larger zero-shot models. [One sentence main contribution]. This paper introduces VoiceTTA, a reinforcement learning-based test-time adaptation method that optimizes learnable prefixes in flow-matching TTS models using a composite reward of prosodic variation and speaker similarity. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The work represents a novel application of LLM-centric RL algorithms (GRPO) to continuous audio generation, offering a parameter-efficient way to adapt zero-shot TTS models to unseen, low-resource styles. While the technical approach is innovative and the results show clear improvements in objective style similarity metrics, the reliance on an internal dataset and the modest gains in perceptual naturalness limit its immediate impact on the broader TTS community. It serves as a proof-of-concept for RL-based TTA in audio, paving the way for more sophisticated reward designs and public benchmarking.
Recent Large Audio Language Models (LALMs) have achieved remarkable progress in audio perceptual tasks across individual acoustic layers, including speech, sound, and music. However, existing benchmarks predominantly evaluate these layers in isolation, overlooking the complex contextual relationships that arise when multiple acoustic sources co-occur in real-world auditory scenes. Real-world auditory interpretation requires Context-Aware Auditory Scene Understanding (CASU): the ability to comprehend the holistic scene by integrating sound layers. To evaluate this capability, we introduce the CASU benchmark, which assesses whether Audio LLMs can interpret auditory scenes composed of speech, acoustic events (e.g., announcements), and background environments (e.g., traffic), and reason about the logical relationships between these layers. We propose a scalable pipeline for constructing time-accurate, semi-synthetic audio streams by composing real-world scene sounds with synthetic speech. Building on this data, we design four tasks that probe scene understanding: contextual question answering, entity extraction from the scene, speaker role inference, and counterfactual reasoning where scene is manipulated. Experiments across multiple LALMs demonstrate that effective auditory scene understanding requires integration over all auditory layers, rather than reliance on speech or sound alone, underscoring the necessity of CASU for advancing complex audio understanding in LALMs.
Primary: University of California Irvine
All Institutions: University of California Irvine, University of Illinois Chicago, Kennesaw State University
This paper presents a significant and timely contribution to the field of audio AI by introducing CASU, a benchmark that rigorously evaluates the ability of Large Audio Language Models to perform context-aware reasoning over complex, multi-layered auditory scenes. By shifting the focus from isolated perception to holistic scene understanding, the authors identify a critical limitation in current state-of-the-art models and provide a scalable, semi-synthetic pipeline to address it, thereby establishing a new standard for evaluating auditory intelligence.
The paper introduces Context-Aware Auditory Scene Understanding (CASU), a novel benchmark and evaluation paradigm designed to assess Large Audio Language Models (LALMs) on their ability to integrate multiple acoustic layers (speech, events, background) for scene-level reasoning. The core methodological contribution is a semi-synthetic data generation pipeline that combines real-world environmental sounds and discrete events with synthetic speech, controlled via structured JSON scripts generated by LLMs. This approach allows for precise manipulation of cross-layer contextual relationships, which is difficult to achieve with naturalistic, unannotated audio. The benchmark defines four specific tasks: Contextual Reasoning, Entity Extraction, Role Inference, and Counterfactual Reasoning. The methodology is sound in its intent to move beyond isolated perception tasks, addressing a genuine gap in current LALM evaluations where models often treat non-speech audio as mere noise or background rather than semantic anchors. The use of an agent-based question generation framework adds a layer of scalability to dataset creation.
The experimental evaluation is comprehensive, benchmarking a wide range of state-of-the-art LALMs, including open-source models (Qwen series, Audio Flamingo, Voxtral, SALMONN, LTU) and closed-source giants (GPT-4o Audio, Gemini 2.0 Flash). The results clearly demonstrate a "Perception-Understanding Gap," where models with high transcription accuracy (low WER) and event detection performance still struggle with tasks requiring logical integration of context. Key findings include the superiority of joint processing (omni-modal models) over cascaded pipelines (transcription + text reasoning) due to information loss in textual descriptions. The ablation studies effectively isolate the contribution of different audio layers, confirming that removing any single layer (speech, event, or background) significantly degrades performance. The error analysis provides valuable insights into whether failures stem from perceptual errors or reasoning flaws.
The paper provides detailed descriptions of the data generation pipeline, including the use of specific TTS tools (Zonos), retrieval datasets (Clotho, ARCA23K), and the matching score formula. The structured JSON script format for ground truth is a strong point for reproducibility, as it allows other researchers to regenerate similar scenes. However, the reliance on proprietary models for question generation and human curation steps introduces some opacity. The code and dataset are not explicitly linked in the provided text (URLs are "none"), which hinders immediate reproducibility, though the methodology is described sufficiently for replication.
The primary limitation is the synthetic nature of the speech component, which, while using high-fidelity TTS, may not fully capture the nuances of natural human speech (prosody, disfluencies, emotional variance) present in real-world recordings. The constraint of audio clips to under 30 seconds limits the complexity of scenes that can be modeled, potentially missing long-range dependencies. Additionally, the current scope is limited to one-person monologues and two-person conversations, excluding more complex multi-party interactions. The reliance on LLMs for script generation and question creation may introduce biases or logical inconsistencies that require significant human filtering, as acknowledged by the authors.
This work has significant implications for the development of more robust and context-aware audio AI systems. By highlighting the "Perception-Understanding Gap," it directs future research towards architectures that can better integrate multimodal signals for reasoning. This is crucial for applications such as autonomous driving (interpreting sirens vs. speech), smart home assistants, and accessibility tools for the hearing impaired. The benchmark provides a standardized way to evaluate progress in this under-explored area, fostering competition and improvement in holistic audio understanding. This paper presents a significant and timely contribution to the field of audio AI by introducing CASU, a benchmark that rigorously evaluates the ability of Large Audio Language Models to perform context-aware reasoning over complex, multi-layered auditory scenes. By shifting the focus from isolated perception to holistic scene understanding, the authors identify a critical limitation in current state-of-the-art models and provide a scalable, semi-synthetic pipeline to address it, thereby establishing a new standard for evaluating auditory intelligence.
Self-supervised learning (SSL) has emerged as an essential paradigm for music information retrieval (MIR). While current SSL models achieve state-of-the-art performance across various MIR tasks, they typically treat audio as 1D sequences, either operating on time-domain waveforms or on flattened time-frequency-domain spectrograms. This discards the rich spatial and structural information in time-frequency representations and overlooks a fundamental intuition in music production. In particular, music is naturally represented as time-frequency grids in MIDI-based workflows, a structure that tightly corresponds to 2D spectrograms and inherently makes many MIR tasks trivial. Motivated by this intuition, we propose PupuJEPA, a visual Joint-Embedding Predictive Architecture (JEPA) that is trained directly on 2D spectrograms. Instead of applying masked language modeling (MLM) to 1D sequences, PupuJEPA learns robust representations by predicting the latent embeddings of masked 2D spectrogram patches from unmasked contexts. To optimally adapt such a visual framework to music signals, we also apply domain-specific modifications to model architecture, training scheme, and inference paradigm, with comprehensive ablation studies showing their effectiveness. Evaluations on the MARBLE benchmark show that PupuJEPA outperforms the 1D sequence-based SSL models across multiple MIR tasks in linear probing. Additionally, case studies of the attention maps also confirm that PupuJEPA captures musically meaningful patterns within the 2D time-frequency domain. Codes and checkpoints are available at: https://www.yichenggu.com/PupuJEPA/.
Primary: The Chinese University of Hong Kong
All Institutions: The Chinese University of Hong Kong, Aalto University, Spellbrush
The paper presents PupuJEPA, a 2D spectrogram-based JEPA model for music representation learning that achieves state-of-the-art results on the MARBLE benchmark by introducing domain-specific architectural modifications and inference strategies. The work is a solid contribution to audio SSL, effectively bridging the gap between visual JEPA successes and music information retrieval needs, although the novelty is somewhat incremental given the existing landscape of 2D audio models.
The paper proposes PupuJEPA, a Joint-Embedding Predictive Architecture (JEPA) adapted for music information retrieval (MIR) by operating directly on 2D Mel-spectrograms. The core methodological contribution lies in adapting the visual JEPA framework to the audio domain through specific architectural and training modifications. Key innovations include: 1) Using an asymmetric patch size (4x16) to maintain high temporal resolution suitable for MIR tasks; 2) Implementing a restricted target encoder that only processes masked patches to prevent shortcut learning, diverging from standard JEPA implementations; 3) Introducing domain-specific masking strategies (blockwise and time-frequency masking) alongside random masking, with a curriculum-based scheduling mechanism; 4) Proposing novel inference paradigms for 2D models, including weighted layer fusion and structure-aware patch aggregation (Time-, Frequency-, and Block-Partitioned) to replace standard Global Average Pooling (GAP). The authors also identify that standard ViT components like DropPath and LayerScale cause representation collapse in this specific audio-SSL context, recommending their removal.
The evaluation is conducted on the MARBLE benchmark, covering a wide range of MIR tasks including emotion recognition, key detection, genre classification, beat tracking, structure analysis, and music tagging. PupuJEPA-Large achieves state-of-the-art (SOTA) or near-SOTA performance across most tasks compared to 1D sequence-based models (MERT, MusicFM, MuQ) and 2D audio models (AudioMAE++, A-JEPA). The ablation studies are comprehensive, validating the necessity of SwiGLU, QK-Norm, the smoothed L1 loss, and the specific masking/inference strategies. The paper demonstrates that 2D modeling preserves structural information beneficial for both global and local tasks. However, the performance gain over strong baselines like A-JEPA (which also uses 2D spectrograms) is modest in some metrics, suggesting that the specific JEPA adaptation provides incremental rather than revolutionary gains over existing 2D audio SSL approaches.
The paper provides detailed implementation details, including hyperparameters, dataset preprocessing (24kHz mono, 10.24s crops), and model configurations. The code and checkpoints are made publicly available. The training setup (500k steps, 32 B200 GPUs) is clearly described. The reproducibility is high, although the reliance on a large in-house dataset (100k hours) for pre-training might limit independent verification of the pre-training phase, though the downstream evaluation is on standard benchmarks.
The paper notes that scaling beyond the Large variant yields diminishing returns, likely due to the limitations of linear probing on highly complex representations. The performance on HookTheory structure analysis is only on par with baselines, indicating that 2D pooling strategies may still struggle with fine-grained local temporal dependencies compared to 1D sequence models in some contexts. The claim that music is "naturally" represented as 2D grids is a strong intuition but may not hold for all musical styles or production techniques, potentially limiting generalizability. Additionally, the comparison with some baselines involves retraining them on the authors' in-house dataset, which introduces a potential bias if the dataset distribution differs from the original training data of the baselines.
This work advances the field of self-supervised learning for audio by demonstrating the efficacy of 2D visual architectures (JEPA) for music processing. It challenges the dominance of 1D sequence models in MIR and provides a robust framework for learning rich musical representations. The findings could influence future model architectures for audio understanding, potentially leading to more efficient and accurate MIR systems. The open-source release contributes to the community by providing a strong baseline and codebase for future research in audio SSL. The paper presents PupuJEPA, a 2D spectrogram-based JEPA model for music representation learning that achieves state-of-the-art results on the MARBLE benchmark by introducing domain-specific architectural modifications and inference strategies. The work is a solid contribution to audio SSL, effectively bridging the gap between visual JEPA successes and music information retrieval needs, although the novelty is somewhat incremental given the existing landscape of 2D audio models.
Recent end-to-end models for EEG-guided target speech extraction report impressive results, underscoring potential for neuro-steered hearing technologies. However, our analysis reveals that high within-trial performance can be driven by trial-specific EEG structure that acts as shortcuts for target selection, leading to poor generalization on unseen trials. To overcome this gap, we propose TRUST-TSE, a two-stage framework to mitigate shortcut learning. By introducing contrastive pretraining with attended-speaker negative sampling, we encourage the EEG encoder to capture fine-grained EEG--speech alignment while suppressing trial-identity cues. We also employ a confidence-weighted extraction objective based on EEG--source similarity to guide extraction using the learned representations. Experiments on KUL and DTU datasets show that TRUST-TSE outperforms end-to-end baselines under strict cross-trial protocols, addressing a key reliability bottleneck of existing approaches.
Primary: Seoul National University
All Institutions: Seoul National University, University of Iowa
This paper presents a critical analysis of shortcut learning in EEG-guided speech extraction and proposes a robust two-stage training framework (TRUST-TSE) that significantly improves cross-trial generalization, addressing a major reliability bottleneck in neuro-steered audio technologies.
The paper proposes TRUST-TSE, a two-stage framework designed to mitigate shortcut learning in EEG-guided target speech extraction (TSE). The core methodological contribution lies in the diagnosis that end-to-end models exploit trial-specific EEG artifacts (trial identity) rather than genuine attention signals. To counter this, Stage 1 employs contrastive pretraining with a novel "attended-speaker negative sampling" strategy. This forces the EEG encoder to align with specific speech segments within the same trial, thereby suppressing trial-level shortcuts. Stage 2 uses a confidence-weighted SI-SDR objective, where the weight is derived from the similarity between the frozen EEG embedding and the audio embeddings of the attended vs. ignored sources. This allows the extractor to handle ambiguous or contradictory guidance segments by weighting gradients accordingly. The approach is theoretically sound and addresses a critical flaw in current evaluation protocols for neuro-steered audio systems.
The authors conduct rigorous experiments on two public datasets, KUL and DTU, under strict cross-trial protocols. They demonstrate that standard end-to-end baselines (NeuroHeed, M3ANet) suffer significant performance drops when evaluated on unseen trials compared to within-trial evaluations, confirming the shortcut hypothesis. TRUST-TSE consistently outperforms these baselines in cross-trial selection accuracy and separation quality (SI-SDR). The paper includes extensive ablation studies validating the components: the specific negative sampling strategy, the confidence weighting mechanism, and the superiority of contrastive embeddings over envelope decoding. Stress tests (EEG shuffling, trial-wise permutation) further confirm that TRUST-TSE relies on meaningful EEG-audio alignment rather than shortcuts. The results are robust across different window lengths and show generalization to unseen subjects.
The paper provides detailed descriptions of the model architectures, training hyperparameters, and data preprocessing steps. The authors explicitly state that the source code is publicly available on GitHub, which significantly enhances reproducibility. The evaluation protocols are clearly defined, including the specific fold constructions to prevent data leakage. The inclusion of supplementary material with additional metrics (PESQ, STOI) and unseen-subject results adds to the transparency.
The primary limitation is the reliance on public datasets which, while standard, are relatively small in scale and diversity compared to large-scale consumer audio datasets. The performance gains, while statistically significant and methodologically important, are modest in absolute terms (e.g., ~15% accuracy gain on KUL). The method assumes a known-subject setting in the main experiments, although unseen-subject results are provided. The confidence weighting mechanism, while effective, introduces a dependency on the quality of the frozen EEG encoder; if Stage 1 fails to capture attention, Stage 2 may struggle.
This work has significant implications for the development of reliable neuro-steered hearing aids and brain-computer interfaces. By highlighting the fragility of current end-to-end models and providing a robust alternative, it pushes the field towards more rigorous evaluation standards and more reliable real-world deployment. It also contributes to the broader understanding of shortcut learning in multimodal representation learning, offering a template for ensuring that models learn task-relevant features rather than spurious correlations. This paper presents a critical analysis of shortcut learning in EEG-guided speech extraction and proposes a robust two-stage training framework (TRUST-TSE) that significantly improves cross-trial generalization, addressing a major reliability bottleneck in neuro-steered audio technologies.
Large Audio-Language Models (LALMs) have been widely used as judge models for the automatic evaluation of generated speech. However, prior approaches predominantly focus on holistic naturalness, leaving fine-grained paralinguistic distinctions underexplored. We introduce ParaPairAudioBench, a pairwise benchmark of 5,175 audio pairs across five paralinguistic dimensions: Style, Rate, Emphasis, Age, and Gender. Our experiments show that current LALM judges still lag behind human judgments by 32%p on average and exhibit severe calibration failures, particularly in Tie cases where the correct decision is to abstain. To further analyze lexical versus acoustic reliance, the benchmark includes both same-transcript and cross-transcript conditions. ParaPairAudioBench enables multi-dimensional, calibration-aware assessment of the reliability of LALM-as-a-Judge for paralinguistic speech evaluation.
Primary: Seoul National University
All Institutions: Hongik University, Seoul National University, NAVER Cloud, KAIST
ParaPairAudioBench provides a critical diagnostic framework for evaluating the reliability and calibration of LALM-as-a-Judge systems in paralinguistic speech tasks, revealing systematic biases and modality dependencies that are invisible to aggregate naturalness metrics.
The paper proposes a novel diagnostic benchmark, ParaPairAudioBench, designed to evaluate Large Audio-Language Models (LALMs) as judges for paralinguistic speech attributes. The methodology is rigorous in its design: it decomposes evaluation into five distinct dimensions (Style, Rate, Emphasis, Age, Gender) and employs a pairwise comparison format with a "Tie" option to assess calibration. A key methodological strength is the control of lexical content through same-transcript and cross-transcript conditions, allowing for an isolation of acoustic versus textual reliance. The use of position-swapping to quantify order bias is also a robust analytical technique. However, the novelty is somewhat limited by the fact that it is a benchmark/evaluation paper rather than a new model architecture or algorithm. It repurposes existing LALMs and public datasets (Expresso, Sonos, LibriTTS, EARS) into a new evaluation protocol. EXPERIMENTAL_EVALATION: The experimental setup is well-structured, evaluating five representative LALMs (Gemini 2.5 Flash, GPT-4o Audio, Kimi-Audio-7B, Qwen2.5-Omni-7B, SpeechJudge-7B) against human baselines. The results provide valuable insights: significant gaps in human-model alignment (32% average lag), severe calibration failures in Tie cases (models forcing preferences), and asymmetric modality dependence (over-reliance on text for Style, better acoustic reliance for Emphasis in cross-transcript settings). The inclusion of human evaluation on a subset with inter-rater reliability metrics adds credibility. The analysis of position bias (up to 29.4% gap) is a critical finding that highlights a systematic flaw in current judge pipelines. The data presentation is clear, though the paper relies on tables that are not fully rendered in the text provided, the described trends are specific and actionable.
The paper provides a GitHub repository URL for data and code, which is a strong positive for reproducibility. The dataset construction methodology is described in detail, including the sources of public corpora and the constraints used for pair selection (label constraints, transcript control). The evaluation protocol is clearly defined, including prompt templates (though not explicitly listed, the structure is described) and decoding settings (greedy vs. majority voting). The use of standard public datasets ensures that other researchers can replicate the data generation process.
The primary limitation is the scope of the benchmark. It covers only five paralinguistic dimensions and uses a limited number of public datasets, which may not capture the full diversity of speech styles and accents. The Tie condition construction is acknowledged as difficult, particularly for Emphasis, leading to its exclusion in some cases. The human evaluation is limited to a 250-item subset, which may not be fully representative of the entire benchmark's difficulty distribution. Furthermore, the study focuses on existing LALMs; it does not propose a new model or fine-tuning strategy to address the identified failures, limiting its immediate technical impact on model development.
This paper has significant broader impact for the field of audio-language model evaluation. By exposing the systematic weaknesses of current LALM judges in paralinguistic tasks, it provides a necessary diagnostic tool for the community. The findings on calibration failures and position bias are critical for the development of reliable automated evaluation systems, which are increasingly used in training generative models (RLHF). The benchmark encourages the community to move beyond holistic naturalness scores to more fine-grained, robust evaluation metrics. It also highlights the need for better acoustic understanding in LALMs, particularly for localized prosodic features. ParaPairAudioBench provides a critical diagnostic framework for evaluating the reliability and calibration of LALM-as-a-Judge systems in paralinguistic speech tasks, revealing systematic biases and modality dependencies that are invisible to aggregate naturalness metrics.
Aligner-Encoders are recently proposed seq2seq end-to-end ASR models that replace decoder attention by predicting the uth token directly from the u-th encoder position, so the encoder must learn the alignment internally without cross-attention or a transducer lattice. In practice, this alignment often forms abruptly in the upper layers, making training sensitive and brittle on long utterances. We propose InterAligner, which adds an intermediate Aligner objective so alignment can form progressively across depth, together with an intermediate CTC loss (InterCTC) to stabilize optimization. On LibriSpeech with a 17-layer Conformer, a final-only Aligner reaches 5.0/7.8 WER (test-clean/other). InterCTC improves to 3.4/6.0, and InterAligner further reduces WER to 3.1/5.6 with the largest gains on long utterances.
Primary: NTT, Inc.
All Institutions: NTT, Inc.
The paper proposes InterAligner, an intermediate supervision method for Aligner-Encoders that progressively builds alignment across network depth, significantly improving robustness on long utterances in ASR tasks.
The paper addresses a specific and well-defined problem in Aligner-Encoder architectures: the brittleness of alignment formation in deep layers for long utterances. The proposed solution, InterAligner, introduces a hierarchical supervision strategy. By attaching an intermediate Aligner loss at an intermediate layer (layer 15) using a finer-grained tokenization (smaller vocabulary size) and an intermediate CTC loss (InterCTC) at an earlier layer (layer 12), the authors aim to create a "curriculum" for alignment. This approach is technically sound and leverages established concepts of intermediate supervision (common in deep learning) and multi-granularity learning. The novelty lies in the specific application to the structural constraints of Aligner-Encoders, where the one-to-one mapping requires careful management of sequence lengths and token granularities. The method is relatively simple to implement, adding auxiliary heads and losses without altering the core encoder architecture significantly.
The experimental evaluation is robust and comprehensive. The authors use standard benchmarks (LibriSpeech and Common Voice English) and a strong baseline (17-layer Conformer Aligner-Encoder). The results show consistent improvements: InterCTC provides a significant boost, and InterAligner provides further gains, particularly on long utterances (>21s), which validates the core hypothesis. The ablation studies are thorough, investigating the impact of vocabulary size, loss weights, and layer placement. The statistical significance testing adds credibility. The attention visualization provides qualitative support for the progressive alignment hypothesis. However, the gains, while consistent, are moderate in absolute terms (e.g., 3.1 vs 5.0 WER on test-clean for the final comparison, but note the baseline reproduction difficulty mentioned). The comparison is primarily internal (ablation), with limited comparison to other state-of-the-art ASR systems like RNN-T or standard AEDs, though the paper claims competitiveness.
The paper provides sufficient detail for reproduction. The architecture (Conformer-L), training hyperparameters (learning rate, warmup, batch size), and dataset details are clearly stated. The specific layer indices for intermediate losses (12 and 15) and the tokenization sizes (256 vs 1024) are provided. The use of model averaging is standard. The code is not explicitly linked, but the methodology is described with enough precision that implementation should be feasible for researchers in the field.
The primary limitation is the incremental nature of the contribution. It improves an existing architecture but does not propose a fundamentally new paradigm. The gains are most pronounced on long utterances, suggesting limited utility for short-form speech. The method adds computational overhead during training due to the auxiliary heads and losses, though inference remains unchanged (using only the final head). The paper acknowledges the difficulty in reproducing the baseline Aligner-Encoder results, which might make the absolute WER numbers less comparable to other works if the baseline was under-optimized.
This work contributes to the field of Automatic Speech Recognition by making Aligner-Encoders more robust and practical, especially for long-form audio. This can benefit applications requiring low-latency or lightweight decoding where Aligner-Encoders are advantageous. The technique of progressive alignment supervision could potentially be applied to other sequence-to-sequence models with similar structural constraints. The paper proposes InterAligner, an intermediate supervision method for Aligner-Encoders that progressively builds alignment across network depth, significantly improving robustness on long utterances in ASR tasks.
Large audio-language models (LALMs) can reason about audio, yet it remains unclear whether they can perform comparative judgments between two speech signals along emotional, environmental, linguistic, prosodic, and interpersonal dimensions. We study this question in the context of speech emotion recognition (SER), where the model determines which utterance exhibits higher arousal, valence, or dominance. We introduce a reasoning-guided ordinal SER framework that conditions an LALM on paired speech inputs. The model is trained using reasoning traces generated from both semantic audio descriptions and acoustic evidence derived from GeMAPS features, enabling interpretable comparative decisions. Beyond direct supervision, we also employ direct preference optimization to encourage stronger separation for emotional differences. Experiments show that the proposed framework improves preference prediction while requiring only 5% of the training data used by conventional ordinal SER systems.
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University, The University of Texas at Dallas, NVIDIA
This paper presents a significant advancement in comparative audio reasoning by introducing a reasoning-guided ordinal SER framework that leverages both semantic and acoustic evidence, demonstrating superior data efficiency and cross-domain robustness compared to conventional SSL-based methods.
The paper proposes a novel framework for adapting Large Audio-Language Models (LALMs) to comparative speech emotion recognition (SER). The core innovation lies in the use of "reasoning-guided" training, where the model generates intermediate reasoning traces based on both semantic audio descriptions and discrete GeMAPS acoustic features before making a pairwise comparison. This approach integrates explicit acoustic evidence (GeMAPS) with high-level semantic reasoning, addressing a gap in current LALMs that often rely on implicit feature representations. The use of Direct Preference Optimization (DPO) with constructed correct and incorrect reasoning traces is a sophisticated application of preference learning to enhance both accuracy and interpretability. However, the reliance on a separate large reasoning model (Qwen3-Next-80B) to generate these traces introduces significant computational overhead and potential error propagation if the reasoning model hallucinates or misinterprets the acoustic features. The methodology is sound and theoretically well-motivated, bridging the gap between low-level acoustic analysis and high-level linguistic reasoning.
The experimental setup is robust, utilizing the MSP-Podcast v2.0 corpus as the primary dataset and evaluating cross-domain generalization on BIIC-Podcast and WHiSER. The comparison against strong self-supervised learning (SSL) baselines (WavLM, HuBERT with RankNet/RankList) is appropriate and highlights the data efficiency of the LALM approach. The results demonstrate that the proposed DPO-CoT method significantly outperforms baselines in preference accuracy, particularly in cross-domain settings. The ablation studies effectively isolate the contributions of SFT, DPO, and reasoning traces. The finding that reasoning traces improve performance specifically under DPO (but not SFT) is an interesting and valuable insight into how preference optimization interacts with chain-of-thought reasoning. The data efficiency claim (5% of training data) is compelling and well-supported by the results.
The paper provides sufficient detail regarding the model architectures (Qwen2.5-Omni-3B, Qwen3-Next-80B), feature extraction (GeMAPS), and training protocols (LoRA ranks, DPO parameters). The dataset partitions and preprocessing steps for GeMAPS features are described. However, the specific prompts used for the reasoning model and the exact criteria for "discretizing" GeMAPS features into qualitative levels (low/medium/high) could be more explicitly defined to ensure exact reproducibility. The reliance on a proprietary or specific version of the "Qwen3-Next" model might also pose reproducibility challenges if the weights or specific inference configurations are not publicly available.
A primary limitation is the computational cost and latency introduced by generating reasoning traces using a large 80B parameter model. This makes the approach less suitable for real-time applications compared to direct SSL-based ranking. The quality of the reasoning traces is dependent on the capability of the reasoning model; if the reasoning model fails to correctly interpret the GeMAPS features or the audio description, the subsequent DPO training might reinforce incorrect reasoning patterns, although the paper attempts to mitigate this by regenerating traces. Furthermore, the evaluation is limited to emotional attributes; while the authors suggest generalizability, the specific mechanisms for handling non-emotional comparative tasks (e.g., speaker identity, environmental noise) are not empirically validated in this work.
This work contributes to the development of more interpretable and robust audio understanding systems. By enabling LALMs to perform comparative reasoning with explicit justifications, it enhances trust and diagnostic capabilities in applications such as mental health monitoring, human-computer interaction, and audio content moderation. The approach of grounding LLM reasoning in structured acoustic features could inspire similar frameworks for other audio domains, such as music analysis or environmental sound classification. However, the potential for bias in the reasoning traces, if the underlying models or GeMAPS feature interpretations are biased, remains a concern that should be addressed in future deployments. This paper presents a significant advancement in comparative audio reasoning by introducing a reasoning-guided ordinal SER framework that leverages both semantic and acoustic evidence, demonstrating superior data efficiency and cross-domain robustness compared to conventional SSL-based methods.
Existing Reinforcement Learning (RL) research for Text-to-Speech (TTS) focuses on large language models (LLMs), leaving Flow-Matching (FM) under-explored. We present FlowTTS-GRPO, an online RL framework for FM-based TTS. By converting ordinary differential equation (ODE) trajectories into stochastic differential equation (SDE) paths, our method enables direct fine-tuning of open-source FM models without auxiliary models. We show that a weighted reward combination converges faster than a probabilistic scheme, and identify three practical optimizations: omitting classifier-free guidance (CFG) during training accelerates convergence; synthesizing hard cases improves robustness; and applying RL to the FM component enhances audio-detail metrics. Experiments on CosyVoice 3.0 and F5-TTS demonstrate objective and subjective preference gains in speaker similarity and perceptual quality, with F5-TTS also improving intelligibility.
Primary: Alibaba Group
All Institutions: Alibaba Group, Tongyi Lab
[One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper successfully adapts Group Relative Policy Optimization (GRPO) to Flow-Matching based TTS by introducing stochasticity via SDE conversion, demonstrating that online RL can significantly enhance speaker similarity and perceptual quality in both hybrid and pure FM architectures without auxiliary models.
The paper proposes FlowTTS-GRPO, an online reinforcement learning framework tailored for Flow-Matching (FM) based Text-to-Speech (TTS) models. The core technical novelty lies in adapting the Group Relative Policy Optimization (GRPO) algorithm, originally designed for Large Language Models (LLMs), to continuous diffusion/flow models. This is achieved by converting the deterministic Ordinary Differential Equation (ODE) sampling trajectory into a Stochastic Differential Equation (SDE) path, thereby introducing the necessary stochasticity for policy gradient estimation. The authors formulate the FM decoding process as a Markov Decision Process (MDP) where actions are velocity predictions. They employ a multi-objective reward structure combining speaker similarity (SS), ASR-based intelligibility (CER/WER), and perceptual quality (DNSMOS). A key methodological contribution is the analysis of reward fusion strategies, demonstrating that weighted combination with standard deviation normalization converges faster and more stably than probabilistic assignment. Additionally, they identify practical optimizations such as omitting Classifier-Free Guidance (CFG) during training to enhance exploration and using hard-case synthesis to improve robustness. The approach is model-agnostic regarding the FM backbone, successfully applied to both LLM-FM hybrid (CosyVoice 3.0) and pure FM (F5-TTS) architectures.
The experimental evaluation is comprehensive, covering two major open-source TTS systems (CosyVoice 3.0 and F5-TTS) and multiple languages (Chinese, English, and several European languages). The authors utilize the Seed-TTS-Eval benchmark, reporting improvements in speaker similarity (surpassing closed-source baselines like Seed-TTS on SS1 for Chinese), perceptual quality (DNSMOS), and intelligibility (WER/CER). The inclusion of subjective A/B preference tests strengthens the claims by correlating objective metrics with human judgment. Ablation studies effectively isolate the impact of reward combination strategies, CFG omission, and hard-case training. The results demonstrate that RL on the FM component primarily enhances acoustic details and timbre, while RL on the LLM component (as seen in comparative baselines) is more critical for semantic alignment, providing valuable architectural insights.
The paper provides sufficient detail for reproduction, including the MDP formulation, SDE conversion equations, reward definitions, and training hyperparameters (LoRA ranks, noise levels, window sizes). The use of widely available models (CosyVoice 3.0, F5-TTS, Whisper, Paraformer) and datasets (WenetSpeech4TTS, LibriTTS) facilitates replication. However, the specific implementation details of the SDE windowing and the exact weighting coefficients for the multi-objective reward are provided, though the code is not publicly linked in the text. The distinction between training and inference CFG usage is clearly explained.
The primary limitation is the computational cost associated with online RL, requiring multiple rollouts per prompt and significant GPU resources (8 GPUs mentioned). The method relies on proxy rewards (DNSMOS, ASR, SS embeddings) which may not perfectly align with all aspects of human perception, although subjective tests mitigate this concern. The improvement in intelligibility for the LLM-FM hybrid (CosyVoice) is limited because the semantic content is determined by the frozen LLM front-end; RL on the FM can only refine acoustic realization, not correct semantic errors. Furthermore, the "hard case" synthesis strategy, while effective, relies on heuristic augmentations that may not cover all edge cases in natural speech.
This work significantly advances the field of generative audio by bridging the gap between discrete token-based RL (used in LLMs) and continuous flow-based generation. It enables the fine-tuning of high-quality, open-source FM models without the need for complex auxiliary reward models or value networks, democratizing access to advanced RL techniques in TTS. The findings on reward conflicts and optimization strategies provide generalizable insights for other continuous generative tasks. The potential for improved voice cloning and natural speech synthesis has broad applications in accessibility, entertainment, and human-computer interaction, though it also raises concerns regarding voice impersonation and deepfakes. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper successfully adapts Group Relative Policy Optimization (GRPO) to Flow-Matching based TTS by introducing stochasticity via SDE conversion, demonstrating that online RL can significantly enhance speaker similarity and perceptual quality in both hybrid and pure FM architectures without auxiliary models.
Speech Language Models achieve reasoning capabilities, but are often hindered by massive parameter counts and a tendency to prioritize linguistic priors over acoustic features. While contrastive decoding enhances grounding by contrasting audio-aware and text-only logits, it increases inference latency. We propose Contrastive Audio-Aware Distillation (CAAD), a framework that internalizes the teacher's contrastive reasoning into the student model's weights. To overcome the high computational training overhead in the dual-path token-by-token contrastive distillation process, we introduce a synchronized teacher-forcing strategy. Anchored by unified Pseudo-Ground Truths, this mechanism enables simultaneous full-sequence generation of the teacher's contrastive distributions, allowing student to distill the audio-aware signal efficiently. Overall, CAAD yields a ~8% relative gain over standard knowledge distillation on Dynamic-SUPERB and successfully reduces linguistic bias in MCR-BENCH.
Primary: National Taiwan University
All Institutions: Graduate Institute of Electrical Engineering, National Taiwan University, Graduate Institute of Communication Engineering, National Taiwan University, NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), National Taiwan University
The paper presents a novel and effective method for distilling contrastive decoding into Speech Language Models, addressing critical efficiency and bias challenges in multimodal AI through a synchronized teacher-forcing strategy anchored by metadata.
The paper proposes Contrastive Audio-Aware Distillation (CAAD), a method to compress Speech Language Models (SLMs) by distilling the benefits of Contrastive Decoding (CD) into a student model's weights. The core innovation is a "synchronized teacher-forcing strategy" that uses a "Pseudo-Ground Truth" (Pseudo-GT) generated from text metadata to anchor both the audio-aware (positive) and text-only (negative) teacher passes. This allows for parallel training, avoiding the sequential bottleneck of standard autoregressive contrastive decoding. The approach effectively transforms a test-time inference technique (CD) into a training-time objective. While the concept of distilling decoding strategies is not entirely new, the specific mechanism of using metadata-anchored pseudo-GT to enable parallel contrastive distillation in SLMs is a novel and practical engineering contribution. It addresses a genuine computational bottleneck in applying CD to large models.
The experimental evaluation is robust, utilizing the Dynamic-SUPERB benchmark and the MCR-BENCH for conflict resolution. The results demonstrate that the CAAD-distilled 3B student model significantly outperforms standard KD and even the greedy decoding baseline of the 8B teacher on several metrics, particularly in paralinguistic tasks (PAR) and conflict resolution (MCR-BENCH Shift). The ablation studies effectively validate the components of the method, showing that metadata-based Pseudo-GT outperforms audio-based synchronization and that the contrastive weight is crucial for mitigating linguistic bias. The comparison against Contrastive Decoding at inference time highlights the efficiency gain (single-path vs. dual-path) while acknowledging the performance gap, which is a fair and honest assessment.
The paper provides sufficient detail regarding the model architectures (Llama-3.2-8B teacher, Llama-3.2-3B student), training configurations (learning rate, optimizer, loss weights), and datasets (DeSTA2, Dynamic-SUPERB, MCR-BENCH). The code repository is linked, which significantly aids reproducibility. The description of the Pseudo-GT generation process is clear enough for replication.
The primary limitation is the dependency on the quality of the Pseudo-GT. If the metadata-derived text is inaccurate or lacks nuance, the distillation signal may be noisy. Additionally, the method assumes that the teacher model's contrastive reasoning is transferable via KL divergence, which may not capture all nuances of the teacher's decision boundary. The performance of the student, while improved, still lags behind the teacher's contrastive decoding performance, indicating that some information is lost in the distillation process. The paper also notes that the efficacy is bounded by the student's capacity.
This work contributes to the democratization of SLMs by enabling efficient, low-latency inference without sacrificing the robustness offered by contrastive methods. By mitigating linguistic bias, it promotes more reliable multimodal AI systems, which is crucial for applications in accessibility, customer service, and interactive agents where audio cues are critical. The method is generalizable to other multimodal LLMs beyond speech. The paper presents a novel and effective method for distilling contrastive decoding into Speech Language Models, addressing critical efficiency and bias challenges in multimodal AI through a synchronized teacher-forcing strategy anchored by metadata.
Continuous Variational Autoencoders (VAEs) serve as the fundamental continuous tokenizer for modern neural audio generation systems, enabling high-fidelity reconstruction while providing a compact, smooth latent space for downstream generative priors. However, continuous VAEs face a fundamental conflict among compression rate, reconstruction fidelity, and latent space topology, which we formalize as the Rate-Distortion-Regularity Trilemma. This trilemma stems from a topological mismatch: the isotropic Gaussian prior in standard VAEs imposes a flat latent geometry that fails to accommodate audio's hierarchical nature, where low-frequency components are structured and compressible while high-frequency components are stochastic and incompressible, leading to disordered information packing in which crucial semantic features are interleaved with high-entropy noise. To address this challenge, we propose Structured Topology-Aware Regularization (STAR), a general training strategy that reshapes latent space geometry by imposing a growth-based constraint field, routing structural and textural information into channel subspaces with matching capacities. STAR is applicable to any VAE architecture and effectively resolves the trilemma, as demonstrated in CNN-based VAEs. We further present STAR-VAE, which combines STAR with a hybrid CNN-Mamba architecture for local feature extraction and linear-complexity global context modeling, and STAR-Gen, an LLM-based Flow Matching framework that leverages STAR-VAE's structured latent space for high-fidelity generation without vector quantization artifacts. Experiments across diverse audio domains show that STAR-VAE achieves state-of-the-art reconstruction fidelity and enhanced semantic information preservation, while the structured latent space improves both traditional diffusion models and STAR-Gen for text-to-audio generation.
Primary: Hong Kong University of Science and Technology
All Institutions: Hong Kong University of Science and Technology, Tongyi Fun Team, Alibaba Group
The paper presents a significant advancement in continuous audio tokenization by introducing Structured Topology-Aware Regularization (STAR), which effectively resolves the Rate-Distortion-Regularity Trilemma in VAEs through a theoretically motivated capacity gradient, leading to state-of-the-art performance in both audio reconstruction and LLM-based flow matching generation.
The paper proposes Structured Topology-Aware Regularization (STAR), a novel regularization strategy for Variational Autoencoders (VAEs) that replaces the standard isotropic Gaussian prior with a channel-wise structured constraint field. The core theoretical contribution is the formalization of the "Rate-Distortion-Regularity Trilemma," arguing that isotropic priors cause "disordered information packing" in audio VAEs. STAR addresses this by imposing a Gamma-Growth function on the KL divergence weights, creating a "capacity gradient" that aligns latent channel capacity with the spectral hierarchy of audio (low-frequency structure vs. high-frequency texture). The authors combine this with a hybrid CNN-Mamba architecture (STAR-VAE) for efficient global context modeling and introduce STAR-Gen, an LLM-based Flow Matching framework for text-to-audio generation. The methodology is well-motivated, theoretically grounded in information theory (power-law decay), and technically sound, offering a generalizable solution to a known problem in continuous tokenization.
The experimental evaluation is comprehensive and robust. The authors provide extensive ablation studies validating the STAR regularization, including comparisons of different growth functions (Step, Linear, Gamma) and hyperparameters. They demonstrate that STAR-VAE achieves state-of-the-art reconstruction fidelity on AudioCaps and Song Describer datasets, outperforming strong baselines like Stable Audio Open and HiFi-VAE. Crucially, they validate the "Reconstruction Drift" phenomenon in high-capacity encoders without STAR, reinforcing their theoretical claims. For generation, STAR-Gen achieves SOTA performance on text-to-audio tasks, significantly outperforming diffusion-based baselines in perceptual quality (FD_openl3) and semantic alignment (CLAP). The inclusion of human evaluation (MOS) and linear probing for semantic information adds significant weight to the empirical claims.
The paper provides detailed implementation specifications, including dataset preprocessing steps, architectural details (ResNet blocks, Mamba dimensions, normalization strategies), and training configurations (loss weights, optimizer settings, hardware). The two-stage training strategy (pre-training with isotropic KL, fine-tuning with STAR) is clearly described. The project page URL is provided, suggesting code or demos may be available, though the GitHub link is not explicitly in the text. The level of detail is sufficient for reproduction by other researchers in the field.
The paper focuses primarily on audio generation and reconstruction. While STAR is claimed to be architecture-agnostic, the empirical validation is limited to audio domains. The integration of Mamba introduces linear complexity but may still face challenges with extremely long sequences compared to sparse attention mechanisms, though this is mitigated by the VAE compression. The STAR-Gen model relies on a large LLM backbone (Qwen3), which may limit its deployment on resource-constrained devices compared to smaller diffusion models. Additionally, the "Gamma-Growth" parameter requires tuning, although the paper suggests a default value.
This work advances the field of neural audio generation by providing a more robust and semantically rich continuous tokenization method. By resolving the trilemma between compression, fidelity, and regularity, STAR-VAE enables higher-quality audio synthesis with fewer artifacts. The integration with LLM-based Flow Matching opens new avenues for scalable and controllable audio generation. The potential applications in creative content production, sound design, and music composition are significant, though the authors appropriately note the risks associated with high-fidelity generative audio, such as misinformation and intellectual property concerns. The paper presents a significant advancement in continuous audio tokenization by introducing Structured Topology-Aware Regularization (STAR), which effectively resolves the Rate-Distortion-Regularity Trilemma in VAEs through a theoretically motivated capacity gradient, leading to state-of-the-art performance in both audio reconstruction and LLM-based flow matching generation.
Generative music systems can now produce impressive audio from text prompts, but audio outputs are difficult to inspect, edit, and diagnose as musical structure. We introduce Libretto, an agent-facing framework for symbolic music generation and revision. Libretto uses an LLM-native grammar with explicit onset slots, voices, and bar-level organization, then evaluates each piece in a corpus-calibrated statistical space over rhythm, harmony, melody, texture, form, and variation. The same structural axes support retrieval, diagnosis, copy-risk control, and iterative self-revision. Across gap filling, reference-guided full-piece generation, gradual morphing, and educational music generation, Libretto turns symbolic music from a raw token sequence into a measurable and editable object for language-model agents.
Primary: University of California, Berkeley
All Institutions: University of California, Berkeley
Libretto presents a structured, agent-centric framework for symbolic music generation that leverages corpus-calibrated structural metrics to enable interpretable diagnosis and iterative self-revision, offering a significant methodological contribution to controllable AI music composition despite lacking perceptual validation.
The paper introduces "Libretto," a framework designed to bridge the gap between generative audio models and symbolic music representation by creating an LLM-native grammar for symbolic music. The core methodological contribution is not a new neural architecture for generation, but rather a structured representation system and an evaluation loop. The grammar explicitly defines onset slots, voices, and bar-level organization, making the symbolic output directly editable and interpretable by an LLM agent. The evaluation mechanism relies on a "corpus-calibrated statistical cloud" comprising 29 structural axes (rhythm, harmony, melody, texture, form, variation) computed from the symbolic representation. These axes are mapped to percentiles against a reference corpus, allowing the agent to diagnose structural deviations (e.g., "too sparse," "harmonically unstable") rather than relying on black-box aesthetic scores. The agent loop involves generation, measurement against these axes, and iterative self-revision based on musician-readable feedback. This approach shifts the focus from end-to-end differentiable generation to a retrieval-augmented, self-correcting agentic workflow.
The authors evaluate the framework across four tasks: gap filling, reference-guided full-piece generation, gradual morphing, and educational music generation. They use a corpus of 314 MIDI files from the Lakh MIDI Dataset. The experiments demonstrate that the structural axes can distinguish between genres (e.g., Jazz vs. Folk on harmonic complexity) and that the agent loop improves pass rates for structural validity (e.g., gap-filling pass rate increased from 12% to significantly higher levels with the loop). The paper provides qualitative examples and quantitative metrics for copy-risk and structural degeneracy. However, the evaluation is largely internal and self-referential; it measures how well the generated pieces fit the *defined* structural axes and avoid copying, rather than assessing musical quality via human listening tests or comparison to state-of-the-art audio generation models (like Suno or Udio) in terms of perceptual quality. The dataset size (314 songs) is small for corpus calibration, though sufficient for the descriptive statistics used.
The paper provides a clear description of the grammar, the 29 axes, and the evaluation gates. The code and project website are linked, which enhances reproducibility. The reliance on a specific LLM (Claude Code with Opus 4.8) for the agent loop is noted, which allows other researchers to replicate the agentic behavior, though the specific prompts and retrieval mechanisms would need to be carefully reconstructed from the text and code. The definition of the structural axes is mathematically precise in the appendix.
A significant limitation is the lack of perceptual evaluation. The system optimizes for structural properties defined by the authors, but there is no evidence that these properties correlate with human judgments of musical quality or "goodness." The small reference corpus (314 songs) limits the generalizability of the statistical cloud, potentially biasing the "idiomatic" norms towards the specific genres present in that small set. The abstraction of the grammar (ignoring velocity, timbre, micro-timing) means it cannot capture expressive performance nuances, limiting its applicability to purely structural composition tasks. Furthermore, the reliance on a proprietary LLM (Claude) for the agent loop raises questions about the accessibility and cost of the method for broader research communities.
Libretto offers a novel perspective on AI-assisted music creation by treating symbolic music as a structured, editable object for LLM agents. This could empower musicians and educators by providing tools for targeted theory practice, gap-filling, and style exploration. It contributes to the broader field of AI creativity by demonstrating how structured representations can enhance the controllability and interpretability of generative models. However, the potential for generating high-quality, commercially viable music is limited by the lack of audio fidelity and expressive nuance in the current symbolic representation. Libretto presents a structured, agent-centric framework for symbolic music generation that leverages corpus-calibrated structural metrics to enable interpretable diagnosis and iterative self-revision, offering a significant methodological contribution to controllable AI music composition despite lacking perceptual validation.
Diffusion models show potential for speech enhancement but lack linguistic guidance. We condition a diffusion-based model on wav2vec 2.0 features from noisy input, injected at the U-Net bottleneck via Feature-wise Linear Modulation (FiLM). Phonetic representations from wav2vec 2.0 features of degraded speech, anchor the reverse diffusion process. While a frozen wav2vec 2.0 encoder extracts features, a learned FiLM generator produces scale and shift parameters modulating the bottleneck with minimal overhead. Motivated by the optimal Bayesian causal estimator under a linear-Gaussian state-space model, FiLM coefficients are aggregated via exponential smoothing for temporal compression. Evaluation on VoiceBank-DEMAND and LibriMix shows competitive performance against the unconditioned baseline in PESQ, STOI, SI-SDR and DNSMOS. We consistently record an improvement of 0.4 on PESQ score, suggesting self-supervised representations effectively condition diffusion-based speech enhancement.
Primary: University of Maryland
All Institutions: University of Maryland
This paper presents a theoretically motivated and empirically effective method for conditioning diffusion-based speech enhancement with self-supervised features, offering a compelling alternative to standard conditioning strategies despite a noted trade-off in source separation metrics.
The paper proposes a novel conditioning mechanism for diffusion-based speech enhancement by integrating self-supervised learning (SSL) features from wav2vec 2.0. The core technical contribution lies in the injection of these phonetic representations into the U-Net bottleneck via Feature-wise Linear Modulation (FiLM). A significant methodological strength is the theoretical derivation of the temporal aggregation strategy for the FiLM coefficients. By modeling the phonetic state as a random walk and the projected coefficients as noisy observations, the authors derive that the optimal causal estimator is a Kalman filter, which simplifies to exponential moving average (EMA) at steady state. This provides a principled, theoretically grounded alternative to ad-hoc pooling methods (like mean pooling) for handling the temporal mismatch between frame-level SSL features and the global context required at the diffusion bottleneck. The choice to apply FiLM only at the bottleneck, supported by ablation studies, demonstrates a nuanced understanding of feature abstraction levels in U-Nets.
The experimental evaluation is conducted on two standard benchmarks: VoiceBank-DEMAND and LibriMix. The results show consistent improvements in perceptual metrics (PESQ, STOI, DNSMOS) compared to the unconditioned StoRM baseline, with a notable 0.4 improvement in PESQ on VB-DEMAND. However, the paper acknowledges a trade-off: a degradation in SI-SDR, attributed to aggressive noise suppression. This is a critical observation; while perceptual quality improves, the objective source separation metric suffers, suggesting the model may be over-smoothing or removing non-speech components that contribute to the separation score. The evaluation includes ablations on the smoothing coefficient and conditioning location, which strengthen the claims. However, the lack of subjective listening tests (MOS) limits the validation of the perceptual gains claimed by the DNSMOS and PESQ improvements. The comparison is primarily against StoRM and a few other diffusion baselines, which is appropriate but could be broader to include recent SSL-enhanced discriminative models.
The paper provides sufficient detail regarding the model architecture (U-Net configuration, wav2vec 2.0 base model), training hyperparameters (learning rate, epochs, optimizer), and datasets. The theoretical derivation of the EMA smoothing is clearly explained. However, the code is not publicly linked in the text provided, and specific details on the noise types and mixing conditions for LibriMix are somewhat generic ("min" mixing mode). The claim of "minimal overhead" is supported by FLOPs analysis, but exact inference latency comparisons would be more useful for reproducibility in real-time applications.
The primary limitation is the trade-off between perceptual quality and source separation fidelity (SI-SDR). The model improves PESQ but reduces SI-SDR, which may be undesirable for applications requiring strict source isolation. Additionally, the reliance on wav2vec 2.0, while effective, ties the method to a specific SSL model; the authors mention future work with WavLM/HuBERT, but the current work does not explore the sensitivity to the choice of SSL encoder. The theoretical derivation assumes a linear-Gaussian state-space model, which is a simplification of the complex, non-linear dynamics of speech and noise. Finally, the evaluation lacks subjective human listening tests, which are the gold standard for speech enhancement quality.
This work contributes to the field of audio processing by bridging self-supervised learning and generative modeling. It demonstrates that linguistic/phonetic information can effectively guide diffusion processes, potentially leading to more robust speech enhancement systems that preserve speech content even in low-SNR conditions. This has implications for telecommunications, hearing aids, and speech recognition preprocessing. The theoretical connection between Kalman filtering and exponential smoothing for feature aggregation is a generalizable insight that could apply to other temporal sequence modeling tasks. This paper presents a theoretically motivated and empirically effective method for conditioning diffusion-based speech enhancement with self-supervised features, offering a compelling alternative to standard conditioning strategies despite a noted trade-off in source separation metrics.
We propose AugCodec, a low-bitrate disentangled neural speech codec that leverages data augmentation to decompose speech into three distinct components: semantic, speaker, and prosody tokens. Specifically, we employ tailored augmenta tion strategies to transform speech into distinct variants, each serving as input for extracting tokens that preserve the target attribute while suppressing others. This disentanglement strategy enables substantial reduction in token rate. Further more, we introduce an augmentation loss that aligns semantic encoder outputs between source and voice-converted speech, encouraging speaker-agnostic embeddings while mitigating the acoustic mismatch induced by voice conversion. Experiments on LibriSpeech test-clean demonstrate that AugCodec significantly outperforms state-of-the-art methods in both reconstruction quality and disentanglement, while operating at only 12.5Hz with three token streams.
Primary: Amazon
All Institutions: Amazon
AugCodec introduces a data-augmentation-driven disentanglement strategy for neural speech codecs, achieving state-of-the-art reconstruction and content preservation at low bitrates by isolating semantic, speaker, and prosody features through tailored input transformations and alignment losses.
The paper proposes AugCodec, a novel approach to disentangled neural speech coding that leverages data augmentation as a primary mechanism for feature separation. The core innovation lies in the input preprocessing strategies: using a voice-converted speech variant for the semantic encoder to enforce speaker invariance, a different utterance from the same speaker for the speaker encoder to isolate identity, and low-frequency STFT components for the prosody encoder to capture F0 and lower harmonics while discarding timbral content. This is complemented by an "augmentation loss" that aligns semantic embeddings between the original and voice-converted speech, explicitly penalizing speaker information leakage. The architecture employs separate streams for semantic, speaker, and prosody tokens, which are quantized independently (using VQ for semantic, FSQ for speaker/prosody) and fused via a learned expansion and modulation mechanism (FiLM) before decoding. The semantic encoder uses a learned compression/expansion scheme to reduce frame rate while preserving dynamics, addressing a key limitation of average pooling.
The authors evaluate AugCodec on LibriSpeech test-clean, comparing against strong baselines including BiCodec, Mimi, Qwen-TTS-Tokenizer-12Hz, and FACodec. The results demonstrate that AugCodec achieves superior reconstruction quality (PESQ, UTMOS) and significantly lower Word Error Rates (WER) compared to baselines at comparable or lower bitrates (e.g., 387.5 bps for AugCodec-2 vs 412.5 bps for Mimi). The disentanglement capability is validated through voice conversion experiments, where AugCodec shows better content preservation (lower WER) than FACodec and BiCodec. The ablation study confirms the importance of the augmentation loss, showing performance degradation when it is removed. The results are robust across different codebook sizes and frame rates.
The paper provides detailed architectural specifications, including layer dimensions, kernel sizes, and loss weights. It specifies the use of open-source components like wav2vec 2.0, ECAPA-TDNN, and Seed-VC. The training setup (optimizer, learning rate, batch size, iterations) is clearly described. However, the specific random seeds and exact data splitting procedures for the 3000 hours of LibriLight-medium data are not fully detailed, which might introduce minor variability. The use of off-the-shelf voice conversion models introduces a dependency that is well-documented but adds complexity to the training pipeline.
The reliance on an external voice conversion model (Seed-VC) for generating semantic training data is a potential limitation, as errors or biases in the VC model could propagate to the codec. The prosody encoder's reliance on low-frequency STFT components might struggle with complex harmonic structures or high-frequency prosodic cues, although the authors argue this minimizes correlation with semantic/speaker features. The method operates at a maximum frame rate of 12.5Hz for semantic tokens, which is low but may not be sufficient for all real-time applications requiring ultra-low latency without further optimization. The paper does not report results on out-of-domain datasets or diverse languages, limiting the generalizability assessment.
AugCodec contributes to the development of efficient, disentangled speech representations, which are foundational for speech language models, voice conversion, and text-to-speech systems. By enabling low-bitrate transmission with high quality and disentanglement, it facilitates more scalable and privacy-preserving speech applications. The emphasis on speaker invariance in semantic tokens has implications for reducing bias and protecting speaker identity in downstream tasks. AugCodec introduces a data-augmentation-driven disentanglement strategy for neural speech codecs, achieving state-of-the-art reconstruction and content preservation at low bitrates by isolating semantic, speaker, and prosody features through tailored input transformations and alignment losses.
Understanding speaker attributes is crucial for voice-related applications, yet conventional approaches rely on fixed categorical labels, lacking semantic richness and zero-shot generalizability. We propose a novel framework for open-set speaker attribute prediction leveraging Large Language Model (LLM) embeddings to represent attributes in a continuous semantic space. To bridge the cross-modal gap, we introduce a keyword-appending strategy that structures broad semantic representations into a compact, discriminative manifold. Furthermore, we employ a top-k negative loss to establish robust decision boundaries in crowded semantic regions. Experimental results on LibriTTS-P demonstrate that our method outperforms closed-set benchmarks and generalizes effectively to unseen synonyms. Geometric analysis suggests that our strategies regularize the embedding manifold, balancing semantic cohesion with predictive clarity.
Primary: Seoul National University
All Institutions: Seoul National University, Artificial Intelligence Institute, Interdisciplinary Program in Artificial Intelligence
This paper presents a well-executed extension of speaker attribute prediction into the open-set domain using LLM embeddings, demonstrating that structural regularization of the semantic manifold can significantly improve both performance and generalizability.
The paper proposes a novel framework for open-set speaker attribute prediction by leveraging LLM embeddings as continuous targets rather than discrete categorical labels. The core methodological contributions are the "keyword-appending strategy" to ground semantic ambiguity in a specific domain (e.g., appending "speech" to "cute") and a "top-k negative penalization" loss to manage semantic crowding in the embedding manifold. The approach effectively bridges the cross-modal gap between acoustic features (ECAPA-TDNN) and textual semantics (GPT-OSS-20B). The geometric analysis providing evidence for manifold regularization is a strong theoretical component. However, the novelty is slightly tempered by the fact that using LLM embeddings for zero-shot/open-set classification is an established paradigm (e.g., CLIP-style alignment), and the specific application here is a logical extension rather than a fundamental algorithmic breakthrough. The "apple" keyword finding is intriguing but suggests the mechanism might be more about structural regularization than semantic grounding, which is an interesting insight but limits the semantic interpretability claim.
The experimental evaluation is robust within the scope of the dataset. The use of LibriTTS-P is appropriate as it is the standard for this specific task. The comparison against the closed-set baseline (Vove) demonstrates clear performance gains in both closed-set F1 scores and zero-shot synonym generalization. The inclusion of a geometric analysis (Center Sim, Total Variance, PCA Log-determinant) adds significant value, providing empirical backing for the claim that the proposed methods structure the embedding space effectively. The ablation studies on different keywords (speech, voice, face, man, apple) are particularly valuable, revealing that the benefit of keyword appending is not strictly semantic but structural. The results are statistically sound and well-presented.
The paper provides sufficient implementation details for reproduction. It specifies the backbone (ECAPA-TDNN), the LLM used (GPT-OSS-20B), the loss functions, hyperparameters (margin, k, weights), and the dataset splits. The reference to the baseline code (https://github.com/jaejunL/vove) is helpful. One minor ambiguity is the exact version or specific prompt used to generate the GPT-OSS-20B embeddings, though this is a minor detail. The weight assignment for intensity levels (1.5, 1.0, 0.5) is clearly defined.
The primary limitation is the reliance on a single dataset (LibriTTS-P), which restricts the generalizability of the findings to other speaker attribute corpora or styles. The authors acknowledge this. Additionally, the use of a proprietary LLM (GPT-OSS-20B, described as "open-weights from OpenAI" which is a contradiction in terms or a specific internal model) might limit reproducibility for others without access to that specific model. The "apple" keyword result, while insightful, highlights that the model might be sensitive to arbitrary structural shifts in the embedding space rather than pure semantic meaning, which could be a vulnerability in noisy real-world applications.
This work contributes to the broader field of interpretable AI in speech processing. By moving from black-box embeddings to semantically rich, explainable attributes, it enables more controllable and transparent voice technologies (e.g., TTS, voice conversion). The open-set capability allows for more flexible user interfaces where users can describe speakers in natural language. The geometric insights into embedding manifolds are also relevant to broader multimodal learning research. This paper presents a well-executed extension of speaker attribute prediction into the open-set domain using LLM embeddings, demonstrating that structural regularization of the semantic manifold can significantly improve both performance and generalizability.
Neural speech codecs efficiently compress speech and have become a foundation for speech generation, but they are typically learned as holistic representations that intertwine linguistic content, speaker identity, and prosody. While this design is effective for zero-shot voice cloning, it hinders downstream tasks that require prosody preservation or transfer, such as voice conversion. To address this, we introduce ProsoCodec, a prosody-oriented speech codec that models prosody as a conditional residual rather than as a disentangled stream. Specifically, by conditioning both the encoder and decoder on text and speaker embeddings as prefix tokens, the discrete bottleneck is encouraged to capture prosodic variation not explained by content and speaker. To further preserve prosody, we use the low-frequency mel band and train the model on paired same-speaker utterances. Experiments on voice conversion show improved prosody preservation and reduced source-timbre leakage.
Primary: Chung-Ang University
All Institutions: KAIST, Chung-Ang University, The Chinese University of Hong Kong
ProsoCodec presents a novel and effective approach to voice conversion by treating prosody as a conditional residual in a discrete speech codec, achieving state-of-the-art performance in prosody preservation and content fidelity through clever architectural conditioning and training strategies.
The paper proposes ProsoCodec, a neural speech codec specifically designed for voice conversion (VC) by modeling prosody as a conditional residual. The core innovation lies in the architectural conditioning: by providing explicit text and speaker embeddings as prefix tokens to both the encoder and decoder, the discrete bottleneck is forced to encode only the "residual" information, which the authors argue corresponds to prosody. This is a clever application of residual learning in the latent space. The use of Binary Spherical Quantization (BSQ) provides a compact discrete representation. The decoder utilizes a Diffusion Transformer (DiT) with flow-matching, which is a modern and effective approach for high-fidelity waveform/spectrogram reconstruction. The "dual-utterance" training strategy is a pragmatic and effective heuristic to prevent prompt-style leakage, where the model might otherwise copy the reference speaker's prosody instead of the source's. The methodology is sound, well-motivated, and builds logically on recent trends in discrete speech representation and diffusion-based generation.
The experimental setup is comprehensive, comparing ProsoCodec against several strong baselines including DDDM-VC, UniAudio, HierSpeech++, FACodec, Seed-VC, and Vevo. The evaluation covers both objective metrics (WER, Speaker Similarity, RMSE of F0, UTMOS) and subjective metrics (MOS for similarity, prosody, and naturalness). The results show that ProsoCodec achieves state-of-the-art performance in terms of content preservation (lowest WER) and prosody preservation (lowest F0 RMSE), while maintaining competitive speaker similarity and naturalness. The ablation studies are particularly valuable, demonstrating the individual contributions of the low-frequency mel input, the dual-utterance training, and the explicit conditioning. The analysis of the trade-off between bitrate and timbre leakage adds depth to the understanding of discrete codecs for VC.
The paper provides detailed implementation information, including dataset (LibriTTS, VCTK), model architecture (Transformer encoder, DiT decoder), hyperparameters (learning rate, batch size, optimizer), and pre-trained models used for ASR and speaker embedding (Qwen3-ASR, CAM++). The use of standard datasets and widely available pre-trained models enhances reproducibility. However, the specific initialization from "TaDiCodec" and the exact details of the binary spherical quantization implementation could benefit from more precise citation or code availability, though the description is generally sufficient for a competent researcher to replicate.
The paper acknowledges that strong reconstruction does not always imply strong conversion, a known issue in discrete speech models. The reliance on external ASR and speaker verification models introduces potential error propagation; if the ASR transcript is incorrect, the text conditioning will be wrong, potentially affecting prosody encoding. The dual-utterance training strategy, while effective, requires paired data from the same speaker, which might limit its applicability in scenarios where such pairs are scarce, although LibriTTS provides ample such data. The focus on prosody preservation might come at the cost of some fine-grained spectral details, as evidenced by the use of low-frequency mel bands, though the diffusion decoder helps mitigate this.
ProsoCodec contributes to the field of speech processing by offering a robust solution for voice conversion, a task with significant applications in entertainment, accessibility, and privacy. By decoupling prosody from timbre and content more effectively than previous holistic codecs, it enables more natural and controllable speech synthesis. The approach of using residual learning with explicit conditioning is a generalizable technique that could be applied to other speech generation tasks requiring fine-grained control over specific acoustic attributes. ProsoCodec presents a novel and effective approach to voice conversion by treating prosody as a conditional residual in a discrete speech codec, achieving state-of-the-art performance in prosody preservation and content fidelity through clever architectural conditioning and training strategies.
Recent speech research involves increasingly large datasets, complex models, and diverse experimental workflows. However, existing frameworks require substantial engineering effort to support such experiments. We present ESPnet3, a speech and audio research framework built on a modular system architecture with configuration-driven dataset composition and unified Python-based workflows. ESPnet3 introduces a DataOrganizer abstraction for flexible dataset integration and dataset sharding for memory-efficient large-scale training, while allowing recipe-specific logic through lightweight stage overrides. In OWSM pre-training experiments, ESPnet3 reduces per-epoch training time by \emph{21.1 minutes} compared to ESPnet2 and achieves \emph{>80\% GPU utilization} in multi-node training. Fine-tuning experiments show that new models and datasets can be integrated with around \emph{46 lines of additional code}. ESPnet3 will be publicly released with model checkpoints and training logs.
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University, Brno University of Technology, Instituto Superior TĂ©cnico, Hanyang University, Hitachi Astemo, Shanghai Jiao Tong University
ESPnet3 represents a crucial evolution in speech research infrastructure, transforming ESPnet from a collection of recipes into a scalable, modular framework capable of supporting the data- and compute-intensive demands of modern foundation models.
The paper presents ESPnet3, a significant architectural refactoring of the ESPnet speech processing framework. The core methodological contribution lies in the decoupling of experiment logic from core infrastructure through a modular system architecture. Key technical innovations include the `DataOrganizer` abstraction for configuration-driven dataset composition using Hydra, and a shard-based iteration mechanism for memory-efficient large-scale training. The shift from shell/Perl-based orchestration in ESPnet2 to a unified Python-based workflow using PyTorch Lightning and HuggingFace Datasets represents a substantial engineering improvement for scalability and maintainability. While the individual components (Hydra, PyTorch Lightning, HuggingFace Datasets) are not novel, their specific integration and abstraction design for the speech domain, particularly the `BaseSystem` pattern, constitute a valid and impactful engineering contribution.
The evaluation focuses on system-level metrics rather than model performance improvements. The authors demonstrate a 21.1-minute reduction in per-epoch training time and >80% GPU utilization in multi-node OWSM pre-training. They provide concrete evidence of reduced engineering effort, citing a reduction from 2,289 lines to 70 lines for the OWSM recipe and a 46-line integration for new datasets. The fine-tuning experiments on Whisper with LoRA and the FalAR dataset serve to validate the framework's extensibility. However, the absence of new SOTA model results means the paper does not demonstrate that ESPnet3 enables *better* models, only *more efficient* development and training. The system-level gains are well-documented and convincing.
The paper promises public release of code, checkpoints, and training logs. The use of standard libraries (PyTorch, Hydra, HuggingFace) enhances reproducibility. The detailed comparison of code lines and memory usage provides clear benchmarks for other developers. The reliance on external large-scale datasets (OWSM, FalAR) and specific hardware (Bridges2, Delta) means exact replication of timing results may vary, but the architectural reproducibility is high.
The primary limitation is that this is an infrastructure paper, not a model architecture paper. It does not propose new neural network architectures or learning algorithms. The novelty is largely in software engineering and system design. The evaluation lacks comparison with other modern speech frameworks like NVIDIA NeMo or HuggingFace Transformers in terms of raw throughput or ease of use for specific tasks, focusing instead on internal comparisons with ESPnet2. The claim of "foundation model era" readiness is supported by scale but not by demonstrating capabilities beyond ASR.
ESPnet3 addresses a critical bottleneck in speech research: the high engineering overhead for large-scale experiments. By lowering the barrier to entry for multi-node, large-dataset training and simplifying the integration of new models and datasets, it has the potential to accelerate research in low-resource languages and multimodal speech systems. The modular design encourages community contributions and long-term maintainability of the ESPnet ecosystem. ESPnet3 represents a crucial evolution in speech research infrastructure, transforming ESPnet from a collection of recipes into a scalable, modular framework capable of supporting the data- and compute-intensive demands of modern foundation models.
Neural networks outperform classical GCC-PHAT for Time-Difference-of-Arrival (TDOA) estimation in noise and reverberation, yet their internal strategy remains unexplored. To uncover it, we turn GCC-PHAT's mathematical steps into diagnostic targets, probing hidden layers of three architectures (MLP, CNN, Transformer) and complementing with gradient attribution and causal frequency masking. We find that cross-power computation consistently emerges across all architectures and conditions, while PHAT whitening, the defining step of GCC-PHAT, fails to emerge. Instead, networks learn a magnitude-aware frequency weighting that preserves per-frequency reliability information discarded by PHAT. This makes PHAT an information bottleneck: removing it from both classical and neural GCC pipelines improves performance under additive noise. On real-world reverberant data, PHAT remains the best classical weighting, but end-to-end networks achieve lower error by learning data-adaptive weighting.
Primary: Institute of Science Tokyo
All Institutions: Institute of Science Tokyo
This paper provides a compelling interpretability study of neural networks for TDOA estimation, demonstrating that networks bypass the classical PHAT whitening step in favor of magnitude-aware weighting, thereby offering actionable insights for improving hybrid audio processing pipelines.
The paper employs a rigorous interpretability framework, specifically representation probing, to analyze the internal representations of neural networks trained for Time-Difference-of-Arrival (TDOA) estimation. By mapping hidden layer activations to specific mathematical steps of the classical GCC-PHAT algorithm (cross-power spectrum, PHAT whitening, phase), the authors create a precise diagnostic tool. The methodology is sound, utilizing linear and nonlinear probes, gradient attribution, and causal frequency masking to disentangle what information is represented versus how it is weighted. The choice of architectures (MLP, CNN, Transformer) with varying frequency connectivity allows for a controlled study of how different inductive biases affect the emergence of these algorithmic components.
The experimental design is comprehensive, covering synthetic data with controlled noise and reverberation, simulated speech data, and real-world recordings from the LOCATA challenge. The results are robust, consistently showing that while cross-power computation emerges reliably across all models and conditions, PHAT whitening does not. Instead, networks learn a magnitude-aware weighting scheme that correlates with signal reliability. The causal masking experiments strongly support the gradient attribution findings, demonstrating that high-energy bins are causally important for performance. The finding that removing PHAT whitening improves performance in additive noise is a significant empirical contribution, challenging a long-standing assumption in the audio signal processing community.
The paper provides a clear description of the network architectures, training procedures (AdamW, Huber loss, etc.), and data generation processes. The code is publicly available on GitHub, and the use of standard libraries (LibriSpeech, pyroomacoustics, LOCATA) ensures that the experimental setup can be replicated. The inclusion of specific hyperparameters and dataset sizes enhances reproducibility.
The study focuses primarily on pairwise TDOA estimation using short-time Fourier transform (STFT) representations. It does not explore waveform-domain models or end-to-end systems that might learn different internal representations. Additionally, the analysis is limited to single-source scenarios; the behavior in multi-source environments with overlapping speech or noise is noted as an open question. The reliance on linear probes, while standard, might miss complex nonlinear interactions, although the authors mitigate this with nonlinear probe controls.
This work has significant implications for the design of hybrid signal processing and neural network systems. By identifying PHAT whitening as an information bottleneck in noisy conditions, it suggests that classical pipelines can be improved by removing this step or replacing it with learned, data-adaptive weighting. It also provides a template for using probing to understand and improve other audio processing algorithms, bridging the gap between classical signal processing theory and modern deep learning practices. The findings encourage a re-evaluation of fixed preprocessing steps in favor of learnable, interpretable alternatives. This paper provides a compelling interpretability study of neural networks for TDOA estimation, demonstrating that networks bypass the classical PHAT whitening step in favor of magnitude-aware weighting, thereby offering actionable insights for improving hybrid audio processing pipelines.
Voice reconstruction using Text-to-Speech (TTS) offers a communication method for people with speech disorders, which aims to retain their speaker identity while improving intelligibility. Previous work generally relies on Mean Opinion Score (MOS) to evaluate naturalness and speaker similarity, but this has limited sensitivity and reliability. We propose an evaluation framework with subjective and objective components. Subjectively, we evaluate perceived intelligibility and speaker identity using Best Worst Scaling (BWS) with situational framing. Objectively, we demonstrate that standard measures fail to predict reconstruction success for highly unintelligible speakers, so we introduce a novel dual-reference distributional measure to assess the trade-off between intelligibility and speaker identity. By evaluating the output of 17 zero-shot TTS systems for 193 speakers, we show that our framework provides a reliable and task-aligned approach for assessing voice reconstruction.
Primary: The Centre for Speech Technology Research, University of Edinburgh
All Institutions: The Centre for Speech Technology Research, University of Edinburgh
This paper presents a rigorous and innovative evaluation framework for TTS voice reconstruction, introducing situational framing in subjective evaluation and a novel dual-reference distributional metric that effectively captures the trade-off between intelligibility and speaker identity, addressing critical gaps in current assessment methodologies for assistive speech technologies.
The paper proposes a comprehensive evaluation framework for Text-to-Speech (TTS) voice reconstruction, a task critical for assisting individuals with speech disorders. The methodology addresses two main gaps in current evaluation practices: the limitations of Mean Opinion Score (MOS) in sensitivity and reliability, and the failure of standard objective metrics to correlate with human perception in this specific domain. Subjectively, the authors employ Best Worst Scaling (BWS) with situational framing to isolate intelligibility from speaker identity reconstruction, a nuanced approach that acknowledges the distinct cognitive tasks involved. Objectively, they introduce a novel dual-reference distributional measure (TTSDS Mean) that combines distances to a high-intelligibility generic corpus and the original disordered speaker's reference. This approach attempts to quantify the trade-off between improving intelligibility and preserving speaker identity, addressing the lack of ground truth in this generative task. The methodological rigor is high, particularly in the experimental design of the subjective study and the innovative application of distributional metrics to a domain where they have not been standardly applied.
The experimental setup is robust, involving 17 zero-shot TTS systems evaluated on 193 speakers from the Speech Accessibility Project (SAP) dataset. The dataset covers diverse speech disorders (Parkinson's, Cerebral Palsy, ALS, Down Syndrome), enhancing the generalizability of the findings. The subjective evaluation involved a significant number of listeners (46-47 per condition) and used rigorous statistical modeling (Plackett-Luce). The results clearly demonstrate that standard objective metrics (WER, PER, Speaker Similarity, UTMOS) fail to predict reconstruction success, particularly for highly unintelligible speakers. The proposed TTSDS Mean metric shows strong correlation with subjective reconstruction rankings (rho=0.81 overall), outperforming speaker similarity (rho=0.75). The analysis of system performance reveals that while some systems (IndexTTS2, Qwen3-TTS) perform well on average, they struggle with severely disordered speech, highlighting the complexity of the task. The large-scale evaluation provides a solid empirical basis for the proposed framework.
The paper provides detailed information on the datasets, systems, and evaluation protocols. The use of the public SAP dataset and well-known TTS systems enhances reproducibility. The authors provide a project page with audio examples and listening test instructions, which is crucial for verifying the subjective findings. However, the code for the specific implementation of the TTSDS Mean metric and the exact configuration of the 17 TTS systems (especially if they are open-source but require specific versions) might need clarification. The description of the BWS experimental design is sufficiently detailed for replication. The lack of a public code repository is a minor drawback, but the availability of stimuli and detailed methodology mitigates this.
The study focuses primarily on intelligibility and speaker identity, leaving out other important dimensions such as prosody, accent similarity, and naturalness, which are acknowledged as future work. The subjective evaluation relies on listeners who are not the end-users (people with speech disorders), which may introduce bias or lack of ecological validity. The distributional measure relies on the quality of the reference datasets (LibriTTS and the original disordered speech), and its performance may vary with different TTS architectures or training data distributions. The generalization of the TTSDS Mean metric to other languages or speech disorders not covered in the study is not tested. Additionally, the use of zero-shot TTS systems limits the scope to cloning-based approaches, excluding fine-tuned or specialized reconstruction models.
This work has significant potential impact on the development of assistive communication technologies. By providing a more reliable and task-aligned evaluation framework, it can guide researchers and developers in creating better voice reconstruction systems for people with speech disorders. The findings challenge the reliance on standard TTS metrics and advocate for more nuanced, use-case-specific evaluation methods. The framework can be adopted by the broader TTS and speech accessibility communities to standardize evaluations and facilitate fair comparisons between systems. Ultimately, this contributes to improving the quality of life for individuals with speech impairments by enabling more effective and personalized communication aids. This paper presents a rigorous and innovative evaluation framework for TTS voice reconstruction, introducing situational framing in subjective evaluation and a novel dual-reference distributional metric that effectively captures the trade-off between intelligibility and speaker identity, addressing critical gaps in current assessment methodologies for assistive speech technologies.
Current text-guided audio editing methods rely on paired training data, predefined operation templates, and separate processing pipelines across speech, music, and sound. We present Bagpiper-Edit to enable open-ended audio editing via free-form natural language instructions. We reformulate audio editing as a rich-caption rewriting task by treating a rich caption as the semantic representation of an audio clip. The user request is translated into an edited caption, which then guides Bagpiper-Edit to generate the target edited audio with the original audio as contextual acoustic anchor. This unlocks the potential of free-form editing, and circumvents the need for paired audio-editing training data, enabling powerful zero-shot editing capabilities. Evaluations across speech, audio, and free-form editing show Bagpiper-Edit maintains good consistency to the original audio and achieves similar performance to other expert models in most cases. Demo: https://bagpiper-edit.github.io, Codes: https://github.com/espnet/espnet/pull/6417 & https://github.com/HsunGong/espnet
Primary: Shanghai Jiao Tong University
All Institutions: Auditory Cognition and Computational Acoustics Lab, Shanghai Jiao Tong University, Carnegie Mellon University, Language Technologies Institute
Bagpiper-Edit introduces a novel self-supervised framework for zero-shot open-ended audio editing by reformulating the task as rich-caption rewriting, effectively eliminating the need for paired editing datasets while maintaining high acoustic consistency.
The paper proposes Bagpiper-Edit, a framework that reformulates open-ended audio editing as a rich-caption rewriting task. The core innovation lies in bypassing the need for paired audio-editing training data by introducing a self-supervised training paradigm. This involves two strategies: audio repetition (to teach timbre/background preservation) and audio segmentation (to teach continuity across adjacent clips). The method leverages a pre-trained audio foundation model (Bagpiper-Base) and uses an external LLM to rewrite the rich caption based on user instructions, which then guides the audio generation conditioned on the original audio as an "acoustic anchor." The approach is theoretically sound and addresses a significant bottleneck in the field: the scarcity of high-quality paired editing datasets. The use of contiguous audio segments for self-supervision is a clever and resource-efficient technique for learning acoustic consistency.
The evaluation covers three domains: speech editing, audio-event editing, and free-form editing. The authors compare Bagpiper-Edit against specialized expert models (CosyVoice-3, Ming-UniAudio-Edit, Step-Audio-EditX) and generative models (AudioLDM2). Results show that Bagpiper-Edit (specifically the Multi-Turn variant) achieves competitive performance, often surpassing expert models in speaker similarity preservation and matching them in naturalness and editing accuracy. The metrics used (WER, SpkSIM, DNSMOS, FAD, CLAP scores) are standard and appropriate. The inclusion of LLM-based human preference scoring adds a layer of subjective evaluation, though reliance on LLMs for judgment can be noisy. The ablation between Single-Turn and Multi-Turn patterns provides useful insight into the training dynamics.
The paper provides links to a demo and a GitHub PR for the code. The training data sources (YODAS, LAION-Audio, etc.) are publicly available. The self-supervised training strategy is clearly described, allowing for potential reproduction. However, the reliance on specific large LLMs (Qwen3-235B, Qwen3-8B) for caption extraction and rewriting introduces external dependencies that might vary in performance depending on the version or API access. The exact hyperparameters for the self-supervised phase are partially detailed, but the full pipeline complexity might make exact replication challenging without the complete codebase.
The paper acknowledges that as a zero-shot model, it may exhibit less stability than expert models trained on massive paired data. It also notes limitations in handling highly complex acoustic environments like multi-speaker separation. The reliance on an external LLM for caption rewriting means that errors in the LLM's understanding or rewriting can propagate to the audio generation. Additionally, the "rich caption" representation, while powerful, may lose some fine-grained acoustic details present in the original audio if the captioning process is not perfectly lossless.
This work significantly lowers the barrier for open-ended audio editing by removing the dependency on expensive paired datasets. It enables a more natural, free-form interface for audio manipulation, which has broad applications in content creation, accessibility, and audio post-production. The self-supervised approach could inspire similar techniques in other modalities where paired data is scarce. However, the ease of editing audio also raises concerns about deepfakes and misinformation, necessitating robust watermarking or detection mechanisms, which are not discussed in the paper. Bagpiper-Edit introduces a novel self-supervised framework for zero-shot open-ended audio editing by reformulating the task as rich-caption rewriting, effectively eliminating the need for paired editing datasets while maintaining high acoustic consistency.
Speaker-decoupled speech codecs can reduce bitrate by separating global speaker attributes from local content and prosody, while supporting voice conversion. Existing speaker-decoupled codecs face a trade-off: methods that explicitly suppress speaker leakage often rely on multi-stage or auxiliary training, whereas simpler designs can leave residual speaker information in local tokens. We propose SDP-Codec, a speaker-decoupled, pitch-injected codec trained with a single-stage optimization pipeline. SDP-Codec derives local tokens from continuous pre-quantization features of a pretrained self-supervised encoder and injects normalized F0 via a pitch encoder-decoder with global-conditioned denormalization and soft-label pitch reconstruction objective. Across 16 kHz and 24 kHz settings, SDP-Codec achieves competitive reconstruction and strong zero-shot voice conversion at comparable bitrates, with the lowest speaker-probing accuracy among compared systems, suggesting reduced speaker leakage.
Primary: Graduate School of Culture Technology, KAIST
All Institutions: Graduate School of Culture Technology, KAIST
SDP-Codec presents a compelling single-stage approach to speaker-decoupled speech coding, effectively balancing reconstruction quality and voice conversion capability through innovative pitch injection and soft-label loss mechanisms. The rigorous evaluation, including speaker probing and comprehensive ablation, strengthens the claim that the method successfully reduces speaker leakage in local tokens while preserving content and prosody, offering a practical solution for low-bitrate neural speech codecs.
The paper proposes SDP-Codec, a speaker-decoupled speech codec designed to minimize bitrate while supporting zero-shot voice conversion. The core methodological contribution lies in the decoupling strategy: it uses a single-stage optimization pipeline where local tokens are derived from continuous pre-quantization features of a pretrained vq-wav2vec encoder, rather than the quantized units themselves. To address the loss of prosodic information inherent in this compression, the authors inject normalized F0 via a dedicated pitch encoder-decoder. A key innovation is the use of a soft-label pitch reconstruction objective (Gaussian-blurred one-hot bins) and global-conditioned denormalization, which allows the global branch to handle speaker-dependent pitch range while the local branch handles content and normalized prosody. The global branch utilizes WavLM features compressed via a perceiver resampler to provide time-invariant speaker embeddings. This design attempts to resolve the trade-off between explicit speaker suppression (complex training) and residual leakage (poor VC).
The evaluation covers reconstruction quality (UTMOS, WER, STOI) and zero-shot voice conversion (SECS, F0 correlation, NMOS, SMOS) at 16 kHz and 24 kHz. The paper compares SDP-Codec against strong baselines including LSCodec, BiCodec, MSRCodec, and EZ-VC. Results indicate that SDP-Codec achieves competitive reconstruction quality, particularly in STOI and F0 correlation, while demonstrating superior speaker similarity in VC tasks compared to other codec-based systems. The speaker-probing accuracy experiment is a strong addition, quantitatively demonstrating reduced speaker leakage in the local tokens compared to competitors. The ablation studies effectively validate the components, showing that the soft-label pitch loss and the use of continuous pre-quantization features are critical for performance.
The paper provides source code and a demo link. Training details are provided, including dataset splits (LibriSpeech, LibriTTS, MLS), training steps, and hardware. However, the use of "internal" data for some baselines (Vevo) and the specific configuration of baselines (e.g., merging thresholds) to match bitrates introduces some variability. The description of the architecture is sufficiently detailed for replication, and the frozen pretrained components (vq-wav2vec, WavLM, FCPE) are standard.
The paper acknowledges that content fidelity (WER) remains a limitation, trading some reconstruction quality for VC performance. The reliance on a pretrained pitch extractor (FCPE) introduces a dependency on external tools for feature extraction, which might not be perfectly aligned with the codec's internal representations. The evaluation is primarily on English datasets (LibriSpeech, LibriTTS, MLS), limiting the assessment of multilingual robustness, although the authors mention Indic languages in the table header, the main text focuses on English. The single-stage training, while simpler, might not achieve the same level of disentanglement as multi-stage methods that explicitly optimize for speaker suppression.
SDP-Codec contributes to the field of efficient speech representation learning, enabling low-bitrate transmission and flexible voice conversion. This has implications for telecommunication, speech language models (SLMs) by providing compact tokens, and creative applications in voice cloning and conversion. The reduced speaker leakage is a positive step towards more ethical and controllable voice conversion systems. However, the potential for misuse in deepfake generation remains a concern, though the low bitrate and specific use case mitigate this slightly compared to high-fidelity generative models. SDP-Codec presents a compelling single-stage approach to speaker-decoupled speech coding, effectively balancing reconstruction quality and voice conversion capability through innovative pitch injection and soft-label loss mechanisms. The rigorous evaluation, including speaker probing and comprehensive ablation, strengthens the claim that the method successfully reduces speaker leakage in local tokens while preserving content and prosody, offering a practical solution for low-bitrate neural speech codecs.
As expressive text-to-speech (TTS) and voice conversion (VC) systems increasingly generate non-verbal vocalizations (NVVs) to enhance naturalness, reliable speaker verification (SV) becomes essential to objectively assess identity consistency across both verbal and non-verbal segments. Yet current SV systems generalize poorly to NVVs, and fine-tuning on NVV data causes catastrophic forgetting of speech performance. We present the first systematic study across 10 NVV types and propose a framework combining frozen Data2Vec self-supervised features with ECAPA-TDNN, enhanced by a Mixture of Experts (MoE) module with learned domain-aware routing. A conditional distillation loss on speech inputs via a pretrained teacher retains speech-to-speech accuracy, while a contrastive loss bridges the speech-NVV domain gap. Our method reduces speech-NVV EER from 38.93% to 22.66% over a pretrained baseline, and improves speech EER from 13.17% to 9.24% via distillation.
Primary: National Taiwan University
All Institutions: University of Michigan, National Taiwan University, Signal Analysis and Interpretation Laboratory (SAIL), University of Southern California, National Taiwan University Artificial Intelligence Center of Research Excellence
This paper presents a novel MoE-based speaker verification framework that effectively bridges the domain gap between verbal and non-verbal vocalizations, significantly improving cross-domain verification performance while mitigating catastrophic forgetting through conditional distillation.
The paper proposes a Mixture of Experts (MoE) framework integrated with a frozen Data2Vec self-supervised learning (SSL) front-end and an ECAPA-TDNN backend to address speaker verification (SV) across verbal and non-verbal vocalizations (NVVs). The core methodological innovation lies in the combination of three components: (1) an Inter-Layer Residual MoE (IR-MoE) architecture that routes features to specialized experts for speech vs. NVV, governed by learned domain-aware constraints (entropy maximization, KL divergence for intra-event consistency, and cosine-margin for inter-event separation); (2) a conditional knowledge distillation loss using a frozen WavLM teacher to prevent catastrophic forgetting of speech capabilities during NVV adaptation; and (3) a supervised contrastive loss that bridges the domain gap by enforcing shared speaker manifolds across speech and NVV pairs. The approach is technically sound, leveraging established SSL representations while introducing architectural and loss-function modifications to handle the heterogeneity of non-phonemic acoustic signals. The use of MoE for domain separation in SV is a novel application, moving beyond standard fine-tuning paradigms.
The experimental evaluation is comprehensive, utilizing the NonverbalTTS dataset with 10 distinct NVV categories. The authors provide a systematic analysis of domain mismatch, demonstrating the significant performance drop of zero-shot SV models on NVVs (EER ~39%). They compare against multiple baselines, including standard SSL models (WavLM, Data2Vec, Voc2Vec) and fusion strategies. The proposed IR-MoE model achieves a substantial reduction in cross-domain EER (NvS) from 38.93% to 22.66% and improves speech EER (SvS) from 13.17% to 9.24% compared to a fine-tuned baseline. The ablation studies effectively isolate the contributions of the MoE architecture and the conditional distillation loss. The results are statistically significant and demonstrate that the proposed method successfully mitigates catastrophic forgetting while improving NVV verification. The inclusion of mDCF metrics adds robustness to the evaluation.
The paper provides detailed implementation specifications, including the optimizer (Adam), learning rate schedule (cosine annealing), batch composition strategy (speaker-balanced with cross-domain prioritization), and specific loss weights. The use of standard, open-source components (Data2Vec, ECAPA-TDNN, WavLM) enhances reproducibility. The description of the progressive training schedule and the specific constraints for the MoE routing (entropy, KL, cosine-margin) is sufficiently detailed for replication. However, the exact codebase for the MoE integration and the specific preprocessing steps for the NonverbalTTS dataset (beyond MFA alignment) might require careful interpretation, though the overall pipeline is clear.
A primary limitation is the reliance on the NonverbalTTS dataset, which, while diverse, may not capture the full spectrum of NVVs found in real-world expressive TTS/VC systems (e.g., complex emotional laughter, gasps, or synthesized artifacts). The MoE architecture introduces additional parameters and complexity compared to standard fine-tuning, which could impact inference latency, although this is not explicitly measured. The "conditional" distillation relies on the model's ability to correctly route inputs during inference; if the router misclassifies an input (e.g., a speech-like NVV), the distillation benefit might not be fully realized or could even be detrimental if the wrong expert is activated. Furthermore, the performance on NVV-NVV (NvN) verification remains relatively high (EER ~27-28%), indicating that while cross-domain generalization is improved, intra-NVV verification is still challenging.
This work has significant implications for the evaluation and development of expressive TTS and Voice Conversion systems, where NVVs are crucial for naturalness. By providing a reliable SV metric for NVVs, it enables better objective evaluation of these systems, potentially accelerating their adoption in virtual assistants, gaming, and accessibility tools. It also contributes to the broader field of multi-modal and multi-domain speaker recognition, offering a template for handling heterogeneous acoustic inputs. The mitigation of catastrophic forgetting is a generalizable technique for continual learning in speech processing. This paper presents a novel MoE-based speaker verification framework that effectively bridges the domain gap between verbal and non-verbal vocalizations, significantly improving cross-domain verification performance while mitigating catastrophic forgetting through conditional distillation.
Speech deepfake countermeasures (CMs) are compared almost exclusively by equal error rate (EER), a metric computed at an oracle threshold chosen on the labeled test set. Deployed CMs enjoy no such oracle: a threshold must be fixed in advance and applied to unlabeled target data. We audit this gap with a frozen state-of-the-art SSL-AASIST detector trained on ASVspoof 2019 LA. While its in-domain EER is 0.21%, transferring its LA-calibrated threshold to the In-the-Wild corpus yields a half total error rate (HTER) of 39.5%, with 78.7% of bona fide speech rejected, even though the In-the-Wild EER (11.2%) appears moderate. We then test whether popular unlabeled test-time corrections close this gap, and first prove a simple proposition: any strictly increasing score transform, including z-norm, temperature/shift calibration, and embedding mean alignment under a frozen linear head, cannot change EER. An audit of seven corrections on In-the-Wild and ASVspoof 2021 DF confirms the proposition empirically and exposes two further failure modes: AS-norm with an unlabeled target cohort collapses (EER 11.2% to 60.2%), and pseudo-label calibration that reduces HTER by 38% relative on In-the-Wild degenerates to 50% HTER on DF21, whose spoof prior is 96%. No audited correction reduces EER by more than 1% relative. We recommend reporting HTER at a transferred threshold alongside EER.
Primary: Xidian University
All Institutions: Xidian University
This paper provides a crucial theoretical and empirical audit of speech deepfake detection evaluation, proving that common unlabeled score calibrations cannot improve EER and demonstrating the severe deployment failures hidden by standard metrics, thereby advocating for more realistic evaluation protocols in the field.
The paper presents a critical methodological audit of speech deepfake countermeasures (CMs), specifically focusing on the discrepancy between evaluation metrics (EER) and deployment reality (HTER at fixed thresholds). The core theoretical contribution is a rigorous proof of monotone invariance: any strictly increasing transformation of scores (including z-norm, temperature scaling, and embedding mean alignment) cannot alter the Equal Error Rate (EER). This is a significant theoretical insight that debunks the efficacy of many popular "unlabeled test-time corrections" often cited in adjacent fields like speaker verification. The methodology involves freezing a state-of-the-art SSL-AASIST model and systematically applying seven different correction strategies (C1-C7) to evaluate their impact on both in-domain and out-of-domain performance. The approach is clean, logically sound, and effectively isolates the operating-point shift from the score distribution shift. EXPERIMENTAL_EVALIGATION: The experimental evaluation is robust and well-designed. The authors use a frozen SSL-AASIST model trained on ASVspoof 2019 LA and test it on two distinct out-of-domain datasets: In-the-Wild (ITW) and ASVspoof 2021 DF. The results starkly illustrate the paper's thesis: while the in-domain EER is near-zero (0.21%), the transferred threshold leads to catastrophic failure on ITW (HTER 39.5%, 78.7% FRR). The audit of seven corrections confirms the theoretical proposition: monotone methods (C1-C3) fail to improve EER and often worsen HTER, while non-monotone methods (C4-C7) show limited improvement or collapse (e.g., AS-norm with unlabeled cohort collapses to 60.2% EER due to cohort contamination). The analysis of Failure Mode III (prior sensitivity) on DF21, where pseudo-label calibration fails due to a 96% spoof prior, provides deep practical insight into the fragility of current calibration techniques.
The paper provides sufficient detail for reproduction. It specifies the model architecture (SSL-AASIST with wav2vec 2.0 front-end), the specific checkpoint used, the data subsets (verified against official protocols), and the exact procedures for each correction method (e.g., L-BFGS settings for temperature scaling, leave-one-out for AS-norm). The code for the audit protocol is likely available given the arXiv submission context, though not explicitly linked in the text provided. The verification of labels against official protocol files adds a layer of trust to the experimental setup.
The primary limitation is the scope of the audit: it focuses on a single model architecture (SSL-AASIST). While the monotone invariance proposition is model-agnostic, the magnitude of errors for non-monotone methods may vary across different architectures (e.g., CNN-based vs. Transformer-based detectors). Additionally, the audit uses specific subsets of the ASVspoof datasets (4/9 shards for LA, 4/80 for DF21), which may not fully represent the entire dataset's variability. The paper also notes that HTER is a single operating point; applications requiring different cost functions (e.g., t-DCF) might see different absolute numbers, although the qualitative conclusions likely hold.
This paper has significant implications for the speech anti-spoofing community. By highlighting that EER is a misleading metric for deployment readiness, it challenges the status quo of benchmarking in the field. The recommendation to report HTER at transferred thresholds alongside EER could lead to more realistic evaluation standards. The proof that many common calibration techniques are theoretically incapable of improving EER saves researchers from pursuing dead ends and directs attention to non-monotone corrections or fundamental model improvements. This work promotes more rigorous and deployment-aware evaluation practices in audio security and deepfake detection. This paper provides a crucial theoretical and empirical audit of speech deepfake detection evaluation, proving that common unlabeled score calibrations cannot improve EER and demonstrating the severe deployment failures hidden by standard metrics, thereby advocating for more realistic evaluation protocols in the field.