Recent full-duplex spoken dialogue models have demonstrated compelling progress toward human-like interaction, enabling agents to respond with low latency, produce backchannels, and handle user barge-ins. Yet these improvements in conversational dynamics often come with weaker reasoning and instruction-following abilities, revealing a potential tension between interactive dynamics and intelligence capability. In this paper, we argue that such an intelligence--dynamics trade-off is not fundamental: conversational dynamics can instead be learned as a separate real-time decision policy from human dialogue data. To this end, we propose DuplexPO, a reinforcement learning (RL) framework that decouples when to speak from what to say. It preserves the semantic response capability of an instruction-tuned assistant, while optimizing its temporal interaction behavior over selected high-impact windows from long human conversations. To quantitatively optimize these dynamics, we formulate the Factorized Conversational Dynamics Reward (FCDR) to enable fine-grained temporal credit assignment for turn initiation, backchanneling, yielding, and regularized participation. The policy is then optimized with a GRPO-style objective. Experiments show that DuplexPO substantially improves full-duplex behaviors, including timely backchannels, smooth turn-taking, and barge-in handling, while maintaining strong reasoning and instruction-following performance. Moreover, improvements in dynamics-oriented metrics are reflected in better user experience, suggesting that optimizing conversational timing as a standalone objective can promote more natural full-duplex interaction.
Primary: NVIDIA
All Institutions: Nanyang Technological University, National Taiwan University, The Hong Kong University of Science and Technology, NVIDIA
This paper presents a significant advancement in full-duplex spoken dialogue systems by introducing a novel RL framework that effectively decouples conversational timing from semantic generation, leading to more natural interactions without compromising reasoning capabilities.
The paper proposes DuplexPO, a reinforcement learning framework designed to decouple conversational dynamics (when to speak) from semantic content (what to say) in full-duplex spoken dialogue models. The core methodological innovation lies in the Factorized Conversational Dynamics Reward (FCDR), which provides fine-grained temporal credit assignment for specific interaction events like turn initiation, backchanneling, and yielding. By restricting policy optimization to "dynamics-critical windows" sampled from long human conversations and using a GRPO-style objective, the method aims to improve real-time interaction behaviors without degrading the model's instruction-following capabilities. This addresses a specific and challenging problem in end-to-end spoken language models: the tension between low-latency responsiveness and reasoning quality. The approach is technically sound, leveraging established RL techniques (GRPO) in a novel context (temporal windowing for audio dialogue).
The evaluation is comprehensive, covering both conversational dynamics metrics (onset MAE, yield rate, voiced interrupt rate) on standard datasets (Fisher, Seamless, Full-Duplex-Bench) and model intelligence benchmarks (QA, instruction following). The results demonstrate that DuplexPO significantly improves dynamic metrics compared to SFT baselines while maintaining or slightly improving intelligence metrics. The inclusion of an LLM-as-a-Judge evaluation for naturalness adds qualitative depth. The ablation studies on window size, reward design, and optimization objectives (GRPO vs. DPO) provide strong empirical support for the design choices. The comparison with commercial models and other open-source full-duplex models places the work in a competitive context.
The paper provides detailed implementation details, including model architecture (Qwen2.5-7B backbone, Parakeet encoder), training stages, and hyperparameters for the RL phase. The data sources (Fisher, Seamless) are public, and the reconstruction process is described. The inclusion of a demo URL suggests the authors are committed to transparency. However, the exact code for the window sampling and FCDR reward calculation is not explicitly linked in the text (though a GitHub link might be implied by the demo page, it is not in the provided text). The detailed appendix sections on data generation and training pipelines enhance reproducibility.
The authors acknowledge that FCDR may miss subtle pragmatic factors like user intent and cultural timing preferences. The reliance on human timing annotations, which are not unique ground truths, is a limitation. Furthermore, restricting optimization to local windows might miss long-range dialogue coherence effects. The evaluation is primarily in English, and the method's generalizability to other languages or highly asynchronous interaction styles is not fully explored.
This work contributes to the development of more natural and responsive voice assistants, which has significant implications for human-computer interaction, accessibility, and AI adoption. The potential for misuse in deceptive practices (e.g., vishing) is noted, highlighting the need for responsible deployment and safeguards. The decoupling of dynamics and semantics offers a generalizable paradigm for optimizing multi-modal agents where timing and content are distinct concerns. This paper presents a significant advancement in full-duplex spoken dialogue systems by introducing a novel RL framework that effectively decouples conversational timing from semantic generation, leading to more natural interactions without compromising reasoning capabilities.
In this report, we introduce Qwen-Music, a powerful music generation model capable of producing highly musical and high-fidelity songs with complete vocal singing. Qwen-Music supports two core tasks: Text to Music Generation, which create entirely new songs from text descriptions, lyrics, and musical attributes, and Cover Song Generation, which reinterprets existing songs with different styles and vocal characteristics. Architecturally, Qwen-Music integrates three core components: Qwen-Music-Tokenizer, Qwen-Music-LLM, and Qwen-Music-Render. Qwen-Music-Tokenizer compresses audio into a 25 Hz single-codebook stream of Music Semantic Tokens that preserve semantic and melodic information for LLM prediction. Based on these tokens, Qwen-Music-LLM performs autoregressive music semantic modeling, with a key novelty being a melody-token-based chain-of-thought (Melody-CoT) mechanism that plans melodies before full-song generation, improving creativity, musicality, structural coherence, and reference-audio-based melody cloning. To overcome the fidelity limitations of discrete semantic tokens, Qwen-Music-Render performs generative stereo rendering, enriching acoustic details and producing high-fidelity stereo waveforms. Finally, we train Qwen-Music-LLM on more than 5 million hours of multilingual music data covering hundreds of languages. We first apply quality-aware pre-training curriculum, then use progressive post-training, comprising supervised initialization, offline DPO, and online GSPO, to further improve musicality and instruction-following ability. Across 600 Chinese and English prompts, Qwen-Music achieves state-of-the-art results in 13 of 16 objective musicality and audio-quality metrics. Professional evaluators also prefer Qwen-Music over leading proprietary systems. For cover song generation, Qwen-Music preserves reference melodies more accurately than leading proprietary systems.
Primary: Qwen Team
All Institutions: Qwen Team
This paper introduces Qwen-Music, a state-of-the-art music generation system that effectively bridges semantic composition and acoustic rendering through a novel Melody-CoT mechanism and a high-fidelity generative renderer, achieving superior performance in both objective metrics and human preference tests.
The paper presents a comprehensive three-stage architecture for music generation: a semantic tokenizer, an autoregressive language model (LLM), and a generative renderer. The core methodological contributions lie in the specific design choices for handling the complexity of full-song generation. First, the **Qwen-Music-Tokenizer** employs a four-stage training recipe (BestRQ pretraining, causal adaptation, multi-task SFT, and VQ insertion) to create a 25 Hz single-codebook stream. This is a significant engineering contribution, as it balances semantic richness with computational tractability for LLMs. Second, the **Melody-CoT** mechanism is a notable innovation. By explicitly planning melody tokens before generating full-mixture semantic tokens, the model addresses the "composition vs. rendering" mismatch. This chain-of-thought approach allows for better structural coherence and enables cover song generation by conditioning on reference melody tokens while allowing style/timbre changes. The use of relative MIDI representations for melody tokens is a smart design to prevent leakage of absolute pitch/key information that might conflict with style tags. Third, the **Qwen-Music-Render** utilizes a Diffusion Transformer (DiT) conditioned on both semantic tokens and textual descriptions, followed by a novel **Spec-VAE** and **Band-Mode Refiner**. The refiner’s frequency-adaptive correction (phase-only for low, joint for mid, magnitude-only for high) is a sophisticated touch to address specific spectral artifacts. The training pipeline is also rigorous, featuring a quality-graded pre-training curriculum (Q1-Q6 buckets) and a progressive post-training alignment (SFT -> Offline DPO -> Online GSPO), which is state-of-the-art practice for aligning generative models with human preference.
The evaluation is extensive and robust. The authors conduct blind A/B preference tests with 50 professional human raters, showing Qwen-Music outperforms leading proprietary systems (Suno V5.5, MiniMax, Mureka) with a 59.1% win rate. They also report performance on the Artificial Analysis leaderboard. Objective evaluations cover 16 metrics across SongBench, SongEval, and AudioBox-Aesthetic, where Qwen-Music leads in 13/16 metrics. The cover song generation evaluation is particularly strong, demonstrating superior melody preservation compared to baselines. The inclusion of a detailed acoustic data quality pipeline for training data filtering (detecting fake stereo, noise floors, cutoffs) adds credibility to the high-fidelity claims. The ablation studies on the renderer and the effectiveness of the Melody-CoT mechanism are well-supported by the results.
The paper provides detailed architectural specifications, including layer counts, parameter sizes, and training objectives. The tokenizer training stages are clearly defined. The post-training pipeline (SFT, DPO, GSPO) is described with sufficient detail for replication by other research groups. However, the specific "quality-aware" reward models and the exact composition of the 5 million-hour dataset are proprietary and not fully disclosed, which limits exact reproducibility of the pre-training phase. The code for the tokenizer and renderer is not explicitly linked in the provided text, though the Qwen team typically releases code for their models.
The paper relies heavily on proprietary baselines (Suno, MiniMax) for comparison, which makes independent verification difficult. The "quality-graded" pre-training assumes the availability of a reliable internal reward model, which may not be generalizable to all settings. The Melody-CoT mechanism, while effective, adds complexity and inference time. The model's performance on non-vocal music or highly abstract musical forms (e.g., free jazz, classical symphonies without lyrics) is less emphasized compared to vocal pop/rock genres, which dominate the training data. The reliance on a single codebook for the tokenizer might limit the expressiveness compared to multi-codebook approaches (like Encodec) in capturing fine-grained timbral details, although the renderer is designed to mitigate this.
Qwen-Music represents a significant step forward in accessible, high-fidelity music generation. By open-sourcing the technical report and potentially the model (implied by the Qwen lineage), it democratizes access to state-of-the-art music AI. The ability to generate cover songs and control musical attributes empowers creators. However, the potential for misuse in generating copyrighted material or deepfake vocals remains a concern, necessitating robust safety filters and watermarking, which are not detailed in this technical report. The high computational requirements for training such large models also contribute to the centralization of AI capabilities. This paper introduces Qwen-Music, a state-of-the-art music generation system that effectively bridges semantic composition and acoustic rendering through a novel Melody-CoT mechanism and a high-fidelity generative renderer, achieving superior performance in both objective metrics and human preference tests.
Audio-language models compress a speech encoder's output through a Querying Transformer (Q-Former) connector before feeding it to a large language model. We identify two failures in this compression. The connector's output vectors collapse to a single direction, and different speakers produce nearly indistinguishable outputs, with paralinguistic cues such as speaker identity, gender, and prosody lost along the way. Our method, ORCA, reverses this collapse by splitting the queries into groups whose outputs are constrained to point in different directions. On SAKURA multi-hop reasoning, ORCA gains 26.4 points over an identically trained 4B baseline, reaching 75.2% (vs. 49.0% for the 8B Audio Flamingo-3). At the connector level, the same change cuts query redundancy by 12x and raises cross-speaker variance by 75x.
Primary: National Taiwan University
All Institutions: National Taiwan University, ASUS AICS
The paper presents a compelling structural solution to query collapse in audio-language model connectors, demonstrating that enforcing orthogonality among query groups significantly enhances the preservation of paralinguistic information and boosts multi-hop reasoning performance without increasing computational cost.
The paper proposes ORCA, a structural modification to the Querying Transformer (Q-Former) connector used in Audio-Language Models (ALMs). The core innovation is the partitioning of learnable queries into $G$ groups, each with its own layer mixture from the audio encoder, and enforcing orthogonality between the mean vectors of these groups. This is a geometric constraint applied to the connector's output space. The approach is theoretically grounded in diagnosing "query collapse" (where output vectors converge to a single direction) and "information loss" (specifically paralinguistic cues like speaker identity). The method is elegant in its simplicity: it requires no additional labeled data for specific attributes, relying solely on the language modeling loss and the geometric regularizer. The separation of shallow, mid, and deep encoder layers across groups emerges naturally from the optimization, suggesting the method successfully forces the model to utilize different representational subspaces.
The experimental design is rigorous, particularly the controlled comparison where only the connector changes between the baseline and ORCA, keeping the encoder, LLM, data, and training recipe identical. This isolates the contribution effectively. The results on the SAKURA benchmark are impressive, showing a +26.4 point gain on multi-hop reasoning for a 4B model, surpassing larger 8B models. The diagnostic metrics (cosine similarity reduction by 12x, cross-speaker variance increase by 75x) strongly support the hypothesis that query collapse was the bottleneck. The robustness analysis under noise adds depth, showing that while ORCA preserves cues that survive noise (like gender), it cannot recover physically masked information (fine prosody in emotion). The comparison with MMAU shows that while ORCA helps paralinguistic reasoning, it doesn't drastically outperform baselines on general audio understanding, which is a nuanced and honest result.
The paper provides sufficient detail for reproduction. It specifies the architecture (Whisper-Large-V3, Qwen3-4B), the connector structure (G=8, J=8, K=64), the loss weights ($inter=0.1$, $intra=0.03$), and the training setup (AQA5M dataset, 5 epochs, Adam optimizer). The code for the Q-Former modification is likely straightforward to implement given the clear mathematical formulation of the inter-group and intra-group loss terms. The use of standard datasets (SAKURA, MMAU, CREMA-D) ensures that results can be compared against existing literature.
The primary limitation is that the improvement is specific to tasks requiring the preservation of diverse acoustic cues (paralinguistics). For general audio understanding (MMAU), the gains are modest. Furthermore, the method cannot recover information that is physically lost in the audio encoder or masked by noise, as acknowledged in the discussion. The reliance on the language modeling objective means that the "orthogonal" subspaces are still biased towards what helps predict text, so purely acoustic tasks without textual grounding might not benefit as much. The method also assumes that the audio encoder provides sufficient diversity in its hidden states, which might not be true for all encoders.
This work has significant implications for the design of multimodal models. By identifying and fixing the "Procrustean Bed" problem in connectors, it demonstrates that architectural changes can yield massive performance gains without increasing model scale or data volume. This is particularly relevant for resource-constrained settings where scaling up is not feasible. It also highlights the importance of paralinguistic information in ALMs, encouraging future research to focus on preserving speaker identity, emotion, and prosody, which are crucial for human-like interaction. The geometric approach to preventing representation collapse could be adapted to other modalities where similar bottlenecks exist. The paper presents a compelling structural solution to query collapse in audio-language model connectors, demonstrating that enforcing orthogonality among query groups significantly enhances the preservation of paralinguistic information and boosts multi-hop reasoning performance without increasing computational cost.
Large audio-language models (LALMs) often underperform on fine-grained, non-semantic attributes of speech, such as a speaker's emotion, despite strong performance on speech content. Improving this without the cost of retraining calls for an effective inference-time intervention, yet most existing methods intervene only after the audio encoder and operate at a relatively coarse granularity. The encoder itself, where acoustic information is first extracted from the waveform, remains largely unexplored, especially at the level of individual neurons. We introduce IAAN, Identifying and Amplifying Acoustic Neurons, a training-free and label-free method that scores each feed-forward neuron in the audio encoder by contrasting its activation on the real waveform with that on a noise reference lacking the real audio's acoustic information. IAAN then amplifies a small set of the highest-scoring neurons at inference. Across ten non-semantic speech attributes, IAAN improves average accuracy by 25.7 points on Audio-Flamingo-3, 21.4 on Qwen2.5-Omni, and 9.7 on Kimi-Audio. It also improves a model already explicitly fine-tuned to prioritize acoustic evidence. In controlled comparisons, both the encoder locus and neuron-level selectivity prove necessary for this gain. Intervening after the encoder, at the decoding side or inside the language model, yields little to no improvement, or even deteriorates accuracy. The improvement also depends on which specific neurons are amplified, not merely on their number, confirming that IAAN's acoustic score succeeds in identifying the neurons that matter. These results show that a small, precisely targeted intervention inside the audio encoder is an effective and largely untapped way to strengthen the acoustic understanding of LALMs, opening a new direction for inference-time methods that improve acoustic perception through neuron-level access to the encoder.
Primary: National Taiwan University
All Institutions: National Taiwan University, NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), ASUS Open Cloud Infrastructure Software Center
[One sentence main contribution]. The paper introduces IAAN, a training-free inference-time intervention that identifies and amplifies specific feed-forward neurons in the audio encoder of LALMs by contrasting their activations on real audio versus noise, significantly improving non-semantic acoustic perception. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. This paper makes a valuable contribution to the field of multimodal machine learning by shifting the focus of inference-time interventions from the language model backbone to the audio encoder. By proposing a simple yet effective contrastive scoring mechanism at the individual neuron level, IAAN addresses a critical gap in how LALMs process fine-grained acoustic information. The rigorous experimental evaluation, including comprehensive ablations against other intervention loci and granularities, strongly validates the necessity of encoder-side, neuron-level targeting. The significant performance gains across multiple state-of-the-art LALMs on a challenging benchmark (VoxParadox) demonstrate the practical utility of this approach. While the computational cost of dual encoder passes is a minor drawback, the method's label-free and training-free nature makes it a highly attractive tool for enhancing the acoustic grounding of existing models.
The paper proposes IAAN (Identifying and Amplifying Acoustic Neurons), a training-free, label-free inference-time intervention method. The core methodology involves contrasting the activation of feed-forward neurons in the audio encoder of Large Audio-Language Models (LALMs) between the real audio waveform and a noise reference (Gaussian noise with matched length/variance). Neurons that activate more strongly on real audio are identified as "acoustic neurons" and their activations are amplified by a gain factor $g$ during inference. The approach is conceptually simple and elegant, leveraging the intuition that acoustic-specific neurons should be sensitive to the presence of structured audio versus unstructured noise. The method operates at the individual neuron level within the audio encoder, which is a finer granularity than previous interventions that typically target the language model backbone or output logits. The technical execution involves standard transformer operations (FFN activations) and does not require backpropagation or external datasets for calibration, relying instead on an internal contrastive signal.
The authors evaluate IAAN on three open-source LALMs (Audio-Flamingo-3, Qwen2.5-Omni, Kimi-Audio) and one specialized model (AF3-PD) using the VoxParadox benchmark, which tests ten non-semantic speech attributes (e.g., emotion, gender, pitch) where semantic content may contradict acoustic cues. The results are significant: IAAN improves average accuracy by 25.7 points on Audio-Flamingo-3, 21.4 on Qwen2.5-Omni, and 9.7 on Kimi-Audio. Crucially, the paper includes rigorous ablation studies. It compares IAAN against other inference-time interventions (Audio-aware decoding, LLM-amplify, Hidden-state steering) and demonstrates that intervening *inside* the encoder is necessary, as post-encoder interventions yield little to no gain. It also shows that the specific selection of neurons (via the acoustic score) is critical, as random selection or whole-layer amplification fails to reproduce the gains. The method also improves a model (AF3-PD) that was already fine-tuned for acoustic tasks, suggesting broad applicability. The use of a held-out development set (LISTEN) for hyperparameter tuning adds credibility, although the transfer of optimal hyperparameters across models is noted to be variable.
The paper provides sufficient detail for reproduction. It defines the neuron definition (FFN hidden activations), the reference signal generation (Gaussian noise with fixed seed), the scoring equation, and the amplification hook. The models and benchmark (VoxParadox) are open-source and widely available. The hyperparameter selection process is clearly described, though the reliance on an external development set (LISTEN) for tuning $(K, g)$ is a minor constraint for fully zero-shot application. The code is not explicitly linked in the text provided, but the method is algorithmic and straightforward to implement given the open-source nature of the base models.
The primary limitation is the computational overhead of running two forward passes through the audio encoder (real vs. noise reference) for every inference step. While the encoder is smaller than the LLM, this still doubles the encoder compute time. Second, the method's effectiveness is somewhat model-dependent; Kimi-Audio showed smaller gains and higher sensitivity to hyperparameters compared to Audio-Flamingo-3. The noise reference, while effective, is a heuristic; it assumes that neurons responding to "non-noise" are the acoustic ones, which might not capture all nuanced acoustic features equally well. Finally, the method is evaluated only on multiple-choice and free-text description tasks; it does not address potential negative impacts on semantic understanding or other modalities, though the authors claim no deterioration in semantic tasks.
This work opens a new direction for improving multimodal models by focusing on the encoder side, which has been largely ignored in favor of LLM-centric interventions. It demonstrates that fine-grained, neuron-level steering within encoders can be highly effective for specific perceptual tasks. This could lead to more efficient and targeted methods for enhancing multimodal capabilities without the cost of retraining. However, the ability to easily steer model behavior via activation amplification also raises questions about robustness and potential adversarial vulnerabilities, although these are not explored in depth here. [One sentence main contribution]. The paper introduces IAAN, a training-free inference-time intervention that identifies and amplifies specific feed-forward neurons in the audio encoder of LALMs by contrasting their activations on real audio versus noise, significantly improving non-semantic acoustic perception. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. This paper makes a valuable contribution to the field of multimodal machine learning by shifting the focus of inference-time interventions from the language model backbone to the audio encoder. By proposing a simple yet effective contrastive scoring mechanism at the individual neuron level, IAAN addresses a critical gap in how LALMs process fine-grained acoustic information. The rigorous experimental evaluation, including comprehensive ablations against other intervention loci and granularities, strongly validates the necessity of encoder-side, neuron-level targeting. The significant performance gains across multiple state-of-the-art LALMs on a challenging benchmark (VoxParadox) demonstrate the practical utility of this approach. While the computational cost of dual encoder passes is a minor drawback, the method's label-free and training-free nature makes it a highly attractive tool for enhancing the acoustic grounding of existing models.
Speech enhancement (SE) can substantially improve perceptual quality, yet enhanced speech does not necessarily improve automatic speech recognition (ASR). Existing remedies, such as retraining the enhancer jointly with recognizer or interpolating enhanced speech with the noisy input, can mitigate this mismatch, but common explanations such as artifacts and over-suppression remain qualitative and do not localize which enhancement component harms recognition. We propose inference time polar projection, a diagnosis for STFT domain enhancement. Given a mask $M=Ae^{jφ}$, polar projection forms $M_{α,γ}=A^αe^{jγφ}$, where $α$ controls magnitude strength and $γ$ controls phase correction. Sweeping these controls on frozen SE and ASR models turns ASR degradation into measurable magnitude and phase effects. Our projection analysis shows that magnitude strength is the operative axis, while estimated phase correction provides no recognition benefit. The optimal magnitude strength is recognizer dependent: waveform-input wav2vec2.0 favors strong correction, whereas log-Mel-input, noise-robust Whisper prefers weaker correction. Finally, the projection provides a simple mitigation for any SE front end in the STFT mask domain, without retraining either the enhancer or the recognizer, making it directly useful for voice assistants and agents that rely on enhanced speech.
Primary: University of Illinois Urbana-Champaign
All Institutions: University of Illinois Urbana-Champaign, Wuhan University
The paper presents a rigorous diagnostic framework for understanding the mismatch between speech enhancement quality and automatic speech recognition performance, demonstrating that magnitude strength is the critical factor and that phase correction is largely superfluous for recognition tasks.
The paper introduces "Inference Time Polar Projection," a diagnostic framework that factorizes STFT-domain complex masks into independent magnitude and phase controls. By parameterizing the mask as $M_{\alpha,\gamma} = A^\alpha e^{j\gamma\phi}$, the authors decouple the effects of magnitude scaling ($\alpha$) and phase correction ($\gamma$) on Automatic Speech Recognition (ASR) performance. This approach is methodologically sound and offers a rigorous way to dissect the black-box nature of Speech Enhancement (SE) models. The derivation of the effective mask for non-multiplicative models (like MossFormerGAN-SE) is a clever technical addition that extends the applicability of the diagnosis. However, the core concept of sweeping mask parameters is not entirely new; it builds on existing work in mask exponentiation and observation adding, though the specific disentanglement of magnitude vs. phase for ASR diagnosis is a distinct contribution.
The experimental evaluation is comprehensive and well-structured. The authors test on the VoiceBank+DEMAND benchmark, which is standard for this domain, using six different SE models (both discriminative and generative) and two distinct ASR backbones (Whisper and wav2vec 2.0). The results clearly demonstrate the SE-ASR mismatch and validate the polar projection hypothesis: magnitude strength is the primary driver of ASR performance, while phase correction provides negligible benefit. The analysis of recognizer dependence (wav2vec 2.0 favoring strong suppression vs. Whisper preferring mild correction) is insightful and aligns with the known robustness differences between waveform-based and feature-based models. The statistical significance testing (paired bootstrap) adds rigor to the claims.
The paper provides sufficient detail for reproduction. It specifies the datasets (VoiceBank+DEMAND), the specific SE models used (with references to ClearerVoice-Studio implementations), and the ASR models. The mathematical formulation of the polar projection is clear. However, the code for the "effective mask" calculation for MossFormerGAN-SE is described conceptually but not provided in the text, which might require some inference to implement exactly. The use of standard benchmarks and publicly available models enhances reproducibility.
The primary limitation is the scope of the evaluation. The experiments are conducted on a single dataset (VoiceBank+DEMAND), which, while standard, may not generalize to more complex real-world conditions with heavy reverberation or diverse noise types not present in the test set. The authors acknowledge this in the limitations section. Additionally, the diagnosis is limited to STFT-domain enhancements; while they extend it to effective masks for other architectures, it may not apply to pure time-domain models that do not operate in the spectral domain. The phase result, while robust for the tested models, is specific to the estimated phase of those enhancers and does not rule out the potential utility of explicitly modeled phase in other contexts.
This work has significant implications for the deployment of SE systems in voice assistants and agents. By showing that phase correction is largely unnecessary for ASR and that magnitude strength is recognizer-dependent, it suggests that SE systems can be simplified or calibrated specifically for machine listening, potentially reducing computational cost and improving robustness. It shifts the paradigm from optimizing for perceptual quality (PESQ, SI-SDR) to optimizing for recognition utility, which is crucial for the next generation of speech-centric AI applications. The paper presents a rigorous diagnostic framework for understanding the mismatch between speech enhancement quality and automatic speech recognition performance, demonstrating that magnitude strength is the critical factor and that phase correction is largely superfluous for recognition tasks.
Zero-shot instrument cloning aims to render an arbitrary [Target MIDI] sequence with the acoustic identity of an unseen instrument given only a short [Reference Audio, Reference MIDI] pair. Existing methods rely on pre-trained embeddings (e.g., CLAP) that compress the reference audio into a fixed-length vector, discarding fine-grained acoustic cues essential for faithful timbre reconstruction. We present Anysynth, an embedding-free neural synthesizer based on in-context flow matching. By conditioning a Diffusion Transformer (DiT) directly on the uncompressed reference audio and target MIDI, our model allows self-attention to dynamically retrieve acoustic details at generation time. Experiments show that \tool outperforms embedding-based and auto-regressive baselines in audio quality, timbre similarity, and melody adherence. Notably, the model exhibits prompt-length scaling: longer reference prompts yield steadily better timbre fidelity, a property absent in embedding-based systems. To optimize controllability, we further propose Asymmetric Hierarchical CFG, which structurally decouples MIDI and reference-timbre guidance based on their natural semantic-acoustic dependency. This asymmetric formulation avoids gradient conflicts and improves both note accuracy and timbre fidelity, pushing the boundary of expressive, zero-shot instrument cloning. Demo audios are available at https://anysynth-demo.github.io/
Primary: Central Media Technology Institute
All Institutions: Central Media Technology Institute, Chinese University of Hong Kong
The paper presents a significant advancement in zero-shot instrument cloning by replacing information-bottleneck embeddings with in-context learning in a DiT architecture, supported by a novel Asymmetric CFG strategy that effectively decouples semantic and acoustic guidance.
The paper proposes "Anysynth," an embedding-free neural synthesizer for zero-shot instrument cloning. The core methodological innovation lies in reformulating the task as an In-Context Learning (ICL) problem using a Diffusion Transformer (DiT) with Flow Matching. Instead of compressing reference audio into a fixed-length embedding (like CLAP), the model conditions directly on the uncompressed reference mel spectrogram and MIDI as a prefix. This allows the self-attention mechanism to dynamically attend to fine-grained acoustic details (transients, room reverb) from the reference. A key technical contribution is the "Asymmetric Hierarchical Classifier-Free Guidance" (CFG). The authors argue that standard symmetric CFG causes gradient conflicts between semantic (MIDI) and acoustic (timbre) conditions. By factorizing the probability $p(c_m, c_r | x_t)$ as $p(c_r | c_m, x_t)p(c_m | x_t)$, they decouple the guidance, applying strong structural guidance first and then refining with timbre. This is a theoretically sound and practically effective modification to standard diffusion/flow matching conditioning strategies. The MIDI encoding is also enhanced with a binary onset channel to better capture note boundaries, addressing a common failure mode in piano roll representations.
The experimental setup is robust, utilizing a mix of synthetic datasets (NSynth, Slakh) and real-world recordings (MAESTRO, GuitarSet). The evaluation metrics are comprehensive, including PANNs for timbre similarity, Onset F1 for melody adherence, MERT for perceptual quality, and Audiobox Aesthetics for overall quality. The results demonstrate that Anysynth outperforms embedding-based baselines (CLAP-conditioned DiT) and autoregressive/token-based baselines (TokenSynth, CTD). A significant finding is the "prompt-length scaling" property: unlike embedding-based models which saturate quickly, Anysynth's performance improves with longer reference prompts, validating the ICL approach. The ablation studies effectively isolate the benefits of in-context conditioning over CLAP embeddings and validate the superiority of Asymmetric CFG over Combined and Symmetric CFG variants. The UMAP visualizations provide qualitative support for the clustering of timbres.
The paper provides detailed implementation specifics, including the DiT architecture (1024 hidden dim, 25 layers, 16 heads), training hyperparameters (batch size, learning rate, warmup), and dataset construction details (filtering rules for NSynth/Slakh). The use of standard components (Vocos vocoder, RoPE, AdaLN) aids reproducibility. However, the code repository is not explicitly linked in the text (only a demo page), which is a minor hurdle for immediate reproduction, though the architectural details are sufficient for a competent researcher to implement.
The paper acknowledges that performance gains from prompt length diminish after 8 seconds, suggesting a limit to the utility of very long prompts for short generation tasks. The model is evaluated only on keyboard, guitar, and bass instruments; generalization to more complex orchestral instruments or vocals is not demonstrated. The reliance on a pre-trained vocoder (Vocos) means that the quality of the final waveform is partially dependent on the vocoder's capability, although this is standard practice. The computational cost of attending to long reference sequences in a DiT might be higher than embedding-based methods during inference, although the authors claim no major overhead due to parallel computation.
This work advances the field of controllable audio generation by providing a more faithful and flexible method for instrument cloning. It addresses a critical bottleneck in current systems (information loss in embeddings) and offers a scalable solution (ICL) that improves with more data/context. This has implications for music production tools, virtual instrument synthesis, and audio restoration. The Asymmetric CFG technique is generalizable to other multimodal generation tasks where semantic and stylistic/acoustic conditions are entangled. The paper presents a significant advancement in zero-shot instrument cloning by replacing information-bottleneck embeddings with in-context learning in a DiT architecture, supported by a novel Asymmetric CFG strategy that effectively decouples semantic and acoustic guidance.
Automated assessment of second language (L2) speaking proficiency relies on large-scale annotated speech data, which remains scarce compared to widely available written learner corpora. A promising direction for addressing this imbalance is to use text-to-speech (TTS) and voice cloning to convert written L2 production into synthetic speech. However, written and spoken L2 differ fundamentally: spontaneous speech includes disfluencies and discourse markers, while writing is more planned and complex. This raises the question of what is required to generate synthetic L2 speech suitable for assessment. We address this through a systematic analysis of speaker-text relationships using COREFL, a publicly available corpus containing paired spoken and written responses from the same L2 learners to the same questions across modalities. In our proposed framework, we first address the structural differences between written and spoken language by transforming written responses into spoken-style transcripts ("speechification") using a large language model. These transcripts are then converted into speech using a TTS/voice-cloning model. To assign a voice to each synthetic response, we investigate different speaker-text pairing strategies based on shared learner attributes (proficiency level, first language, both, or neither). We evaluate our data augmentation techniques on the language assessment task, with improvements shown in both wav2vec2 (audio-based) and ModernBERT (text-based) scoring systems. Results show that matching speakers and texts by proficiency level yields the most robust synthetic speech. Moreover, raw written text leads to a strong mismatch with spoken language, while speechification substantially reduces this gap and improves grading performance.
Primary: University of Cambridge
All Institutions: University of Cambridge
The paper presents a systematic framework for augmenting L2 speaking assessment data by speechifying written texts and synthesizing speech, demonstrating that matching proficiency levels and modeling spoken linguistic structures significantly improves downstream grading performance.
The paper proposes a data augmentation framework for L2 speaking assessment by converting written learner essays into synthetic speech. The core methodological contribution is the "speechification" step, where an LLM (GPT-4.1) rewrites written text into spoken-style transcripts incorporating disfluencies, hesitations, and learner errors, conditioned on CEFR proficiency levels. This is followed by zero-shot voice cloning using OmniVoice. The study systematically evaluates speaker-text pairing strategies (matching by CEFR, L1, both, or neither) to determine the optimal conditions for generating synthetic data that mimics real L2 speech characteristics. The approach is well-motivated by the scarcity of annotated L2 speech data and the structural differences between written and spoken language.
The authors utilize the COREFL corpus, which provides paired written and spoken responses, allowing for a controlled analysis of modality gaps. They evaluate the synthetic data using two downstream grading models: wav2vec2 (audio-based) and ModernBERT (text-based). The experiments demonstrate that speechification significantly improves performance, particularly for the text-based grader, by reducing the linguistic mismatch. Matching speakers and texts by proficiency level was found to be the most robust strategy. The results show consistent improvements in PCC, SRC, and RMSE metrics when using augmented data. However, the evaluation relies heavily on automatic metrics (WER, PCC, SRC) and speaker verification cosine similarity, lacking subjective human evaluation (e.g., MOS) for speech naturalness or assessment accuracy, which is a notable gap for TTS research.
The paper provides detailed descriptions of the pipeline, including the LLM prompt for speechification, the TTS model (OmniVoice), and the grading architectures. The COREFL dataset is publicly available. The code for the grading systems is not explicitly linked, but the model architectures (wav2vec2, ModernBERT) are standard. The specific hyperparameters are mentioned to be in a table (referenced but not fully detailed in the snippet provided, though standard practices are implied). The cost of LLM usage is noted, adding transparency. Overall, the methodology is clear enough for reproduction, assuming access to the proprietary OmniVoice model or a suitable open-source alternative.
The reliance on GPT-4.1 for speechification introduces a dependency on a specific, potentially costly, and non-deterministic component. The synthetic speech, while linguistically improved via speechification, may still lack the acoustic variability and prosodic nuances of real spontaneous speech, as evidenced by the performance gap between real and synthetic data. The study focuses on English L2 learners, limiting generalizability to other languages without further validation. The lack of human evaluation for the synthetic speech quality and the grading performance limits the confidence in the "realism" of the data from a pedagogical perspective. The use of zero-shot TTS means the synthetic voices are clones but may not capture the full idiosyncratic speech patterns of the original speakers.
This work has significant potential to address the data scarcity problem in automated language assessment, particularly for L2 speaking. By enabling the generation of large-scale, annotated synthetic speech data from abundant written corpora, it could accelerate the development of more robust and fair assessment tools. However, ethical considerations regarding the use of synthetic data in high-stakes testing environments must be addressed, including issues of bias, transparency, and the potential for over-reliance on synthetic data that may not fully represent human speech variability. The paper presents a systematic framework for augmenting L2 speaking assessment data by speechifying written texts and synthesizing speech, demonstrating that matching proficiency levels and modeling spoken linguistic structures significantly improves downstream grading performance.
Generative streaming models for Target Speaker Extraction (TSE) commonly exhibit a quality--intelligibility trade-off, wherein naive optimization for perceptual audio quality tends to degrade speech intelligibility, and conversely. We reveal that this trade-off arises not from the constraints of streaming architectures, but from an inappropriate choice of optimization anchor. Directly optimizing against audio quality metrics induces catastrophic reward hacking, where content critical to pronunciation and intelligibility is systematically erased to maximize a proxy score. To break this bottleneck, we propose two complementary improvements: an enlarged Conformer convolution kernel for richer local spectro-temporal modeling, and WavLM-anchored Direct Preference Optimization (DPO) fine-tuning strategy. DPO preference pairs are ranked by WavLM cosine similarity, a deep acoustic feature encoding both phonetic structure and speaker identity, providing an optimization anchor that resists hacking. Under a 560 ms streaming chunk size, the proposed method achieves a 10.9% relative intelligibility improvement (word error rate: 0.138 to 0.123), with marginal simultaneous gains in audio quality and speaker similarity.
Primary: Tsinghua University
All Institutions: Tsinghua University, The Chinese University of Hong Kong, SenseTime
The paper makes a significant contribution to streaming TSE by identifying and resolving the reward hacking problem in DPO through deep-feature anchoring, achieving a Pareto improvement in quality and intelligibility.
The paper proposes a novel approach to the quality-intelligibility trade-off in streaming Target Speaker Extraction (TSE). The core methodological contribution is the identification and mitigation of "reward hacking" in Direct Preference Optimization (DPO) when applied to generative speech models. The authors correctly diagnose that optimizing for perceptual metrics (DNSMOS) leads to intelligibility collapse, while optimizing for Word Error Rate (WER) fails to improve quality. Their solution involves two steps: architectural (enlarging Conformer kernel size to $k=15$ for better local context) and algorithmic (using WavLM cosine similarity as the preference anchor for DPO). The separation of the semantic pathway (optimized via DPO) and acoustic pathway (frozen) is a sound engineering decision to prevent semantic drift during acoustic refinement. The use of WavLM, which encodes both phonetic and speaker information, as a proxy for "good" speech is a clever heuristic that effectively balances the competing objectives.
The experimental setup is rigorous, utilizing the Libri2Mix dataset and standard metrics (DNSMOS, WER, Speaker Similarity). The ablation studies are particularly strong, clearly demonstrating the failure modes of $DPO_{DNSMOS}$ (intelligibility collapse) and $DPO_{WER}$ (no gain, quality drop), thereby validating the necessity of the $DPO_{WavLM}$ approach. The results show a statistically significant improvement in WER (10.9% relative) with simultaneous gains in quality and speaker similarity. The inclusion of Inference Success Rate (ISR) is a valuable addition for streaming contexts. The evaluation covers multiple KL penalty coefficients and checkpoints, providing a robust analysis of hyperparameter sensitivity.
The paper provides sufficient implementation details, including model architecture (Conformer kernel size, SELM/ARLM layers), training hyperparameters (learning rate, optimizer), and dataset specifications. The use of standard libraries (FunCodec, Whisper, WavLM) aids reproducibility. However, the exact code for the "Historical Context Refinement" mechanism and the specific stochastic sampling procedure for generating preference pairs could be more detailed. The claim of "catastrophic reward hacking" is well-supported by the provided analysis (spectrograms and metric trends), which enhances trust in the reproducibility of the findings.
The primary limitation is the reliance on WavLM as a proxy for human preference. While effective, WavLM is a self-supervised model trained on large-scale speech data, and its alignment with human perceptual quality is not perfect. The method is evaluated only on Libri2Mix, a relatively clean, two-speaker mixture dataset. Real-world scenarios with heavy noise, reverberation, and more than two speakers are not addressed. Furthermore, the streaming latency is 560ms, which is acceptable for many applications but not for strict real-time (<200ms) interaction, as noted by the authors. The computational cost of generating 16 candidates for DPO training is also non-trivial.
This work has significant implications for the deployment of generative AI in real-time communication systems, such as teleconferencing and voice assistants. By breaking the quality-intelligibility trade-off, it enables systems that are both natural-sounding and easy to understand. The insight regarding reward hacking in generative speech models is broadly applicable to other text-to-speech and audio generation tasks. It highlights the danger of using naive proxy metrics for optimization and advocates for more semantically aware anchors. The paper makes a significant contribution to streaming TSE by identifying and resolving the reward hacking problem in DPO through deep-feature anchoring, achieving a Pareto improvement in quality and intelligibility.
Melody skeleton extraction aims to derive a shorter melody that preserves structural notes while removing ornaments. Prior methods rely on hand-crafted reduction rules or note-wise salience classifiers trained with heuristically or procedurally generated pseudo-labels. Such supervision can inherit generator bias and does not explicitly optimize a coherent reduced melody. We introduce MeloBottleneck, a self-supervised framework that represents a skeleton as a length-controlled, order-preserving latent subsequence. A hard-bottleneck extractor selects note events, a rhythmic-closure operator produces a self-consistent skeleton, and a re-ornamentation decoder reconstructs the input melody. Training combines reconstruction, a frozen autoregressive melody prior, ornament-invariant consistency across procedurally ornamented views, and ornament exclusion. We evaluate three regimes: synthetic out-of-distribution ornament-to-skeleton, TAVERN variation-to-theme, and Jiugong ornamented-to-gongche. A matched pseudo-label classifier excels on the synthetic benchmark, while MeloBottleneck transfers better, achieving competitive selection quality on TAVERN and Jiugong. Skeletonized melodies also improve BM25-based fragment retrieval, boosting Recall@K and MRR while reducing query time. Overall, the results suggest that learning skeletons as latent subsequences yields more robust transfer than pseudo-label imitation.
Primary: Beijing University of Posts and Telecommunications
All Institutions: Beijing University of Posts and Telecommunications, Central Conservatory of Music
MeloBottleneck introduces a novel self-supervised framework for melody skeleton extraction that models skeletons as latent subsequences, demonstrating superior cross-domain transfer and practical utility in music retrieval compared to prior pseudo-label or heuristic methods.
The paper proposes MeloBottleneck, a self-supervised framework for melody skeleton extraction. The core innovation lies in formulating the skeleton not as a set of independent note-wise labels, but as a length-controlled, order-preserving latent subsequence. This is a significant methodological shift from prior work that relies on heuristic rules or pseudo-label classifiers. The architecture employs a Transformer encoder to select top-K notes, followed by a deterministic "rhythmic closure" operator that converts the sparse subsequence into a standalone melody by absorbing deleted time into preceding notes. Training is multi-objective: it includes reconstruction loss (re-ornamentation), a frozen autoregressive melody prior for musicality, and ornament-invariant consistency losses to ensure stability under procedural ornamentation. The approach effectively decouples note selection from rhythmic coherence, addressing a key limitation in previous discrete latent compression methods for music.
The evaluation is comprehensive, covering three distinct regimes: synthetic out-of-distribution (OOD) ornament-to-skeleton, cross-domain variation-to-theme (TAVERN), and in-domain ornamented-to-gongche (Jiugong). The results demonstrate a clear trade-off: while a pseudo-label classifier excels on synthetic data where it directly imitates the generator, MeloBottleneck shows superior transferability to zero-shot and cross-domain settings. This supports the central claim that learning structural representations is more robust than imitation. Additionally, the paper provides a practical downstream application, showing that skeletonized melodies improve BM25-based fragment retrieval by reducing noise and query time. The ablation studies rigorously validate the contribution of each loss component, particularly the ornament exclusion and consistency losses.
The paper provides detailed implementation specifics, including the SimpleMono event representation, hyperparameters for the Transformer backbone (6 encoder/3 decoder layers), and specific loss coefficients. The authors also provide a link to supplementary materials including code and a demo web page, which significantly enhances reproducibility. The use of standard symbolic music representations and clear algorithmic descriptions of the rhythmic closure and selection mechanisms makes the method straightforward to implement.
The authors acknowledge several limitations. First, the method relies on a procedural ornamenter for training views; if this ornamenter does not match the target style, the invariance signal may be suboptimal. Second, the evaluation of selection quality uses an oracle compression ratio, meaning free-length prediction is only indirectly tested. Third, the rhythmic closure operator is simple (absorbing time into the preceding note), which may not preserve musical phrasing as well as more complex closure strategies. Finally, the focus is strictly on monophonic melodies, limiting immediate applicability to polyphonic music without significant extension.
This work has significant implications for symbolic music analysis, retrieval, and compression. By providing a robust method to extract structural skeletons, it facilitates better theme tracing, variation recognition, and efficient indexing of large music corpora. The self-supervised nature of the approach reduces reliance on expensive human annotations, making it scalable to diverse musical traditions. The improved retrieval performance suggests practical utility in digital library systems and music information retrieval tools. MeloBottleneck introduces a novel self-supervised framework for melody skeleton extraction that models skeletons as latent subsequences, demonstrating superior cross-domain transfer and practical utility in music retrieval compared to prior pseudo-label or heuristic methods.
Phone segmentation and recognition are inherently related tasks, yet modern approaches typically model them separately. We argue that phonetic structure is already latent in the representations of self-supervised speech models (S3Ms), and one only needs to steer them to solve both tasks. We leverage S3M-based Phonological Activation Mapping (SPAM), which maps each S3M representation frame to a vector of phonological feature activations, such as voicing and nasality. On top of SPAM, we introduce two simple but effective lightweight, gradient-descent-free prediction heads: a recognition head and a segmentation head. Our method requires less than a minute of phonetic transcriptions, and generalizes to unseen phones during training. Across a diverse range of datasets, our approach attains strong segmentation and recognition performance.
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University, University of Texas at Austin, University of Tokyo, University of British Columbia
This paper presents a highly innovative, sample-efficient approach to phone segmentation and recognition by steering self-supervised speech models through phonological activation mapping, achieving strong generalization with minimal supervision.
The paper proposes a novel, gradient-descent-free framework for phone segmentation and recognition called SPAM (S3M-based Phonological Activation Mapping). The core innovation lies in leveraging the latent phonetic structure of Self-Supervised Speech Models (S3Ms), specifically WavLM, by projecting frame-level representations onto learned phonological feature vectors (e.g., voicing, nasality) derived from the PanPhon library. Instead of training a deep classifier, the authors design two simple heads: a recognition head that performs nearest-neighbor matching against canonical PanPhon vectors, and a segmentation head that uses an ensemble of dissimilarity signals (multi-scale differences, backward contrasts, mel-spectrogram differences) to detect boundaries. This approach is theoretically elegant, as it treats phonetic structure as a geometric property of the S3M space rather than a classification target requiring massive supervision. The method is highly interpretable and sample-efficient.
The evaluation is comprehensive, covering phone segmentation (R-value) and recognition (PFER) across a diverse set of in-domain (TIMIT, Buckeye) and out-of-domain datasets (GTIMIT-S/T, TORGO, SSNCE, Voxangeles, GTIMIT-Thai). The results demonstrate that SPAM significantly outperforms standard supervised baselines (CTC, FCE, BCE) trained on TIMIT, particularly in low-resource and multilingual settings. For instance, on the challenging Voxangeles dataset, SPAM achieves a PFER of 24.3 compared to 40.2 for FCE. The paper also includes rigorous ablation studies on sample efficiency (showing stability with <1 minute of data), segmentation signal contributions, and S3M layer/model choices. The comparison against strong toplines (KoelLabs-XLSR, PhoneticXeus) highlights SPAM's competitive performance despite using negligible training data and compute.
The authors provide code for modeling, evaluation metrics, and the benchmark suite. The methodology relies on standard libraries (PanPhon, MFA) and publicly available S3Ms (WavLM). The deterministic nature of the gradient-descent-free heads enhances reproducibility. However, the exact hyperparameters for the ensembling weights in the segmentation head are not fully detailed in the text, though the logic is clearly explained. The reliance on specific S3M layers (final layer of WavLM) is well-specified.
The method is currently limited to phones with defined consonantal values in PanPhon, excluding tones and potentially some complex phonological phenomena. The segmentation head, while effective, relies on heuristic ensembling of dissimilarity signals which may not generalize as robustly as learned boundary detectors in highly noisy or non-speech audio. The recognition head's performance is bottlenecked by the segmentation head; as shown in the oracle ablation, perfect segmentation drastically reduces error, indicating that the boundary detection is the primary area for improvement. Additionally, the method is not end-to-end differentiable in its current form, limiting potential joint optimization.
This work has significant implications for low-resource speech processing, clinical speech assessment, and the documentation of endangered languages. By requiring less than a minute of labeled data and generalizing to unseen phones and languages, SPAM lowers the barrier to entry for phonetic analysis. It promotes the use of interpretable, linguistically grounded representations in deep learning, moving away from black-box sequence-to-sequence models. This could facilitate more accessible tools for linguists and clinicians. This paper presents a highly innovative, sample-efficient approach to phone segmentation and recognition by steering self-supervised speech models through phonological activation mapping, achieving strong generalization with minimal supervision.
Large language model (LLM)-based audio-visual speech recognition (LLM-AVSR) has recently demonstrated strong robustness in adverse acoustic environments by leveraging complementary audio and visual information. Existing approaches typically employ independently pretrained acoustic and visual encoders, whose outputs are projected and fused as soft prompts to condition an LLM for speech recognition. However, most methods perform multimodal fusion without explicitly addressing the representational discrepancy between audio, visual and text modalities, potentially limiting the effectiveness of cross-modal integration. In this paper, we propose an optimal transport (OT)-based semantic alignment framework for LLM-AVSR. The proposed method explicitly bridges the modality gap by aligning the acoustic and visual representations with reference to the linguistic embedding space of the LLM before multimodal fusion. Specifically, OT is used to estimate probabilistic coupling matrices that characterize structured correspondences between modality-specific features and linguistic embeddings. The resulting OT couplings are further utilized as soft pseudo-labels to supervise contrastive learning, encouraging the extraction of semantically coherent and cross-modal consistent audio-visual representations. By anchoring multimodal features to the linguistic space of the LLM, the proposed framework facilitates more effective multimodal fusion and decoding. We implement the proposed framework using a Whisper-based acoustic encoder, an AV-HuBERT-based visual encoder, and a LLaMA3.2-3B decoder. Experiments conducted on the LRS3-TED benchmark demonstrate consistent improvements over strong baselines and achieve state-of-the-art performance under both clean and noisy evaluation conditions across a wide range of signal-to-noise ratios (SNRs).
Primary: National Institute of Information and Communications Technology
All Institutions: National Institute of Information and Communications Technology, Research Center for Information Technology Innovation, Academia Sinica
The paper presents a method for aligning audio-visual features with linguistic embeddings in LLM-based ASR using Optimal Transport, demonstrating improved robustness and state-of-the-art results on LRS3-TED.
The paper proposes an Optimal Transport (OT)-based semantic alignment framework for LLM-based Audio-Visual Speech Recognition (LLM-AVSR). The core methodological contribution is the use of entropy-regularized OT to compute probabilistic coupling matrices between acoustic/visual features and linguistic token embeddings. These couplings are then used as soft pseudo-labels to supervise a contrastive learning objective, aiming to bridge the representational gap between modality-specific encoders (Whisper, AV-HuBERT) and the LLM decoder (LLaMA3.2-3B). The approach also introduces "virtual buckets" in the OT formulation to handle non-informative frames (noise/silence). While the application of OT to multimodal alignment is not entirely new, its specific integration into the pre-fusion stage of LLM-AVSR to align with the linguistic embedding space is a distinct methodological choice. The use of OT couplings as supervision for contrastive learning is a clever mechanism to enforce semantic consistency without requiring hard alignments. However, the novelty is somewhat tempered by the fact that OT-based alignment has been explored in other multimodal contexts, and the specific adaptation here is incremental rather than foundational.
The experimental evaluation is conducted on the LRS3-TED benchmark, which is a standard dataset for AVSR. The authors evaluate performance under clean and noisy conditions (various SNRs). They compare against strong baselines and ablate different fusion strategies (concatenation, addition, cross-attention, Q-former) and hyperparameters. The results show consistent improvements over baselines, achieving state-of-the-art performance on LRS3-TED. The inclusion of noisy evaluation is crucial for AVSR and demonstrates the robustness of the method. However, the paper lacks comparison with some very recent LLM-AVSR baselines that might use different architectural designs (e.g., direct projection vs. complex fusion). The ablation studies are present but could be more extensive regarding the sensitivity of the OT parameters. The results are solid but not overwhelmingly superior to the baselines, suggesting that while effective, the gains are marginal in some settings.
The paper provides detailed descriptions of the model architecture, including the encoders (Whisper, AV-HuBERT), the LLM (LLaMA3.2-3B), and the OT alignment module. The loss functions and optimization strategies (Adam, LoRA) are specified. The dataset splits and preprocessing steps (lip cropping, audio downsampling) are described. However, the code is not publicly available (no project URL provided), and some hyperparameter tuning details are empirical. The "virtual bucket" mechanism is described but the exact implementation details (e.g., how the similarity margin $s$ is dynamically adjusted or fixed) could be clearer. Overall, reproducibility is moderate to high given the detailed mathematical formulation, but the lack of code release hinders immediate verification.
The method introduces additional computational overhead due to the OT calculation (Sinkhorn iterations) during training. The paper acknowledges that hyperparameter tuning for the OT and contrastive learning components is non-trivial and sensitive. The "virtual bucket" mechanism, while effective, adds complexity to the OT formulation. The evaluation is limited to the LRS3-TED dataset; performance on other AVSR benchmarks (e.g., LRS2, VoxCeleb) is not reported, limiting the generalizability claims. The method relies on pretrained encoders, so improvements are constrained by the quality of these base models.
This work contributes to the field of multimodal learning, specifically in robust speech recognition. By improving the alignment between audio, visual, and linguistic modalities, it enhances the reliability of AVSR systems in adverse environments, which has practical applications in assistive technologies, secure communications, and human-computer interaction. The use of OT for semantic alignment could inspire similar approaches in other multimodal tasks where cross-modal correspondence is challenging. The paper presents a method for aligning audio-visual features with linguistic embeddings in LLM-based ASR using Optimal Transport, demonstrating improved robustness and state-of-the-art results on LRS3-TED.
Multimodal Large Language Models (LLMs) have remarkable semantic audio understanding, yet they remain "spatially agnostic" due to their reliance on mono-channel audio representations. Currently, spatial audio perception methods mainly focus on complex room simulations and custom-trained, geometry-aware stereo encoders, which limits their accessibility and generalizability. In this paper, we introduce the Dual-BEATs architecture, in which the left and right audio channels are routed independently through two identical semantic encoders as an alternative to specialized spatial modules. To circumvent the architectural bottleneck where internal normalization otherwise erases the inter-channel variance of stereo audio, we inject a static, uncorrelated dithering noise floor prior to encoding. This dithering intervention establishes a macro-variance floor that "smuggles" spatial geometry across the normalization layers. Evaluated on a ternary directional classification task (Left, Center, Right), we demonstrate that dithered models achieve exceptional spatial resolution--reaching up to 97.2% localization accuracy even on subtle 0.5 panning amplitudes--and demonstrates robust, zero-shot generalization to entirely unseen spatial configurations. Our results suggest that with the appropriate acoustic regularization, standard multimodal models are natively capable of generalized stereo audio understanding.
Primary: Institute of Information Science, Academia Sinica
All Institutions: Institute of Information Science, Academia Sinica
The paper introduces Dual-BEATs, a novel architecture that unlocks zero-shot stereo perception in Audio LLMs by using uncorrelated dithering noise to bypass normalization-induced spatial information loss, achieving high-resolution spatial localization without specialized encoders.
The paper proposes a novel architectural intervention called "Dual-BEATs" to address the "spatial agnosticism" of standard Multimodal LLMs. The core insight is that standard normalization layers (LayerNorm/RMSNorm) act as compressors that erase subtle inter-channel amplitude differences (ICLD) essential for stereo perception. To bypass this, the authors inject static, uncorrelated Gaussian dithering noise into the left and right audio channels before encoding. This creates a "macro-variance floor" that allows the normalization layers to preserve the relative amplitude delta between channels, effectively "smuggling" spatial geometry into the LLM's latent space. The approach is theoretically grounded in Stochastic Resonance and avoids the need for complex, custom-trained spatial encoders or 3D room simulations. The methodology is elegant, leveraging existing frozen semantic encoders (BEATs) and standard LLM backbones (Gemma, OLMo) with minimal modification (QLoRA adapters).
The evaluation is rigorous and well-designed. The authors construct a controlled dataset using AudioSet samples, applying deterministic amplitude panning to create stereo pairs with varying panning amplitudes (PA). They evaluate on a ternary classification task (Left, Center, Right) and, crucially, test zero-shot generalization across unseen panning amplitudes. The results are compelling: undithered models suffer a "Center-Pan Crash" (performance drops to random chance for center-panned audio), while dithered models maintain high accuracy (>97%) across the entire spectrum, including subtle 0.5 PA shifts. The ablation studies on correlated vs. uncorrelated noise effectively demonstrate that the model is learning true spatial geometry rather than exploiting noise artifacts. The trade-off between spatial resolution and semantic retention ("Semantic Tax") is also quantified, showing a manageable ~5% drop in semantic F1.
The paper provides extensive implementation details, including specific hyperparameters for dithering amplitude (DA=0.05), hardware specifications (V100/H100), and optimization settings (QLoRA, AdamW). The dataset curation process is described in detail, referencing specific snapshots of AudioSet. The strict isolation of random seeds for training and evaluation is a strong point for reproducibility and validity, preventing shortcut learning. The code is not publicly linked, but the method is simple enough to be reimplemented.
The primary limitation is the reliance on simple amplitude panning (ICLD) rather than full binaural rendering (which includes Interaural Time Differences - ITD and Head-Related Transfer Functions - HRTFs). This limits the ecological validity of the spatial cues compared to real-world binaural recordings. Additionally, the evaluation is restricted to single-source scenarios; multi-source separation and localization are not addressed. The "Semantic Tax" indicates that while spatial perception is unlocked, it comes at the cost of some semantic fidelity, which may be a concern for downstream tasks requiring high-precision semantic understanding.
This work has significant implications for the development of spatially-aware AI assistants, accessibility tools for the visually impaired, and immersive media applications. By demonstrating that standard LLMs can be endowed with spatial perception through simple signal-level interventions, it lowers the barrier to entry for spatial audio understanding. However, the dual-use nature of such technology (e.g., enhanced surveillance or tracking) is acknowledged. The approach also highlights a fundamental architectural weakness in current multimodal models, potentially guiding future model designs to better preserve physical signal properties. The paper introduces Dual-BEATs, a novel architecture that unlocks zero-shot stereo perception in Audio LLMs by using uncorrelated dithering noise to bypass normalization-induced spatial information loss, achieving high-resolution spatial localization without specialized encoders.
End-to-end neural audio models achieve high-fidelity compression and generation. We might read that performance as evidence they directly represent interpretable features such as pitch and timbre, but a model can produce plausible outputs without doing so. A model may encode these features in any reachable basis, but regardless of which, the features are well described as compositions of time-frequency-localized primitives. Whether state-of-the-art encoders preserve access to these primitives, and thus to compositions of them, remains unclear. Through theoretical analysis and controlled experiments, we show that several state-of-the-art strided convolutional encoders impose two structural bottlenecks, both predictable from architecture and signal structure, on access to these primitives: (1) they collapse primitives into alias equivalence classes, establishing a bound on representational capacity, and (2) they limit the frequency resolution available to learned filters, restricting separability. For well structured data, we find collapse rates of 31-35% and filter bandwidths 10-35x above the theoretical resolution bound, confirming that both bottlenecks arise under realistic signal conditions. We then introduce Gabor Latent Refactorization (GLRF), a lightweight post-hoc intervention that re-expresses encoder latents in a frequency-localized basis, reducing filter bandwidths from 10-35x to 1.5-3x of the theoretical resolution bound while preserving reconstruction fidelity and improving control over attributes like pitch. These results show that the encoders in question predictably degrade access to frequency-localized primitives, entangling the features that depend on them, and that a lightweight, retraining-free intervention can recover much of that access, improving steerability and interpretability.
Primary: Yale University
All Institutions: Yale University
This paper provides a compelling theoretical and empirical analysis of the representational limitations of strided convolutional audio encoders, introducing a novel post-hoc refactorization method that significantly improves frequency-specific controllability without retraining.
The paper proposes a rigorous theoretical and empirical framework to analyze the representational geometry of end-to-end audio encoders (specifically strided convolutional networks like EnCodec, DAC, and Stable Audio). The core methodological contribution is the identification of two structural bottlenecks: (1) Injectivity Failure, where downsampling causes distinct frequency components to collapse into alias equivalence classes, and (2) Separability Failure, where the receptive field limits the frequency resolution of learned filters, preventing independent access to these components. The authors derive analytical bounds for these phenomena based on signal structure and architecture parameters (stride, kernel size, dilation). To address the separability bottleneck, they introduce Gabor Latent Refactorization (GLRF), a lightweight, post-hoc intervention that re-expresses encoder latents in a frequency-localized Gabor basis using a learned linear mapping. This approach is theoretically grounded in signal processing principles (uncertainty principle, aliasing theory) and applied to modern deep learning architectures.
The experimental evaluation is comprehensive and well-controlled. The authors use synthetic signals with known ground-truth frequency components to precisely measure collapse rates and filter bandwidths, avoiding the ambiguity of natural audio analysis. They demonstrate a high correlation ($r=0.99$) between predicted and observed collapse rates across 643 signal configurations and three state-of-the-art models. The separability analysis shows that learned filters operate significantly above the theoretical resolution bound (10-35x), confirming the bottleneck. The GLRF intervention is evaluated on reconstruction fidelity (preserved) and controllability (improved). The controllability tests, including targeted component substitution and interpolation on both synthetic and real (NSynth) data, show that GLRF enables precise, frequency-specific manipulation that is absent in the original latent space. The results are robust and clearly support the claims.
The paper provides detailed descriptions of the models, signal generation procedures, and the GLRF method. The appendix includes proofs, simulation details, and code availability statements. The theoretical models for injectivity and separability are deterministic and depend only on architecture parameters, making them highly reproducible. The GLRF linear mapping is fit via closed-form ridge regression, which is stable and reproducible. The use of synthetic signals for primary validation ensures that results are not dependent on specific dataset splits or annotation errors.
The paper acknowledges several limitations. The analysis assumes signals are locally well-approximated as sums of narrowband components, which may not hold for highly transient or broadband sounds (e.g., percussion). The GLRF intervention addresses separability but not injectivity failure, which is inherent to the downsampling process and requires architectural changes. The linear mapping is fit on simple harmonic signals, and its generalization to complex, transient-rich content is not quantitatively assessed. The study focuses on strided convolutional encoders and does not evaluate models with explicit frequency-selective front-ends (e.g., SincNet).
This work has significant implications for the interpretability and controllability of neural audio systems. By demonstrating that state-of-the-art encoders systematically degrade access to interpretable frequency primitives, it challenges the assumption that high-fidelity reconstruction implies learned interpretability. The GLRF method offers a practical, retraining-free way to recover this structure, enabling better control over pitch and timbre in generation and compression tasks. This could facilitate applications in music production, speech synthesis, and scientific audio analysis. The paper also raises safety concerns regarding enhanced voice cloning capabilities, suggesting potential mitigations. This paper provides a compelling theoretical and empirical analysis of the representational limitations of strided convolutional audio encoders, introducing a novel post-hoc refactorization method that significantly improves frequency-specific controllability without retraining.
Existing methods for automatic music transcription are often limited to single-instrument recordings or fail on complex, real music mixes. Although previous work utilizes synthetic training data, the resulting models generalize poorly, leading to largely unusable transcription output in realistic, multi-instrument settings. In this work, we analyze the effectiveness of synthetic data for pre-training while combining it with fine-tuning on real music audio and post-training using reinforcement learning. We further introduce conditioning on instrument presence to customize transcriptions. Finally, we release MuScriptor, an open-weight multi-instrument music transcription model that works on real-world music recordings from across a diverse range of musical genres.
Primary: IRCAM-CNRS Sorbonne Univ
All Institutions: IRCAM-CNRS Sorbonne Univ
The paper presents MuScriptor, an open-weight multi-instrument music transcription model that achieves state-of-the-art performance on real-world music by combining synthetic pre-training, large-scale real-data fine-tuning, and reinforcement learning alignment, addressing the critical domain shift problem in automatic music transcription.
The paper proposes a practical and effective pipeline for multi-instrument automatic music transcription (AMT) using a decoder-only Transformer architecture. The core methodological contribution lies in the training strategy rather than architectural novelty: it combines large-scale synthetic pre-training (1.45M MIDIs) with fine-tuning on a newly collected dataset of 170k real-world recordings, followed by Reinforcement Learning (RL) post-training using a GRPO-like algorithm. The introduction of instrument presence conditioning at inference time is a valuable engineering choice that stabilizes predictions. The application of RL (specifically GRPO without KL penalties) to align transcription outputs with high-quality human annotations is a novel adaptation of techniques typically seen in LLMs to the audio-symbolic domain. However, the base architecture is a standard adaptation of existing sequence-to-sequence models (like MT3), limiting its theoretical novelty.
The experimental evaluation is comprehensive and robust. The authors conduct extensive ablation studies demonstrating the necessity of each training stage (synthetic pre-training, real fine-tuning, RL post-training). They show significant improvements over state-of-the-art baselines (YourMT3+, MT3) across multiple metrics (F1, Onset F1, Offset F1, Multi F1) on a diverse test set. The analysis of synthetic data effectiveness is particularly strong, providing clear evidence that synthetic data helps when real data is scarce but is insufficient on its own. The inclusion of a large, diverse real-world dataset (11k hours) addresses a critical gap in the field. The results are consistent and the improvements are substantial, particularly in generalization to real-world mixes.
The paper provides significant details for reproducibility. The authors release the model weights and inference code via GitHub. They describe the data collection process, including the use of audio-symbolic synchronization and filtering criteria. The hyperparameters for training and RL are specified. However, the exact source and licensing of the 170k real-world recordings are not fully detailed in the text (referred to as "internal dataset"), which may pose challenges for full independent replication of the data pipeline, although the model weights are available. The RL implementation details are clear enough for implementation.
The model struggles with overlapping notes of the same instrument and pitch, a common issue in dense music, which the authors acknowledge requires different tokenization. The performance on onset/offset precision for certain genres (like choral music) remains lower, indicating domain-specific challenges. The reliance on a specific instrument taxonomy (36 subgroups) limits the granularity of transcription. The RL post-training is performed on a small subset (300 tracks), which may limit the generalization of the RL benefits compared to the broader fine-tuning data.
MuScriptor represents a significant step forward for open-source music information retrieval tools. By providing a high-quality, open-weight model that works on real-world music, it lowers the barrier for researchers and practitioners to use AMT in downstream applications like music generation, education, and archival. The release of the large-scale dataset also contributes to the community, although the licensing details are crucial for broader adoption. The work highlights the importance of real-world data and alignment techniques in bridging the gap between synthetic benchmarks and practical applications. The paper presents MuScriptor, an open-weight multi-instrument music transcription model that achieves state-of-the-art performance on real-world music by combining synthetic pre-training, large-scale real-data fine-tuning, and reinforcement learning alignment, addressing the critical domain shift problem in automatic music transcription.
Training target speaker extraction (TSE) models for real conversational mixtures remains challenging because large-scale training corpora and clean target speech for supervision are unavailable. We present PS4, a proxy-supervised training framework for TSE in real conversational mixtures, with two main contributions. First, we construct a large-scale corpus of 71,771 training samples derived from four public datasets, covering both Chinese and English scenarios. Each sample contains an overlapping speech mixture, per-speaker enrollment audio, a ground-truth transcript, and frame-level voice activity labels. Second, we propose a proxy-supervised joint training strategy that fine-tunes a BSRNN-based TSE model using four complementary differentiable objectives: ASR cross-entropy, speaker similarity, frame-level voice activity detection, and perceptual audio quality. Starting from a publicly available pre-trained checkpoint, only the BSRNN separator is updated during fine-tuning. On the REAL-T challenge leaderboard, PS4 ranks 2nd overall, achieving the best speaker similarity and timing F1 among all submitted systems.
Primary: Nanjing University
All Institutions: Yijiahe AI, Nanjing, China; Tianjin University, Tianjin, China; Nanjing University, Nanjing, China
This paper presents a practical and effective proxy-supervised training framework for target speaker extraction, successfully leveraging multiple auxiliary signals to achieve state-of-the-art performance on real-world conversational speech benchmarks.
The paper proposes PS4, a proxy-supervised training framework for Target Speaker Extraction (TSE) in real conversational scenarios. The core methodological contribution is the design of four complementary differentiable loss functions to replace the unavailable clean target speech signal: ASR cross-entropy (using Whisper), speaker similarity (cosine margin ranking), frame-level Voice Activity Detection (VAD), and perceptual audio quality (DNSMOS). The approach leverages a pre-trained BSRNN-ECAPA model and fine-tunes only the separator. While the individual components (Whisper loss, speaker embedding losses) are known in the literature, their specific combination and application to the "real-world" TSE problem without clean references is a coherent and practical engineering contribution. The novelty lies in the systematic integration of these proxies to handle the domain gap between simulated and real data. EXPERIMENTAL_EVALUTION: The authors construct a new dataset, REAL-PS4, by curating four public datasets (AISHELL-4, AliMeeting, AMI, CHiME-6) to create overlapping speech mixtures with ground-truth transcripts and VAD labels. Experiments are conducted on the REAL-T benchmark. The results show consistent improvements over the official REAL-T baselines (BSRNN_TFMAP and BSRNN_EMB) across all four evaluation metrics (TER, SIM, DNSMOS OVRL, Timing F1). Notably, the system achieved 2nd place on the REAL-T challenge leaderboard, with the best speaker similarity and timing F1 scores among all submitted systems. The ablation and per-dataset analysis provide reasonable insights into where the model succeeds (cleaner recordings like AMI) and struggles (noisy/far-field like AISHELL-4).
The paper provides a link to the code repository (GitHub) and the dataset (HuggingFace). The methodology is described in sufficient detail, including the specific loss formulations and the use of frozen pre-trained models (Whisper, ECAPA-TDNN). The training setup (fine-tuning only the separator) is clearly defined. This ensures a high degree of reproducibility for other researchers.
The primary limitation is the reliance on the quality of the source datasets for the REAL-PS4 corpus. If the original diarization or transcript annotations contain errors, these propagate into the training data. Additionally, the use of black-box proxies (Whisper, DNSMOS) introduces potential misalignment; for instance, Whisper might hallucinate content or fail on heavy noise, and DNSMOS is a heuristic metric that may not perfectly correlate with human perception of extraction quality versus just audio fidelity. The paper does not include a detailed ablation study showing the marginal gain of each specific loss component, making it hard to isolate the contribution of the perceptual or VAD losses versus the linguistic/speaker losses.
This work addresses a critical bottleneck in speech processing: the lack of large-scale, real-world training data for TSE. By demonstrating that proxy supervision can effectively bridge the gap between simulated and real domains, it enables more robust deployment of TSE systems in real-world applications like meeting transcription and hearing aids. The release of the REAL-PS4 dataset also benefits the community by providing a standardized, large-scale resource for training and evaluating TSE models in realistic conditions. This paper presents a practical and effective proxy-supervised training framework for target speaker extraction, successfully leveraging multiple auxiliary signals to achieve state-of-the-art performance on real-world conversational speech benchmarks.
Audio-Visual speech recognition systems often degrade in real-world scenarios due to signal corruption and distribution shifts. To address this, we propose a unified uncertainty-modeling framework, namely the uncertainty-aware Bayesian gating network (UBG-Net). UBG-Net features a Modality Uncertainty-aware Bayesian Fusion (MUBF) mechanism that injects signal-level aleatoric uncertainty into a Bayesian network to model epistemic uncertainty, thereby ensuring robust fusion of pre-trained backbone features. For inference, we introduce Distribution Uncertainty-aware Hierarchical Voting (DUHV) to select transcripts from Monte Carlo samples, prioritizing frequency and using inference scores in case of a tie. Experiments on the AVCocktail and LRS2 datasets demonstrate the overall superiority of UBG-Net compared to SOTA baselines. Ablation studies confirm that MUBF and DUHV effectively filter noise, enhancing fusion and decoding robustness.
Primary: University of Science and Technology of China
All Institutions: University of Science and Technology of China
The paper presents a well-motivated and technically sound framework for robust audio-visual speech recognition by integrating aleatoric and epistemic uncertainty modeling through a novel Bayesian gating mechanism and hierarchical voting strategy, demonstrating significant improvements on challenging real-world and simulated benchmarks.
The paper proposes UBG-Net, an Audio-Visual Speech Recognition (AVSR) framework that integrates uncertainty modeling into the fusion process. The core methodological contribution is the Modality Uncertainty-aware Bayesian Fusion (MUBF), which attempts to bridge aleatoric (data) and epistemic (model) uncertainty. Specifically, it extracts signal-level aleatoric uncertainty (mean and variance) from audio and visual streams and concatenates them as a context vector to guide a Bayesian Gating Network (BGN). The BGN uses variational inference to model weight uncertainty, aiming to produce robust fusion weights. For inference, the authors introduce Distribution Uncertainty-aware Hierarchical Voting (DUHV), a post-processing step that selects the final transcript from Monte Carlo samples by prioritizing frequency (majority vote) and using inference scores for tie-breaking. While the integration of these components is coherent, the novelty is moderate. Bayesian fusion in multimodal learning is not new; the specific twist of using aleatoric parameters as context for epistemic gating is a logical and sensible design choice, but it does not represent a fundamental shift in uncertainty modeling theory. The approach relies on standard variational inference techniques (ELBO, reparameterization) applied to a specific architectural bottleneck.
The experimental evaluation is conducted on two benchmarks: LRS2 (with simulated noise at various SNRs) and AVCocktail (real-world multi-party conversations). The results demonstrate consistent improvements over a strong baseline (AV-HuBERT based) and other SOTA methods, particularly in low-SNR conditions (-5 dB) and in the challenging AVCocktail fixed-chunk setting. The ablation studies effectively isolate the contributions of MUBF and DUHV, showing that combining both uncertainty types and the hierarchical voting strategy yields the best performance. The inclusion of general-purpose models like Whisper and Qwen3-Omni provides good context for the difficulty of the task. However, the evaluation is limited to Word Error Rate (WER). There is no analysis of the calibration of the uncertainty estimates themselves (e.g., Expected Calibration Error or reliability diagrams), which is crucial for a paper claiming to model uncertainty. Additionally, the claim of "7.9% improvement" on AVCocktail seems high and warrants scrutiny regarding the baseline comparison, though the table data (implied) supports significant gains. The lack of LRS3 evaluation is noted but acceptable given access issues.
The paper provides detailed implementation details, including network architectures (2-layer MLPs, Bayesian linear layers), initialization schemes (Xavier-uniform, bias -5.0), hyperparameters (learning rate, batch size, weight decay), and training strategies (two-stage, DoRA). The use of a publicly available baseline repository (AVSRCocktail) aids reproducibility. The Monte Carlo sampling count (K=5) is specified. However, the code is not explicitly linked in the provided text (though a baseline repo is mentioned), and the specific random seeds for the 10 independent runs are not detailed, which is important for stochastic Bayesian methods. Overall, the description is sufficient for a competent researcher to reproduce the results.
The paper acknowledges a slight performance decline at 10 dB and clean conditions, attributing it to "over-enhancement." This suggests the model may be overly aggressive in suppressing features even when they are reliable, indicating a potential lack of robustness in high-quality regimes. The reliance on Monte Carlo sampling for inference increases latency compared to deterministic baselines, although the authors argue K=5 is a good trade-off. The method assumes that the pre-trained backbone features are sufficiently expressive, which might not hold for out-of-domain data not seen during the fine-tuning stage. Furthermore, the complexity of the Bayesian layers adds computational overhead during training and inference.
This work contributes to the robustness of multimodal AI systems, which is critical for real-world deployment in noisy environments (e.g., autonomous vehicles, smart homes, hearing aids). By providing a framework that quantifies and mitigates uncertainty, it offers a pathway to more reliable AVSR systems. The techniques could be extended to other multimodal tasks where sensor reliability varies dynamically. The paper presents a well-motivated and technically sound framework for robust audio-visual speech recognition by integrating aleatoric and epistemic uncertainty modeling through a novel Bayesian gating mechanism and hierarchical voting strategy, demonstrating significant improvements on challenging real-world and simulated benchmarks.
Neural audio codecs (NACs) enable efficient audio compression and have achieved success in downstream tasks such as speech synthesis. However, their discrete representations consistently underperform traditional spectral features in automatic speaker verification (ASV). We empirically demonstrate that speaker cues are implicitly preserved in discrete tokens but remain underutilized by conventional ASV training paradigms. To address this, we propose a Cross-Feature Knowledge Distillation (CFKD) framework. By guiding the codec-based student to mimic the embedding space of a strong Fbank-based teacher, CFKD provides structured supervision for effective utilization of speaker information in tokens. Experiments on the VoxCeleb benchmarks show that CFKD substantially improves the ASV performance of codec-based systems, allowing them to approach the accuracy of Fbank-based teacher models and highlighting the potential of discrete audio tokens for diverse speech tasks.
Primary: The Hong Kong Polytechnic University
All Institutions: The Hong Kong Polytechnic University
The paper presents a practical and effective method for leveraging discrete neural audio tokens in Automatic Speaker Verification through cross-feature knowledge distillation, demonstrating that discrete representations can achieve performance comparable to traditional spectral features.
The paper proposes Cross-Feature Knowledge Distillation (CFKD) to address the performance gap between discrete neural audio codec tokens and continuous spectral features (Fbanks) in Automatic Speaker Verification (ASV). The core methodology involves training a student model (using EnCodec tokens) to mimic the embedding space of a teacher model (using Fbanks) via cosine similarity loss at the embedding level. The authors also conduct an ablation study on backbone architectures (1D vs 2D), concluding that 1D architectures (ECAPA-TDNN) are more suitable for discrete tokens due to the lack of spatial adjacency in the token latent space. The approach is conceptually sound and addresses a clear problem: discrete tokens from generative codecs are underutilized in discriminative tasks. However, the novelty is moderate as knowledge distillation is a standard technique, and applying it across different feature modalities (cross-feature) is a logical but not radically new extension. The specific insight regarding the inductive bias of 1D vs 2D networks for discrete tokens is a valuable empirical contribution.
The experiments are conducted on standard ASV benchmarks (VoxCeleb1 and VoxCeleb2). The results show that CFKD significantly improves the performance of token-based systems, narrowing the gap with Fbank-based systems. For instance, on Vox1-O with ECAPA-TDNN, the EER drops from 3.38% (baseline) to 2.25% (CFKD), approaching the teacher's 2.21%. The paper includes error analysis showing complementary error sets between teacher and student, which supports the claim that tokens preserve distinct speaker cues. The evaluation is thorough, covering different bitrates and backbones. However, the comparison is primarily against the Fbank baseline and a prior work (Codec-ASV). The improvement is substantial but the absolute performance of the token-based system still lags slightly behind the Fbank teacher in some settings (e.g., minDCF 0.231 vs 0.236 is very close, but in other configs, the teacher is better). The results are convincing but not revolutionary.
The paper provides detailed descriptions of the experimental setup, including the EnCodec configuration (24kHz, 32 quantizers), backbone architectures, training hyperparameters (learning rate, batch size, optimizer), and data augmentation (MUSAN, RIR). The use of standard frameworks (Wespeaker) and datasets (VoxCeleb) enhances reproducibility. The specific distillation weight ($\alpha=40$) is noted as unusually high, which is a specific hyperparameter choice that might require tuning for other datasets. Overall, the method is reproducible.
The primary limitation is that the token-based system, even with CFKD, does not consistently outperform the Fbank teacher; it merely approaches it. This suggests that while the gap is closed, the discrete representation may still have inherent limitations in capturing fine-grained speaker details compared to continuous spectral features. Additionally, the method relies on a pre-trained Fbank teacher, which adds complexity to the training pipeline (two models instead of one). The paper does not explore the impact of different codec types or bitrates on the distillation efficacy beyond the standard EnCodec configuration. The claim that tokens "preserve sufficient speaker-discriminative information" is supported, but the mechanism of *how* the distillation extracts this is somewhat opaque (black-box embedding alignment).
This work has significant implications for the field of speech processing, particularly in bridging the gap between generative audio models (which use discrete tokens) and discriminative tasks (like ASV). It suggests that discrete representations can be effectively used for identity verification, which is crucial for applications requiring low-bitrate transmission or integration with large language models that operate on discrete tokens. It challenges the dominance of spectral features in ASV and opens avenues for unified audio models that can handle both generation and recognition tasks efficiently. The paper presents a practical and effective method for leveraging discrete neural audio tokens in Automatic Speaker Verification through cross-feature knowledge distillation, demonstrating that discrete representations can achieve performance comparable to traditional spectral features.
Video-to-audio (V2A) generation aims to synthesize realistic audio that is both semantically consistent with and temporally synchronized to a silent video. Despite recent progress, many methods still rely on multi-stage training, resulting in high computational costs and long runtimes, or transform visual input into text to leverage pretrained text-to-audio models, sacrificing fine-grained temporal cues. To overcome these limitations, we propose Flowley, an end-to-end, single-stage training architecture that produces soundtracks by combining visual features with textual prompts. Crucially, we introduce Progressive Soft-masked Cross-Attention, which embeds audio-visual synchronization directly within its attention mechanism, adding zero additional computational cost compared to standard attention layers. We further observe that existing V2A benchmarks lack sound-oriented descriptive captions, which can potentially degrade the quality of the synthesized audio. To remedy this, we propose SoundCap, a plug-and-play pipeline for creating detailed, sound-aware captions that guide the model. Remarkably, without integrating any pretrained audio-visual alignment modules, Flowley achieves state-of-the-art performance on VGGSound across multiple metrics. Moreover, by incorporating SoundCap, we further exceed the performance of the strongest existing close-sourced methods in terms of audio quality in the zero-shot setting.
Primary: FPT Software AI Center
All Institutions: FPT Software AI Center, NVIDIA Corporation
[Flowley introduces a novel end-to-end flow-based architecture with Progressive Soft-masked Cross-Attention for precise video-to-audio generation, achieving state-of-the-art performance on VGGSound and outperforming larger closed-source models in zero-shot settings when augmented with its SoundCap captioning pipeline.] This paper presents a significant advancement in V2A generation by eliminating the need for multi-stage training and external alignment modules, offering a more efficient and effective solution for synthesizing temporally synchronized and semantically consistent audio from video.
The paper proposes Flowley, an end-to-end flow-based architecture for Video-to-Audio (V2A) generation. The core technical contribution is the Progressive Soft-masked Cross-Attention (PSCA) mechanism, which embeds temporal alignment directly into the attention layers of a DiT-like backbone. This avoids the need for external synchronization modules or multi-stage training pipelines common in prior works (e.g., MMAudio, VinTAGe). The method combines visual features (CLIP), textual prompts (FLAN-T5), and audio latents in a multi-stream to single-stream architecture. Additionally, the authors introduce SoundCap, a pipeline using AV-LLMs to generate detailed, sound-aware captions to address the lack of rich textual annotations in V2A datasets. The approach is technically sound, leveraging recent advances in flow matching and multimodal transformers. The PSCA mechanism is a clever, low-cost modification to standard cross-attention that explicitly models the temporal relationship between video frames and audio tokens.
The evaluation is comprehensive, covering distribution matching (KAD, FAD), audio quality (IS), semantic alignment (IB-Score, LB-Score), and synchronization (Align Acc). Flowley achieves state-of-the-art results on the VGGSound benchmark, outperforming larger models like MMAudio and FoleyCrafter in most metrics. The inclusion of SoundCap further boosts performance, particularly in zero-shot settings on the Movie Gen Audio benchmark, where it surpasses the closed-source Movie Gen Audio model in quality despite being significantly smaller and trained on less data. The subjective human evaluation supports the objective metrics, showing a preference for Flowley's synchronization and quality. The ablation studies effectively validate the contributions of PSCA, the dual cross-attention, and the SoundCap pipeline.
The paper provides detailed implementation details, including architecture dimensions, hyperparameters for PSCA (window sizes, fade zones), training schedules, and evaluation metrics. The use of standard datasets (VGGSound) and established baselines facilitates comparison. The code link is provided. The description of the SoundCap pipeline is sufficiently detailed for reproduction, assuming access to the specified AV-LLMs and VLMs.
The paper acknowledges that while Flowley excels in quality and distribution matching, it slightly lags behind some baselines in the Align Acc metric, although human preference favors its synchronization. This suggests a potential discrepancy between the automated Align Acc metric and human perception of temporal alignment. The reliance on AV-LLMs for SoundCap introduces a dependency on large, potentially expensive models for data generation, although the final VLM is lightweight. The method's performance on highly complex scenes with multiple simultaneous sound sources or off-screen sounds might be challenging, as indicated by the noise handling strategies in SoundCap.
This work contributes to the field of generative AI for multimedia, specifically in creating realistic and synchronized audio-visual experiences. It has applications in film production (Foley), gaming, virtual reality, and accessibility. The SoundCap pipeline offers a generalizable solution for improving text-audio grounding in other multimodal tasks. The open-source nature of the work (implied by the project page) encourages further research in efficient, end-to-end V2A generation. [Flowley introduces a novel end-to-end flow-based architecture with Progressive Soft-masked Cross-Attention for precise video-to-audio generation, achieving state-of-the-art performance on VGGSound and outperforming larger closed-source models in zero-shot settings when augmented with its SoundCap captioning pipeline.] This paper presents a significant advancement in V2A generation by eliminating the need for multi-stage training and external alignment modules, offering a more efficient and effective solution for synthesizing temporally synchronized and semantically consistent audio from video.
Speech large language models (Speech LLMs) typically encode speech into sequences far longer than text, creating a major efficiency bottleneck during autoregressive decoding. A common remedy is to compress the speech sequence at the adapter level to remove temporal redundancy before it enters the LLM; however, such early downsampling risks discarding fine-grained information that cannot be recovered. We propose SpeechKV, which applies a learned pooling to the KV cache of speech tokens inside the LLM. This design allows the LLM to fuse speech and text internally while directly accelerating decoding. Trained on 71K hours of speech data, SpeechKV compresses the speech to approximately text-level granularity yet maintains performance on par with or even slightly better than the uncompressed baseline, with relative gains of 6.6% on out-of-domain entity recognition and 2.3% on OpenASR, while delivering at least 1.49 times decoding speedup that scales with audio length.
Primary: Microsoft
All Institutions: Microsoft, National Taiwan University
SpeechKV introduces a novel in-LLM KV cache compression technique for Speech LLMs that achieves competitive performance with significant decoding speedups by leveraging learned pooling on intermediate layer representations.
The paper proposes SpeechKV, a novel approach to compressing the Key-Value (KV) cache of speech tokens within the internal layers of a Speech Large Language Model (Speech LLM). Unlike traditional methods that compress speech embeddings before they enter the LLM (adapter-level compression), SpeechKV applies a learned pooling operation to the KV cache at an intermediate layer ($l_0$). This design preserves full-resolution queries and the residual stream, allowing the model to fuse speech and text representations internally while reducing the sequence length for subsequent attention computations. The method is theoretically sound, leveraging the observation that speech representations become redundant in deeper layers. The approach is simple yet effective, avoiding the need for complex architectural changes or external modules.
The authors evaluate SpeechKV on a Qwen3-1.7B backbone trained on 71K hours of ASR data. They compare against adapter-level compression (Concat-MLP, Window Q-Former) and an alternative in-LLM hidden-state compression (HS Compression). Results show that SpeechKV maintains or slightly improves upon the uncompressed baseline performance on both in-domain (OpenASR) and out-of-domain (In-house ER, In-house ASR) benchmarks. Notably, it achieves a 6.6% relative gain in entity recognition and a 2.3% gain in OpenASR WER compared to the baseline, while providing a 1.49x to 2.02x decoding speedup. The ablation studies on compression ratios ($R=4, 8$) and starting layers ($l_0$) provide valuable insights into the trade-offs between compression aggressiveness and performance retention. The attention analysis further supports the claim that SpeechKV leads to more focused attention patterns.
The paper provides detailed descriptions of the model architecture, training setup (optimizer, learning rate, batch size), and evaluation metrics. The use of open-source components (Qwen3-1.7B, Conformer encoder) and standard ASR benchmarks enhances reproducibility. However, the mention of "proprietary test sets" for In-house ASR and ER limits the ability of external researchers to fully replicate the out-of-domain evaluation. The code is not explicitly linked in the text provided, which is a minor hindrance to immediate reproducibility.
The primary limitation is the reliance on proprietary data for a significant portion of the evaluation, which prevents independent verification of the out-of-domain generalization claims. Additionally, the speedup, while significant, is modest (1.49x-2.02x) and may not justify the added training complexity for all use cases, especially given that feed-forward layers and text KV cache remain bottlenecks at higher compression ratios. The method is specific to Speech LLMs and may not generalize directly to other multimodal settings without adaptation.
This work contributes to the efficiency and accessibility of Speech LLMs, which are increasingly important for real-time applications like voice assistants and transcription services. By enabling faster decoding without significant performance loss, SpeechKV could facilitate the deployment of more capable speech models on resource-constrained devices. The insights into layer-wise redundancy in speech representations also offer broader implications for understanding how multimodal models process continuous signals. SpeechKV introduces a novel in-LLM KV cache compression technique for Speech LLMs that achieves competitive performance with significant decoding speedups by leveraging learned pooling on intermediate layer representations.
Voice anonymisation aims to protect speaker identity. Currently, its empirical privacy evaluation heavily relies on the Equal Error Rate (EER). Originally designed for biometric verification, EER aggregates scores globally, implicitly assuming an attacker is only trying to verify if two specific voice samples match (a 1-to-1 comparison). This introduces a threat model mismatch with real-world database linkage attacks, where an attacker searches across a fixed set of N enrolled identities (a 1-to-N closed-set search), allowing global averages to obscure localised privacy failures. While recent 1-to-N metrics address this aggregation issue, they abstract away the magnitude of the biometric evidence. In this paper, we propose a modular, information-theoretic evaluation framework explicitly designed for the 1-to-N linkage threat model. Within this framework, our core metric, Local Information Disclosure (LID), quantifies the exact privacy loss of a single trial utterance in bits by calibrating its raw similarity scores into the attacker's posterior confidence for each enrolled identity. Evaluating top-performing systems from the VoicePrivacy 2024 Challenge reveals that systems exhibiting near-perfect EERs (48 %) can still suffer from localised vulnerabilities with worst-case disclosures reaching 1 bit per trial utterance (effectively doubling the attacker's success rate over a random guess). We demonstrate that adopting localised privacy metrics is essential for capturing worst-case risks and aligning with strict privacy regulations.
Primary: Aalto University
All Institutions: Aalto University, Inria, Nokia, Macquarie University
The paper introduces a novel information-theoretic metric, Local Information Disclosure (LID), to evaluate voice anonymisation systems under 1-to-N linkage threats, revealing that systems with good global privacy metrics (EER) can still suffer from severe local vulnerabilities, thereby providing a more accurate and regulation-aligned assessment of privacy risks.
The paper proposes a rigorous, information-theoretic evaluation framework for voice anonymisation, specifically targeting the mismatch between standard 1-to-1 verification metrics (like EER) and real-world 1-to-N linkage threats. The core contribution is the Local Information Disclosure (LID) metric, which quantifies privacy loss in bits by calibrating raw similarity scores into posterior probabilities using logistic regression on row-normalized scores. The methodology is mathematically sound, leveraging Pointwise Mutual Information (PMI) to measure the reduction in attacker uncertainty. The framework is modular, allowing for various global aggregations (ALID, PDR, NDR, LID_max) that provide a more granular view of system vulnerabilities than global averages. The approach correctly identifies that global metrics can obscure local failures where specific utterances are highly linkable despite good average performance.
The authors evaluate their framework against systems from the VoicePrivacy 2024 Challenge (VPC 2024), including baselines and top submissions. They use the standard LibriSpeech dev/eval splits. The experiments demonstrate a critical finding: systems with near-perfect EERs (e.g., 0.48, indicating high privacy in 1-to-1 terms) can still exhibit worst-case local disclosures of ~1 bit, effectively doubling the attacker's success rate over random chance for specific trials. The results are presented clearly through CCDFs and tables comparing EER, C_llr, and the new LID-based metrics. The analysis of the calibration weight $w$ provides additional insight into the discriminative power of the anonymised embeddings. The empirical evidence strongly supports the claim that 1-to-1 metrics are insufficient for assessing 1-to-N privacy risks.
The paper provides detailed mathematical formulations for the LID metric, including the row-wise z-normalization and logistic calibration steps. It specifies the datasets (LibriSpeech) and the attacker model (semi-informed ECAPA-TDNN). However, the specific implementation details of the calibration (e.g., exact hyperparameters for the logistic regression, handling of edge cases) are somewhat abstract. The code is not explicitly linked in the provided text, though the VPC 2024 data is public. Reproducibility is moderate; a researcher could implement the method, but might need to infer some implementation details.
The primary limitation is the reliance on logistic regression for score calibration, which assumes a specific distribution of scores (normality after z-normalization). The authors acknowledge this and suggest it may not hold for all anonymisation methods. Additionally, the framework is evaluated only on the semi-informed threat model; while this is the most realistic, the framework's behavior under ignorant or lazy-informed models is not explored. The evaluation is limited to voice data; while the framework is generalizable, its applicability to other modalities (e.g., face, fingerprint) is not demonstrated. The "random" baseline comparison is useful but simplistic; a more nuanced baseline might involve a dummy classifier with similar score distributions but no identity information.
This work has significant implications for the deployment of voice anonymisation systems in privacy-sensitive applications (e.g., healthcare, legal, customer service). By providing a metric that aligns with regulatory concerns about "singling out" and "linkability" (e.g., GDPR), it offers a more robust tool for auditors and developers. It shifts the focus from average performance to worst-case risks, which is crucial for compliance. The information-theoretic framing also contributes to the broader field of privacy-preserving machine learning by offering a concrete way to quantify information leakage in biometric systems. The paper introduces a novel information-theoretic metric, Local Information Disclosure (LID), to evaluate voice anonymisation systems under 1-to-N linkage threats, revealing that systems with good global privacy metrics (EER) can still suffer from severe local vulnerabilities, thereby providing a more accurate and regulation-aligned assessment of privacy risks.