Full-length song generation must preserve coherence and musicality, render detailed vocal and accompaniment acoustics, and follow lyrics and prompts. Existing language model-based systems face a structural trade-off: mixed-token modeling preserves vocal-instrument coordination but obscures track-specific details, whereas dual-track prediction improves acoustics but requires longer sequences and weakens global planning. We present LeVo 2, a hybrid LLM-Diffusion framework for controllable full-length song generation. LeVo 2 formulates this trade-off as hierarchical modeling: LeLM first predicts mixed tokens for semantic planning, then predicts vocal and accompaniment tokens in parallel for track-specific refinement, while a diffusion-based Music Codec reconstructs full-length waveforms. A central contribution of this extended version is an aesthetics-guided training schedule for alignment. During pre-training, an automated music aesthetic evaluation framework assigns musicality-tier conditions to large-scale data, providing musicality priors before preference alignment. Progressive post-training applies SFT, large-scale offline DPO, and closed-loop semi-online DPO to separately improve generation quality, controllability, and musicality. Modular extension then trains the Track-Specific LM for acoustic refinement while preserving the aligned semantic planner. This schedule separates musicality learning, controllability alignment, and acoustic refinement, mitigating optimization conflict and the limitations of static offline preference pairs. Expert listening tests and objective evaluations show that LeVo 2 outperforms open-source baselines across six subjective dimensions, and approaches leading commercial systems on several listening metrics. Ablations validate the effects of the training strategy, aesthetics guidance, scaling, and hierarchical architecture.
Primary: Tsinghua University
All Institutions: Tsinghua University, Tencent, Wuhan University, Hong Kong Polytechnic University
LeVo 2 introduces a hierarchical LLM-Diffusion framework with a decoupled, aesthetics-guided progressive post-training strategy that effectively balances global musical coherence with track-specific acoustic fidelity, achieving state-of-the-art performance among open-source models.
The paper proposes LeVo 2, a hybrid LLM-Diffusion architecture for full-length song generation. The core methodological contribution is a hierarchical representation modeling strategy that decouples global semantic planning (via a Mixed Semantic LM predicting mixed tokens) from track-specific acoustic refinement (via a Track-Specific LM predicting parallel vocal and accompaniment tokens). This addresses the trade-off in existing LLM-based systems between maintaining vocal-instrument harmony and capturing fine-grained acoustic details. A significant portion of the contribution lies in the training paradigm: an aesthetics-guided three-stage process involving pre-training with musicality-tier conditions, decoupled progressive post-training (SFT, offline DPO for controllability, semi-online DPO for musicality), and modular extension. The use of an automated music aesthetic evaluation framework to guide data filtering and preference alignment is a notable technical innovation, aiming to mitigate gradient conflicts and reward hacking common in multi-objective alignment.
The evaluation is comprehensive, featuring both subjective (expert MOS across six dimensions) and objective (PER, Gemini-based prompt alignment) metrics. The paper compares LeVo 2 against leading commercial systems (Suno v5, Mureka v8) and open-source baselines (YuE, DiffRhythm 2, ACE-Step 1.5). Results indicate that LeVo 2 outperforms all open-source baselines and approaches commercial systems in key metrics like Melody and Arrangement. The ablation studies effectively validate the contribution of each training stage, particularly demonstrating that the decoupled progressive post-training strategy yields better results than single-dimension or mixed multi-dimension optimization. The inclusion of a semi-online DPO stage is well-supported by ablations showing its ability to push performance beyond static offline data limits.
The authors provide inference code and full model weights on GitHub, which significantly enhances reproducibility. The paper details the model architecture (4B parameters for LeLM, 700M for diffusion decoder), training steps, and hardware setup (64 H20 GPUs). The use of open-source components like MuCodec and Qwen2-tokenizer aids in replication. However, the specific implementation details of the "automated music aesthetic evaluation framework" and the exact filtering thresholds for the semi-online DPO loop are somewhat high-level, which might require careful engineering to replicate exactly.
The paper acknowledges that commercial systems still hold an advantage in some metrics, suggesting room for improvement. The reliance on an automated aesthetic evaluation framework for training data filtering and DPO reward modeling introduces potential biases or inaccuracies inherent in the evaluator model (MuQ-based). Furthermore, the "semi-online" DPO, while effective, still relies on periodic updates rather than true online RL, which may limit the extent of distribution shift handling. The model's performance on non-Chinese/non-English languages is not explicitly detailed, though English is tested.
LeVo 2 represents a significant step forward in controllable, high-fidelity song generation, bridging the gap between open-source research and commercial capabilities. By providing open weights and code, it fosters further research in music AI. The decoupled alignment strategy offers a generalizable framework for multi-objective optimization in generative models, potentially applicable to other multimodal domains. However, the ease of generating high-quality songs also raises concerns regarding copyright infringement, deepfakes, and the displacement of human musicians, necessitating responsible deployment guidelines. LeVo 2 introduces a hierarchical LLM-Diffusion framework with a decoupled, aesthetics-guided progressive post-training strategy that effectively balances global musical coherence with track-specific acoustic fidelity, achieving state-of-the-art performance among open-source models.
Traditional emotional voice conversion (EVC) conditions generation on explicit target emotions like labels or references, defining the target affective state but omitting the direction or nature of the transition. We introduce instruction-guided relative emotional voice conversion, a task where natural-language instructions specify source-conditioned affective transformations (e.g., "make the speech slightly calmer" or "sound noticeably more confident") instead of fixed targets. To support this task, we construct TRACE-Instruct, a dataset of relative emotion instructions covering categorical transitions, intensity modifications, and open-ended affective changes. We propose TRACE-EVC, a zero-shot framework built around Emo-Compass, a module that models each conversion as a source-anchored rectified flow. Rather than conditioning on an explicit target, it predicts the direction and degree of the affective change. Experiments demonstrate that TRACE-EVC accurately follows relative emotion instructions while preserving speaker identity, linguistic content, and speech quality, and remains competitive with conventional EVC systems on standard categorical emotion conversion.
Primary: Johns Hopkins University
All Institutions: Johns Hopkins University, Center for Language and Speech Processing (CLSP)
This paper presents a significant methodological advance in emotional voice conversion by shifting from absolute target conditioning to relative, instruction-guided transformation using source-anchored rectified flows, offering a more intuitive and flexible control interface for expressive speech synthesis.
The paper introduces a novel task formulation: instruction-guided relative emotional voice conversion (EVC). Unlike traditional EVC which maps to absolute emotional states (labels or references), this work conditions generation on natural language instructions describing the *change* in emotion relative to the source (e.g., "make it calmer"). The core technical innovation is "Emo-Compass," a module that models this transformation as a source-anchored rectified flow. Instead of predicting an absolute emotion embedding from noise, it predicts the velocity vector (displacement) in a continuous affective space (combining emotion2vec, VAD, and prosodic features) from the source anchor. This is a sophisticated application of continuous normalizing flows/rectified flows to affective computing. The synthesis module adapts DurFlex-EVC to use these continuous embeddings. The approach is theoretically sound and addresses a genuine gap in fine-grained, directional control of speech emotion.
The authors evaluate on two datasets: ESD (inter-emotion) and MEAD (intra-emotion intensity). They compare against strong baselines including label-based (SGEVC, DurFlex-EVC) and reference-based (ZEST, VEVO) systems. Results show TRACE-EVC achieves competitive or superior performance in emotion similarity (EECS), classification accuracy, and speaker similarity (SECS) compared to baselines, while maintaining high speech quality (UTMOS/NISQA). Crucially, they introduce a human evaluation metric for "Instruction Following" (IF), which is essential for this task. The ablation studies on Emo-Compass components (VAD, prosody) are thorough and validate the design choices. The inclusion of unseen-speaker zero-shot evaluation on MEAD strengthens the claim of generalization.
The paper provides detailed implementation details, including model architectures (6-layer Transformer for Emo-Compass), training hyperparameters (AdamW, LR 1e-4), and datasets. They release code and demos. The dataset construction process (using LLMs to generate instructions from paired speech and VAD differences) is described, though the specific prompts and filtering rules could be more detailed for perfect reproducibility. However, the core code release mitigates this.
The reliance on LLMs for dataset construction introduces potential biases or hallucinations in the instruction generation, although rule-based filtering helps. The evaluation is limited to English. The "zero-shot" claim relies on the disentanglement capabilities of the underlying DurFlex-EVC backbone; if the source speaker's emotion is very strong, the relative control might struggle to override it completely, a common issue in EVC. The comparison with other prompt-based EVC methods is limited by the lack of public implementations, which is a fair point but leaves a gap in direct comparison with similar prompt-based approaches.
This work enables more natural and intuitive human-computer interaction in speech synthesis, allowing users to specify nuanced emotional adjustments via natural language. It has applications in audiobooks, dubbing, and assistive communication. The dataset TRACE-Instruct is a valuable resource for the community. Potential misuse includes generating deceptive emotional speech, though this is a general risk in EVC. This paper presents a significant methodological advance in emotional voice conversion by shifting from absolute target conditioning to relative, instruction-guided transformation using source-anchored rectified flows, offering a more intuitive and flexible control interface for expressive speech synthesis.
Strong speech-to-text (S2T) LLMs already provide robust speech perception and text reasoning, but adding speech-to-speech (S2S) output is challenging: fine-tuning the backbone can degrade the original S2T performance, while attaching a downstream talker reintroduces a serial text-to-speech bottleneck. We present PRIME-Speech, a frozen-backbone S2S conversion framework that trains only speech-generation modules. PRIME-Speech synchronizes a causal audio post-decoder with intermediate hidden states of the frozen backbone, so codec tokens are generated from the model's evolving reasoning trajectory rather than from completed text chunks. The post-decoder uses mixed hidden-state, text, and audio-history conditioning, and a training-time packing strategy with turn-level audio KV-cache and position reset stabilizes multi-turn spoken interaction without additional multi-turn S2S training data. Multi-token prediction further reduces the effective codec prediction rate and improves first-audio latency without modifying the reasoning path. Across speech translation, spoken QA, speech understanding, and multi-turn dialogue, PRIME-Speech preserves the S2T behavior of the frozen backbone while producing accurate, low-WER spoken responses.
Primary: Microsoft
All Institutions: Microsoft
This paper presents a significant advancement in Speech-to-Speech generation by introducing a frozen-backbone framework that effectively balances the preservation of reasoning capabilities with the generation of low-latency, high-quality speech, addressing key limitations in current S2S architectures.
The paper proposes PRIME-Speech, a novel framework for Speech-to-Speech (S2S) generation that addresses the critical trade-off between preserving the reasoning capabilities of a frozen Speech-to-Text (S2T) backbone and generating high-quality speech output. The core innovation lies in "hidden-state synchronization," where a trainable causal audio post-decoder is attached to intermediate hidden states of the frozen backbone, allowing speech tokens to be generated in parallel with text tokens from the evolving reasoning trajectory. This avoids the serial bottleneck of traditional cascade systems and the catastrophic forgetting associated with full fine-tuning. The methodology includes a sophisticated multi-token prediction (MTP) strategy for efficiency and a specific multi-turn cache policy (accumulating text KV-cache while resetting audio KV-cache) to prevent acoustic drift across turns. The approach is technically sound, leveraging the strengths of large language models while introducing a clean interface for audio generation.
The experimental evaluation is comprehensive, covering speech translation (FLEURS, CoVoST-2), spoken question answering (UltraEval-Audio), speech understanding (BigBench-Audio), and multi-turn dialogue. The authors demonstrate that PRIME-Speech preserves the S2T performance of the backbone (Phi-4-MM-7B) while achieving competitive S2S performance. Ablation studies effectively isolate the contributions of the frozen backbone, the audio post-decoder, and the MTP module. The results show that freezing the backbone prevents the degradation of reasoning capabilities seen in other S2S models that fine-tune the entire network. The efficiency gains from MTP are also well-documented, showing significant reductions in Time-to-First-Audio (TTFA) and Real-Time Factor (RTF) with minimal impact on task correctness.
The paper provides detailed descriptions of the model architecture, training curriculum (two-stage training), and dataset composition. The use of standard codecs (CosyVoice2) and clear definitions of the conditioning mechanisms enhance reproducibility. However, the reliance on proprietary backbone models (Phi-4-MM-7B) and specific internal data synthesis procedures may limit exact replication. The cache reset policy and MTP implementation details are sufficiently described for implementation.
The primary limitation is the reliance on a strong, pre-existing S2T backbone, which may not be available or open-source for all use cases. The system's performance is inherently bounded by the reasoning capabilities of the frozen backbone. Additionally, while the paper reports UTMOS scores for fluency, it does not provide extensive subjective human evaluation (MOS) for naturalness or speaker consistency, which are crucial for S2S systems. The current evaluation focuses on transcript-level correctness and WER, which may not fully capture the perceptual quality of the generated speech, especially regarding prosody and emotion.
PRIME-Speech contributes to the development of more natural and efficient human-computer interaction systems by enabling direct speech-to-speech interaction without intermediate text bottlenecks. By preserving the reasoning capabilities of S2T models, it offers a pathway to robust, intelligent voice assistants. The approach could influence the design of future multimodal LLMs, encouraging modular designs that separate reasoning from modality-specific generation. However, the potential for misuse in generating realistic voice clones or deceptive audio content remains a concern, necessitating responsible deployment guidelines. This paper presents a significant advancement in Speech-to-Speech generation by introducing a frozen-backbone framework that effectively balances the preservation of reasoning capabilities with the generation of low-latency, high-quality speech, addressing key limitations in current S2S architectures.
Accent normalization (AN) seeks to convert non-native (L2) accented speech into standard (L1) speech while preserving speaker identity. The current techniques either require naturally recorded parallel L1-L2 speech for training, or suffer from quality degradation when supervised by synthesized targets. In this paper, we present TokAN, a token-based accent normalization framework that operates on self-supervised discrete speech tokens extracted from a L1-L2 jointly trained vector-quantization (VQ) tokenizer, without the need of synthetic supervisory speech. An autoregressive encoder-decoder model performs token-to-token conversion, translating L2-accented token sequences into the tokens of standard voice. We also introduce reinforcement learning (RL) post-training based on Group Relative Policy Optimization (GRPO), using word error rate and accent classifier confidence as complementary rewards. A non-autoregressive flow-matching synthesizer recovers the Mel-spectrogram from the converted tokens, conditioned on the source speaker embedding. We also develop a flow-matching duration predictor that supports total-duration-aware synthesis, making TokAN applicable to duration-critical tasks such as voice dubbing and live casting. Experiments on seven English accents demonstrate that TokAN reduced the word error rate from 12.40% to 9.89% after supervised fine-tuning, and further to 9.23% after RL post-training, consistently outperforming frame-to-frame, direct flow-matching, and prompt-based token-conversion baselines in terms of accent reduction and intelligibility.
Primary: Nanjing University
All Institutions: Nanjing University, The Chinese University of Hong Kong, Shenzhen (CUHKSZ), Tencent Ethereal Audio Lab, Shanghai Jiao Tong University, Shenzhen Loop Area Institute, Shenzhen Research Institute of Big Data
TokAN introduces a novel token-based accent normalization framework that leverages a jointly trained VQ tokenizer and RL post-training with GRPO to achieve superior accent reduction and intelligibility without requiring naturally recorded parallel speech data.
The paper proposes TokAN, a token-based accent normalization framework that addresses the data scarcity and quality degradation issues in existing methods. The core methodological contributions are threefold: (1) A jointly trained Vector Quantization (VQ) tokenizer that co-optimizes phonetic content representation with speech synthesis and ASR supervision, replacing static K-Means clustering. This ensures the discrete tokens are more discriminative for accent-related phonetic features. (2) An autoregressive encoder-decoder conversion model using Rotary Positional Encodings (RoPE) and a self-attention-only decoder, trained with BART-style pre-training and supervised fine-tuning on semi-synthetic pairs. (3) A novel Reinforcement Learning (RL) post-training stage using Group Relative Policy Optimization (GRPO). This is a significant technical innovation in the speech domain, as it directly optimizes task-level objectives (Word Error Rate and Accent Classifier Confidence) rather than relying solely on token-level cross-entropy loss. The use of GRPO eliminates the need for a value network, making it computationally efficient for sequence generation. The system also includes a flow-matching synthesizer and a duration predictor for total-duration-aware synthesis, enhancing applicability to dubbing.
The evaluation is comprehensive, covering seven English accents from the L2-ARCTIC dataset. The authors compare TokAN against strong baselines: FramAN (frame-to-frame), CosyAccent (direct flow-matching), and VEVO (token-based). Results show TokAN significantly reduces Word Error Rate (WER) from 12.40% (SFT only) to 9.23% (after RL), outperforming all baselines in intelligibility and accent reduction (L1-Prob). Subjective evaluations (MUSHRA) confirm higher naturalness and accentedness reduction. The ablation studies effectively isolate the contributions of the joint tokenizer training, BART pre-training, SFT, and RL post-training. The phonemic distribution analysis provides deep insight into how the model corrects specific accent-related errors (e.g., dental-alveolar fricative confusion for Chinese speakers). The inclusion of both duration-free and source-length modes adds practical value.
The paper provides extensive implementation details, including hyperparameters for all training stages, dataset descriptions (LibriTTS-R, Emilia-EN, L2-ARCTIC, GLOBE), and model architectures. The code and demo samples are publicly available on GitHub and a dedicated webpage, which significantly enhances reproducibility. The use of standard components (WavLM, HiFT, Whisper) and open-source RL libraries (TRL) further supports reproducibility.
The reliance on semi-synthetic data for SFT means the model's performance is still somewhat tied to the quality of the TTS system used to generate native targets. While RL mitigates this by optimizing on real data, the initial mapping is learned from synthetic pairs. Additionally, the speaker similarity is slightly lower than some baselines (like VEVO), indicating a trade-off between accent normalization and voice preservation that might be critical for some applications. The method assumes access to a pre-trained accent classifier and ASR model for rewards, which introduces potential bias if these models are not robust to the specific accents being normalized.
TokAN has significant potential for improving accessibility and communication for non-native speakers, aiding language learning, and enhancing the quality of automated dubbing and live translation services. By reducing the reliance on parallel L1-L2 data, it lowers the barrier to creating accent normalization systems for low-resource languages or less common accents. The use of RL for speech generation also opens new avenues for optimizing speech models against complex, non-differentiable metrics. TokAN introduces a novel token-based accent normalization framework that leverages a jointly trained VQ tokenizer and RL post-training with GRPO to achieve superior accent reduction and intelligibility without requiring naturally recorded parallel speech data.
Training automatic speech recognition (ASR) models for low-resource languages is challenging due to limited data and highly variable supervision quality. In particular, Pacific Indigenous speech corpora often exhibit heterogeneous acoustic conditions, transcript inconsistencies, and varying degrees of acoustic-text alignment reliability, making standard fine-tuning approaches sensitive to noisy or misleading supervision signals. In this work, we propose QuaSR, a simple yet effective weighting framework that combines data-side reliability with model-side learnability to improve ASR adaptation. Specifically, we estimate data reliability from acoustic, transcription, and alignment, while measuring learnability using training loss from the model. These two complementary signals are integrated into a unified sample utility score to produce training weights for the samples. We also evaluated across four Pacific Indigenous languages, which shows that the proposed utility scores reliably correlate with adaptation performance. Furthermore, QuaSR consistently improves ASR adaptation over standard fine-tuning and alternative data selection strategies, highlighting a new way to leverage difficulty scores for low-resource speech learning.
Primary: The University of Melbourne
All Institutions: The University of Melbourne, Wuhan University
QuaSR presents a robust, multi-dimensional quality-aware reweighting framework that effectively improves ASR adaptation for low-resource Pacific Indigenous languages by distinguishing between data reliability and model learnability, offering a scalable solution for noisy supervision in under-resourced domains.
The paper proposes QuaSR, a sample reweighting framework for Automatic Speech Recognition (ASR) adaptation in low-resource settings. The core methodology involves calculating a "sample utility" score by multiplying a data-side reliability term (derived from acoustic quality, transcript stability, and alignment reliability) with a model-side learnability term (derived from teacher-forced loss). This approach attempts to disentangle "hard but learnable" samples from "noisy/unreliable" samples, which is a critical distinction in low-resource domains where data quality is heterogeneous. The methodology is well-motivated and addresses a genuine gap in current PEFT (Parameter-Efficient Fine-Tuning) practices for ASR, which often treat all samples equally or rely solely on loss-based filtering. However, the novelty is moderate as sample reweighting based on quality heuristics is not entirely new in the broader ML community, though its specific application and combination for Pacific Indigenous languages is a novel contribution. The technical implementation details, particularly the alignment reliability metric, are somewhat opaque in the text (referencing undefined variables like 'm' and 'pho' in the raw text), suggesting a need for clearer formalization.
The evaluation is conducted on four Pacific Indigenous languages (Bislama, Nafsan, Lelepa, Nguna) using Whisper-small fine-tuned with LoRA. The results demonstrate that QuaSR consistently improves WER and CER compared to standard unweighted fine-tuning and hard filtering baselines. The gains are most significant for languages with lower data volume and higher quality heterogeneity (Lelepa, Nguna), which aligns with the paper's hypothesis. The analysis of why certain reliability signals (e.g., transcript-alignment vs. acoustic-alignment) work better for specific languages adds valuable empirical insight. However, the comparison is limited to Whisper-small and LoRA; it does not explore other foundation models or adaptation techniques. The ablation studies on the reliability components are helpful but could be more extensive. The claim of "largest gains" is supported by the data, but the absolute performance metrics remain relatively high, indicating the inherent difficulty of the task.
The paper provides sufficient detail on the experimental setup, including model architecture (Whisper-small, LoRA ranks, learning rates), dataset sources (PARADISEC), and the specific metrics used for reliability estimation (SQUIM/STOI, MMS forced alignment). The use of standard tools (SQUIM, MMS) enhances reproducibility. However, the raw text contains placeholders and undefined variables in the equations section (e.g., "what is pho here?", "m is just a symbol"), which suggests that the final published version must address these clarity issues. The code is not explicitly linked, which hinders immediate reproducibility.
The primary limitation is the dependency on external tools for reliability estimation (SQUIM, MMS), which may not be available or accurate for all low-resource languages. The method assumes that the "unweighted baseline" run is a good proxy for learnability, which might not hold if the baseline fails to converge. The paper does not address the computational overhead of computing these reliability scores, which could be significant for very large datasets. Furthermore, the "transcript stability" metric relies on heuristic counting of irregularities, which may not capture all forms of transcription error. The evaluation is restricted to four languages, limiting the generalizability of the findings to other low-resource language families.
This work has significant positive impact for the preservation and technological empowerment of Indigenous communities in the Pacific region. By improving ASR performance on under-resourced languages, it facilitates better access to technology and documentation for these communities. The framework also offers a generalizable approach for handling noisy, low-quality data in other domains beyond speech, such as text classification or computer vision, where data quality is a major bottleneck. The emphasis on data quality over quantity shifts the narrative in low-resource ML towards more sustainable and ethical data practices. QuaSR presents a robust, multi-dimensional quality-aware reweighting framework that effectively improves ASR adaptation for low-resource Pacific Indigenous languages by distinguishing between data reliability and model learnability, offering a scalable solution for noisy supervision in under-resourced domains.
Crossmodal correspondences between sound and taste are well established in psychology and neuroscience, but largely absent from content-based multimedia retrieval. We formalise taste-from-audio prediction as a content-based music information retrieval benchmark over a perceptually validated multi-source corpus, comparing ten frozen audio encoders from the four HEAR families under a shared multi-task regression head, with gated late-fusion as a configurable variant. In order to assess the effectiveness of the models, we compute absolute error and rank correlation. The strongest systems predict the five tastes within a macro RMSE of 0.134; on held-out real music their error is less than half a single rater's deviation from the consensus (RMSE 0.13 vs. 0.28), so the model tracks the group consensus more closely than an average human rater, and well below the previous state of the art baseline (0.219). On absolute error the encoders are statistically flat, with a single VGGish matching the best fusion, but gated late-fusion's advantage is confined to rank correlation (macro Pearson r 0.724 vs. 0.666). Operationalised as a content-based retrieval index, the predicted taste space ranks a 309-item pool far more faithfully than a CLAP-text baseline, which sits at chance; ridge probes and an audio-bandstop knockout read the strongest representations against documented sound-taste correspondences.
Primary: University of Padua
All Institutions: University of Padua
The paper presents a well-executed benchmark for taste-from-audio prediction, demonstrating that frozen audio encoders can predict taste intensities with accuracy exceeding average human raters, while providing valuable insights into the spectral features underlying crossmodal correspondences.
The paper proposes a systematic benchmark for predicting five basic taste intensities from audio embeddings. The methodology is straightforward: it freezes ten pre-trained audio encoders (spanning supervised, self-supervised, multimodal, and music-specific families) and trains a lightweight multi-task regression head (MLP) on top. A key methodological contribution is the "gated late-fusion" mechanism to combine complementary encoders. The approach is technically sound but not architecturally novel; it applies standard transfer learning and ensemble techniques to a new domain (crossmodal taste prediction). The use of masked MSE loss to handle partial annotations is a practical and necessary detail. The interpretability layer using ridge probes and bandstop knockouts is a strong methodological addition, linking model behavior to psychophysical literature.
The experimental setup is rigorous for the domain. The authors evaluate on a perceptually validated corpus, comparing against a previous SOTA. They provide a detailed breakdown of performance by taste and by data source (real vs. generated music). The comparison against human inter-rater agreement is a compelling metric, showing the model outperforms average human raters on absolute error for real music. The retrieval experiments, while limited by the small pool size (309 items), effectively demonstrate the utility of the predicted taste space for content-based retrieval, showing it outperforms text-based baselines. The statistical analysis includes bootstrap tests, adding credibility to the encoder rankings.
The paper provides significant detail for reproducibility. It specifies the encoders, the architecture of the head (hidden size, dropout, activation), training hyperparameters (AdamW, LR, batch size), and the fusion strategy. Crucially, it provides code and dataset links. The use of frozen encoders and cached embeddings makes the computational cost low and the setup easy to replicate. The five-seed averaging for reported metrics is a good practice for small datasets.
The primary limitation is the small dataset size (269 train, 40 test clips), which restricts the generalizability of the findings and the complexity of the models that can be effectively trained. The authors acknowledge this and frame the work as a benchmark rather than a scalable solution. The crossmodal mappings are largely based on Western psychophysics, potentially limiting cross-cultural applicability. The "spicy" label is noted as less constrained by prior evidence. The retrieval evaluation is limited to a small pool, which may not reflect performance on large-scale collections.
This work opens a new avenue for content-based music retrieval and recommendation by introducing "taste" as a semantic axis. It bridges the gap between MIR and psychophysics/neuroscience, providing a computational framework for studying crossmodal correspondences. The potential applications in sonic seasoning, multisensory design, and personalized audio experiences are significant. It also highlights the limitations of current text-based multimodal models (like CLAP-text) in capturing non-linguistic perceptual qualities. The paper presents a well-executed benchmark for taste-from-audio prediction, demonstrating that frozen audio encoders can predict taste intensities with accuracy exceeding average human raters, while providing valuable insights into the spectral features underlying crossmodal correspondences.
While Large Multimodal Models excel in comprehension, high-throughput inference engines lack native support for multimodal generation. This is severe in Speech Language Models, where generating multi-layered audio tokens via decoupled AR+NAR or synchronous Multi-Token Prediction (MTP) with delay-pattern interleaving conflicts with standard single-stream loops. We present a vLLM-based inference pipeline for unified speech understanding and generation. We extend autoregressive decoding to natively execute delay-pattern de-interleaving and coordinated multi-stream sampling, integrating an on-GPU acoustic decoder for end-to-end waveform synthesis. Crucially, we overcome the shared intuition that Classifier-Free Guidance (CFG) halves throughput. By co-scheduling paired conditional and unconditional requests within a continuous batch, our CFG implementation sustains 80% of non-CFG throughput, absorbing dual-request and logit merging overheads. We open-source our framework.
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University, Shanghai Jiao Tong University
This paper presents a significant systems contribution to the deployment of Speech Language Models by extending the vLLM inference engine to natively support multi-token audio generation and efficient Classifier-Free Guidance. By introducing paired request co-scheduling, the authors overcome the traditional throughput penalty of CFG, enabling high-fidelity audio synthesis at scale, which is a critical step for making unified audio understanding and generation models practically viable in real-world applications.
The paper addresses a critical infrastructure gap in Speech Language Models (SpeechLMs): the lack of high-throughput inference support for multi-token generation and Classifier-Free Guidance (CFG) in existing engines like vLLM. The proposed methodology involves extending the vLLM architecture to handle "delay-pattern" interleaving common in Residual Vector Quantization (RVQ) based audio codecs. The core technical innovation is the "Paired Request Co-Scheduling" mechanism for CFG. Instead of treating conditional and unconditional passes as separate requests (which fragments KV cache and prevents joint batching), the authors fuse them into a single scheduling unit. This allows the transformer backbone to process both branches in a single forward pass (sharing the KV cache computation for the shared prefix), significantly reducing overhead. The integration of an on-GPU acoustic decoder further unifies the pipeline. The approach is technically sound and leverages low-level engine optimizations (PagedAttention, continuous batching) effectively.
The evaluation is conducted on three representative SpeechLM architectures (Bagpiper, OpusLM, OpusLM-Dialogue) on a single H100 GPU. The authors demonstrate massive throughput gains (two orders of magnitude) over sequential PyTorch baselines. Crucially, they address the CFG bottleneck, showing that their co-scheduling method sustains 80% of non-CFG throughput, whereas naive implementations would suffer significant degradation. Numerical correctness is verified against reference implementations, showing strict alignment in FP32 and acceptable divergence in BF16 that does not impact end-to-end quality metrics (WER, UTMOS). The experiments are comprehensive for a systems paper, covering throughput, hardware utilization (MFU), and output quality.
The authors state they open-source their framework, which is a strong positive for reproducibility. The paper provides detailed descriptions of the architecture, including the phase state machine for mixed-modality handling and the specific logit merging logic for CFG. The experimental setup is clearly defined (H100, FlashAttention-3, specific models). However, the lack of a provided URL in the metadata requires verification of the actual repository link, though the claim of open-sourcing is present in the abstract.
The paper focuses heavily on the inference engine and does not propose new model architectures or training methods. The performance gains are specific to the vLLM architecture and RVQ-based SpeechLMs; generalization to other codec structures or non-autoregressive models is not discussed. The CFG optimization is specific to the "paired" request pattern and may not generalize to more complex guidance schemes (e.g., iterative refinement). The evaluation is limited to a single GPU (H100), so scalability to multi-GPU setups is not demonstrated.
This work significantly lowers the barrier to deploying high-fidelity SpeechLMs by making them computationally viable for real-time or high-concurrency applications. By enabling efficient CFG, it improves the quality of generated speech without prohibitive latency costs. This contributes to the broader field of multimodal AI by bridging the gap between model capability and system efficiency. This paper presents a significant systems contribution to the deployment of Speech Language Models by extending the vLLM inference engine to natively support multi-token audio generation and efficient Classifier-Free Guidance. By introducing paired request co-scheduling, the authors overcome the traditional throughput penalty of CFG, enabling high-fidelity audio synthesis at scale, which is a critical step for making unified audio understanding and generation models practically viable in real-world applications.
Humans can selectively attend to a target sound and estimate its direction in complex scenarios, whereas such selective localization remains challenging for current deep learning-based systems. Sound source localization (SSL) has achieved remarkable success with deep learning, yet most methods localize all active sources without selectivity. Conversely, target sound extraction (TSE) extracts sources using multimodal prompts but typically fails to preserve the multichannel spatial information required for accurate localization. To bridge this gap, we formulate the task of prompt-guided selective target sound localization and propose SelectTSL, an end-to-end architecture that localizes only the user-specified target in multi-source acoustic scenes. Specifically, we design a target-aware selective localization strategy that employs a Prompt-Guided Selective Attention Module (PGSA) to generate prompt-informed embeddings. These embeddings guide an inter-channel phase difference (IPD) enhancer to refine raw phase cues, fusing with target magnitudes to jointly estimate direction of arrival (DoA) and target-source cardinality, i.e., the number of target sound sources. This coupled design effectively focuses on the user-specified target spatial cues for selective localization and also handles time-varying numbers of target sources. Extensive experiments on both synthetic data and real-world recordings demonstrate that our proposed method consistently outperforms other baselines and exhibits robust generalization to real acoustic environments.
Primary: Fudan University
All Institutions: Fudan University
[One sentence main contribution]. This paper introduces SelectTSL, an end-to-end framework for prompt-guided selective target sound localization that jointly estimates DoA and source cardinality using prompt-informed spatial cue refinement. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The proposed method effectively addresses the challenge of localizing specific sound sources in complex multi-source environments by integrating semantic guidance from multimodal prompts with precise spatial cue enhancement. The novel combination of a PGSA module for target extraction and an IPD enhancer conditioned on extraction-informed embeddings allows for robust and selective localization, outperforming existing methods in both static and dynamic metrics. The joint prediction of DoA and cardinality provides a flexible solution for handling varying numbers of active sources, making it a significant advancement in the intersection of target sound extraction and sound source localization.
The paper proposes SelectTSL, a novel framework for prompt-guided selective target sound localization (SSL). The core innovation lies in decoupling semantic selection from spatial estimation. It employs a Prompt-Guided Selective Attention (PGSA) module that uses multimodal prompts (text or audio) to generate extraction-informed embeddings (EIEs). These EIEs condition an Inter-Channel Phase Difference (IPD) enhancer to refine spatial cues, which are then fused with target magnitudes for Direction of Arrival (DoA) estimation. A key technical contribution is the joint prediction of DoA posteriorgrams and source cardinality, allowing the system to handle time-varying numbers of active target sources without relying on fixed track slots. The architecture integrates a DPRNN-based extraction network with a specialized DoA estimator using depthwise-separable convolutions and temporal modeling (TCN/BiGRU). The approach effectively bridges the gap between Target Sound Extraction (TSE) and SSL, addressing the "cocktail party problem" by enabling users to specify *which* source to localize.
The authors conduct extensive experiments on a large-scale synthetic dataset (288.9 hours of training data) generated using dynamic Room Impulse Responses (RIRs) and real-world recordings from the TAU-SRIR dataset. They compare SelectTSL against a comprehensive taxonomy of baselines, including track-wise SSL methods (IPDNet, EINV2), SELD methods (SELDnet, SELDT), pure DoA methods (SRP-DNN, FN-SSL), and prompt-based localization (SEL). Results demonstrate that SelectTSL significantly outperforms all baselines, achieving an MAE of 0.98° and a MOTA of 91.57% on the synthetic test set, compared to the next best prompt-based method (SEL) which achieved 2.78° MAE and 16.25% MOTA. The paper includes detailed ablation studies validating the contributions of the PGSA module, IPD enhancement, cardinality head, and training scheme. The robustness to noise and reverberation is also evaluated.
The paper provides detailed implementation details, including STFT parameters, network architecture dimensions, loss function weights, and training hyperparameters (Adam optimizer, learning rate, early stopping). The dataset generation process is clearly described, including room dimensions, RIR simulation tools (GPURIR), and source/noise datasets. The authors state that the code and dataset will be released, which enhances reproducibility. The synthetic data generation protocol is sufficiently detailed for replication.
The primary limitation is the reliance on synthetic data for training and primary evaluation. While real-world evaluation is performed, it is limited to a subset of TAU-SRIR rooms with fixed source positions (simulated movement via static RIRs), which may not fully capture the complexities of real-time moving source localization with dynamic acoustic changes. The dual-microphone setup limits the spatial resolution and front-back ambiguity handling compared to larger arrays. The method's performance on highly complex, multi-source scenarios with more than two concurrent targets is not explicitly evaluated (cardinality head is limited to 0, 1, or 2).
This work has significant implications for human-computer interaction, particularly in smart speakers, hearing aids, and robotics, where selective auditory attention is crucial. By enabling users to specify a target sound via text or audio, the system can improve speech enhancement, far-field ASR, and spatial audio perception in noisy environments. It advances the field of audio AI by integrating semantic understanding with precise spatial sensing. [One sentence main contribution]. This paper introduces SelectTSL, an end-to-end framework for prompt-guided selective target sound localization that jointly estimates DoA and source cardinality using prompt-informed spatial cue refinement. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The proposed method effectively addresses the challenge of localizing specific sound sources in complex multi-source environments by integrating semantic guidance from multimodal prompts with precise spatial cue enhancement. The novel combination of a PGSA module for target extraction and an IPD enhancer conditioned on extraction-informed embeddings allows for robust and selective localization, outperforming existing methods in both static and dynamic metrics. The joint prediction of DoA and cardinality provides a flexible solution for handling varying numbers of active sources, making it a significant advancement in the intersection of target sound extraction and sound source localization.
We propose a lightweight multi-path alignment network (LMPAN) for on-device joint acoustic echo cancellation (AEC) and noise suppression (NS) in full-duplex spoken dialogue systems. To address hardware-induced distortions and dynamic acoustic conditions, we introduce three core innovations: (1) a multi-path alignment stage correcting temporal and energy mismatches across reference, linear AEC (LAEC) output, and microphone signals; (2) an attention-based mechanism that dynamically integrates enhanced LAEC and microphone features under varying acoustic scenarios; (3) a post-filtering module with a dynamic target generation strategy for downstream tasks (ASR, VAD). Furthermore, we adopt a two-stage training framework leveraging self-supervised learning representations to enhance perceptual quality. Experiments show that LMPAN, with only 480K parameters and 126 MACs, achieves performance comparable to the state-of-the-art lightweight model DeepVQE-S, while ensuring real-time inference capability.
Primary: TongYi AI Lab of Alibaba Group
All Institutions: Qwen Business Unit of Alibaba, TongYi AI Lab of Alibaba Group
[One sentence main contribution]. LMPAN introduces a lightweight multi-path alignment network with SSL-guided training to achieve robust joint AEC and NS on edge devices. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a well-engineered hybrid solution that addresses the critical challenge of hardware-induced misalignments in full-duplex systems. By combining explicit temporal/energy alignment with attention-based fusion and perceptual SSL losses, it achieves a favorable balance between model size, computational cost, and performance. While not theoretically groundbreaking, the systematic integration of these components and the rigorous evaluation on downstream tasks make it a valuable contribution to practical audio signal processing for on-device AI.
The paper proposes LMPAN, a lightweight network for joint Acoustic Echo Cancellation (AEC) and Noise Suppression (NS). The core methodological contribution lies in the explicit handling of temporal and energy misalignments between the reference, microphone, and Linear AEC (LAEC) outputs via a multi-path alignment stage. This is followed by an attention-based fusion module and a dynamic target adaptation strategy. The use of Self-Supervised Learning (SSL) representations (WavLM) in a two-stage training framework is a notable technical choice, aiming to preserve perceptual quality and semantic fidelity. However, the individual components (LAEC hybrid, attention fusion, SSL loss) are incremental combinations of existing techniques rather than fundamentally new algorithmic breakthroughs. The "multi-path alignment" is a practical engineering solution to a known problem (hardware latency) but lacks deep theoretical novelty in the context of modern end-to-end learning.
The experimental setup is rigorous, utilizing standard benchmarks (AEC Challenge 2023) and a self-collected real-world dataset from smartphones. The inclusion of downstream task evaluation (ASR, VAD, TIR) is a strong point, as it demonstrates the practical utility of the enhanced audio. The results show that LMPAN achieves performance comparable to DeepVQE-S with significantly fewer parameters (480K vs 820K) and lower computational cost (126M vs 315M MACs). The ablation studies effectively isolate the contributions of the alignment module, fusion module, and SSL training. The trade-off analysis of the Dynamic Target Adaptation (DTA) parameters provides valuable insight into balancing objective metrics (ERLE, PESQ) with subjective/perceptual quality (WER, MOS).
The paper provides sufficient implementation details, including STFT parameters, optimizer settings, and dataset augmentation strategies. The use of standard libraries (WavLM, Paraformer) aids reproducibility. However, the specific hyperparameters for the dynamic target generation and the exact architecture of the GTCRN-based refinement branches could be more explicitly defined. The self-collected dataset details are described but not publicly available, which limits independent verification on real-world hardware variations.
The primary limitation is the incremental nature of the contributions. The method relies heavily on a pre-computed LAEC stage, which, while common in hybrid systems, contradicts the trend towards fully end-to-end neural AEC. The performance gains, while consistent, are modest in absolute terms compared to the massive leaps seen in generative audio models. Furthermore, the "lightweight" claim is relative; 480K parameters is small for speech models but not negligible for extremely constrained edge devices. The reliance on SSL features adds inference overhead (though frozen) and complexity to the training pipeline.
This work contributes to the deployment of robust full-duplex spoken dialogue systems on edge devices, which is critical for the widespread adoption of always-on AI assistants. By improving the robustness of AEC and NS under hardware-induced distortions, it enhances the user experience and the reliability of downstream AI tasks like ASR and VAD. The emphasis on lightweight models supports sustainable AI by reducing computational resources required for real-time audio processing. [One sentence main contribution]. LMPAN introduces a lightweight multi-path alignment network with SSL-guided training to achieve robust joint AEC and NS on edge devices. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a well-engineered hybrid solution that addresses the critical challenge of hardware-induced misalignments in full-duplex systems. By combining explicit temporal/energy alignment with attention-based fusion and perceptual SSL losses, it achieves a favorable balance between model size, computational cost, and performance. While not theoretically groundbreaking, the systematic integration of these components and the rigorous evaluation on downstream tasks make it a valuable contribution to practical audio signal processing for on-device AI.
Real-time binaural speech enhancement is constrained by latency, computational cost, and inter-device communication, yet existing efficient solutions predominantly address single-channel settings. In this paper, we introduce RT-Tango, a real-time distributed binaural speech enhancement framework designed for streaming on resource-constrained platforms and specifically for hearing aids. RT-Tango relies on a two-stage distributed architecture combining perceptually motivated ERB feature compression, lightweight grouped recurrent mask estimation, and temporal sparsification to reduce computational cost. Stringent latency constraints are addressed by decoupling spectral resolution from algorithmic delay using an asymmetric STFT, together with causal recurrent inference and online estimation of spatial statistics. Experimental results show that RT-Tango achieves competitive speech enhancement while significantly reducing MACs operations and functioning at ultra-low latencies as low as 8 ms.
Primary: Université Paris-Saclay, CEA, List
All Institutions: Université Paris-Saclay, CEA, List; Université de Lorraine, CNRS, Inria, LORIA
This paper presents a practical and well-engineered solution for real-time binaural speech enhancement on hearing aids, effectively balancing computational efficiency, latency, and performance through a combination of perceptual feature compression, grouped recurrent modeling, and asymmetric streaming STFT.
The paper proposes RT-Tango, a distributed binaural speech enhancement framework specifically optimized for the stringent latency and computational constraints of hearing aids. The core technical contribution lies in the system-level integration of several efficiency-driven components: (1) Perceptually motivated ERB feature compression to reduce input dimensionality; (2) Grouped Recurrent Neural Networks (GRNNs) to parallelize processing and reduce the quadratic complexity of full-band recurrent layers; (3) Temporal sparsification (Fixed-Rate Skipping and learned gating) to reduce inference frequency; and (4) Asymmetric STFT configurations to decouple spectral resolution from algorithmic latency, enabling ultra-low latency (8 ms) streaming. The approach adapts the existing Tango architecture by replacing its CNN-based components with lightweight recurrent and grouped variants, focusing on hardware-aware efficiency rather than purely algorithmic novelty. The methodology is sound and directly addresses the trade-off between performance and resource consumption in embedded audio systems.
The experimental evaluation compares RT-Tango against the original Tango, a causal RNN variant (Tango-RNN), and a lightweight CNN baseline (GTCRN). Results are reported on a simulated binaural dataset using standard metrics (SI-SDR, SI-SIR, SI-SAR, STOI, PESQ). The paper demonstrates that RT-Tango achieves competitive performance with significantly lower computational cost (MACs/s) compared to GTCRN at similar frame rates, and lower cost than Tango-RNN while supporting higher frame rates. The ablation studies effectively isolate the contributions of grouping, temporal sparsification, and asymmetric STFT. However, the evaluation relies entirely on simulated data with measured room impulse responses, lacking real-world hardware deployment validation or subjective listening tests, which are critical for hearing aid applications. The comparison with GTCRN is somewhat uneven as GTCRN is a monaural/single-node model adapted to the setting, whereas RT-Tango leverages binaural spatial cues.
The paper provides sufficient detail regarding the architecture (group sizes, STFT parameters, hop sizes) and training setup (optimizer, loss function, dataset generation protocol). The use of standard datasets (LibriSpeech, BinauRec) and metrics aids reproducibility. However, the specific implementation details of the learned skip gating mechanism and the exact hyperparameters for the online SCM estimation convergence are described qualitatively or with limited quantitative precision, which may hinder exact replication. The code is not publicly available (URL: none), which is a significant barrier to full reproducibility.
The primary limitation is the reliance on simulated data, which may not capture the full complexity of real-world acoustic environments and hearing aid hardware imperfections (e.g., microphone mismatch, wind noise). The lack of subjective listening tests (MOS/MUSHRA) is a notable gap for a hearing aid paper, as objective metrics do not always correlate perfectly with user-perceived quality and comfort. Additionally, the "online" variant (RT-Tango-OS) shows a performance drop in SI-SDR/SI-SAR compared to the offline version, and the paper does not extensively discuss the impact of this degradation on user experience. The comparison with other distributed binaural methods is limited, as the field is still emerging.
This work has significant potential impact on the field of assistive listening devices. By demonstrating that high-quality binaural speech enhancement is feasible on low-power, resource-constrained hearing aids with ultra-low latency, it paves the way for more effective real-time noise reduction for hearing aid users. The techniques for efficient distributed processing and low-latency streaming are also relevant to other edge-Audio applications, such as smart speakers and wearable audio devices. The focus on perceptual metrics and interaural balance aligns well with the clinical requirements of hearing rehabilitation. This paper presents a practical and well-engineered solution for real-time binaural speech enhancement on hearing aids, effectively balancing computational efficiency, latency, and performance through a combination of perceptual feature compression, grouped recurrent modeling, and asymmetric streaming STFT.
Estimating a speaker's head orientation from audio can provide valuable information in smart environments, meetings, and driver monitoring. We propose a novel approach that leverages the phase component of the short-time Fourier transform from a single microphone array as input to a deep neural network combining convolutional, recurrent, and self-attention layers. Unlike prior methods that use physics-informed handcrafted features or raw waveform inputs, our approach enables robust learning from simulated and real data. Trained on a large-scale dataset generated with voice directivity patterns and fine-tuned on real recordings, our model achieves state-of-the-art accuracy, outperforming baselines under both clean and noisy conditions. Personalization experiments further demonstrate significant gains, reaching a mean angular error of 11.3 degrees when adapting to individual users and environments.
Primary: Tampere University
All Institutions: Tampere University
This paper presents a robust deep learning approach for speaker head orientation estimation using STFT phase features, demonstrating superior performance over raw audio and handcrafted features in noisy and reverberant environments, with significant gains achieved through user-specific personalization.
The paper proposes a deep learning framework for speaker head orientation estimation using a single microphone array. The core methodological contribution is the use of the phase component of the Short-Time Fourier Transform (STFT) as the primary input feature, rather than raw waveforms or handcrafted features like GCC-PHAT. The architecture combines 2D Convolutional Neural Networks (CNNs) for spatial feature extraction, Bidirectional Gated Recurrent Units (GRUs) for temporal modeling, and Multi-Head Self-Attention mechanisms. The input representation involves stacking the sine and cosine of the phase from all microphone channels. The model predicts orientation in a continuous regression format using sine/cosine representation to handle circular continuity. The approach is technically sound, leveraging established deep learning components in a novel configuration for this specific acoustic task. The use of phase-only features is the key differentiator, justified by the hypothesis that phase carries robust directional cues, particularly in reverberant environments.
The evaluation is comprehensive, covering both simulated and real-world data. The simulated dataset is generated using Voice Directivity Patterns (VDP) and the VCTK corpus, with room acoustics simulated via the image source method. This allows for controlled testing across various noise levels (clean, moderate, high SNR) and reverberation conditions. The paper compares the proposed method against three baselines: Soundr (CNN+LSTM on raw audio), a method based on ITD/ILD features, and a physics-informed feature-based method. Results indicate that the phase-based model outperforms baselines in noisy and reverberant conditions, which is a significant finding given the sensitivity of phase to environment. Personalization experiments (fine-tuning on user/room data) show significant performance gains, reducing Mean Angular Error (MAE) to 11.3 degrees. The evaluation on a real-world dataset further validates the generalization capability, although the absolute performance metrics on real data are not explicitly detailed in the text provided (referenced as Table REF). The ablation of reverberation effects (anechoic vs. reverberant) provides valuable insight into the model's behavior.
The paper provides sufficient detail for reproduction. The STFT parameters (4ms window, 2ms stride), network architecture (3 conv layers, 2 GRU layers, 2 attention blocks), and training details (Adam optimizer, MSE loss, 200k iterations) are specified. The dataset generation process is clearly described, including the use of VCTK, VDPs, and Pyroomacoustics for simulation. The noise augmentation strategy (phase-randomized monophonic noise) is also detailed. However, the specific hyperparameters for the personalization fine-tuning (learning rate, number of epochs) are not explicitly stated, which might require some trial and error for exact reproduction. The code is not explicitly linked, but the methodology is clear enough to implement.
The paper acknowledges several limitations. First, the model's generalization to unseen users and environments is limited, necessitating personalization/fine-tuning for optimal performance. This suggests the model relies heavily on specific acoustic characteristics of the training data. Second, the reliance on simulated data for pre-training, while effective, introduces a domain gap that requires real-world fine-tuning. Third, the phase-only input might discard magnitude information that could be beneficial in certain conditions, although the authors claim no benefit was found. Finally, the evaluation is limited to azimuthal orientation; elevation estimation is not addressed. The performance in highly non-stationary noise or with significant speaker movement is not evaluated.
Accurate and privacy-preserving head orientation estimation has significant implications for human-computer interaction, smart homes, meeting transcription systems, and driver monitoring. By using audio-only sensors, this technology avoids the privacy concerns associated with cameras. The ability to work with compact microphone arrays makes it deployable in consumer devices. The focus on robustness to noise and reverberation enhances its practical utility in real-world environments. The personalization aspect highlights the need for adaptive systems in user-centric AI. This paper presents a robust deep learning approach for speaker head orientation estimation using STFT phase features, demonstrating superior performance over raw audio and handcrafted features in noisy and reverberant environments, with significant gains achieved through user-specific personalization.
Modern automatic speaker verification (ASV) systems are vulnerable to adversarial perturbations. Diffusion-based purification has recently shown strong effectiveness against such perturbations, but its reverse denoising process requires iterative sampling and leads to high inference latency. We find that the forward noising process provides most of the robustness gain. Motivated by this observation, we reformulate adversarial purification as a learnable noising problem, and propose the Positive-Incentive Noise Predictor (PnP), the first framework that explicitly introduces positive-incentive noise (π-noise) into the purification task. PnP learns input-adaptive π-noise and mixes it with the input to improve the robustness of downstream ASV systems. Experiments on four advanced ASV backbones show that PnP effectively defends against adversarial attacks while preserving performance on natural speech. Compared with representative purification baselines, the proposed framework provides a competitive balance among defense effectiveness, impact on genuine utterances, and inference efficiency under white-box, black-box, and defender-aware adaptive attacks, with a real-time factor as low as 0.014. Moreover, PnP can be cascaded with a diffusion denoiser to further improve the perceptual quality of purified utterances. Code and purified audio examples are available at https://eurecom-asp.github.io/pnp/
Primary: EURECOM
All Institutions: EURECOM, The University of Sydney, Northwestern Polytechnical University, China Telecom (TeleAI), Research and Development Institute of Northwestern Polytechnical University in Shenzhen
The paper presents a significant and well-executed contribution to adversarial robustness in speaker verification by reformulating diffusion-based purification as a learnable forward noising problem, achieving a superior balance between defense effectiveness, inference efficiency, and audio quality.
The paper proposes a novel paradigm shift in adversarial purification for Automatic Speaker Verification (ASV). Instead of relying on the computationally expensive reverse denoising process of diffusion models, the authors hypothesize that the forward noising process provides the majority of the robustness gain. They introduce the Positive-Incentive Noise Predictor (PnP), which learns an input-adaptive noise pattern ($\pi$-noise) that is task-beneficial (i.e., it preserves speaker identity while suppressing adversarial perturbations). The methodology involves training a U-Net based noise predictor using a variational lower bound of mutual information, instantiated as a hinge loss on ASV similarity scores. The framework includes variants like PnP-Gaussian (simple additive) and PnP-Diff (diffusion-style schedule). The approach is theoretically grounded in information theory and practically motivated by the inefficiency of current diffusion-based purifiers.
The experimental evaluation is comprehensive and rigorous. The authors test on four state-of-the-art ASV backbones (ECAPA-TDNN, CAM++, ResNet, SimAMResNet) and under three attack settings: white-box (MI-FGSM, PGD), black-box (FAKEBOB), and defender-aware adaptive attacks. They compare against strong baselines including DAP, AudioPure, and neural codecs. Key findings include: 1) PnP-Diff achieves state-of-the-art robustness with a very low Real-Time Factor (RTF) of 0.014, significantly faster than iterative diffusion methods. 2) The forward-process-only hypothesis is validated, showing minimal performance drop compared to full diffusion pipelines. 3) Cascading PnP with a diffusion denoiser improves perceptual quality (WB-PESQ, SI-SDR) without significantly compromising robustness. The ablation studies on hyperparameters and purification steps add depth to the analysis.
The paper provides detailed descriptions of the architecture, loss functions, and training procedures. The code and purified audio examples are available via the provided URL, which greatly enhances reproducibility. The datasets (VoxCeleb, LibriSpeech) are standard and accessible. The use of open-source toolkits (WeSpeaker, torchattacks) further supports reproducibility.
The primary limitation is that PnP-Gaussian, while fast, degrades audio quality significantly (low WB-PESQ, high WER), making it less suitable for applications where perceptual quality is critical. The PnP-Diff variant is better balanced but still introduces some distortion compared to the clean signal. The method is specifically tailored for ASV; its generalizability to other audio tasks (e.g., speech recognition, emotion recognition) is not explored in depth, although the core idea might transfer. The adaptive attack evaluation is limited to gradient-based attacks through the purifier; more complex adaptive attacks (e.g., black-box adaptive) are not fully explored.
This work has significant implications for the security of biometric systems. By providing a lightweight, effective defense against adversarial attacks, it enhances the trustworthiness of ASV systems in real-world applications. The insight that forward noising can be optimized for robustness opens new avenues for efficient adversarial defense in other domains using generative models. However, the dual-use nature of adversarial attacks and defenses means that improved defenses may also spur more sophisticated attacks, necessitating continuous research. The paper presents a significant and well-executed contribution to adversarial robustness in speaker verification by reformulating diffusion-based purification as a learnable forward noising problem, achieving a superior balance between defense effectiveness, inference efficiency, and audio quality.
While prior work has explored emotion control in hybrid text-to-speech systems, the geometric properties of these modules, and their implications for steerability, remain poorly understood. We present the first comparative study of speech language model (SLM) and conditional flow-matching (CFM) modules as activation steering sites for mixed emotion speech synthesis. We first characterize emotion representations using linear probing and local intrinsic dimensionality (LID), and then evaluate single-site and joint steering for mixed-emotion synthesis. Our results show that SLM offers a clean, low-dimensional emotion-specific subspace with strong speaker--emotion disentanglement, while CFM exhibitspoor cross-speaker generalization due to speaker--emotion entanglement. Joint steering increases emotion intensity but degrades proportional control and speech quality on in-distribution data. These findings provide practical guidance for multi-site activation steering in hybrid TTS systems and highlight the importance of representation geometry in controllable speech generation.
Primary: The University of Melbourne
All Institutions: The University of Melbourne, Monash University
This paper presents the first comparative geometric analysis of activation steering in hybrid TTS models, revealing that Speech Language Models offer cleaner, more disentangled emotion subspaces than Conditional Flow-Matching modules, thereby providing crucial insights for designing effective and interpretable emotion control mechanisms in speech synthesis.
The paper employs a rigorous geometric analysis framework to compare two distinct activation steering sites (SLM and CFM) within a hybrid Text-to-Speech (TTS) architecture. The methodology is sound and well-structured, utilizing linear probing for discriminability and Local Intrinsic Dimensionality (LID) to characterize the manifold structure of emotion representations. The extraction of steering vectors via mean subtraction and the application of weighted summation for mixed emotions are standard but effectively applied in this context. The core methodological contribution lies in the systematic correlation between geometric properties (LID, discriminability gaps) and steering performance (proportional control, speaker fidelity), providing a mechanistic explanation for why certain steering sites work better than others. The analysis of joint steering interference is particularly insightful, attributing degradation to distribution shift and speaker entanglement.
The experimental setup is comprehensive, covering three major emotion datasets (ESD, CREMA-D, RAVDESS) and evaluating both emotion control metrics (E-SIM, TEP, Proportional Control, H-Rt) and speech quality metrics (S-SIM, WER). The results clearly demonstrate the trade-offs: SLM offers better proportional control and speaker preservation, while CFM offers higher intensity but suffers from speaker entanglement. The use of objective metrics like WER and S-SIM alongside emotion-specific embeddings adds robustness. However, the reliance on objective metrics for emotion perception (E-SIM, TEP) rather than human subjective evaluation (MOS/MUSHRA) is a limitation, though common in early-stage steering papers. The ablation on steering strength and the comparison of single vs. joint steering provide strong empirical evidence for the geometric claims.
The paper provides sufficient detail for reproduction, including the backbone model (CosyVoice2), specific layers for steering (SLM layers 14, 17; CFM every 5th layer), and the datasets used. The mathematical definitions for LID and steering vector extraction are clear. However, the code is not explicitly linked in the text provided, and some hyperparameters for the linear probes and LID estimation (e.g., K for nearest neighbors) are referenced via citations rather than detailed in the main text, which might slightly hinder immediate reproducibility without accessing the cited works or supplementary material.
The primary limitation is the lack of human subjective evaluation for emotion perception and speech quality, relying solely on proxy metrics. The study is confined to a single hybrid TTS architecture (CosyVoice2), limiting the generalizability of the geometric findings to other architectures (e.g., end-to-end diffusion TTS or different hybrid designs). The joint steering analysis is limited to in-distribution data; out-of-distribution generalization of the interference effects is not explored. Additionally, the LID analysis, while insightful, is computationally expensive and sensitive to the choice of K, which is not fully ablated.
This work significantly advances the understanding of controllable speech generation by linking representation geometry to steering efficacy. It provides practical guidelines for developers of hybrid TTS systems, suggesting that SLM is a superior site for precise emotional mixing, while CFM is better for intensity but risky for speaker identity. The findings on speaker-emotion entanglement in flow-matching modules have broader implications for interpretability and control in generative models. By highlighting the geometric properties of latent spaces, the paper contributes to the growing field of mechanistic interpretability in audio generation. This paper presents the first comparative geometric analysis of activation steering in hybrid TTS models, revealing that Speech Language Models offer cleaner, more disentangled emotion subspaces than Conditional Flow-Matching modules, thereby providing crucial insights for designing effective and interpretable emotion control mechanisms in speech synthesis.
Multichannel Deep Neural Networks (DNNs) have significantly improved speech enhancement performance; however, they typically remain constrained by reliance on fixed microphone array geometries, leading to poor generalization on unseen or irregular configurations. Current array-agnostic approaches often rely on high-complexity architectures or massive, diverse datasets, yet they still struggle to generalize to out-of-distribution layouts. In this paper, we present an in-depth analysis of AmbiDrop, a recently proposed framework that achieves geometry independence by leveraging ideal Ambisonics as the DNN input. By employing a channel-wise dropout layer during training to simulate Ambisonics encoding errors, AmbiDrop decouples the learning process from the physical sensor arrangement. During inference, microphone signals from arbitrary array configurations are transformed into the Ambisonics domain via Ambisonics Signal Matching (ASM) before processing. Extensive experiments demonstrate that AmbiDrop maintains high robustness across a diverse suite of unseen simulated arrays and real-world recordings. Furthermore, our results show that the framework is resilient to sensor failures and remains effective even with reduced network scales, making it highly suitable for deployment on resource-constrained edge devices and versatile wearable hardware.
Primary: Ben-Gurion University of the Negev
All Institutions: Ben-Gurion University of the Negev, Reality Labs Research at Meta
[One sentence main contribution]. [The paper presents AmbiDrop, an array-agnostic speech enhancement framework that leverages Ambisonics domain transformation and channel-wise dropout to achieve robust generalization across diverse and unseen microphone array geometries, validated on both simulated and real-world wearable hardware].
The paper proposes AmbiDrop, a framework for array-agnostic speech enhancement. The core innovation lies in decoupling the neural network training from specific microphone geometries by transforming inputs into the Ambisonics domain. Specifically, it uses ideal Ambisonics signals for training and employs a channel-wise dropout layer to simulate the encoding errors that occur when using Ambisonics Signal Matching (ASM) on physical arrays during inference. This is a clever and relatively simple mechanism to handle domain shift between ideal training data and imperfect real-world encoding. The approach is architecture-agnostic, demonstrated with FT-JNF and IC-ConvTasNet. While the concept of using spherical harmonics for array invariance is not entirely new (e.g., eigenbeam features), the specific combination of ASM for inference and dropout for robustness to encoding errors is a distinct and practical contribution. It avoids the complexity of learnable permutation-invariant layers or massive meta-learning datasets.
The experimental evaluation is comprehensive and rigorous. It covers: 1. **Simulated Data:** Extensive testing on 20 different simulated arrays (1D, 2D, 3D, free-field, rigid-sphere) including unseen test geometries. This directly addresses the generalization claim. 2. **Real-World Data:** Evaluation on Project Aria glasses, a real wearable device. This is a significant strength, moving beyond simulation. It includes tests with normal and mispositioned glasses, adding practical relevance. 3. **Ablation Studies:** Detailed analysis of dropout strategies (uniform vs. per-channel), resilience to microphone failures, and network complexity scaling. 4. **Baselines:** Comparison against standard geometry-dependent baselines and other array-agnostic approaches mentioned in the intro. The results clearly show that while baseline models fail on unseen arrays, AmbiDrop maintains performance. The drop in performance on real-world data compared to simulation is analyzed and attributed to ATF modeling inaccuracies and environmental factors, which is a honest and insightful discussion.
The paper provides detailed mathematical formulations for ASM and the Ambisonics encoding. It specifies the DNN architectures (FT-JNF, IC-ConvTasNet) and their parameters. The dataset generation process (image method, WSJ0 speech) is described. However, the exact code for ASM implementation and the specific random seeds for the simulations are not explicitly linked in the text (though often available in supplementary materials or future releases). The reliance on specific ATFs (simulated vs. measured) for the Aria glasses is well-documented. Overall, reproducibility is high given the standard nature of the components, though the specific ASM filter design details are crucial.
1. **ATF Dependency:** The performance is heavily dependent on the accuracy of the Ambisonics Signal Matching (ASM) filters. If the assumed ATF (e.g., rigid sphere) deviates significantly from the physical reality (e.g., due to head-related transfer functions not accounted for, or mispositioning), performance degrades. The paper acknowledges this but the gap between simulated and real-world ATF performance is notable. 2. **Order Limitation:** The method is demonstrated with 2nd-order Ambisonics (9 channels). Higher orders might capture more spatial detail but require more microphones and are more sensitive to spatial aliasing. 3. **Computational Overhead:** While the DNN can be small, the ASM step adds a computational burden at inference time, which might be non-trivial for very low-latency applications, although the paper argues it is suitable for edge devices. 4. **Generalization to Extreme OOD:** While it generalizes to unseen *geometries*, it assumes the sound field can be reasonably approximated by the ASM model. Highly irregular or non-spherical arrays might still pose challenges if the ASM error is too high.
This work has significant implications for wearable audio devices (hearables, smart glasses) where microphone arrays are small, irregular, and subject to movement/occlusion. By enabling a single model to work across different hardware configurations, it reduces the need for device-specific model training and deployment. This promotes interoperability and robustness in consumer electronics. It also contributes to the broader field of robust speech processing by providing a new perspective on handling sensor variability. [One sentence main contribution]. [The paper presents AmbiDrop, an array-agnostic speech enhancement framework that leverages Ambisonics domain transformation and channel-wise dropout to achieve robust generalization across diverse and unseen microphone array geometries, validated on both simulated and real-world wearable hardware].
Multimodal large language models (MLLMs) have emerged as a promising approach for improving the accuracy, transferability, and explainability of automatic dementia classification (ADC) systems from voice recordings. Yet it remains unclear whether their reasoning capabilities are beneficial for ADC, and how such capabilities should be leveraged. In this paper, we conduct a careful evaluation of reasoning MLLMs for ADC and show that naive strategies, such as relying on text-based rationales, can lead to hallucinated and inconsistent rationales for diagnosis and yield inferior ADC performance compared with LLM-free baselines. To overcome this limitation, we propose \textbf{De}mentia \textbf{T}hinker with Nonlinear \textbf{A}daptor and Re\textbf{i}nforcement \textbf{L}earning (DeTAiL), an adaptor-based framework that exploits the internal representations of reasoning MLLMs for improved dementia classification. Across two dementia datasets with distinct test formats and label granularities, DeTAiL consistently outperforms strong baselines and methods that rely on text-based rationales. Code and demo will be released upon acceptance.
Primary: MIT CSAIL
All Institutions: MIT CSAIL, Massachusetts General Hospital, Harvard Medical School
The paper presents a rigorous evaluation of reasoning MLLMs for dementia classification, proposing DeTAiL to leverage internal representations for improved accuracy and transferability, offering valuable insights into the utility of reasoning traces in medical speech analysis.
The paper proposes a novel framework, DeTAiL, to investigate whether reasoning capabilities in Multimodal Large Language Models (MLLMs) are beneficial for Automatic Dementia Classification (ADC). The core methodological contribution is the "nonlinear adaptor" stage, which extracts hidden representations from the MLLM conditioned on generated rationales, rather than relying solely on the textual output. This is combined with a distillation stage (using a teacher LLM to generate rationales) and a Reinforcement Learning stage (GRPO) to align the model. The approach attempts to bridge the gap between the generative reasoning capabilities of LLMs and the discriminative needs of speech classification, addressing the issue of hallucinated or unfaithful rationales by probing internal states. The methodology is sound and addresses a relevant gap in understanding how reasoning traces interact with downstream classification tasks in medical speech analysis.
The evaluation is conducted on two datasets: ADReSS (binary classification) and LEADS (fine-grained classification). The experiments cover in-domain performance, cross-domain transfer, and ablation studies on input modalities and layer selection. The results demonstrate that while naive reasoning strategies (text-based) can underperform or hallucinate, the proposed DeTAiL framework consistently outperforms strong baselines, including LoRA-based adaptation and text-only MLLM approaches. The cross-domain analysis is particularly valuable, showing that the hidden-state adaptor offers better transferability than LoRA in some settings. However, the paper notes that the MLP adaptor can overfit to dataset-specific patterns, which is a critical finding. The inclusion of a reliability analysis of linguistic evidence types adds depth to the evaluation.
The paper provides detailed descriptions of the datasets, model architectures (Qwen-2.5-VL), and training hyperparameters (LoRA rank, GRPO group size, learning rates). The use of open-source models and standard toolkits (ms-swift) enhances reproducibility. The authors state that code and demo will be released upon acceptance, which is standard for arXiv submissions. The specific details regarding the MLP adaptor structure and the distillation process are sufficiently described to allow replication.
The paper acknowledges several limitations. First, the LEADS dataset is private, which limits independent verification and broader community benchmarking. Second, the ASR transcripts used for LEADS may introduce noise, affecting the quality of rationales and hidden states. Third, the cross-domain transfer performance is not uniform; while DeTAiL helps, it does not fully solve the domain shift problem. Fourth, the reliability of the generated rationales is still an open question, as the paper suggests they may not always be faithful clinical explanations. Finally, the study is limited to specific MLLMs (Qwen family) and datasets, so generalizability to other models or languages is not fully established.
This work has significant implications for the development of explainable and robust AI systems for healthcare, specifically in neurodegenerative disease screening. By demonstrating that internal representations of reasoning MLLMs can be more reliable than their textual outputs for classification, it provides a pathway for building systems that are both accurate and interpretable. However, the caution regarding hallucinated rationales highlights the need for rigorous validation in clinical settings. The findings contribute to the broader understanding of how to effectively leverage large models for specialized, high-stakes tasks where reasoning is often assumed to be beneficial but may not be directly transferable. The paper presents a rigorous evaluation of reasoning MLLMs for dementia classification, proposing DeTAiL to leverage internal representations for improved accuracy and transferability, offering valuable insights into the utility of reasoning traces in medical speech analysis.
Spoken language models (SLMs) extend LLMs to speech input and output. Existing SLMs represent speech at fixed frame rates (e.g., 25 or 12.5 Hz), ignoring the time-varying information density of speech and offering no flexibility to trade off quality for speed at inference time. Recent audio tokenizer research has proposed dynamic frame rate speech coding, which exploits this non-uniformity and enables two new capabilities: very low average frame rates and frame rate controllability. However, this technique has not yet been applied to SLMs. We introduce Flexible Spoken Language Model (FlexiSLM), the first SLM that supports dynamic and controllable frame rates on both speech input and output. Using dynamic frame rate representations, FlexiSLM outperforms fixed-frame-rate 7B models including Qwen2.5-Omni and Kimi-Audio at its high-quality operating points. We further verify that FlexiSLM can be accurately steered down to 4.0 Hz; at 6.25 Hz, it roughly halves inference time relative to 12.5 Hz while retaining strong speech-to-speech quality. Audio samples are available at https://flexislm.github.io .
Primary: The Chinese University of Hong Kong, Shenzhen
All Institutions: The Chinese University of Hong Kong, Shenzhen, ByteDance
FlexiSLM introduces the first spoken language model with dynamic and controllable frame rates, achieving superior efficiency and competitive performance compared to fixed-rate 7B baselines by integrating dynamic speech tokenization with direct inference-time frame rate conditioning.
The paper proposes FlexiSLM, a spoken language model (SLM) that integrates dynamic frame rate tokenization into both the input and output streams. The core methodological contribution is the adaptation of FlexiCodec, a dynamic-rate speech tokenizer, into a full SLM pipeline. Key technical innovations include: 1) A "Thinker-Talker" architecture where the Talker module predicts dynamic-frame-rate speech tokens and their associated frame lengths. 2) Direct frame rate conditioning using sinusoidal positional encoding, allowing users to specify the target average frame rate at inference time, which is a significant improvement over the indirect threshold-based control used in prior tokenizer work. 3) A bidirectional connection between the Talker and Thinker, enabled in the final fine-tuning stage, to provide the language model with explicit knowledge of generated speech tokens. The approach effectively bridges the gap between efficient audio tokenization and end-to-end language modeling, addressing the inefficiency of fixed-rate representations in SLMs.
The authors conduct comprehensive evaluations against strong 7B baselines (Qwen2.5-Omni, Kimi-Audio, Mimo-Audio) and larger models (Gemini 2.5-Pro). Results on Kimi-Audio-Evalkit show that FlexiSLM-7B outperforms fixed-rate 7B models at 12.5 Hz and maintains strong performance at 6.25 Hz, achieving roughly half the inference time (RTF) with minimal quality degradation. The paper includes ablation studies validating the necessity of dynamic merging, the effectiveness of direct frame rate control, and the benefits of the Talker-to-Thinker connection. Additional evaluations on audio understanding (LLaSO-Eval) and ASR (LibriSpeech) further demonstrate the model's robustness. The experimental design is rigorous, covering multiple operating points and comparing against state-of-the-art systems.
The paper provides detailed descriptions of the architecture, training stages (pre-training, LoRA fine-tuning, full fine-tuning), and hyperparameters. The authors commit to releasing code and data. The use of open-source components (Qwen2.5-7B, FlexiCodec, Vocos) enhances reproducibility. However, the construction of the proprietary "FlexiSLM-Data" involves distillation from a 30B model and specific filtering pipelines which may be complex to replicate exactly without access to the intermediate models or detailed filtering scripts.
The authors acknowledge several limitations: 1) The model is not yet streaming-capable, limiting its use in real-time interactive scenarios. 2) Post-training alignment techniques like RLHF or DPO have not been explored, which could improve response quality. 3) The training data lacks reasoning-intensive tasks and multi-turn dialogues, potentially limiting generalization. 4) Performance degrades significantly at very low frame rates (4.0-5.0 Hz), indicating a need for better robustness in extreme compression regimes.
FlexiSLM demonstrates a practical path toward more efficient and flexible spoken language models. By enabling dynamic frame rates, it allows for better trade-offs between quality and compute resources, which is crucial for deploying SLMs on edge devices or in bandwidth-constrained environments. This work contributes to the broader goal of making multimodal AI more accessible and efficient. FlexiSLM introduces the first spoken language model with dynamic and controllable frame rates, achieving superior efficiency and competitive performance compared to fixed-rate 7B baselines by integrating dynamic speech tokenization with direct inference-time frame rate conditioning.
Foundation models have transformed vision and language processing by providing rich, reusable representations that transfer across diverse tasks. Sheet music, as a visual encoding of musical language, lacks such a strong domain-specific backbone. We introduce MuSViT (Music Score Vision Transformer): the first foundation vision model for sheet music representation -- a ViT encoder pre-trained via Masked Autoencoders on 9.7 million pages from the IMSLP. To handle the complexity of real-world scores, we adopt a two-stage curriculum: a synthetic warm-up on typeset scores followed by large-scale training on the full IMSLP corpus. We evaluate MuSViT on four downstream tasks -- full-page and staff-level music score recognition, music symbol detection, and score difficulty classification -- under two scenarios: linear probing (frozen encoder) and fine-tuning. Under linear probing, MuSViT consistently outperforms modern vision encoders, revealing that general-purpose representations, regardless of scale, fall systematically short on the structured symbolic properties of musical notation. Under fine-tuning, MuSViT generally improves upon task-specific state-of-the-art methods. An additional embedding-transcription consistency analysis reveals that MuSViT encodes symbolic musical structure directly in its representation space -- unlike other encoders, whose embeddings do not correlate with music notation content. These results establish MuSViT as a foundation backbone for sheet music understanding.
Primary: University of Alicante
All Institutions: University of Alicante
MuSViT is a significant contribution to domain-specific foundation models, demonstrating that large-scale self-supervised pre-training on structured symbolic data yields representations that are semantically aligned with the domain's logic, outperforming general-purpose vision encoders and enabling state-of-the-art performance on key music document analysis tasks.
The paper proposes MuSViT, a Vision Transformer (ViT) pre-trained via Masked Autoencoders (MAE) on 9.7 million sheet music pages from the IMSLP. The methodology is sound and follows established self-supervised learning paradigms (MAE) adapted for the specific constraints of sheet music (fine-grained patches, 2D positional encodings). The introduction of a two-stage curriculum learning strategy—starting with synthetic data to prevent dimensional collapse before moving to real-world scans—is a significant methodological contribution. This addresses a specific failure mode (average patch prediction) observed when training directly on heterogeneous real-world data. The architecture choices (ViT-B/S variants, 2D PE) are well-justified for the structured, symbolic nature of the domain. However, the core algorithmic novelty is incremental; it applies known SOTA vision techniques (MAE, ViT) to a new, under-explored domain rather than proposing a fundamentally new architectural or learning mechanism.
The evaluation is comprehensive and rigorous. The authors assess MuSViT on four diverse downstream tasks: full-page recognition, staff-level recognition, symbol detection, and difficulty classification. They employ two standard protocols for foundation models: linear probing (to test representation quality) and fine-tuning (to test adaptability). The results show that MuSViT consistently outperforms general-purpose vision encoders (DINOv3, Qwen3-VL, PaliGemma, Kosmos-2.5) in linear probing, highlighting the inadequacy of generic visual features for symbolic music. In fine-tuning, it matches or exceeds task-specific state-of-the-art methods. The inclusion of an "embedding-transcription consistency analysis" is a strong point, providing qualitative and quantitative evidence that the learned representations align with symbolic musical structure, unlike general-purpose models. The use of large-scale, real-world data (IMSLP) adds significant weight to the empirical claims.
The paper provides substantial detail for reproduction. The dataset source (IMSLP) is public, and the authors release code, pre-training scripts, and evaluation scripts via a project URL. The architecture details, hyperparameters, and training protocols are described in the main text and supplementary material. The two-stage curriculum and specific masking ratios are clearly defined. The use of standard libraries (Hugging Face for baselines) and clear evaluation metrics (SER, mAP, Accuracy) ensures that the results can be verified.
The primary limitation is that the model is a vision-only encoder. While effective for OMR-related tasks, it does not inherently capture the temporal or harmonic structure of music without downstream symbolic processing (e.g., via a transcription head). The reliance on IMSLP data, while large, introduces biases towards public domain and historically preserved scores, potentially affecting performance on contemporary or highly stylized modern scores not well-represented in the corpus. Additionally, the "synthetic warm-up" stage relies on generated data, which may not fully capture the noise and degradation of real-world scans, although the authors argue this is necessary for stability. The paper does not explore multimodal extensions (e.g., audio alignment) which could further enhance the utility of the representations.
This work establishes a foundational backbone for sheet music understanding, addressing a significant gap in the intersection of computer vision and music information retrieval. By providing a strong, reusable representation, it lowers the barrier for developing new applications in musicology, education, and archival digitization. The finding that general-purpose vision models fail to capture symbolic structure has broader implications for other structured visual domains (e.g., chemical structures, mathematical notation). The public release of the model and code accelerates research in this niche but culturally significant area. MuSViT is a significant contribution to domain-specific foundation models, demonstrating that large-scale self-supervised pre-training on structured symbolic data yields representations that are semantically aligned with the domain's logic, outperforming general-purpose vision encoders and enabling state-of-the-art performance on key music document analysis tasks.
Text-based singing voice editing (SVE) aims to revise sung lyrics while preserving the original melody, total duration, and non-edited regions. In this paper, we propose MeloDISinger, a flow-matching-based SVE model for melody-aware and duration-preserving editing. Its core module, MeloDRP, predicts fixed-budget duration ratios, enabling explicit span-wise duration control. For melody-aware duration allocation, MeloDRP fuses phonetic cues with pseudo-MIDI melodic context through cross-attention, while temporal-overlap supervision encourages soft phoneme--note correspondences. We further use a flow-matching mel decoder for audio infilling to synthesize edited regions while preserving surrounding context. In addition, we introduce a duration-aware edited-lyric generation pipeline using WhisperX and an LLM to construct feasible evaluation scenarios. Experiments demonstrate state-of-the-art performance in both objective and subjective evaluations.
Primary: Graduate School of Artificial Intelligence, KAIST
All Institutions: Graduate School of Artificial Intelligence, KAIST, Graduate School of Culture Technology, KAIST
[This paper presents MeloDISinger, a novel flow-matching-based singing voice editing model that introduces melody-aware duration ratio prediction to ensure strict temporal synchronization and high-quality audio infilling, achieving state-of-the-art performance in both objective and subjective evaluations.]
The paper proposes MeloDISinger, a flow-matching-based architecture for text-based Singing Voice Editing (SVE). The core technical novelty lies in the "MeloDRP" (Melody-aware Duration Ratio Predictor) module. Unlike previous methods that predict absolute durations or reuse original phoneme durations (which fails when phoneme counts change), MeloDRP predicts duration *ratios* within a fixed budget for each edit span. This ensures strict total duration preservation, a critical constraint for synchronization with accompaniment. The method fuses phonetic cues with pseudo-MIDI melodic context via cross-attention to inform these ratios, addressing the strong link between melody and rhythm in singing. The audio generation uses a flow-matching mel decoder with an infilling strategy, conditioning on the predicted durations, pitch, and original context to seamlessly replace edited regions. The use of pseudo-MIDI derived from F0 rather than score annotations is a pragmatic and effective choice for real-world singing voice editing where pitch deviations are common.
The evaluation is comprehensive, covering six distinct editing scenarios (insertion, deletion, mixed, and three types of replacement based on phoneme/syllable matching). The authors construct a novel, duration-aware evaluation dataset using WhisperX and an LLM to ensure temporal feasibility, addressing a significant gap in prior SVE benchmarks where generated edits often violated timing constraints. Objective metrics (WER, CER, Duration Consistency, F0 Pearson Correlation) and subjective MOS scores demonstrate state-of-the-art performance against baselines like EditSinger and Vevo2. The ablation studies effectively isolate the contributions of melody conditioning, guided-attention loss, and duration ratio prediction. The results clearly show that explicit duration ratio prediction significantly outperforms methods that do not account for the fixed budget, particularly in complex replacement scenarios.
The paper provides detailed implementation details, including model architectures (Transformer layers, hidden sizes), training hyperparameters (Adam optimizer, learning rate schedule), and preprocessing steps (MFA alignment, g2p-en, Parselmouth for F0). The dataset (GTSinger-En) is publicly available. However, the code is not explicitly linked in the provided text (only a demo page is listed), and the baseline "EditSinger" was reproduced from the paper rather than using a public repository, which may introduce slight implementation variances. The use of proprietary LLMs (Gemini-2.5-flash) for data generation limits full reproducibility of the evaluation dataset construction, though the pipeline is described in detail.
The method relies on accurate pseudo-MIDI extraction from F0; poor F0 estimation or highly vibrato-heavy sections could degrade the melodic context input to the duration predictor. The assumption that a fixed budget can be strictly allocated via ratios may struggle with extreme lyrical changes where the semantic content requires significantly different rhythmic phrasing than the original, potentially leading to unnatural "speech-like" timing if the melody conditioning is insufficient. The evaluation is limited to English singing voices (GTSinger-En), and the generalizability to other languages or singing styles (e.g., rap, which has different rhythmic constraints) is not demonstrated. Additionally, the reliance on WhisperX for alignment introduces potential errors in onset/offset detection, which could affect the syllable capacity calculation.
This work advances the field of audio generation and music production tools by providing a robust solution for precise singing voice editing. It enables more natural and efficient post-production workflows for musicians and producers. The proposed evaluation pipeline offers a new standard for assessing temporal fidelity in SVE systems. However, the technology also raises ethical concerns regarding the potential for deepfake singing voices and the misappropriation of artists' vocal styles, necessitating responsible use guidelines. [This paper presents MeloDISinger, a novel flow-matching-based singing voice editing model that introduces melody-aware duration ratio prediction to ensure strict temporal synchronization and high-quality audio infilling, achieving state-of-the-art performance in both objective and subjective evaluations.]
Speech conveys rich emotional information. As Speech Emotion Recognition (SER) is usually deployed in privacy-sensitive and reliability-critical environments, adversarial attacks on SER have attracted increasing attention. Existing sparse attacks control the number of perturbed elements, yet, they often lack explainability guidance and explicit measures of explanation consistency. A unified treatment of sparsity and magnitude constraints is also uncommon. In addition, transferability across attack families and target models remains limited. Hence, we propose a SalIency-Guided sparse Mask Attack (SIGMA). On self-supervised speech features, we use post-hoc explainable artificial intelligence (XAI) techniques to produce saliency maps and identify the scope of the mask, and then restrict magnitude-bounded updates to this mask. The mask is computed once and can be reused across models and different sparsity attacks to amortise cost. We evaluate on the IEMOCAP and TESS datasets. Under matched budgets and across multiple sparse-attack settings, SIGMA maintains competitive attack success rates, navigating a conscious trade-off between attack efficacy and explanation consistency. SIGMA therefore provides an efficient and interpretable framework for analysing the vulnerability and explanation behaviour of SER models under structured perturbations.
Primary: Imperial College London
All Institutions: Imperial College London, Hunan University, Technical University of Munich, Munich Data Science Institute, Munich Center for Machine Learning, Konrad Zuse School of Excellence in Reliable AI, Shenzhen Research Institute
SIGMA introduces a novel saliency-guided sparse masking mechanism for adversarial attacks on SER models, effectively balancing attack efficacy with explanation consistency and offering a reusable framework for analyzing model vulnerabilities in latent feature spaces.
The paper proposes SIGMA, a framework for generating sparse adversarial attacks on Speech Emotion Recognition (SER) models. The core innovation lies in using post-hoc Explainable AI (XAI) techniques (Gradient x Input, Integrated Gradients, LIME) to generate a saliency map on a surrogate model, which is then used to create a binary mask. This mask restricts the support of the adversarial perturbation to only the most salient feature elements in the latent space of self-supervised speech encoders (e.g., Emotion2Vec, WavLM, HuBERT). The authors integrate this mask into standard iterative attack algorithms (PGD, Frank-Wolfe, Sparsefool). The methodology is technically sound and addresses a specific gap in adversarial robustness research: the lack of explainability-guided sparsity constraints. By operating in the latent feature space, the method isolates the vulnerability of the classifier head to perturbations in semantically critical regions identified by XAI. The approach is modular and pluggable, allowing reuse of the mask across different target models, which is a practical advantage for transferability studies.
The experimental evaluation is comprehensive, covering two standard SER datasets (IEMOCAP and TESS) and multiple SSL encoders and classifier architectures. The authors provide rigorous white-box comparisons against baseline sparse attacks (PGD, FW, Sparsefool) under matched sparsity and magnitude budgets. They also evaluate transferability (white-box cross-model) and black-box zero-query transfer. Key metrics include Attack Success Rate (ASR), sparsity, and novel explanation consistency metrics (Top-k Intersection, Kendall’s Tau, Total Variation Distance). The results demonstrate that SIGMA maintains competitive ASR while significantly improving explanation consistency (i.e., the perturbed input's saliency map remains closer to the clean input's map). The ablation studies on XAI methods and sparsity rates provide valuable insights into the trade-offs between computational cost (LIME is slow, GI is fast) and performance. The statistical significance testing adds robustness to the claims.
The paper provides detailed descriptions of the datasets, model architectures, training hyperparameters, and attack parameters. The authors state that code and models will be released. The experimental setup is clear, including the specific SSL checkpoints and classifier designs. The inclusion of algorithm pseudocode and detailed metric definitions enhances reproducibility. However, as an arXiv preprint, the lack of immediate code availability is a minor hurdle, though the description is sufficient for implementation.
The primary limitation is the operational domain: attacks are conducted in the latent feature space of SSL encoders, not on the raw waveform. While the authors argue this is a useful analytical proxy, it does not directly address the challenge of generating perceptually valid adversarial audio in the time domain, which is the ultimate goal for many real-world threats. Additionally, the method relies on the accuracy of the XAI techniques; if the saliency maps are noisy or misleading, the mask may not effectively guide the attack or ensure consistency. The computational cost of XAI pre-computation (especially for LIME) is noted as a bottleneck for real-time single-sample attacks, although amortization across targets mitigates this.
This work contributes to the field of adversarial machine learning and explainable AI, specifically in the audio domain. By linking adversarial robustness with explanation consistency, it provides a framework for auditing SER models not just for their vulnerability to misclassification, but for the stability of their interpretability. This is crucial for high-stakes applications like mental health screening, where both accurate emotion detection and trustworthy explanations are required. The findings suggest that current SER models may be vulnerable to subtle perturbations in semantically critical features, highlighting the need for more robust training methods that consider attribution stability. SIGMA introduces a novel saliency-guided sparse masking mechanism for adversarial attacks on SER models, effectively balancing attack efficacy with explanation consistency and offering a reusable framework for analyzing model vulnerabilities in latent feature spaces.
Variable frame rate (VFR) coding has recently emerged in neural speech codecs, allocating fewer frames to redundant regions and more frames to rapidly changing speech. VFR must transmit side information about retained time steps, but prior gains are either not rigorously addressed or often minor once these overhead bits are included in total bitrate. We present Dynamic Token Masking (DTM)-Codec, a neural speech codec that demonstrates clear gains over fixed-frame-rate baselines under a strict matched-total-bitrate protocol. DTM keeps selected encoder tokens, fills masked positions with a learned
Primary: Graduate School of Cultural Technology, KAIST
All Institutions: Graduate School of Cultural Technology, KAIST
DTM-Codec introduces a novel dynamic token masking mechanism and a linear-time boundary selector for variable frame rate speech coding, demonstrating significant reconstruction quality improvements over fixed-rate baselines under strict matched-total-bitrate evaluations. The paper makes a valuable contribution to the field of neural audio codecs by addressing the critical issue of fair bitrate comparison and providing a practical, efficient solution for adaptive temporal resolution in speech tokenization.
The paper proposes DTM-Codec, a neural speech codec that integrates Variable Frame Rate (VFR) coding via Dynamic Token Masking (DTM) and a linear-time boundary selector called Path Length Equalization (PLE). The core methodological contribution is the combination of a masking-based token retention strategy (preserving original feature vectors rather than pooling/merging) with a computationally efficient, content-adaptive boundary selection algorithm. The approach addresses a specific gap in the literature: the lack of rigorous, matched-total-bitrate comparisons that account for side-information overhead in VFR codecs. The use of a learnable `
The experimental evaluation is a strong point of this paper. The authors conduct a comprehensive set of experiments on LibriSpeech and MLS, comparing DTM-Codec against several state-of-the-art baselines (FlexiCodec, VARSTok, BigCodec, etc.) under strict matched-total-bitrate protocols. They include both objective metrics (UTMOS, PESQ, STOI, WER) and subjective listening tests (MUSHRA). The results consistently show that DTM-Codec outperforms fixed-frame-rate baselines and competitive VFR baselines, particularly at lower bitrates. The ablation studies on the boundary selector (PLE vs. DP vs. Clustering) provide valuable insights into the trade-off between computational complexity and reconstruction quality. The inclusion of semantic evaluation (ARCH benchmark) adds depth, although the results there are mixed, highlighting that VFR benefits reconstruction more than global semantic retention.
The paper provides sufficient implementation details, including model architecture (TAAE backbone, STFT/iSTFT front-end/back-end), training hyperparameters (AdamW, batch size, steps), and the specific VQ codebook size. The GitHub repository link is provided. The strict bitrate accounting methodology is clearly defined, which aids in reproducing the fair comparisons. The linear-time PLE algorithm is simple to implement.
The primary limitation is that the model is evaluated primarily on English speech (LibriSpeech) and a small set of non-English utterances (MLS). Generalization to other languages or highly diverse acoustic environments is not thoroughly demonstrated. Additionally, while PLE is efficient, it is a heuristic; the paper acknowledges that Dynamic Programming (DP) yields slightly better quality but is slower. The semantic evaluation results suggest that for tasks requiring global context (like emotion classification), VFR might not always be superior to FFR with a larger codebook, which is an important nuance for downstream applications.
This work contributes to the efficient transmission and processing of speech data, which is crucial for low-bandwidth communication, streaming services, and efficient tokenization for Speech Language Models (SLMs). By demonstrating that VFR can provide clear gains even with side-information overhead, it encourages further research into adaptive-rate codecs for AI-driven audio applications. DTM-Codec introduces a novel dynamic token masking mechanism and a linear-time boundary selector for variable frame rate speech coding, demonstrating significant reconstruction quality improvements over fixed-rate baselines under strict matched-total-bitrate evaluations. The paper makes a valuable contribution to the field of neural audio codecs by addressing the critical issue of fair bitrate comparison and providing a practical, efficient solution for adaptive temporal resolution in speech tokenization.
In long-form multi-party conversations, highly imbalanced speaker activity and frequent overlap make it difficult to identify "who spoke when and what". Sliding-window continuous speech separation (CSS) mitigates sparse supervision, but often suffers from cross-window speaker inconsistency and residual crosstalk, which in practice requires diarization for reliable speaker attribution. Motivated by the stability of speakers' directions of arrival (DOAs) in meetings, we propose PATSE, a multi-channel Position-Aware Target Speaker Extraction front-end that uses DOA as a spatial prior to directly extract the speech of each target speaker. PATSE combines a DOA-guided spatial encoder and conditioner to generate speaker-attributed streams, from which speaker activity can be inferred via simple post-processing (e.g., VAD) without explicit diarization. Experiments on both replayed and real conversations show consistent ASR gains outperforming CSS and diarization-based pipelines.
Primary: Kyoto University
All Institutions: Kyoto University
This paper presents a practical and effective framework for diarization-free target speaker extraction using DOA priors, demonstrating significant ASR gains in multi-party conversations through the novel integration of spatial conditioning into continuous speech separation.
The paper proposes PATSE, a Position-Aware Target Speaker Extraction framework that leverages Direction of Arrival (DOA) as a spatial prior to condition a separation backbone (TIGER). The core methodological contribution is the integration of a DOA-guided spatial encoder and conditioner (using FiLM modulation) into a continuous speech separation pipeline. This allows the model to extract specific speaker streams directly, bypassing the need for explicit speaker diarization. The approach is technically sound, combining established multi-channel features (IPD, TPD) with modern deep separation architectures. However, the novelty is moderate as DOA-conditioned extraction is a known paradigm in the speech processing community; the primary innovation lies in its specific application to long-form, diarization-free ASR pipelines and the integration with the TIGER backbone.
The experimental evaluation is robust and addresses a significant gap in the field: the lack of real-world datasets with ground-truth DOA labels. The authors introduce LibriReplay-DOA, a replayed dataset, and evaluate on TEIDAN, a real-world conversational dataset. Results demonstrate consistent Word Error Rate (WER) improvements over strong baselines including CSS (TIGER), Sortformer+GSS, and FastMNMF. The comparison against CSS with oracle speaker assignment is particularly compelling, highlighting the inherent instability of sliding-window separation without spatial priors. The evaluation covers various angular configurations and overlap ratios, providing a comprehensive view of performance under different acoustic conditions.
The paper provides detailed architectural descriptions, including the specific implementation of the spatial encoder, conditioner, and loss functions. The authors release the LibriReplay-DOA dataset and a demo page, which significantly aids reproducibility. The use of standard components like TIGER and Silero-VAD also supports reproducibility. However, the exact hyperparameters for the training of the PATSE module on top of TIGER (e.g., learning rate schedules, specific optimizer settings beyond the initial LR) could be more detailed.
The method relies on the availability of accurate DOA information. While DOAs are stable in meeting scenarios, they may vary in more dynamic environments. The performance on LibriReplay-DOA, while strong, is based on replayed audio, which does not fully capture the complex reverberation and noise characteristics of real spontaneous conversations, although TEIDAN results mitigate this concern. The approach assumes speakers are stationary or move slowly enough for DOA estimation to remain valid during the extraction window.
This work has significant implications for automatic speech recognition in multi-party settings, such as meeting transcription systems. By eliminating the need for explicit diarization, it simplifies the pipeline and improves robustness to diarization errors. The release of LibriReplay-DOA provides a valuable resource for the community to benchmark DOA-based methods on real-room recordings, fostering further research in spatial audio processing. This paper presents a practical and effective framework for diarization-free target speaker extraction using DOA priors, demonstrating significant ASR gains in multi-party conversations through the novel integration of spatial conditioning into continuous speech separation.
Noise-robust bandwidth expansion aims to reconstruct high-fidelity wideband speech from noisy low-resolution inputs. While flow matching has shown strong performance in speech generation, accurately recovering clean speech from noisy inputs remains challenging due to the ambiguity of velocity estimation under noise. In this work, we propose VeRe-Flow, a clean-guided flow matching framework that introduces multi-level clean supervision to guide the generative process toward clean speech. At the velocity level, we introduce velocity contrastive regularization, which attracts the predicted velocity toward the clean trajectory while repelling it from noisy trajectories. At the representation level, we incorporate representation alignment that aligns intermediate features with clean self-supervised learning representations. The results demonstrate that the proposed method achieves the lowest LSD and highest DNSMOS OVRL among all baselines, and the highest MOS among generative baselines.
Primary: KAIST
All Institutions: MAGO, KAIST
The paper presents VeRe-Flow, a flow matching framework for noise-robust bandwidth expansion that introduces velocity contrastive regularization and representation alignment to guide the generative process toward clean speech manifolds. While the methodological novelty is incremental compared to the broader landscape of generative audio, the empirical results demonstrate a clear improvement in objective and subjective metrics, making it a solid contribution to the specific subfield of speech enhancement and bandwidth expansion.
The paper proposes VeRe-Flow, a flow matching framework for noise-robust bandwidth expansion (NR-BWE). The core technical contributions are two regularization terms: Velocity Contrastive Regularization (VeCoR) and Representation Alignment. VeCoR attempts to guide the velocity field by attracting it toward clean trajectories and repelling it from noisy ones. Representation Alignment uses a projection head to align intermediate transformer features with clean self-supervised learning (SSL) embeddings (specifically from XEUS). The architecture combines Convolutional ResBlocks and Transformer blocks, conditioned on noisy low-resolution mel-spectrograms and SSL features. While the integration of SSL features is established in recent speech literature, the specific application of contrastive regularization on the velocity field of a flow matching model for this specific task is a novel methodological contribution. However, the theoretical grounding for why velocity contrastive learning is superior to standard conditional flow matching or diffusion-based noise modeling in this specific context is not deeply explored mathematically.
The experiments are conducted on the Valentini-Botinhao dataset, a standard benchmark for NR-BWE. The authors compare against generative baselines (FLowHigh, NU-Wave2) and non-generative methods. They report objective metrics (LSD, DNSMOS) and subjective metrics (MOS). The results indicate that VeRe-Flow outperforms baselines in LSD and DNSMOS OVRL. The ablation studies provide insight into the contribution of each component (Conv ResBlocks, XEUS, REPA, VeCoR). The evaluation is thorough for the scope of the paper, covering both spectral fidelity and perceptual quality. The use of DNSMOS is appropriate for speech enhancement tasks. However, the comparison with non-generative baselines is limited to reported numbers from other papers, which may introduce inconsistencies in evaluation protocols (e.g., vocoder differences, though BigVGAN is used for the proposed method and FLowHigh).
The paper provides sufficient implementation details, including dataset preprocessing (Chebyshev filter parameters), model architecture (Conv ResBlock structure, transformer depth), training hyperparameters (optimizer, learning rate, batch size, loss weights), and the specific SSL model used (XEUS). The use of publicly available components (BigVGAN, XEUS, Valentini-Botinhao) enhances reproducibility. The code is not explicitly linked in the text provided (only a demo URL), which is a minor drawback for immediate reproducibility, but the description is detailed enough for a competent researcher to implement.
The paper does not discuss the computational cost or inference speed of VeRe-Flow compared to baselines. Flow matching models can be sensitive to the choice of ODE solvers and number of function evaluations (NFE); while they mention testing different settings, the optimal trade-off between quality and speed is not analyzed. The reliance on SSL features (XEUS) introduces a dependency on an external model, which might not be available or compatible with all deployment scenarios. Furthermore, the "repulsion" term in VeCoR requires careful tuning of the temperature or margin parameter; the paper reports a fixed weight but does not discuss the sensitivity of this hyperparameter. The claim of being the "first to apply velocity contrastive regularization to speech generation" is strong and should be verified against recent diffusion-based contrastive works.
This work contributes to the field of speech processing by improving the quality of bandwidth expansion in noisy environments, which has applications in telecommunications, hearing aids, and audio restoration. By leveraging flow matching, it offers a potentially faster alternative to diffusion models for high-quality speech generation. The integration of SSL representations highlights the trend of using self-supervised features to guide generative processes, which can be generalized to other audio tasks. The paper presents VeRe-Flow, a flow matching framework for noise-robust bandwidth expansion that introduces velocity contrastive regularization and representation alignment to guide the generative process toward clean speech manifolds. While the methodological novelty is incremental compared to the broader landscape of generative audio, the empirical results demonstrate a clear improvement in objective and subjective metrics, making it a solid contribution to the specific subfield of speech enhancement and bandwidth expansion.
Audio-Visual Speech Recognition takes two input modalities, acoustic and visual streams, where visual information from lip movements aids recognition when audio is noisy. Recently, LLM-based AVSR models have emerged as a promising paradigm by connecting pre-trained audio-visual encoders to an LLM, achieving strong results in clean conditions. However, these models are predominantly optimized for clean acoustic conditions, with limited attention to making the LLM backbone robust to noise. No explicit mechanism is employed to produce stable representations under corrupted audio, leading to performance degradation in noisy environments. To address this, we propose VIB-AVSR, which integrates Variational Information Bottleneck layers at targeted positions within the LLM backbone to regularize representations. VIB-AVSR reduces degradation under noisy conditions across multiple SNR levels and noise types, without requiring architectural modifications or additional training data.
Primary: Imperial College London
All Institutions: Imperial College London, NatWest AI Research
VIB-AVSR introduces Variational Information Bottleneck layers into the LLM backbone of AVSR models to regularize audio representations, demonstrating that variational compression can improve noise robustness and generalization without additional training data or architectural redesign.
The paper proposes VIB-AVSR, a method to enhance the noise robustness of LLM-based Audio-Visual Speech Recognition (AVSR) models. The core innovation is the integration of Variational Information Bottleneck (VIB) layers into the intermediate layers of the LLM backbone (Llama-3.2-1B). Specifically, the method applies a variational compression objective to the audio hidden states ($H_a$) while leaving visual ($H_v$) and text ($H_t$) representations uncompressed. This is motivated by the observation that pre-trained LLMs, fine-tuned via LoRA, lack intrinsic mechanisms to filter out acoustic noise, relying solely on encoders which may not fully disentangle noise from speech features. The VIB module parameterizes the posterior distribution of the compressed representation as a diagonal Gaussian and uses a learnable prior, optimizing a lower bound on the IB objective. The approach is theoretically sound, applying a well-established information-theoretic principle to a modern multimodal architecture. However, the novelty is somewhat limited by the fact that VIB has been applied in various contexts before; the specific application to the *internal* representations of an LLM backbone for AVSR is the key contribution, but it is an incremental architectural modification rather than a new algorithmic breakthrough.
The experimental evaluation is conducted on the LRS2 dataset using Whisper-medium and AV-HuBERT encoders. The authors evaluate under two training paradigms: "Noisy" (noise augmentation during training) and "Clean" (no noise augmentation). Results are reported across multiple SNR levels (-10 to 5 dB) and noise types (Babble, Speech). The results show consistent Word Error Rate (WER) reductions for VIB-AVSR compared to the Llama-AVSR baseline, particularly in low-SNR regimes. A significant finding is that VIB-AVSR trained on *clean* data still outperforms the baseline on noisy test data, suggesting that the variational compression acts as a regularizer that improves generalization to unseen noise distributions. The ablation studies on layer placement, regularization strength, and interpolation coefficients provide good empirical grounding. However, the improvements, while consistent, are modest (e.g., Avg WER reduction from 18.85 to 17.39 in one setting). The paper lacks comparison with other robustness techniques (e.g., adversarial training, specific noise-robust encoders like Wav2Vec 2.0 with masking) which would better contextualize the gain.
The paper provides sufficient implementation details, including the architecture of the VIB module (2-layer MLP), the use of LoRA, and the specific layers for bottleneck insertion. The code is available on GitHub. The use of standard datasets (LRS2, MUSAN) and models (Whisper, Llama-3.2) enhances reproducibility. The description of the training paradigms and hyperparameters is clear.
The primary limitation is the modest magnitude of improvement. While statistically significant, the WER reductions are not transformative. The method adds computational overhead during training (sampling from the posterior) and slight complexity, though inference is unaffected. The approach assumes that noise is the primary source of variance to be discarded, which might risk discarding subtle acoustic features if the compression is too aggressive (though the interpolation term mitigates this). The evaluation is limited to LRS2; performance on more challenging, real-world datasets with diverse speaking styles and backgrounds is not reported. Furthermore, the "Clean" training paradigm's success relies on the assumption that noise robustness can be learned via representation compression alone, which might not hold for all noise types or severe distortions.
This work contributes to the broader goal of making multimodal AI systems more robust and reliable in real-world, uncontrolled environments. By improving the noise robustness of LLM-based AVSR, it paves the way for more accessible speech recognition systems for users with hearing impairments or in noisy environments. It also highlights the importance of representation regularization in large foundation models when adapting them to noisy sensory inputs. VIB-AVSR introduces Variational Information Bottleneck layers into the LLM backbone of AVSR models to regularize audio representations, demonstrating that variational compression can improve noise robustness and generalization without additional training data or architectural redesign.