Recent full-duplex spoken dialogue models have demonstrated compelling progress toward human-like interaction, enabling agents to respond with low latency, produce backchannels, and handle user barge-ins. Yet these improvements in conversational dynamics often come with weaker reasoning and instruction-following abilities, revealing a potential tension between interactive dynamics and intelligence capability. In this paper, we argue that such an intelligence--dynamics trade-off is not fundamental: conversational dynamics can instead be learned as a separate real-time decision policy from human dialogue data. To this end, we propose DuplexPO, a reinforcement learning (RL) framework that decouples when to speak from what to say. It preserves the semantic response capability of an instruction-tuned assistant, while optimizing its temporal interaction behavior over selected high-impact windows from long human conversations. To quantitatively optimize these dynamics, we formulate the Factorized Conversational Dynamics Reward (FCDR) to enable fine-grained temporal credit assignment for turn initiation, backchanneling, yielding, and regularized participation. The policy is then optimized with a GRPO-style objective. Experiments show that DuplexPO substantially improves full-duplex behaviors, including timely backchannels, smooth turn-taking, and barge-in handling, while maintaining strong reasoning and instruction-following performance. Moreover, improvements in dynamics-oriented metrics are reflected in better user experience, suggesting that optimizing conversational timing as a standalone objective can promote more natural full-duplex interaction.
Primary: NVIDIA
All Institutions: Nanyang Technological University, National Taiwan University, The Hong Kong University of Science and Technology, NVIDIA
This paper presents a significant advancement in full-duplex spoken dialogue systems by introducing a novel RL framework that effectively decouples conversational timing from semantic generation, leading to more natural interactions without compromising reasoning capabilities.
The paper proposes DuplexPO, a reinforcement learning framework designed to decouple conversational dynamics (when to speak) from semantic content (what to say) in full-duplex spoken dialogue models. The core methodological innovation lies in the Factorized Conversational Dynamics Reward (FCDR), which provides fine-grained temporal credit assignment for specific interaction events like turn initiation, backchanneling, and yielding. By restricting policy optimization to "dynamics-critical windows" sampled from long human conversations and using a GRPO-style objective, the method aims to improve real-time interaction behaviors without degrading the model's instruction-following capabilities. This addresses a specific and challenging problem in end-to-end spoken language models: the tension between low-latency responsiveness and reasoning quality. The approach is technically sound, leveraging established RL techniques (GRPO) in a novel context (temporal windowing for audio dialogue).
The evaluation is comprehensive, covering both conversational dynamics metrics (onset MAE, yield rate, voiced interrupt rate) on standard datasets (Fisher, Seamless, Full-Duplex-Bench) and model intelligence benchmarks (QA, instruction following). The results demonstrate that DuplexPO significantly improves dynamic metrics compared to SFT baselines while maintaining or slightly improving intelligence metrics. The inclusion of an LLM-as-a-Judge evaluation for naturalness adds qualitative depth. The ablation studies on window size, reward design, and optimization objectives (GRPO vs. DPO) provide strong empirical support for the design choices. The comparison with commercial models and other open-source full-duplex models places the work in a competitive context.
The paper provides detailed implementation details, including model architecture (Qwen2.5-7B backbone, Parakeet encoder), training stages, and hyperparameters for the RL phase. The data sources (Fisher, Seamless) are public, and the reconstruction process is described. The inclusion of a demo URL suggests the authors are committed to transparency. However, the exact code for the window sampling and FCDR reward calculation is not explicitly linked in the text (though a GitHub link might be implied by the demo page, it is not in the provided text). The detailed appendix sections on data generation and training pipelines enhance reproducibility.
The authors acknowledge that FCDR may miss subtle pragmatic factors like user intent and cultural timing preferences. The reliance on human timing annotations, which are not unique ground truths, is a limitation. Furthermore, restricting optimization to local windows might miss long-range dialogue coherence effects. The evaluation is primarily in English, and the method's generalizability to other languages or highly asynchronous interaction styles is not fully explored.
This work contributes to the development of more natural and responsive voice assistants, which has significant implications for human-computer interaction, accessibility, and AI adoption. The potential for misuse in deceptive practices (e.g., vishing) is noted, highlighting the need for responsible deployment and safeguards. The decoupling of dynamics and semantics offers a generalizable paradigm for optimizing multi-modal agents where timing and content are distinct concerns. This paper presents a significant advancement in full-duplex spoken dialogue systems by introducing a novel RL framework that effectively decouples conversational timing from semantic generation, leading to more natural interactions without compromising reasoning capabilities.
Audio-language models compress a speech encoder's output through a Querying Transformer (Q-Former) connector before feeding it to a large language model. We identify two failures in this compression. The connector's output vectors collapse to a single direction, and different speakers produce nearly indistinguishable outputs, with paralinguistic cues such as speaker identity, gender, and prosody lost along the way. Our method, ORCA, reverses this collapse by splitting the queries into groups whose outputs are constrained to point in different directions. On SAKURA multi-hop reasoning, ORCA gains 26.4 points over an identically trained 4B baseline, reaching 75.2% (vs. 49.0% for the 8B Audio Flamingo-3). At the connector level, the same change cuts query redundancy by 12x and raises cross-speaker variance by 75x.
Primary: National Taiwan University
All Institutions: National Taiwan University, ASUS AICS
The paper presents a compelling structural solution to query collapse in audio-language model connectors, demonstrating that enforcing orthogonality among query groups significantly enhances the preservation of paralinguistic information and boosts multi-hop reasoning performance without increasing computational cost.
The paper proposes ORCA, a structural modification to the Querying Transformer (Q-Former) connector used in Audio-Language Models (ALMs). The core innovation is the partitioning of learnable queries into $G$ groups, each with its own layer mixture from the audio encoder, and enforcing orthogonality between the mean vectors of these groups. This is a geometric constraint applied to the connector's output space. The approach is theoretically grounded in diagnosing "query collapse" (where output vectors converge to a single direction) and "information loss" (specifically paralinguistic cues like speaker identity). The method is elegant in its simplicity: it requires no additional labeled data for specific attributes, relying solely on the language modeling loss and the geometric regularizer. The separation of shallow, mid, and deep encoder layers across groups emerges naturally from the optimization, suggesting the method successfully forces the model to utilize different representational subspaces.
The experimental design is rigorous, particularly the controlled comparison where only the connector changes between the baseline and ORCA, keeping the encoder, LLM, data, and training recipe identical. This isolates the contribution effectively. The results on the SAKURA benchmark are impressive, showing a +26.4 point gain on multi-hop reasoning for a 4B model, surpassing larger 8B models. The diagnostic metrics (cosine similarity reduction by 12x, cross-speaker variance increase by 75x) strongly support the hypothesis that query collapse was the bottleneck. The robustness analysis under noise adds depth, showing that while ORCA preserves cues that survive noise (like gender), it cannot recover physically masked information (fine prosody in emotion). The comparison with MMAU shows that while ORCA helps paralinguistic reasoning, it doesn't drastically outperform baselines on general audio understanding, which is a nuanced and honest result.
The paper provides sufficient detail for reproduction. It specifies the architecture (Whisper-Large-V3, Qwen3-4B), the connector structure (G=8, J=8, K=64), the loss weights ($inter=0.1$, $intra=0.03$), and the training setup (AQA5M dataset, 5 epochs, Adam optimizer). The code for the Q-Former modification is likely straightforward to implement given the clear mathematical formulation of the inter-group and intra-group loss terms. The use of standard datasets (SAKURA, MMAU, CREMA-D) ensures that results can be compared against existing literature.
The primary limitation is that the improvement is specific to tasks requiring the preservation of diverse acoustic cues (paralinguistics). For general audio understanding (MMAU), the gains are modest. Furthermore, the method cannot recover information that is physically lost in the audio encoder or masked by noise, as acknowledged in the discussion. The reliance on the language modeling objective means that the "orthogonal" subspaces are still biased towards what helps predict text, so purely acoustic tasks without textual grounding might not benefit as much. The method also assumes that the audio encoder provides sufficient diversity in its hidden states, which might not be true for all encoders.
This work has significant implications for the design of multimodal models. By identifying and fixing the "Procrustean Bed" problem in connectors, it demonstrates that architectural changes can yield massive performance gains without increasing model scale or data volume. This is particularly relevant for resource-constrained settings where scaling up is not feasible. It also highlights the importance of paralinguistic information in ALMs, encouraging future research to focus on preserving speaker identity, emotion, and prosody, which are crucial for human-like interaction. The geometric approach to preventing representation collapse could be adapted to other modalities where similar bottlenecks exist. The paper presents a compelling structural solution to query collapse in audio-language model connectors, demonstrating that enforcing orthogonality among query groups significantly enhances the preservation of paralinguistic information and boosts multi-hop reasoning performance without increasing computational cost.
Multimodal Large Language Models (LLMs) have remarkable semantic audio understanding, yet they remain "spatially agnostic" due to their reliance on mono-channel audio representations. Currently, spatial audio perception methods mainly focus on complex room simulations and custom-trained, geometry-aware stereo encoders, which limits their accessibility and generalizability. In this paper, we introduce the Dual-BEATs architecture, in which the left and right audio channels are routed independently through two identical semantic encoders as an alternative to specialized spatial modules. To circumvent the architectural bottleneck where internal normalization otherwise erases the inter-channel variance of stereo audio, we inject a static, uncorrelated dithering noise floor prior to encoding. This dithering intervention establishes a macro-variance floor that "smuggles" spatial geometry across the normalization layers. Evaluated on a ternary directional classification task (Left, Center, Right), we demonstrate that dithered models achieve exceptional spatial resolution--reaching up to 97.2% localization accuracy even on subtle 0.5 panning amplitudes--and demonstrates robust, zero-shot generalization to entirely unseen spatial configurations. Our results suggest that with the appropriate acoustic regularization, standard multimodal models are natively capable of generalized stereo audio understanding.
Primary: Institute of Information Science, Academia Sinica
All Institutions: Institute of Information Science, Academia Sinica
The paper introduces Dual-BEATs, a novel architecture that unlocks zero-shot stereo perception in Audio LLMs by using uncorrelated dithering noise to bypass normalization-induced spatial information loss, achieving high-resolution spatial localization without specialized encoders.
The paper proposes a novel architectural intervention called "Dual-BEATs" to address the "spatial agnosticism" of standard Multimodal LLMs. The core insight is that standard normalization layers (LayerNorm/RMSNorm) act as compressors that erase subtle inter-channel amplitude differences (ICLD) essential for stereo perception. To bypass this, the authors inject static, uncorrelated Gaussian dithering noise into the left and right audio channels before encoding. This creates a "macro-variance floor" that allows the normalization layers to preserve the relative amplitude delta between channels, effectively "smuggling" spatial geometry into the LLM's latent space. The approach is theoretically grounded in Stochastic Resonance and avoids the need for complex, custom-trained spatial encoders or 3D room simulations. The methodology is elegant, leveraging existing frozen semantic encoders (BEATs) and standard LLM backbones (Gemma, OLMo) with minimal modification (QLoRA adapters).
The evaluation is rigorous and well-designed. The authors construct a controlled dataset using AudioSet samples, applying deterministic amplitude panning to create stereo pairs with varying panning amplitudes (PA). They evaluate on a ternary classification task (Left, Center, Right) and, crucially, test zero-shot generalization across unseen panning amplitudes. The results are compelling: undithered models suffer a "Center-Pan Crash" (performance drops to random chance for center-panned audio), while dithered models maintain high accuracy (>97%) across the entire spectrum, including subtle 0.5 PA shifts. The ablation studies on correlated vs. uncorrelated noise effectively demonstrate that the model is learning true spatial geometry rather than exploiting noise artifacts. The trade-off between spatial resolution and semantic retention ("Semantic Tax") is also quantified, showing a manageable ~5% drop in semantic F1.
The paper provides extensive implementation details, including specific hyperparameters for dithering amplitude (DA=0.05), hardware specifications (V100/H100), and optimization settings (QLoRA, AdamW). The dataset curation process is described in detail, referencing specific snapshots of AudioSet. The strict isolation of random seeds for training and evaluation is a strong point for reproducibility and validity, preventing shortcut learning. The code is not publicly linked, but the method is simple enough to be reimplemented.
The primary limitation is the reliance on simple amplitude panning (ICLD) rather than full binaural rendering (which includes Interaural Time Differences - ITD and Head-Related Transfer Functions - HRTFs). This limits the ecological validity of the spatial cues compared to real-world binaural recordings. Additionally, the evaluation is restricted to single-source scenarios; multi-source separation and localization are not addressed. The "Semantic Tax" indicates that while spatial perception is unlocked, it comes at the cost of some semantic fidelity, which may be a concern for downstream tasks requiring high-precision semantic understanding.
This work has significant implications for the development of spatially-aware AI assistants, accessibility tools for the visually impaired, and immersive media applications. By demonstrating that standard LLMs can be endowed with spatial perception through simple signal-level interventions, it lowers the barrier to entry for spatial audio understanding. However, the dual-use nature of such technology (e.g., enhanced surveillance or tracking) is acknowledged. The approach also highlights a fundamental architectural weakness in current multimodal models, potentially guiding future model designs to better preserve physical signal properties. The paper introduces Dual-BEATs, a novel architecture that unlocks zero-shot stereo perception in Audio LLMs by using uncorrelated dithering noise to bypass normalization-induced spatial information loss, achieving high-resolution spatial localization without specialized encoders.
Generative streaming models for Target Speaker Extraction (TSE) commonly exhibit a quality--intelligibility trade-off, wherein naive optimization for perceptual audio quality tends to degrade speech intelligibility, and conversely. We reveal that this trade-off arises not from the constraints of streaming architectures, but from an inappropriate choice of optimization anchor. Directly optimizing against audio quality metrics induces catastrophic reward hacking, where content critical to pronunciation and intelligibility is systematically erased to maximize a proxy score. To break this bottleneck, we propose two complementary improvements: an enlarged Conformer convolution kernel for richer local spectro-temporal modeling, and WavLM-anchored Direct Preference Optimization (DPO) fine-tuning strategy. DPO preference pairs are ranked by WavLM cosine similarity, a deep acoustic feature encoding both phonetic structure and speaker identity, providing an optimization anchor that resists hacking. Under a 560 ms streaming chunk size, the proposed method achieves a 10.9% relative intelligibility improvement (word error rate: 0.138 to 0.123), with marginal simultaneous gains in audio quality and speaker similarity.
Primary: Tsinghua University
All Institutions: Tsinghua University, The Chinese University of Hong Kong, SenseTime
The paper makes a significant contribution to streaming TSE by identifying and resolving the reward hacking problem in DPO through deep-feature anchoring, achieving a Pareto improvement in quality and intelligibility.
The paper proposes a novel approach to the quality-intelligibility trade-off in streaming Target Speaker Extraction (TSE). The core methodological contribution is the identification and mitigation of "reward hacking" in Direct Preference Optimization (DPO) when applied to generative speech models. The authors correctly diagnose that optimizing for perceptual metrics (DNSMOS) leads to intelligibility collapse, while optimizing for Word Error Rate (WER) fails to improve quality. Their solution involves two steps: architectural (enlarging Conformer kernel size to $k=15$ for better local context) and algorithmic (using WavLM cosine similarity as the preference anchor for DPO). The separation of the semantic pathway (optimized via DPO) and acoustic pathway (frozen) is a sound engineering decision to prevent semantic drift during acoustic refinement. The use of WavLM, which encodes both phonetic and speaker information, as a proxy for "good" speech is a clever heuristic that effectively balances the competing objectives.
The experimental setup is rigorous, utilizing the Libri2Mix dataset and standard metrics (DNSMOS, WER, Speaker Similarity). The ablation studies are particularly strong, clearly demonstrating the failure modes of $DPO_{DNSMOS}$ (intelligibility collapse) and $DPO_{WER}$ (no gain, quality drop), thereby validating the necessity of the $DPO_{WavLM}$ approach. The results show a statistically significant improvement in WER (10.9% relative) with simultaneous gains in quality and speaker similarity. The inclusion of Inference Success Rate (ISR) is a valuable addition for streaming contexts. The evaluation covers multiple KL penalty coefficients and checkpoints, providing a robust analysis of hyperparameter sensitivity.
The paper provides sufficient implementation details, including model architecture (Conformer kernel size, SELM/ARLM layers), training hyperparameters (learning rate, optimizer), and dataset specifications. The use of standard libraries (FunCodec, Whisper, WavLM) aids reproducibility. However, the exact code for the "Historical Context Refinement" mechanism and the specific stochastic sampling procedure for generating preference pairs could be more detailed. The claim of "catastrophic reward hacking" is well-supported by the provided analysis (spectrograms and metric trends), which enhances trust in the reproducibility of the findings.
The primary limitation is the reliance on WavLM as a proxy for human preference. While effective, WavLM is a self-supervised model trained on large-scale speech data, and its alignment with human perceptual quality is not perfect. The method is evaluated only on Libri2Mix, a relatively clean, two-speaker mixture dataset. Real-world scenarios with heavy noise, reverberation, and more than two speakers are not addressed. Furthermore, the streaming latency is 560ms, which is acceptable for many applications but not for strict real-time (<200ms) interaction, as noted by the authors. The computational cost of generating 16 candidates for DPO training is also non-trivial.
This work has significant implications for the deployment of generative AI in real-time communication systems, such as teleconferencing and voice assistants. By breaking the quality-intelligibility trade-off, it enables systems that are both natural-sounding and easy to understand. The insight regarding reward hacking in generative speech models is broadly applicable to other text-to-speech and audio generation tasks. It highlights the danger of using naive proxy metrics for optimization and advocates for more semantically aware anchors. The paper makes a significant contribution to streaming TSE by identifying and resolving the reward hacking problem in DPO through deep-feature anchoring, achieving a Pareto improvement in quality and intelligibility.
Melody skeleton extraction aims to derive a shorter melody that preserves structural notes while removing ornaments. Prior methods rely on hand-crafted reduction rules or note-wise salience classifiers trained with heuristically or procedurally generated pseudo-labels. Such supervision can inherit generator bias and does not explicitly optimize a coherent reduced melody. We introduce MeloBottleneck, a self-supervised framework that represents a skeleton as a length-controlled, order-preserving latent subsequence. A hard-bottleneck extractor selects note events, a rhythmic-closure operator produces a self-consistent skeleton, and a re-ornamentation decoder reconstructs the input melody. Training combines reconstruction, a frozen autoregressive melody prior, ornament-invariant consistency across procedurally ornamented views, and ornament exclusion. We evaluate three regimes: synthetic out-of-distribution ornament-to-skeleton, TAVERN variation-to-theme, and Jiugong ornamented-to-gongche. A matched pseudo-label classifier excels on the synthetic benchmark, while MeloBottleneck transfers better, achieving competitive selection quality on TAVERN and Jiugong. Skeletonized melodies also improve BM25-based fragment retrieval, boosting Recall@K and MRR while reducing query time. Overall, the results suggest that learning skeletons as latent subsequences yields more robust transfer than pseudo-label imitation.
Primary: Beijing University of Posts and Telecommunications
All Institutions: Beijing University of Posts and Telecommunications, Central Conservatory of Music
MeloBottleneck introduces a novel self-supervised framework for melody skeleton extraction that models skeletons as latent subsequences, demonstrating superior cross-domain transfer and practical utility in music retrieval compared to prior pseudo-label or heuristic methods.
The paper proposes MeloBottleneck, a self-supervised framework for melody skeleton extraction. The core innovation lies in formulating the skeleton not as a set of independent note-wise labels, but as a length-controlled, order-preserving latent subsequence. This is a significant methodological shift from prior work that relies on heuristic rules or pseudo-label classifiers. The architecture employs a Transformer encoder to select top-K notes, followed by a deterministic "rhythmic closure" operator that converts the sparse subsequence into a standalone melody by absorbing deleted time into preceding notes. Training is multi-objective: it includes reconstruction loss (re-ornamentation), a frozen autoregressive melody prior for musicality, and ornament-invariant consistency losses to ensure stability under procedural ornamentation. The approach effectively decouples note selection from rhythmic coherence, addressing a key limitation in previous discrete latent compression methods for music.
The evaluation is comprehensive, covering three distinct regimes: synthetic out-of-distribution (OOD) ornament-to-skeleton, cross-domain variation-to-theme (TAVERN), and in-domain ornamented-to-gongche (Jiugong). The results demonstrate a clear trade-off: while a pseudo-label classifier excels on synthetic data where it directly imitates the generator, MeloBottleneck shows superior transferability to zero-shot and cross-domain settings. This supports the central claim that learning structural representations is more robust than imitation. Additionally, the paper provides a practical downstream application, showing that skeletonized melodies improve BM25-based fragment retrieval by reducing noise and query time. The ablation studies rigorously validate the contribution of each loss component, particularly the ornament exclusion and consistency losses.
The paper provides detailed implementation specifics, including the SimpleMono event representation, hyperparameters for the Transformer backbone (6 encoder/3 decoder layers), and specific loss coefficients. The authors also provide a link to supplementary materials including code and a demo web page, which significantly enhances reproducibility. The use of standard symbolic music representations and clear algorithmic descriptions of the rhythmic closure and selection mechanisms makes the method straightforward to implement.
The authors acknowledge several limitations. First, the method relies on a procedural ornamenter for training views; if this ornamenter does not match the target style, the invariance signal may be suboptimal. Second, the evaluation of selection quality uses an oracle compression ratio, meaning free-length prediction is only indirectly tested. Third, the rhythmic closure operator is simple (absorbing time into the preceding note), which may not preserve musical phrasing as well as more complex closure strategies. Finally, the focus is strictly on monophonic melodies, limiting immediate applicability to polyphonic music without significant extension.
This work has significant implications for symbolic music analysis, retrieval, and compression. By providing a robust method to extract structural skeletons, it facilitates better theme tracing, variation recognition, and efficient indexing of large music corpora. The self-supervised nature of the approach reduces reliance on expensive human annotations, making it scalable to diverse musical traditions. The improved retrieval performance suggests practical utility in digital library systems and music information retrieval tools. MeloBottleneck introduces a novel self-supervised framework for melody skeleton extraction that models skeletons as latent subsequences, demonstrating superior cross-domain transfer and practical utility in music retrieval compared to prior pseudo-label or heuristic methods.
Phone segmentation and recognition are inherently related tasks, yet modern approaches typically model them separately. We argue that phonetic structure is already latent in the representations of self-supervised speech models (S3Ms), and one only needs to steer them to solve both tasks. We leverage S3M-based Phonological Activation Mapping (SPAM), which maps each S3M representation frame to a vector of phonological feature activations, such as voicing and nasality. On top of SPAM, we introduce two simple but effective lightweight, gradient-descent-free prediction heads: a recognition head and a segmentation head. Our method requires less than a minute of phonetic transcriptions, and generalizes to unseen phones during training. Across a diverse range of datasets, our approach attains strong segmentation and recognition performance.
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University, University of Texas at Austin, University of Tokyo, University of British Columbia
This paper presents a highly innovative, sample-efficient approach to phone segmentation and recognition by steering self-supervised speech models through phonological activation mapping, achieving strong generalization with minimal supervision.
The paper proposes a novel, gradient-descent-free framework for phone segmentation and recognition called SPAM (S3M-based Phonological Activation Mapping). The core innovation lies in leveraging the latent phonetic structure of Self-Supervised Speech Models (S3Ms), specifically WavLM, by projecting frame-level representations onto learned phonological feature vectors (e.g., voicing, nasality) derived from the PanPhon library. Instead of training a deep classifier, the authors design two simple heads: a recognition head that performs nearest-neighbor matching against canonical PanPhon vectors, and a segmentation head that uses an ensemble of dissimilarity signals (multi-scale differences, backward contrasts, mel-spectrogram differences) to detect boundaries. This approach is theoretically elegant, as it treats phonetic structure as a geometric property of the S3M space rather than a classification target requiring massive supervision. The method is highly interpretable and sample-efficient.
The evaluation is comprehensive, covering phone segmentation (R-value) and recognition (PFER) across a diverse set of in-domain (TIMIT, Buckeye) and out-of-domain datasets (GTIMIT-S/T, TORGO, SSNCE, Voxangeles, GTIMIT-Thai). The results demonstrate that SPAM significantly outperforms standard supervised baselines (CTC, FCE, BCE) trained on TIMIT, particularly in low-resource and multilingual settings. For instance, on the challenging Voxangeles dataset, SPAM achieves a PFER of 24.3 compared to 40.2 for FCE. The paper also includes rigorous ablation studies on sample efficiency (showing stability with <1 minute of data), segmentation signal contributions, and S3M layer/model choices. The comparison against strong toplines (KoelLabs-XLSR, PhoneticXeus) highlights SPAM's competitive performance despite using negligible training data and compute.
The authors provide code for modeling, evaluation metrics, and the benchmark suite. The methodology relies on standard libraries (PanPhon, MFA) and publicly available S3Ms (WavLM). The deterministic nature of the gradient-descent-free heads enhances reproducibility. However, the exact hyperparameters for the ensembling weights in the segmentation head are not fully detailed in the text, though the logic is clearly explained. The reliance on specific S3M layers (final layer of WavLM) is well-specified.
The method is currently limited to phones with defined consonantal values in PanPhon, excluding tones and potentially some complex phonological phenomena. The segmentation head, while effective, relies on heuristic ensembling of dissimilarity signals which may not generalize as robustly as learned boundary detectors in highly noisy or non-speech audio. The recognition head's performance is bottlenecked by the segmentation head; as shown in the oracle ablation, perfect segmentation drastically reduces error, indicating that the boundary detection is the primary area for improvement. Additionally, the method is not end-to-end differentiable in its current form, limiting potential joint optimization.
This work has significant implications for low-resource speech processing, clinical speech assessment, and the documentation of endangered languages. By requiring less than a minute of labeled data and generalizing to unseen phones and languages, SPAM lowers the barrier to entry for phonetic analysis. It promotes the use of interpretable, linguistically grounded representations in deep learning, moving away from black-box sequence-to-sequence models. This could facilitate more accessible tools for linguists and clinicians. This paper presents a highly innovative, sample-efficient approach to phone segmentation and recognition by steering self-supervised speech models through phonological activation mapping, achieving strong generalization with minimal supervision.
Large language model (LLM)-based audio-visual speech recognition (LLM-AVSR) has recently demonstrated strong robustness in adverse acoustic environments by leveraging complementary audio and visual information. Existing approaches typically employ independently pretrained acoustic and visual encoders, whose outputs are projected and fused as soft prompts to condition an LLM for speech recognition. However, most methods perform multimodal fusion without explicitly addressing the representational discrepancy between audio, visual and text modalities, potentially limiting the effectiveness of cross-modal integration. In this paper, we propose an optimal transport (OT)-based semantic alignment framework for LLM-AVSR. The proposed method explicitly bridges the modality gap by aligning the acoustic and visual representations with reference to the linguistic embedding space of the LLM before multimodal fusion. Specifically, OT is used to estimate probabilistic coupling matrices that characterize structured correspondences between modality-specific features and linguistic embeddings. The resulting OT couplings are further utilized as soft pseudo-labels to supervise contrastive learning, encouraging the extraction of semantically coherent and cross-modal consistent audio-visual representations. By anchoring multimodal features to the linguistic space of the LLM, the proposed framework facilitates more effective multimodal fusion and decoding. We implement the proposed framework using a Whisper-based acoustic encoder, an AV-HuBERT-based visual encoder, and a LLaMA3.2-3B decoder. Experiments conducted on the LRS3-TED benchmark demonstrate consistent improvements over strong baselines and achieve state-of-the-art performance under both clean and noisy evaluation conditions across a wide range of signal-to-noise ratios (SNRs).
Primary: National Institute of Information and Communications Technology
All Institutions: National Institute of Information and Communications Technology, Research Center for Information Technology Innovation, Academia Sinica
The paper presents a method for aligning audio-visual features with linguistic embeddings in LLM-based ASR using Optimal Transport, demonstrating improved robustness and state-of-the-art results on LRS3-TED.
The paper proposes an Optimal Transport (OT)-based semantic alignment framework for LLM-based Audio-Visual Speech Recognition (LLM-AVSR). The core methodological contribution is the use of entropy-regularized OT to compute probabilistic coupling matrices between acoustic/visual features and linguistic token embeddings. These couplings are then used as soft pseudo-labels to supervise a contrastive learning objective, aiming to bridge the representational gap between modality-specific encoders (Whisper, AV-HuBERT) and the LLM decoder (LLaMA3.2-3B). The approach also introduces "virtual buckets" in the OT formulation to handle non-informative frames (noise/silence). While the application of OT to multimodal alignment is not entirely new, its specific integration into the pre-fusion stage of LLM-AVSR to align with the linguistic embedding space is a distinct methodological choice. The use of OT couplings as supervision for contrastive learning is a clever mechanism to enforce semantic consistency without requiring hard alignments. However, the novelty is somewhat tempered by the fact that OT-based alignment has been explored in other multimodal contexts, and the specific adaptation here is incremental rather than foundational.
The experimental evaluation is conducted on the LRS3-TED benchmark, which is a standard dataset for AVSR. The authors evaluate performance under clean and noisy conditions (various SNRs). They compare against strong baselines and ablate different fusion strategies (concatenation, addition, cross-attention, Q-former) and hyperparameters. The results show consistent improvements over baselines, achieving state-of-the-art performance on LRS3-TED. The inclusion of noisy evaluation is crucial for AVSR and demonstrates the robustness of the method. However, the paper lacks comparison with some very recent LLM-AVSR baselines that might use different architectural designs (e.g., direct projection vs. complex fusion). The ablation studies are present but could be more extensive regarding the sensitivity of the OT parameters. The results are solid but not overwhelmingly superior to the baselines, suggesting that while effective, the gains are marginal in some settings.
The paper provides detailed descriptions of the model architecture, including the encoders (Whisper, AV-HuBERT), the LLM (LLaMA3.2-3B), and the OT alignment module. The loss functions and optimization strategies (Adam, LoRA) are specified. The dataset splits and preprocessing steps (lip cropping, audio downsampling) are described. However, the code is not publicly available (no project URL provided), and some hyperparameter tuning details are empirical. The "virtual bucket" mechanism is described but the exact implementation details (e.g., how the similarity margin $s$ is dynamically adjusted or fixed) could be clearer. Overall, reproducibility is moderate to high given the detailed mathematical formulation, but the lack of code release hinders immediate verification.
The method introduces additional computational overhead due to the OT calculation (Sinkhorn iterations) during training. The paper acknowledges that hyperparameter tuning for the OT and contrastive learning components is non-trivial and sensitive. The "virtual bucket" mechanism, while effective, adds complexity to the OT formulation. The evaluation is limited to the LRS3-TED dataset; performance on other AVSR benchmarks (e.g., LRS2, VoxCeleb) is not reported, limiting the generalizability claims. The method relies on pretrained encoders, so improvements are constrained by the quality of these base models.
This work contributes to the field of multimodal learning, specifically in robust speech recognition. By improving the alignment between audio, visual, and linguistic modalities, it enhances the reliability of AVSR systems in adverse environments, which has practical applications in assistive technologies, secure communications, and human-computer interaction. The use of OT for semantic alignment could inspire similar approaches in other multimodal tasks where cross-modal correspondence is challenging. The paper presents a method for aligning audio-visual features with linguistic embeddings in LLM-based ASR using Optimal Transport, demonstrating improved robustness and state-of-the-art results on LRS3-TED.
End-to-end neural audio models achieve high-fidelity compression and generation. We might read that performance as evidence they directly represent interpretable features such as pitch and timbre, but a model can produce plausible outputs without doing so. A model may encode these features in any reachable basis, but regardless of which, the features are well described as compositions of time-frequency-localized primitives. Whether state-of-the-art encoders preserve access to these primitives, and thus to compositions of them, remains unclear. Through theoretical analysis and controlled experiments, we show that several state-of-the-art strided convolutional encoders impose two structural bottlenecks, both predictable from architecture and signal structure, on access to these primitives: (1) they collapse primitives into alias equivalence classes, establishing a bound on representational capacity, and (2) they limit the frequency resolution available to learned filters, restricting separability. For well structured data, we find collapse rates of 31-35% and filter bandwidths 10-35x above the theoretical resolution bound, confirming that both bottlenecks arise under realistic signal conditions. We then introduce Gabor Latent Refactorization (GLRF), a lightweight post-hoc intervention that re-expresses encoder latents in a frequency-localized basis, reducing filter bandwidths from 10-35x to 1.5-3x of the theoretical resolution bound while preserving reconstruction fidelity and improving control over attributes like pitch. These results show that the encoders in question predictably degrade access to frequency-localized primitives, entangling the features that depend on them, and that a lightweight, retraining-free intervention can recover much of that access, improving steerability and interpretability.
Primary: Yale University
All Institutions: Yale University
This paper provides a compelling theoretical and empirical analysis of the representational limitations of strided convolutional audio encoders, introducing a novel post-hoc refactorization method that significantly improves frequency-specific controllability without retraining.
The paper proposes a rigorous theoretical and empirical framework to analyze the representational geometry of end-to-end audio encoders (specifically strided convolutional networks like EnCodec, DAC, and Stable Audio). The core methodological contribution is the identification of two structural bottlenecks: (1) Injectivity Failure, where downsampling causes distinct frequency components to collapse into alias equivalence classes, and (2) Separability Failure, where the receptive field limits the frequency resolution of learned filters, preventing independent access to these components. The authors derive analytical bounds for these phenomena based on signal structure and architecture parameters (stride, kernel size, dilation). To address the separability bottleneck, they introduce Gabor Latent Refactorization (GLRF), a lightweight, post-hoc intervention that re-expresses encoder latents in a frequency-localized Gabor basis using a learned linear mapping. This approach is theoretically grounded in signal processing principles (uncertainty principle, aliasing theory) and applied to modern deep learning architectures.
The experimental evaluation is comprehensive and well-controlled. The authors use synthetic signals with known ground-truth frequency components to precisely measure collapse rates and filter bandwidths, avoiding the ambiguity of natural audio analysis. They demonstrate a high correlation ($r=0.99$) between predicted and observed collapse rates across 643 signal configurations and three state-of-the-art models. The separability analysis shows that learned filters operate significantly above the theoretical resolution bound (10-35x), confirming the bottleneck. The GLRF intervention is evaluated on reconstruction fidelity (preserved) and controllability (improved). The controllability tests, including targeted component substitution and interpolation on both synthetic and real (NSynth) data, show that GLRF enables precise, frequency-specific manipulation that is absent in the original latent space. The results are robust and clearly support the claims.
The paper provides detailed descriptions of the models, signal generation procedures, and the GLRF method. The appendix includes proofs, simulation details, and code availability statements. The theoretical models for injectivity and separability are deterministic and depend only on architecture parameters, making them highly reproducible. The GLRF linear mapping is fit via closed-form ridge regression, which is stable and reproducible. The use of synthetic signals for primary validation ensures that results are not dependent on specific dataset splits or annotation errors.
The paper acknowledges several limitations. The analysis assumes signals are locally well-approximated as sums of narrowband components, which may not hold for highly transient or broadband sounds (e.g., percussion). The GLRF intervention addresses separability but not injectivity failure, which is inherent to the downsampling process and requires architectural changes. The linear mapping is fit on simple harmonic signals, and its generalization to complex, transient-rich content is not quantitatively assessed. The study focuses on strided convolutional encoders and does not evaluate models with explicit frequency-selective front-ends (e.g., SincNet).
This work has significant implications for the interpretability and controllability of neural audio systems. By demonstrating that state-of-the-art encoders systematically degrade access to interpretable frequency primitives, it challenges the assumption that high-fidelity reconstruction implies learned interpretability. The GLRF method offers a practical, retraining-free way to recover this structure, enabling better control over pitch and timbre in generation and compression tasks. This could facilitate applications in music production, speech synthesis, and scientific audio analysis. The paper also raises safety concerns regarding enhanced voice cloning capabilities, suggesting potential mitigations. This paper provides a compelling theoretical and empirical analysis of the representational limitations of strided convolutional audio encoders, introducing a novel post-hoc refactorization method that significantly improves frequency-specific controllability without retraining.
Existing methods for automatic music transcription are often limited to single-instrument recordings or fail on complex, real music mixes. Although previous work utilizes synthetic training data, the resulting models generalize poorly, leading to largely unusable transcription output in realistic, multi-instrument settings. In this work, we analyze the effectiveness of synthetic data for pre-training while combining it with fine-tuning on real music audio and post-training using reinforcement learning. We further introduce conditioning on instrument presence to customize transcriptions. Finally, we release MuScriptor, an open-weight multi-instrument music transcription model that works on real-world music recordings from across a diverse range of musical genres.
Primary: IRCAM-CNRS Sorbonne Univ
All Institutions: IRCAM-CNRS Sorbonne Univ
The paper presents MuScriptor, an open-weight multi-instrument music transcription model that achieves state-of-the-art performance on real-world music by combining synthetic pre-training, large-scale real-data fine-tuning, and reinforcement learning alignment, addressing the critical domain shift problem in automatic music transcription.
The paper proposes a practical and effective pipeline for multi-instrument automatic music transcription (AMT) using a decoder-only Transformer architecture. The core methodological contribution lies in the training strategy rather than architectural novelty: it combines large-scale synthetic pre-training (1.45M MIDIs) with fine-tuning on a newly collected dataset of 170k real-world recordings, followed by Reinforcement Learning (RL) post-training using a GRPO-like algorithm. The introduction of instrument presence conditioning at inference time is a valuable engineering choice that stabilizes predictions. The application of RL (specifically GRPO without KL penalties) to align transcription outputs with high-quality human annotations is a novel adaptation of techniques typically seen in LLMs to the audio-symbolic domain. However, the base architecture is a standard adaptation of existing sequence-to-sequence models (like MT3), limiting its theoretical novelty.
The experimental evaluation is comprehensive and robust. The authors conduct extensive ablation studies demonstrating the necessity of each training stage (synthetic pre-training, real fine-tuning, RL post-training). They show significant improvements over state-of-the-art baselines (YourMT3+, MT3) across multiple metrics (F1, Onset F1, Offset F1, Multi F1) on a diverse test set. The analysis of synthetic data effectiveness is particularly strong, providing clear evidence that synthetic data helps when real data is scarce but is insufficient on its own. The inclusion of a large, diverse real-world dataset (11k hours) addresses a critical gap in the field. The results are consistent and the improvements are substantial, particularly in generalization to real-world mixes.
The paper provides significant details for reproducibility. The authors release the model weights and inference code via GitHub. They describe the data collection process, including the use of audio-symbolic synchronization and filtering criteria. The hyperparameters for training and RL are specified. However, the exact source and licensing of the 170k real-world recordings are not fully detailed in the text (referred to as "internal dataset"), which may pose challenges for full independent replication of the data pipeline, although the model weights are available. The RL implementation details are clear enough for implementation.
The model struggles with overlapping notes of the same instrument and pitch, a common issue in dense music, which the authors acknowledge requires different tokenization. The performance on onset/offset precision for certain genres (like choral music) remains lower, indicating domain-specific challenges. The reliance on a specific instrument taxonomy (36 subgroups) limits the granularity of transcription. The RL post-training is performed on a small subset (300 tracks), which may limit the generalization of the RL benefits compared to the broader fine-tuning data.
MuScriptor represents a significant step forward for open-source music information retrieval tools. By providing a high-quality, open-weight model that works on real-world music, it lowers the barrier for researchers and practitioners to use AMT in downstream applications like music generation, education, and archival. The release of the large-scale dataset also contributes to the community, although the licensing details are crucial for broader adoption. The work highlights the importance of real-world data and alignment techniques in bridging the gap between synthetic benchmarks and practical applications. The paper presents MuScriptor, an open-weight multi-instrument music transcription model that achieves state-of-the-art performance on real-world music by combining synthetic pre-training, large-scale real-data fine-tuning, and reinforcement learning alignment, addressing the critical domain shift problem in automatic music transcription.
Training target speaker extraction (TSE) models for real conversational mixtures remains challenging because large-scale training corpora and clean target speech for supervision are unavailable. We present PS4, a proxy-supervised training framework for TSE in real conversational mixtures, with two main contributions. First, we construct a large-scale corpus of 71,771 training samples derived from four public datasets, covering both Chinese and English scenarios. Each sample contains an overlapping speech mixture, per-speaker enrollment audio, a ground-truth transcript, and frame-level voice activity labels. Second, we propose a proxy-supervised joint training strategy that fine-tunes a BSRNN-based TSE model using four complementary differentiable objectives: ASR cross-entropy, speaker similarity, frame-level voice activity detection, and perceptual audio quality. Starting from a publicly available pre-trained checkpoint, only the BSRNN separator is updated during fine-tuning. On the REAL-T challenge leaderboard, PS4 ranks 2nd overall, achieving the best speaker similarity and timing F1 among all submitted systems.
Primary: Nanjing University
All Institutions: Yijiahe AI, Nanjing, China; Tianjin University, Tianjin, China; Nanjing University, Nanjing, China
This paper presents a practical and effective proxy-supervised training framework for target speaker extraction, successfully leveraging multiple auxiliary signals to achieve state-of-the-art performance on real-world conversational speech benchmarks.
The paper proposes PS4, a proxy-supervised training framework for Target Speaker Extraction (TSE) in real conversational scenarios. The core methodological contribution is the design of four complementary differentiable loss functions to replace the unavailable clean target speech signal: ASR cross-entropy (using Whisper), speaker similarity (cosine margin ranking), frame-level Voice Activity Detection (VAD), and perceptual audio quality (DNSMOS). The approach leverages a pre-trained BSRNN-ECAPA model and fine-tunes only the separator. While the individual components (Whisper loss, speaker embedding losses) are known in the literature, their specific combination and application to the "real-world" TSE problem without clean references is a coherent and practical engineering contribution. The novelty lies in the systematic integration of these proxies to handle the domain gap between simulated and real data. EXPERIMENTAL_EVALUTION: The authors construct a new dataset, REAL-PS4, by curating four public datasets (AISHELL-4, AliMeeting, AMI, CHiME-6) to create overlapping speech mixtures with ground-truth transcripts and VAD labels. Experiments are conducted on the REAL-T benchmark. The results show consistent improvements over the official REAL-T baselines (BSRNN_TFMAP and BSRNN_EMB) across all four evaluation metrics (TER, SIM, DNSMOS OVRL, Timing F1). Notably, the system achieved 2nd place on the REAL-T challenge leaderboard, with the best speaker similarity and timing F1 scores among all submitted systems. The ablation and per-dataset analysis provide reasonable insights into where the model succeeds (cleaner recordings like AMI) and struggles (noisy/far-field like AISHELL-4).
The paper provides a link to the code repository (GitHub) and the dataset (HuggingFace). The methodology is described in sufficient detail, including the specific loss formulations and the use of frozen pre-trained models (Whisper, ECAPA-TDNN). The training setup (fine-tuning only the separator) is clearly defined. This ensures a high degree of reproducibility for other researchers.
The primary limitation is the reliance on the quality of the source datasets for the REAL-PS4 corpus. If the original diarization or transcript annotations contain errors, these propagate into the training data. Additionally, the use of black-box proxies (Whisper, DNSMOS) introduces potential misalignment; for instance, Whisper might hallucinate content or fail on heavy noise, and DNSMOS is a heuristic metric that may not perfectly correlate with human perception of extraction quality versus just audio fidelity. The paper does not include a detailed ablation study showing the marginal gain of each specific loss component, making it hard to isolate the contribution of the perceptual or VAD losses versus the linguistic/speaker losses.
This work addresses a critical bottleneck in speech processing: the lack of large-scale, real-world training data for TSE. By demonstrating that proxy supervision can effectively bridge the gap between simulated and real domains, it enables more robust deployment of TSE systems in real-world applications like meeting transcription and hearing aids. The release of the REAL-PS4 dataset also benefits the community by providing a standardized, large-scale resource for training and evaluating TSE models in realistic conditions. This paper presents a practical and effective proxy-supervised training framework for target speaker extraction, successfully leveraging multiple auxiliary signals to achieve state-of-the-art performance on real-world conversational speech benchmarks.
Audio-Visual speech recognition systems often degrade in real-world scenarios due to signal corruption and distribution shifts. To address this, we propose a unified uncertainty-modeling framework, namely the uncertainty-aware Bayesian gating network (UBG-Net). UBG-Net features a Modality Uncertainty-aware Bayesian Fusion (MUBF) mechanism that injects signal-level aleatoric uncertainty into a Bayesian network to model epistemic uncertainty, thereby ensuring robust fusion of pre-trained backbone features. For inference, we introduce Distribution Uncertainty-aware Hierarchical Voting (DUHV) to select transcripts from Monte Carlo samples, prioritizing frequency and using inference scores in case of a tie. Experiments on the AVCocktail and LRS2 datasets demonstrate the overall superiority of UBG-Net compared to SOTA baselines. Ablation studies confirm that MUBF and DUHV effectively filter noise, enhancing fusion and decoding robustness.
Primary: University of Science and Technology of China
All Institutions: University of Science and Technology of China
The paper presents a well-motivated and technically sound framework for robust audio-visual speech recognition by integrating aleatoric and epistemic uncertainty modeling through a novel Bayesian gating mechanism and hierarchical voting strategy, demonstrating significant improvements on challenging real-world and simulated benchmarks.
The paper proposes UBG-Net, an Audio-Visual Speech Recognition (AVSR) framework that integrates uncertainty modeling into the fusion process. The core methodological contribution is the Modality Uncertainty-aware Bayesian Fusion (MUBF), which attempts to bridge aleatoric (data) and epistemic (model) uncertainty. Specifically, it extracts signal-level aleatoric uncertainty (mean and variance) from audio and visual streams and concatenates them as a context vector to guide a Bayesian Gating Network (BGN). The BGN uses variational inference to model weight uncertainty, aiming to produce robust fusion weights. For inference, the authors introduce Distribution Uncertainty-aware Hierarchical Voting (DUHV), a post-processing step that selects the final transcript from Monte Carlo samples by prioritizing frequency (majority vote) and using inference scores for tie-breaking. While the integration of these components is coherent, the novelty is moderate. Bayesian fusion in multimodal learning is not new; the specific twist of using aleatoric parameters as context for epistemic gating is a logical and sensible design choice, but it does not represent a fundamental shift in uncertainty modeling theory. The approach relies on standard variational inference techniques (ELBO, reparameterization) applied to a specific architectural bottleneck.
The experimental evaluation is conducted on two benchmarks: LRS2 (with simulated noise at various SNRs) and AVCocktail (real-world multi-party conversations). The results demonstrate consistent improvements over a strong baseline (AV-HuBERT based) and other SOTA methods, particularly in low-SNR conditions (-5 dB) and in the challenging AVCocktail fixed-chunk setting. The ablation studies effectively isolate the contributions of MUBF and DUHV, showing that combining both uncertainty types and the hierarchical voting strategy yields the best performance. The inclusion of general-purpose models like Whisper and Qwen3-Omni provides good context for the difficulty of the task. However, the evaluation is limited to Word Error Rate (WER). There is no analysis of the calibration of the uncertainty estimates themselves (e.g., Expected Calibration Error or reliability diagrams), which is crucial for a paper claiming to model uncertainty. Additionally, the claim of "7.9% improvement" on AVCocktail seems high and warrants scrutiny regarding the baseline comparison, though the table data (implied) supports significant gains. The lack of LRS3 evaluation is noted but acceptable given access issues.
The paper provides detailed implementation details, including network architectures (2-layer MLPs, Bayesian linear layers), initialization schemes (Xavier-uniform, bias -5.0), hyperparameters (learning rate, batch size, weight decay), and training strategies (two-stage, DoRA). The use of a publicly available baseline repository (AVSRCocktail) aids reproducibility. The Monte Carlo sampling count (K=5) is specified. However, the code is not explicitly linked in the provided text (though a baseline repo is mentioned), and the specific random seeds for the 10 independent runs are not detailed, which is important for stochastic Bayesian methods. Overall, the description is sufficient for a competent researcher to reproduce the results.
The paper acknowledges a slight performance decline at 10 dB and clean conditions, attributing it to "over-enhancement." This suggests the model may be overly aggressive in suppressing features even when they are reliable, indicating a potential lack of robustness in high-quality regimes. The reliance on Monte Carlo sampling for inference increases latency compared to deterministic baselines, although the authors argue K=5 is a good trade-off. The method assumes that the pre-trained backbone features are sufficiently expressive, which might not hold for out-of-domain data not seen during the fine-tuning stage. Furthermore, the complexity of the Bayesian layers adds computational overhead during training and inference.
This work contributes to the robustness of multimodal AI systems, which is critical for real-world deployment in noisy environments (e.g., autonomous vehicles, smart homes, hearing aids). By providing a framework that quantifies and mitigates uncertainty, it offers a pathway to more reliable AVSR systems. The techniques could be extended to other multimodal tasks where sensor reliability varies dynamically. The paper presents a well-motivated and technically sound framework for robust audio-visual speech recognition by integrating aleatoric and epistemic uncertainty modeling through a novel Bayesian gating mechanism and hierarchical voting strategy, demonstrating significant improvements on challenging real-world and simulated benchmarks.
Neural audio codecs (NACs) enable efficient audio compression and have achieved success in downstream tasks such as speech synthesis. However, their discrete representations consistently underperform traditional spectral features in automatic speaker verification (ASV). We empirically demonstrate that speaker cues are implicitly preserved in discrete tokens but remain underutilized by conventional ASV training paradigms. To address this, we propose a Cross-Feature Knowledge Distillation (CFKD) framework. By guiding the codec-based student to mimic the embedding space of a strong Fbank-based teacher, CFKD provides structured supervision for effective utilization of speaker information in tokens. Experiments on the VoxCeleb benchmarks show that CFKD substantially improves the ASV performance of codec-based systems, allowing them to approach the accuracy of Fbank-based teacher models and highlighting the potential of discrete audio tokens for diverse speech tasks.
Primary: The Hong Kong Polytechnic University
All Institutions: The Hong Kong Polytechnic University
The paper presents a practical and effective method for leveraging discrete neural audio tokens in Automatic Speaker Verification through cross-feature knowledge distillation, demonstrating that discrete representations can achieve performance comparable to traditional spectral features.
The paper proposes Cross-Feature Knowledge Distillation (CFKD) to address the performance gap between discrete neural audio codec tokens and continuous spectral features (Fbanks) in Automatic Speaker Verification (ASV). The core methodology involves training a student model (using EnCodec tokens) to mimic the embedding space of a teacher model (using Fbanks) via cosine similarity loss at the embedding level. The authors also conduct an ablation study on backbone architectures (1D vs 2D), concluding that 1D architectures (ECAPA-TDNN) are more suitable for discrete tokens due to the lack of spatial adjacency in the token latent space. The approach is conceptually sound and addresses a clear problem: discrete tokens from generative codecs are underutilized in discriminative tasks. However, the novelty is moderate as knowledge distillation is a standard technique, and applying it across different feature modalities (cross-feature) is a logical but not radically new extension. The specific insight regarding the inductive bias of 1D vs 2D networks for discrete tokens is a valuable empirical contribution.
The experiments are conducted on standard ASV benchmarks (VoxCeleb1 and VoxCeleb2). The results show that CFKD significantly improves the performance of token-based systems, narrowing the gap with Fbank-based systems. For instance, on Vox1-O with ECAPA-TDNN, the EER drops from 3.38% (baseline) to 2.25% (CFKD), approaching the teacher's 2.21%. The paper includes error analysis showing complementary error sets between teacher and student, which supports the claim that tokens preserve distinct speaker cues. The evaluation is thorough, covering different bitrates and backbones. However, the comparison is primarily against the Fbank baseline and a prior work (Codec-ASV). The improvement is substantial but the absolute performance of the token-based system still lags slightly behind the Fbank teacher in some settings (e.g., minDCF 0.231 vs 0.236 is very close, but in other configs, the teacher is better). The results are convincing but not revolutionary.
The paper provides detailed descriptions of the experimental setup, including the EnCodec configuration (24kHz, 32 quantizers), backbone architectures, training hyperparameters (learning rate, batch size, optimizer), and data augmentation (MUSAN, RIR). The use of standard frameworks (Wespeaker) and datasets (VoxCeleb) enhances reproducibility. The specific distillation weight ($\alpha=40$) is noted as unusually high, which is a specific hyperparameter choice that might require tuning for other datasets. Overall, the method is reproducible.
The primary limitation is that the token-based system, even with CFKD, does not consistently outperform the Fbank teacher; it merely approaches it. This suggests that while the gap is closed, the discrete representation may still have inherent limitations in capturing fine-grained speaker details compared to continuous spectral features. Additionally, the method relies on a pre-trained Fbank teacher, which adds complexity to the training pipeline (two models instead of one). The paper does not explore the impact of different codec types or bitrates on the distillation efficacy beyond the standard EnCodec configuration. The claim that tokens "preserve sufficient speaker-discriminative information" is supported, but the mechanism of *how* the distillation extracts this is somewhat opaque (black-box embedding alignment).
This work has significant implications for the field of speech processing, particularly in bridging the gap between generative audio models (which use discrete tokens) and discriminative tasks (like ASV). It suggests that discrete representations can be effectively used for identity verification, which is crucial for applications requiring low-bitrate transmission or integration with large language models that operate on discrete tokens. It challenges the dominance of spectral features in ASV and opens avenues for unified audio models that can handle both generation and recognition tasks efficiently. The paper presents a practical and effective method for leveraging discrete neural audio tokens in Automatic Speaker Verification through cross-feature knowledge distillation, demonstrating that discrete representations can achieve performance comparable to traditional spectral features.
Video-to-audio (V2A) generation aims to synthesize realistic audio that is both semantically consistent with and temporally synchronized to a silent video. Despite recent progress, many methods still rely on multi-stage training, resulting in high computational costs and long runtimes, or transform visual input into text to leverage pretrained text-to-audio models, sacrificing fine-grained temporal cues. To overcome these limitations, we propose Flowley, an end-to-end, single-stage training architecture that produces soundtracks by combining visual features with textual prompts. Crucially, we introduce Progressive Soft-masked Cross-Attention, which embeds audio-visual synchronization directly within its attention mechanism, adding zero additional computational cost compared to standard attention layers. We further observe that existing V2A benchmarks lack sound-oriented descriptive captions, which can potentially degrade the quality of the synthesized audio. To remedy this, we propose SoundCap, a plug-and-play pipeline for creating detailed, sound-aware captions that guide the model. Remarkably, without integrating any pretrained audio-visual alignment modules, Flowley achieves state-of-the-art performance on VGGSound across multiple metrics. Moreover, by incorporating SoundCap, we further exceed the performance of the strongest existing close-sourced methods in terms of audio quality in the zero-shot setting.
Primary: FPT Software AI Center
All Institutions: FPT Software AI Center, NVIDIA Corporation
[Flowley introduces a novel end-to-end flow-based architecture with Progressive Soft-masked Cross-Attention for precise video-to-audio generation, achieving state-of-the-art performance on VGGSound and outperforming larger closed-source models in zero-shot settings when augmented with its SoundCap captioning pipeline.] This paper presents a significant advancement in V2A generation by eliminating the need for multi-stage training and external alignment modules, offering a more efficient and effective solution for synthesizing temporally synchronized and semantically consistent audio from video.
The paper proposes Flowley, an end-to-end flow-based architecture for Video-to-Audio (V2A) generation. The core technical contribution is the Progressive Soft-masked Cross-Attention (PSCA) mechanism, which embeds temporal alignment directly into the attention layers of a DiT-like backbone. This avoids the need for external synchronization modules or multi-stage training pipelines common in prior works (e.g., MMAudio, VinTAGe). The method combines visual features (CLIP), textual prompts (FLAN-T5), and audio latents in a multi-stream to single-stream architecture. Additionally, the authors introduce SoundCap, a pipeline using AV-LLMs to generate detailed, sound-aware captions to address the lack of rich textual annotations in V2A datasets. The approach is technically sound, leveraging recent advances in flow matching and multimodal transformers. The PSCA mechanism is a clever, low-cost modification to standard cross-attention that explicitly models the temporal relationship between video frames and audio tokens.
The evaluation is comprehensive, covering distribution matching (KAD, FAD), audio quality (IS), semantic alignment (IB-Score, LB-Score), and synchronization (Align Acc). Flowley achieves state-of-the-art results on the VGGSound benchmark, outperforming larger models like MMAudio and FoleyCrafter in most metrics. The inclusion of SoundCap further boosts performance, particularly in zero-shot settings on the Movie Gen Audio benchmark, where it surpasses the closed-source Movie Gen Audio model in quality despite being significantly smaller and trained on less data. The subjective human evaluation supports the objective metrics, showing a preference for Flowley's synchronization and quality. The ablation studies effectively validate the contributions of PSCA, the dual cross-attention, and the SoundCap pipeline.
The paper provides detailed implementation details, including architecture dimensions, hyperparameters for PSCA (window sizes, fade zones), training schedules, and evaluation metrics. The use of standard datasets (VGGSound) and established baselines facilitates comparison. The code link is provided. The description of the SoundCap pipeline is sufficiently detailed for reproduction, assuming access to the specified AV-LLMs and VLMs.
The paper acknowledges that while Flowley excels in quality and distribution matching, it slightly lags behind some baselines in the Align Acc metric, although human preference favors its synchronization. This suggests a potential discrepancy between the automated Align Acc metric and human perception of temporal alignment. The reliance on AV-LLMs for SoundCap introduces a dependency on large, potentially expensive models for data generation, although the final VLM is lightweight. The method's performance on highly complex scenes with multiple simultaneous sound sources or off-screen sounds might be challenging, as indicated by the noise handling strategies in SoundCap.
This work contributes to the field of generative AI for multimedia, specifically in creating realistic and synchronized audio-visual experiences. It has applications in film production (Foley), gaming, virtual reality, and accessibility. The SoundCap pipeline offers a generalizable solution for improving text-audio grounding in other multimodal tasks. The open-source nature of the work (implied by the project page) encourages further research in efficient, end-to-end V2A generation. [Flowley introduces a novel end-to-end flow-based architecture with Progressive Soft-masked Cross-Attention for precise video-to-audio generation, achieving state-of-the-art performance on VGGSound and outperforming larger closed-source models in zero-shot settings when augmented with its SoundCap captioning pipeline.] This paper presents a significant advancement in V2A generation by eliminating the need for multi-stage training and external alignment modules, offering a more efficient and effective solution for synthesizing temporally synchronized and semantically consistent audio from video.
Speech large language models (Speech LLMs) typically encode speech into sequences far longer than text, creating a major efficiency bottleneck during autoregressive decoding. A common remedy is to compress the speech sequence at the adapter level to remove temporal redundancy before it enters the LLM; however, such early downsampling risks discarding fine-grained information that cannot be recovered. We propose SpeechKV, which applies a learned pooling to the KV cache of speech tokens inside the LLM. This design allows the LLM to fuse speech and text internally while directly accelerating decoding. Trained on 71K hours of speech data, SpeechKV compresses the speech to approximately text-level granularity yet maintains performance on par with or even slightly better than the uncompressed baseline, with relative gains of 6.6% on out-of-domain entity recognition and 2.3% on OpenASR, while delivering at least 1.49 times decoding speedup that scales with audio length.
Primary: Microsoft
All Institutions: Microsoft, National Taiwan University
SpeechKV introduces a novel in-LLM KV cache compression technique for Speech LLMs that achieves competitive performance with significant decoding speedups by leveraging learned pooling on intermediate layer representations.
The paper proposes SpeechKV, a novel approach to compressing the Key-Value (KV) cache of speech tokens within the internal layers of a Speech Large Language Model (Speech LLM). Unlike traditional methods that compress speech embeddings before they enter the LLM (adapter-level compression), SpeechKV applies a learned pooling operation to the KV cache at an intermediate layer ($l_0$). This design preserves full-resolution queries and the residual stream, allowing the model to fuse speech and text representations internally while reducing the sequence length for subsequent attention computations. The method is theoretically sound, leveraging the observation that speech representations become redundant in deeper layers. The approach is simple yet effective, avoiding the need for complex architectural changes or external modules.
The authors evaluate SpeechKV on a Qwen3-1.7B backbone trained on 71K hours of ASR data. They compare against adapter-level compression (Concat-MLP, Window Q-Former) and an alternative in-LLM hidden-state compression (HS Compression). Results show that SpeechKV maintains or slightly improves upon the uncompressed baseline performance on both in-domain (OpenASR) and out-of-domain (In-house ER, In-house ASR) benchmarks. Notably, it achieves a 6.6% relative gain in entity recognition and a 2.3% gain in OpenASR WER compared to the baseline, while providing a 1.49x to 2.02x decoding speedup. The ablation studies on compression ratios ($R=4, 8$) and starting layers ($l_0$) provide valuable insights into the trade-offs between compression aggressiveness and performance retention. The attention analysis further supports the claim that SpeechKV leads to more focused attention patterns.
The paper provides detailed descriptions of the model architecture, training setup (optimizer, learning rate, batch size), and evaluation metrics. The use of open-source components (Qwen3-1.7B, Conformer encoder) and standard ASR benchmarks enhances reproducibility. However, the mention of "proprietary test sets" for In-house ASR and ER limits the ability of external researchers to fully replicate the out-of-domain evaluation. The code is not explicitly linked in the text provided, which is a minor hindrance to immediate reproducibility.
The primary limitation is the reliance on proprietary data for a significant portion of the evaluation, which prevents independent verification of the out-of-domain generalization claims. Additionally, the speedup, while significant, is modest (1.49x-2.02x) and may not justify the added training complexity for all use cases, especially given that feed-forward layers and text KV cache remain bottlenecks at higher compression ratios. The method is specific to Speech LLMs and may not generalize directly to other multimodal settings without adaptation.
This work contributes to the efficiency and accessibility of Speech LLMs, which are increasingly important for real-time applications like voice assistants and transcription services. By enabling faster decoding without significant performance loss, SpeechKV could facilitate the deployment of more capable speech models on resource-constrained devices. The insights into layer-wise redundancy in speech representations also offer broader implications for understanding how multimodal models process continuous signals. SpeechKV introduces a novel in-LLM KV cache compression technique for Speech LLMs that achieves competitive performance with significant decoding speedups by leveraging learned pooling on intermediate layer representations.
Voice anonymisation aims to protect speaker identity. Currently, its empirical privacy evaluation heavily relies on the Equal Error Rate (EER). Originally designed for biometric verification, EER aggregates scores globally, implicitly assuming an attacker is only trying to verify if two specific voice samples match (a 1-to-1 comparison). This introduces a threat model mismatch with real-world database linkage attacks, where an attacker searches across a fixed set of N enrolled identities (a 1-to-N closed-set search), allowing global averages to obscure localised privacy failures. While recent 1-to-N metrics address this aggregation issue, they abstract away the magnitude of the biometric evidence. In this paper, we propose a modular, information-theoretic evaluation framework explicitly designed for the 1-to-N linkage threat model. Within this framework, our core metric, Local Information Disclosure (LID), quantifies the exact privacy loss of a single trial utterance in bits by calibrating its raw similarity scores into the attacker's posterior confidence for each enrolled identity. Evaluating top-performing systems from the VoicePrivacy 2024 Challenge reveals that systems exhibiting near-perfect EERs (48 %) can still suffer from localised vulnerabilities with worst-case disclosures reaching 1 bit per trial utterance (effectively doubling the attacker's success rate over a random guess). We demonstrate that adopting localised privacy metrics is essential for capturing worst-case risks and aligning with strict privacy regulations.
Primary: Aalto University
All Institutions: Aalto University, Inria, Nokia, Macquarie University
The paper introduces a novel information-theoretic metric, Local Information Disclosure (LID), to evaluate voice anonymisation systems under 1-to-N linkage threats, revealing that systems with good global privacy metrics (EER) can still suffer from severe local vulnerabilities, thereby providing a more accurate and regulation-aligned assessment of privacy risks.
The paper proposes a rigorous, information-theoretic evaluation framework for voice anonymisation, specifically targeting the mismatch between standard 1-to-1 verification metrics (like EER) and real-world 1-to-N linkage threats. The core contribution is the Local Information Disclosure (LID) metric, which quantifies privacy loss in bits by calibrating raw similarity scores into posterior probabilities using logistic regression on row-normalized scores. The methodology is mathematically sound, leveraging Pointwise Mutual Information (PMI) to measure the reduction in attacker uncertainty. The framework is modular, allowing for various global aggregations (ALID, PDR, NDR, LID_max) that provide a more granular view of system vulnerabilities than global averages. The approach correctly identifies that global metrics can obscure local failures where specific utterances are highly linkable despite good average performance.
The authors evaluate their framework against systems from the VoicePrivacy 2024 Challenge (VPC 2024), including baselines and top submissions. They use the standard LibriSpeech dev/eval splits. The experiments demonstrate a critical finding: systems with near-perfect EERs (e.g., 0.48, indicating high privacy in 1-to-1 terms) can still exhibit worst-case local disclosures of ~1 bit, effectively doubling the attacker's success rate over random chance for specific trials. The results are presented clearly through CCDFs and tables comparing EER, C_llr, and the new LID-based metrics. The analysis of the calibration weight $w$ provides additional insight into the discriminative power of the anonymised embeddings. The empirical evidence strongly supports the claim that 1-to-1 metrics are insufficient for assessing 1-to-N privacy risks.
The paper provides detailed mathematical formulations for the LID metric, including the row-wise z-normalization and logistic calibration steps. It specifies the datasets (LibriSpeech) and the attacker model (semi-informed ECAPA-TDNN). However, the specific implementation details of the calibration (e.g., exact hyperparameters for the logistic regression, handling of edge cases) are somewhat abstract. The code is not explicitly linked in the provided text, though the VPC 2024 data is public. Reproducibility is moderate; a researcher could implement the method, but might need to infer some implementation details.
The primary limitation is the reliance on logistic regression for score calibration, which assumes a specific distribution of scores (normality after z-normalization). The authors acknowledge this and suggest it may not hold for all anonymisation methods. Additionally, the framework is evaluated only on the semi-informed threat model; while this is the most realistic, the framework's behavior under ignorant or lazy-informed models is not explored. The evaluation is limited to voice data; while the framework is generalizable, its applicability to other modalities (e.g., face, fingerprint) is not demonstrated. The "random" baseline comparison is useful but simplistic; a more nuanced baseline might involve a dummy classifier with similar score distributions but no identity information.
This work has significant implications for the deployment of voice anonymisation systems in privacy-sensitive applications (e.g., healthcare, legal, customer service). By providing a metric that aligns with regulatory concerns about "singling out" and "linkability" (e.g., GDPR), it offers a more robust tool for auditors and developers. It shifts the focus from average performance to worst-case risks, which is crucial for compliance. The information-theoretic framing also contributes to the broader field of privacy-preserving machine learning by offering a concrete way to quantify information leakage in biometric systems. The paper introduces a novel information-theoretic metric, Local Information Disclosure (LID), to evaluate voice anonymisation systems under 1-to-N linkage threats, revealing that systems with good global privacy metrics (EER) can still suffer from severe local vulnerabilities, thereby providing a more accurate and regulation-aligned assessment of privacy risks.
Speaker embeddings, or x-vectors, are widely used to represent speaker identity and speaker-related attributes, but existing embedding extractors are typically descriptive rather than generative: they map an observed speech segment to an x-vector, which is then used for downstream applications. We introduce ProPS, Prompted Profile Synthesis, a framework for generating distributions of speaker embeddings conditioned on natural language prompts such as "a thirties male speaker with an Indian accent". ProPS converts human-written profile descriptions into sentence embeddings and uses a mixture density network trained on a large-scale dataset to predict a Gaussian mixture model in the x-vector space. The model is trained by maximizing the likelihood that real speaker embeddings match the requested profile, and its generated distributions are evaluated by negative log-likelihood on held-out x-vectors and by attribute classification accuracies on sampled synthetic x-vectors. Experiments show that ProPS produces profile-conditioned distributions and generates x-vectors that preserve requested speaker attributes such as age, gender, accent, and prosodic characteristics. This design enables controllable speaker-profile synthesis for speech generation systems like Text-To-Speech (TTS) or Voice Conversion (VC) while anchoring generated distributions in observed speaker-embedding structure.
Primary: Johns Hopkins University
All Institutions: Johns Hopkins University
The paper introduces ProPS, a novel framework for natural language-conditioned generation of speaker embedding distributions using Mixture Density Networks initialized from profile-specific GMMs, enabling controllable and diverse synthetic speaker creation for downstream speech applications.
The paper proposes ProPS, a framework for generating speaker embedding distributions conditioned on natural language prompts. The core methodology involves using a Sentence-BERT encoder to map text prompts to embeddings, which are then fed into a Mixture Density Network (MDN). The MDN predicts the parameters (weights, means, diagonal covariances) of a Gaussian Mixture Model in the x-vector space. A key technical contribution is the initialization strategy: instead of random initialization, the authors pre-compute GMMs for each speaker profile in the dataset (CapSpeech) to create a "component bank," which initializes the MDN's Gaussian parameters. The training occurs in two stages: pre-training the MLP to map GPT-generated descriptions to the correct profile GMMs, and fine-tuning end-to-end with human-written descriptions to align with real x-vector distributions. This approach is technically sound and leverages established components (SBERT, MDN, ECAPA-TDNN) in a novel configuration for speaker profile synthesis.
The experiments are conducted on the CapSpeech dataset, which provides structured metadata and human-written descriptions. The evaluation metrics include Negative Log-Likelihood (NLL) on held-out x-vectors and attribute classification accuracy on sampled synthetic x-vectors. The ablation study effectively demonstrates the contribution of each training stage. Results show that ProPS successfully preserves demographic attributes (gender, accent, age) and some prosodic features, though prosodic control (pitch, pace, tone) is less reliable. The LDA visualizations provide qualitative support for the distributional match. The evaluation is rigorous within the scope of embedding-space generation, though it lacks downstream waveform generation tests which are mentioned as future work.
The paper provides a GitHub link for the code (anonymized version). The dataset (CapSpeech) is publicly available. The methodology is described in sufficient detail, including hyperparameters (learning rates, epochs, K=16 components) and model architectures (SBERT all-MiniLM-L6-v2, ECAPA-TDNN). The use of GPT-4 (referred to as gpt-5.4-mini-2026-03-17, likely a typo or future-dated placeholder in the anonymized submission) for generating synthetic descriptions is noted, which aids reproducibility of the training data generation.
The authors acknowledge several limitations: reliance on CapSpeech's label quality and coverage, potential dataset biases, and restriction to English data. The model struggles with fine-grained prosodic characteristics, which is partly inherent to the x-vector representation itself. The use of GPT-generated descriptions for pre-training introduces a potential distribution shift if the synthetic descriptions do not perfectly capture the semantic nuance of human descriptions, although the fine-tuning stage mitigates this.
ProPS enables controllable speaker synthesis for TTS and Voice Conversion systems without requiring reference audio, which has significant implications for accessibility, entertainment, and privacy (anonymization). By generating distributions rather than single points, it allows for diversity in generated voices, reducing the risk of generating only stereotypical or "average" voices. However, the ability to generate realistic speaker embeddings from text prompts also raises concerns about deepfake audio generation and impersonation, necessitating careful ethical guidelines and watermarking in downstream applications. The paper introduces ProPS, a novel framework for natural language-conditioned generation of speaker embedding distributions using Mixture Density Networks initialized from profile-specific GMMs, enabling controllable and diverse synthetic speaker creation for downstream speech applications.
End-to-end ASR models transcribe in a single pass, leaving no room for the decoder to revisit hard inputs. We propose LatentASR, a parameter-efficient method that adds continuous latent test-time scaling to a frozen ASR backbone. Two small trainable modules drive it: a Latent Adapter that iteratively refines a few latent prefix positions through bounded, stabilized updates, and a Value Head that predicts whether extra computation will help and halts the loop early. The Qwen3-ASR-0.6B backbone stays fully frozen, and we train only ~4M extra parameters. We activate this loop with a deliberately small, diverse 500-utterance training set. Under this minimal-data regime, standard adaptation methods all regress: full fine-tuning, LoRA, and prompt tuning each increase WER. LatentASR is the only tested method that reduces WER on both clean benchmarks (FLEURS -2.54% and VoxPopuli -0.47% relative). The reductions are concentrated on intrinsically hard inputs. On accented and code-switched speech (ASCEND), LatentASR achieves a 16.0% relative CER reduction. Across 30 FLEURS languages (23,049 utterances), the multilingual WER decreases uniformly across resource tiers, confirming that the adapter generalizes without overfitting. Dynamic halting preserves most of the clean-set reduction at a fraction of the compute, skipping roughly half of all utterances at the entry gate. Our results show that a small, carefully chosen activation set can switch on test-time scaling inside a frozen ASR model without corrupting the model itself, converting fixed per-utterance compute into input-dependent compute where it is most needed.
Primary: National Taiwan University
All Institutions: National Taiwan University, ASUS, National University of Singapore
LatentASR introduces a parameter-efficient, continuous latent test-time scaling mechanism for frozen ASR models, demonstrating that minimal-data activation can unlock significant robustness gains on challenging speech inputs without corrupting the backbone's zero-shot capabilities.
The paper proposes LatentASR, a method for continuous latent test-time scaling in frozen ASR backbones. The core innovation lies in the "Latent Adapter," which iteratively refines a small number of latent prefix positions using bounded, stabilized updates (L2-normalization, sigmoid gating, fixed embedding anchor). This design is technically sophisticated, addressing the critical issue of representation collapse when injecting continuous states into a frozen decoder. The inclusion of a "Value Head" for dynamic halting adds a layer of efficiency and utility prediction. The methodology is well-grounded in the literature of latent test-time scaling (e.g., Coconut, Quiet-STaR) but adapts it uniquely for the direct transcription task where no intermediate reasoning trajectory exists. The stabilization mechanisms are novel and rigorously ablated, showing that each component addresses a distinct failure mode.
The experimental evaluation is comprehensive and convincing. The authors demonstrate that under a minimal-data regime (500 utterances), standard adaptation methods (LoRA, prompt tuning) regress, whereas LatentASR improves performance. This "minimal-data activation" hypothesis is supported by ablation studies on activation set size. Results on clean benchmarks (FLEURS, VoxPopuli) show modest but consistent WER reductions. More significantly, the method shows substantial gains on challenging data: ASCEND (accented/code-switched) and noise-corrupted speech. The dynamic halting mechanism effectively allocates compute, skipping easy utterances. The multilingual evaluation confirms generalization. The use of CER for ASCEND and WER for others is appropriate. The statistical significance of the small WER gains on clean data is not explicitly tested, but the consistency across benchmarks and the large gains on hard data provide strong evidence.
The paper provides detailed hyperparameters, model architecture details, and training procedures. The use of a frozen backbone (Qwen3-ASR-0.6B) and specific training data sources (Common Voice, FLEURS, etc.) enhances reproducibility. The ablation studies are well-described. However, the exact composition of the 500-utterance training set is not fully public (only described as a mixture from specific sources with a seed), which might slightly hinder exact replication, though the general recipe is clear. The code is not linked, but the description is sufficient for a competent researcher to implement.
The primary limitation is the modest gain on clean, high-resource benchmarks (e.g., -2.54% relative on FLEURS). While statistically significant, the absolute WER reduction is small. The method's effectiveness is highly dependent on the "minimal-data" activation set; deviating from this size or composition can lead to regression. The method adds inference latency for the subset of utterances that pass the Value Head gate, although dynamic halting mitigates this. The approach is specific to ASR and may not generalize directly to other modalities without adaptation.
This work contributes to the field of efficient and robust ASR. By enabling test-time scaling on frozen models, it reduces the need for expensive full fine-tuning and preserves zero-shot generalization. This is particularly impactful for low-resource and challenging acoustic conditions (accents, noise, code-switching). The method promotes sustainable AI by reducing the carbon footprint associated with training large models for every downstream task. It also opens new avenues for applying latent test-time scaling to other sequence-to-sequence tasks where intermediate reasoning is not natural. LatentASR introduces a parameter-efficient, continuous latent test-time scaling mechanism for frozen ASR models, demonstrating that minimal-data activation can unlock significant robustness gains on challenging speech inputs without corrupting the backbone's zero-shot capabilities.
Modern automated audio captioning systems pair a frozen audio encoder with a large language model (LLM) via a trainable projector, incurring the encoder's inference cost and bottlenecking the model through its fixed acoustic features. We present CARD, an encoder-free audio captioning model that removes the encoder at inference: a 13.2M projector feeds a frozen LLM with merged LoRA adapters, while the teacher used to train it is discarded. CARD distills a pretrained audio teacher (CLAP-HTSAT) into the model, but rather than injecting it into the LLM alone, it routes the teacher's representations across components: perceptual stages to the projector and semantic stages to the LLM. This placement improves CIDEr-D by +12.18 over an LLM-only distilled model on AudioCaps and by +5.21 on Clotho, reaching 55.4 against a 66.4 encoder-kept upper bound with no encoder at inference, showing that where a teacher's knowledge is placed matters as much as its presence.
Primary: University of Essex
All Institutions: University of Essex, Institute for Analytics and Data Science
The paper presents CARD, a novel encoder-free audio captioning model that utilizes cross-component knowledge distillation to route teacher representations to appropriate student components, achieving competitive performance with encoder-based baselines while eliminating inference-time encoder costs.
The paper proposes CARD, an encoder-free audio captioning framework that eliminates the need for a dedicated audio encoder at inference time. The core methodological innovation is a "cross-component distillation" strategy. Instead of distilling teacher knowledge solely into the Large Language Model (LLM) or the projector, CARD routes the teacher's (CLAP-HTSAT) representations based on semantic abstraction: early perceptual stages ($t_0, t_1$) supervise the audio projector, while later semantic stages ($t_2, t_3$) supervise the LLM's LoRA adapters. This design choice is theoretically sound, aligning the level of feature abstraction with the functional role of each student component. The use of a two-phase training procedure (distillation followed by captioning fine-tuning) and the merging of LoRA adapters for efficient inference are well-structured technical decisions. However, the novelty is somewhat incremental, as the concept of distilling multimodal knowledge into LLMs without encoders (e.g., VoRA, AuRA) has been explored in vision and speech domains; the specific application to audio captioning with this particular component-aware routing is the primary differentiator.
The experimental evaluation is comprehensive and rigorous. The authors compare CARD against a strong encoder-based baseline (SLAM-AAC with CLAP) and several ablation variants of their own method (No Distill, LLM Distill, Proj_Full, Proj_Early). The results clearly demonstrate that the proposed cross-component distillation (CARD$^*$) significantly outperforms LLM-only distillation (+12.18 CIDEr-D on AudioCaps) and approaches the performance of the encoder-kept baseline (55.4 vs 66.4). The ablation studies effectively isolate the contribution of projector supervision versus LLM supervision, providing strong empirical evidence for the hypothesis that "where a teacher's knowledge is placed matters." The use of standard benchmarks (AudioCaps, Clotho) and metrics (CIDEr-D, SPIDEr, etc.) ensures comparability with existing literature. The inclusion of a "No Distill" baseline highlights the critical role of distillation in encoder-free models.
The paper provides sufficient implementation details for reproduction. It specifies the backbone model (Qwen3-4B), the teacher model (CLAP-HTSAT), the projector architecture (strided 1D convolutions), and the distillation loss formulation (cosine similarity). Training hyperparameters such as learning rate, batch size, LoRA rank, and optimization steps are clearly listed. The two-phase training protocol is well-defined. The only potential ambiguity lies in the exact interleaving strategy for text-only instruction data during Phase 1, but the general approach is clear. The code is not provided, which is a minor hindrance but the method is described with enough precision to be implemented.
A significant limitation is the performance gap between the encoder-free CARD model (55.4 CIDEr-D) and the encoder-based baseline (66.4 CIDEr-D). While the paper argues that removing the encoder is beneficial for efficiency, the ~11-point drop in performance is substantial. The paper does not extensively analyze the specific types of errors or the scenarios where the encoder-free model fails most severely. Additionally, the reliance on a specific teacher (CLAP) means the distillation is bounded by the teacher's capabilities and biases. The evaluation is limited to two standard datasets; performance on more diverse or noisy real-world audio might differ. The "Qwen3-4B" reference might be a typo for Qwen2-1.5B or Qwen2-7B or a very new model, which could cause confusion regarding the exact baseline capabilities.
This work contributes to the trend of efficient, integrated multimodal models that reduce inference latency and memory footprint by removing dedicated encoders. This has practical implications for deploying audio understanding systems on edge devices or in low-latency applications. By demonstrating that knowledge distillation can effectively transfer acoustic understanding into LLMs, it opens avenues for similar encoder-free architectures in other modalities. However, the reduction in performance compared to encoder-based systems suggests that for high-accuracy applications, the trade-off may not yet be favorable. The paper presents CARD, a novel encoder-free audio captioning model that utilizes cross-component knowledge distillation to route teacher representations to appropriate student components, achieving competitive performance with encoder-based baselines while eliminating inference-time encoder costs.
Speaker embeddings aggregate frame-level acoustic features into compact representations for speaker recognition. Recent uncertainty-aware speaker modeling approaches further characterize the reliability of speaker embeddings by estimating their associated uncertainty. However, existing methods often suffer from inaccurate uncertainty estimation and uncertainty miscalibration under domain shifts. To address these challenges, we propose a robust uncertainty modeling framework from both estimation and adaptation perspectives. Specifically, we introduce an Inter- and Intra-Speaker-Aware Uncertainty Softmax that incorporates both inter-speaker separability and intra-speaker variability into uncertainty learning, enabling uncertainty estimates to better capture the reliability of speaker embeddings. Furthermore, we propose an Uncertainty-Calibrated Domain Adaptation (UCDA) framework to mitigate uncertainty miscalibration caused by domain mismatch. Extensive experiments on both in-domain and cross-domain benchmarks demonstrate that the proposed approach consistently improves uncertainty reliability and speaker recognition robustness.
Primary: The Hong Kong Polytechnic University
All Institutions: The Hong Kong Polytechnic University, The University of Melbourne
The paper presents a robust uncertainty-aware speaker modeling framework that integrates inter- and intra-speaker variability into uncertainty learning and employs a novel domain adaptation strategy to mitigate uncertainty miscalibration under domain shifts, demonstrating consistent improvements in speaker verification performance on standard benchmarks.
The paper proposes two main contributions: an Inter- and Intra-Speaker-Aware Uncertainty Softmax (IISA-Uncertainty) and an Uncertainty-Calibrated Domain Adaptation (UCDA) framework. The IISA-Uncertainty method modifies the uncertainty-aware scaling factor in the loss function to incorporate both inter-speaker separability (hardness) and intra-speaker compactness. This is a logical extension of previous uncertainty-aware softmax losses (like UAAM-Softmax), aiming to provide a more nuanced supervision signal for uncertainty estimation. The UCDA framework addresses domain shift by aligning the distribution of target-domain uncertainty estimates with a source-domain prior using a negative log-likelihood objective, freezing most model parameters to preserve speaker-discriminative features. While the theoretical motivation is sound and addresses a real gap (uncertainty miscalibration under domain shift), the novelty is incremental. The concept of using hardness-aware scaling is not new in metric learning, and applying it to uncertainty is a straightforward adaptation. The domain adaptation strategy is also a standard application of distribution alignment, albeit focused on the uncertainty space rather than the embedding space.
The experiments are conducted on standard speaker verification benchmarks: VoxCeleb1 (in-domain) and CNCeleb (cross-domain). The baseline comparisons include strong systems like ECAPA-TDNN, CAM++, and Gemini SD-ResNet38, as well as prior uncertainty-aware methods like $U^3$-xi. The results show consistent improvements in Equal Error Rate (EER) and minimum Detection Cost Function (minDCF) across various settings. The paper provides a detailed ablation study on the proposed uncertainty scaling terms. However, the cross-domain results show some instability in minDCF with standard cosine scoring, though uncertainty-aware scoring mitigates this. The analysis of uncertainty reliability via correlation with discrimination quality is a strong point, providing empirical evidence that the learned uncertainty is meaningful. The use of a fixed source-domain prior for UCDA is a practical choice for unsupervised adaptation but limits the ability to adapt to significant distribution shifts if the prior is not representative.
The paper provides sufficient details regarding the training pipeline (using WeSpeaker toolkit), hyperparameters (scale, margin, epochs), and data augmentation (MUSAN, RIRs). The mathematical formulations are clear. However, the code is not explicitly linked, and some implementation details of the "detach" operations and specific hyperparameter tuning for the stabilizing constants are mentioned but not fully exposed in a way that guarantees out-of-the-box reproducibility without significant effort. The reliance on the WeSpeaker toolkit helps, but the specific modifications to the loss function and the UCDA module would need to be implemented from scratch or adapted from the cited $U^3$-xi code.
The proposed method relies on the assumption that intra-speaker variability can be effectively captured by the similarity to the most similar class prototype, which might be noisy in open-set scenarios. The UCDA framework's performance is dependent on the quality of the source-domain prior; if the source and target domains are too dissimilar, the alignment might not be optimal. The paper acknowledges that minDCF can degrade with standard scoring after UCDA, suggesting a trade-off that isn't fully resolved. Furthermore, the novelty is limited to incremental modifications of existing uncertainty-aware frameworks.
This work contributes to the robustness of speaker recognition systems, which is critical for real-world biometric authentication. By improving uncertainty estimation, it enables better decision-making in low-reliability scenarios (e.g., rejecting uncertain samples), potentially reducing false acceptance/rejection rates in adverse conditions. This has implications for security and user experience in voice-based systems. The paper presents a robust uncertainty-aware speaker modeling framework that integrates inter- and intra-speaker variability into uncertainty learning and employs a novel domain adaptation strategy to mitigate uncertainty miscalibration under domain shifts, demonstrating consistent improvements in speaker verification performance on standard benchmarks.
Large Audio-Language Models (LALMs) reason fluently about sound yet struggle to localize precisely when events occur, while classical Sound Event Detection attains frame-level precision only over a closed label set. At the intersection of these paradigms lies the task of Open-Vocabulary Audio Event Grounding: predicting all time intervals of a target sound event described by an arbitrary natural language query. While this task is crucial for real-world audio understanding and LALM adaptation, it is bottlenecked by data scarcity. Few large-scale resources provide open-vocabulary onset/offset supervision, and manual temporal annotation is prohibitively expensive. To address this, we introduce Auto-AEG, a scalable pipeline that constructs such supervision by automatic data construction and model fine-tuning. It pairs programmatically synthesized clips, which carry exact ground-truth intervals for supervised cold-start, with multi-model pseudo-labels on real-world audio that supply the reward signal for reinforcement learning. Training with this pipeline yields promising performance gains on both the DESED SED benchmark and AEGBench, an independent difficulty-stratified benchmark we release. Our results show that automatically constructed data, coupled with interval-aware reward function design, is an effective data-side route to expanding the temporal localization capability of LALMs.
Primary: Zhejiang University
All Institutions: Zhejiang University, Tsinghua University
This paper presents a novel and effective pipeline for automatically constructing open-vocabulary audio event grounding data, combining synthetic data for cold-start SFT with real-world pseudo-labels for RL fine-tuning, thereby significantly advancing the temporal localization capabilities of Large Audio-Language Models.
The paper proposes Auto-AEG, a pipeline for constructing open-vocabulary audio event grounding (AEG) data without manual annotation. The methodology is two-fold: Stage 1 uses programmatic synthesis of synthetic audio clips with exact ground-truth intervals to perform Supervised Fine-Tuning (SFT) on Large Audio-Language Models (LALMs). Stage 2 uses a multi-model pipeline (Gemini for labeling, PE A-Frame for localization, CLAP for label cleaning) to generate pseudo-labels on real-world audio, which are then used to fine-tune the model via Group Relative Policy Optimization (GRPO). The GRPO reward function is specifically designed to be interval-aware, combining F1-IoU, format rewards, and penalties for over-prediction. The approach is technically sound, leveraging the noise tolerance of RL to correct imperfect pseudo-labels, a strategy increasingly common in LLM alignment but novel in this specific audio grounding context. The separation of semantic identification (LALM) and temporal localization (frame-level model) in the annotation pipeline is a clever engineering choice to mitigate individual model weaknesses.
The authors evaluate their method on a newly introduced benchmark, AEGBench, which is difficulty-stratified and independently annotated. They also report results on the standard DESED SED benchmark. The results show significant improvements in mIoU and Event F1 over zero-shot baselines. The ablation studies effectively demonstrate the contribution of each stage: SFT provides a necessary cold-start, and GRPO further refines the temporal boundaries. The analysis of hard cases (e.g., long duration, polyphonic overlap) provides valuable insights into the current limitations of LALMs. The comparison with existing benchmarks highlights the unique contribution of AEGBench in terms of difficulty stratification and real-world relevance. The results on DESED confirm that the grounding capabilities transfer to closed-set SED tasks.
The paper provides detailed descriptions of the data construction pipeline, including the synthesis parameters, the multi-model annotation steps, and the GRPO training hyperparameters. The code for the synthesis and annotation pipeline is not explicitly linked in the text provided, but the methodology is described with sufficient detail for reproduction. The release of AEGBench is a significant contribution to reproducibility in this area. The use of open-source models (Qwen-Omni, PE A-Frame, CLAP) facilitates replication.
The synthetic data in Stage 1 suffers from a domain gap compared to real-world audio (room acoustics, source interaction). The pseudo-labels in Stage 2 are inherently noisy, although the RL component is designed to be robust to this. The performance on long-duration events is limited by the encoder window size of the underlying LALMs. The reliance on external models (Gemini, PE A-Frame) for annotation introduces potential failure modes if these models make errors, although the paper argues that the RL process mitigates this. The benchmark, while high-quality, is still limited in size (3,427 items) compared to some large-scale audio datasets.
This work addresses a critical bottleneck in audio understanding: the lack of large-scale, open-vocabulary temporal grounding data. By providing a scalable data construction pipeline and a new benchmark, it enables further research into LALMs for precise audio localization. This has implications for applications such as audio search, video editing, and assistive technologies for the visually impaired. The emphasis on open-vocabulary capabilities aligns with the broader trend towards more flexible and generalizable AI systems. This paper presents a novel and effective pipeline for automatically constructing open-vocabulary audio event grounding data, combining synthetic data for cold-start SFT with real-world pseudo-labels for RL fine-tuning, thereby significantly advancing the temporal localization capabilities of Large Audio-Language Models.
Autoregressive (AR) text-to-speech (TTS) models generate discrete speech tokens sequentially, which makes inference slow and can degrade robustness by propagating local errors and hallucinations. This limitation stems from their left-to-right AR commitment: each token must be determined before future speech-token context is available. However, such ordering is not an inherent requirement for TTS, as the full input text is available before synthesis. In this paper, we introduce DELTA-TTS, a lightweight LoRA-based adaptation framework that converts a pretrained AR TTS model into a discrete diffusion language model (dLLM) for confidence-ordered speech-token decoding. To better capture the local structure of speech, DELTA-TTS incorporates a convolution module that injects local acoustic context, together with a $1/t$-weighted training objective and a time-shifted inference schedule that defer low-confidence positions to later steps. Trained on only $585$ hours of LibriTTS, DELTA-TTS achieves a $\textbf{1.75}\%$ WER on Seed-TTS test-en, outperforming its AR backbone while generating tokens $\textbf{3.3}\times$ faster. Further analysis shows that DELTA-TTS produces sharper text--speech alignment, increases overall decoding confidence, and mitigates hallucinations observed in AR generation.
Primary: Sungkyunkwan University
All Institutions: Sungkyunkwan University, University of Seoul
DELTA-TTS effectively bridges the gap between autoregressive and diffusion-based speech synthesis by adapting a pretrained AR model into a discrete diffusion language model with speech-aware modifications, achieving state-of-the-art speed-quality trade-offs and robustness against hallucinations.
The paper proposes DELTA-TTS, a framework that converts a pretrained Autoregressive (AR) Text-to-Speech (TTS) model into a discrete Diffusion Language Model (dLLM). The core methodological contribution lies in adapting the AR-to-dLLM conversion technique, previously established for text, to the speech domain. Key innovations include: 1) Replacing causal attention with bidirectional attention to allow access to future context; 2) Introducing a Conformer-style convolution module to inject local acoustic inductive biases that bidirectional self-attention lacks; 3) Utilizing a $1/t$-weighted training objective and a time-shifted inference schedule to defer low-confidence token predictions to later steps. The approach is parameter-efficient, using LoRA adapters on top of a frozen CosyVoice3 backbone. The methodology is sound and addresses a known bottleneck in AR TTS (inference speed and error propagation) by leveraging the parallel generation capabilities of diffusion models.
The evaluation is comprehensive, comparing DELTA-TTS against a wide range of state-of-the-art AR and Non-Autoregressive (NAR) baselines, including Seed-TTS, CosyVoice series, FireRedTTS, and F5-TTS. The primary metric is Word Error Rate (WER) on Seed-TTS test-en and LibriSpeech-PC test-clean. DELTA-TTS achieves a WER of 1.75% on Seed-TTS, outperforming the AR backbone (CosyVoice3, 2.02%) and all listed NAR baselines. It also demonstrates superior Real-Time Factor (RTF), achieving a 3.3x speedup. The paper includes ablation studies validating the necessity of the convolution module and the time-shifted schedule. Furthermore, it provides qualitative analysis showing sharper text-speech alignment and mitigation of hallucinations (catastrophic failures) observed in the AR baseline. The use of both objective metrics (WER, SIM, UTMOS) and subjective evaluations (CMOS, SMOS) strengthens the claims.
The paper provides sufficient implementation details for reproduction. It specifies the backbone (CosyVoice3, Qwen2-0.5B), the adapter size (94M parameters), training data (585 hours LibriTTS), optimizer settings (AdamW, lr $10^{-4}$), and specific architectural choices (kernel size 31 for convolution). The training data size is small and publicly available, facilitating reproducibility. However, the code is not explicitly linked in the provided text, and the "rule-based length" prediction method is described but might require careful implementation to match results. The reliance on specific baselines' checkpoints (e.g., CosyVoice3) is noted, which is standard but requires access to those resources.
The primary limitation is the training data scale (585 hours), which is significantly smaller than the multi-hundred-thousand-hour datasets used by many baselines. While this highlights the efficiency of the adaptation, it may limit the model's generalization to out-of-distribution speakers or languages compared to larger models. The paper acknowledges this and notes the model is English-only. Additionally, the target length prediction is rule-based, which is a known weakness in dLLM-based TTS systems. The speedup is measured on token generation only; end-to-end latency including the flow-matching decoder is not fully detailed, though the decoder is frozen and shared.
This work contributes to the democratization of high-quality TTS by enabling faster inference and potentially lower training costs through efficient adaptation. It reduces the computational barrier for deploying TTS systems. The mitigation of hallucinations improves reliability. However, like all generative audio technologies, it carries risks related to misuse for deepfakes, though the paper frames it as academic research. The ability to generate speech faster and more robustly has positive implications for assistive technologies and interactive voice assistants. DELTA-TTS effectively bridges the gap between autoregressive and diffusion-based speech synthesis by adapting a pretrained AR model into a discrete diffusion language model with speech-aware modifications, achieving state-of-the-art speed-quality trade-offs and robustness against hallucinations.