Audio-language models compress a speech encoder's output through a Querying Transformer (Q-Former) connector before feeding it to a large language model. We identify two failures in this compression. The connector's output vectors collapse to a single direction, and different speakers produce nearly indistinguishable outputs, with paralinguistic cues such as speaker identity, gender, and prosody lost along the way. Our method, ORCA, reverses this collapse by splitting the queries into groups whose outputs are constrained to point in different directions. On SAKURA multi-hop reasoning, ORCA gains 26.4 points over an identically trained 4B baseline, reaching 75.2% (vs. 49.0% for the 8B Audio Flamingo-3). At the connector level, the same change cuts query redundancy by 12x and raises cross-speaker variance by 75x.
Primary: National Taiwan University
All Institutions: National Taiwan University, ASUS AICS
The paper presents a compelling structural solution to query collapse in audio-language model connectors, demonstrating that enforcing orthogonality among query groups significantly enhances the preservation of paralinguistic information and boosts multi-hop reasoning performance without increasing computational cost.
The paper proposes ORCA, a structural modification to the Querying Transformer (Q-Former) connector used in Audio-Language Models (ALMs). The core innovation is the partitioning of learnable queries into $G$ groups, each with its own layer mixture from the audio encoder, and enforcing orthogonality between the mean vectors of these groups. This is a geometric constraint applied to the connector's output space. The approach is theoretically grounded in diagnosing "query collapse" (where output vectors converge to a single direction) and "information loss" (specifically paralinguistic cues like speaker identity). The method is elegant in its simplicity: it requires no additional labeled data for specific attributes, relying solely on the language modeling loss and the geometric regularizer. The separation of shallow, mid, and deep encoder layers across groups emerges naturally from the optimization, suggesting the method successfully forces the model to utilize different representational subspaces.
The experimental design is rigorous, particularly the controlled comparison where only the connector changes between the baseline and ORCA, keeping the encoder, LLM, data, and training recipe identical. This isolates the contribution effectively. The results on the SAKURA benchmark are impressive, showing a +26.4 point gain on multi-hop reasoning for a 4B model, surpassing larger 8B models. The diagnostic metrics (cosine similarity reduction by 12x, cross-speaker variance increase by 75x) strongly support the hypothesis that query collapse was the bottleneck. The robustness analysis under noise adds depth, showing that while ORCA preserves cues that survive noise (like gender), it cannot recover physically masked information (fine prosody in emotion). The comparison with MMAU shows that while ORCA helps paralinguistic reasoning, it doesn't drastically outperform baselines on general audio understanding, which is a nuanced and honest result.
The paper provides sufficient detail for reproduction. It specifies the architecture (Whisper-Large-V3, Qwen3-4B), the connector structure (G=8, J=8, K=64), the loss weights ($inter=0.1$, $intra=0.03$), and the training setup (AQA5M dataset, 5 epochs, Adam optimizer). The code for the Q-Former modification is likely straightforward to implement given the clear mathematical formulation of the inter-group and intra-group loss terms. The use of standard datasets (SAKURA, MMAU, CREMA-D) ensures that results can be compared against existing literature.
The primary limitation is that the improvement is specific to tasks requiring the preservation of diverse acoustic cues (paralinguistics). For general audio understanding (MMAU), the gains are modest. Furthermore, the method cannot recover information that is physically lost in the audio encoder or masked by noise, as acknowledged in the discussion. The reliance on the language modeling objective means that the "orthogonal" subspaces are still biased towards what helps predict text, so purely acoustic tasks without textual grounding might not benefit as much. The method also assumes that the audio encoder provides sufficient diversity in its hidden states, which might not be true for all encoders.
This work has significant implications for the design of multimodal models. By identifying and fixing the "Procrustean Bed" problem in connectors, it demonstrates that architectural changes can yield massive performance gains without increasing model scale or data volume. This is particularly relevant for resource-constrained settings where scaling up is not feasible. It also highlights the importance of paralinguistic information in ALMs, encouraging future research to focus on preserving speaker identity, emotion, and prosody, which are crucial for human-like interaction. The geometric approach to preventing representation collapse could be adapted to other modalities where similar bottlenecks exist. The paper presents a compelling structural solution to query collapse in audio-language model connectors, demonstrating that enforcing orthogonality among query groups significantly enhances the preservation of paralinguistic information and boosts multi-hop reasoning performance without increasing computational cost.
Traditional emotional voice conversion (EVC) conditions generation on explicit target emotions like labels or references, defining the target affective state but omitting the direction or nature of the transition. We introduce instruction-guided relative emotional voice conversion, a task where natural-language instructions specify source-conditioned affective transformations (e.g., "make the speech slightly calmer" or "sound noticeably more confident") instead of fixed targets. To support this task, we construct TRACE-Instruct, a dataset of relative emotion instructions covering categorical transitions, intensity modifications, and open-ended affective changes. We propose TRACE-EVC, a zero-shot framework built around Emo-Compass, a module that models each conversion as a source-anchored rectified flow. Rather than conditioning on an explicit target, it predicts the direction and degree of the affective change. Experiments demonstrate that TRACE-EVC accurately follows relative emotion instructions while preserving speaker identity, linguistic content, and speech quality, and remains competitive with conventional EVC systems on standard categorical emotion conversion.
Primary: Johns Hopkins University
All Institutions: Johns Hopkins University, Center for Language and Speech Processing (CLSP)
This paper presents a significant methodological advance in emotional voice conversion by shifting from absolute target conditioning to relative, instruction-guided transformation using source-anchored rectified flows, offering a more intuitive and flexible control interface for expressive speech synthesis.
The paper introduces a novel task formulation: instruction-guided relative emotional voice conversion (EVC). Unlike traditional EVC which maps to absolute emotional states (labels or references), this work conditions generation on natural language instructions describing the *change* in emotion relative to the source (e.g., "make it calmer"). The core technical innovation is "Emo-Compass," a module that models this transformation as a source-anchored rectified flow. Instead of predicting an absolute emotion embedding from noise, it predicts the velocity vector (displacement) in a continuous affective space (combining emotion2vec, VAD, and prosodic features) from the source anchor. This is a sophisticated application of continuous normalizing flows/rectified flows to affective computing. The synthesis module adapts DurFlex-EVC to use these continuous embeddings. The approach is theoretically sound and addresses a genuine gap in fine-grained, directional control of speech emotion.
The authors evaluate on two datasets: ESD (inter-emotion) and MEAD (intra-emotion intensity). They compare against strong baselines including label-based (SGEVC, DurFlex-EVC) and reference-based (ZEST, VEVO) systems. Results show TRACE-EVC achieves competitive or superior performance in emotion similarity (EECS), classification accuracy, and speaker similarity (SECS) compared to baselines, while maintaining high speech quality (UTMOS/NISQA). Crucially, they introduce a human evaluation metric for "Instruction Following" (IF), which is essential for this task. The ablation studies on Emo-Compass components (VAD, prosody) are thorough and validate the design choices. The inclusion of unseen-speaker zero-shot evaluation on MEAD strengthens the claim of generalization.
The paper provides detailed implementation details, including model architectures (6-layer Transformer for Emo-Compass), training hyperparameters (AdamW, LR 1e-4), and datasets. They release code and demos. The dataset construction process (using LLMs to generate instructions from paired speech and VAD differences) is described, though the specific prompts and filtering rules could be more detailed for perfect reproducibility. However, the core code release mitigates this.
The reliance on LLMs for dataset construction introduces potential biases or hallucinations in the instruction generation, although rule-based filtering helps. The evaluation is limited to English. The "zero-shot" claim relies on the disentanglement capabilities of the underlying DurFlex-EVC backbone; if the source speaker's emotion is very strong, the relative control might struggle to override it completely, a common issue in EVC. The comparison with other prompt-based EVC methods is limited by the lack of public implementations, which is a fair point but leaves a gap in direct comparison with similar prompt-based approaches.
This work enables more natural and intuitive human-computer interaction in speech synthesis, allowing users to specify nuanced emotional adjustments via natural language. It has applications in audiobooks, dubbing, and assistive communication. The dataset TRACE-Instruct is a valuable resource for the community. Potential misuse includes generating deceptive emotional speech, though this is a general risk in EVC. This paper presents a significant methodological advance in emotional voice conversion by shifting from absolute target conditioning to relative, instruction-guided transformation using source-anchored rectified flows, offering a more intuitive and flexible control interface for expressive speech synthesis.
Video-to-audio (V2A) generation aims to synthesize realistic audio that is both semantically consistent with and temporally synchronized to a silent video. Despite recent progress, many methods still rely on multi-stage training, resulting in high computational costs and long runtimes, or transform visual input into text to leverage pretrained text-to-audio models, sacrificing fine-grained temporal cues. To overcome these limitations, we propose Flowley, an end-to-end, single-stage training architecture that produces soundtracks by combining visual features with textual prompts. Crucially, we introduce Progressive Soft-masked Cross-Attention, which embeds audio-visual synchronization directly within its attention mechanism, adding zero additional computational cost compared to standard attention layers. We further observe that existing V2A benchmarks lack sound-oriented descriptive captions, which can potentially degrade the quality of the synthesized audio. To remedy this, we propose SoundCap, a plug-and-play pipeline for creating detailed, sound-aware captions that guide the model. Remarkably, without integrating any pretrained audio-visual alignment modules, Flowley achieves state-of-the-art performance on VGGSound across multiple metrics. Moreover, by incorporating SoundCap, we further exceed the performance of the strongest existing close-sourced methods in terms of audio quality in the zero-shot setting.
Primary: FPT Software AI Center
All Institutions: FPT Software AI Center, NVIDIA Corporation
[Flowley introduces a novel end-to-end flow-based architecture with Progressive Soft-masked Cross-Attention for precise video-to-audio generation, achieving state-of-the-art performance on VGGSound and outperforming larger closed-source models in zero-shot settings when augmented with its SoundCap captioning pipeline.] This paper presents a significant advancement in V2A generation by eliminating the need for multi-stage training and external alignment modules, offering a more efficient and effective solution for synthesizing temporally synchronized and semantically consistent audio from video.
The paper proposes Flowley, an end-to-end flow-based architecture for Video-to-Audio (V2A) generation. The core technical contribution is the Progressive Soft-masked Cross-Attention (PSCA) mechanism, which embeds temporal alignment directly into the attention layers of a DiT-like backbone. This avoids the need for external synchronization modules or multi-stage training pipelines common in prior works (e.g., MMAudio, VinTAGe). The method combines visual features (CLIP), textual prompts (FLAN-T5), and audio latents in a multi-stream to single-stream architecture. Additionally, the authors introduce SoundCap, a pipeline using AV-LLMs to generate detailed, sound-aware captions to address the lack of rich textual annotations in V2A datasets. The approach is technically sound, leveraging recent advances in flow matching and multimodal transformers. The PSCA mechanism is a clever, low-cost modification to standard cross-attention that explicitly models the temporal relationship between video frames and audio tokens.
The evaluation is comprehensive, covering distribution matching (KAD, FAD), audio quality (IS), semantic alignment (IB-Score, LB-Score), and synchronization (Align Acc). Flowley achieves state-of-the-art results on the VGGSound benchmark, outperforming larger models like MMAudio and FoleyCrafter in most metrics. The inclusion of SoundCap further boosts performance, particularly in zero-shot settings on the Movie Gen Audio benchmark, where it surpasses the closed-source Movie Gen Audio model in quality despite being significantly smaller and trained on less data. The subjective human evaluation supports the objective metrics, showing a preference for Flowley's synchronization and quality. The ablation studies effectively validate the contributions of PSCA, the dual cross-attention, and the SoundCap pipeline.
The paper provides detailed implementation details, including architecture dimensions, hyperparameters for PSCA (window sizes, fade zones), training schedules, and evaluation metrics. The use of standard datasets (VGGSound) and established baselines facilitates comparison. The code link is provided. The description of the SoundCap pipeline is sufficiently detailed for reproduction, assuming access to the specified AV-LLMs and VLMs.
The paper acknowledges that while Flowley excels in quality and distribution matching, it slightly lags behind some baselines in the Align Acc metric, although human preference favors its synchronization. This suggests a potential discrepancy between the automated Align Acc metric and human perception of temporal alignment. The reliance on AV-LLMs for SoundCap introduces a dependency on large, potentially expensive models for data generation, although the final VLM is lightweight. The method's performance on highly complex scenes with multiple simultaneous sound sources or off-screen sounds might be challenging, as indicated by the noise handling strategies in SoundCap.
This work contributes to the field of generative AI for multimedia, specifically in creating realistic and synchronized audio-visual experiences. It has applications in film production (Foley), gaming, virtual reality, and accessibility. The SoundCap pipeline offers a generalizable solution for improving text-audio grounding in other multimodal tasks. The open-source nature of the work (implied by the project page) encourages further research in efficient, end-to-end V2A generation. [Flowley introduces a novel end-to-end flow-based architecture with Progressive Soft-masked Cross-Attention for precise video-to-audio generation, achieving state-of-the-art performance on VGGSound and outperforming larger closed-source models in zero-shot settings when augmented with its SoundCap captioning pipeline.] This paper presents a significant advancement in V2A generation by eliminating the need for multi-stage training and external alignment modules, offering a more efficient and effective solution for synthesizing temporally synchronized and semantically consistent audio from video.
Voice anonymisation aims to protect speaker identity. Currently, its empirical privacy evaluation heavily relies on the Equal Error Rate (EER). Originally designed for biometric verification, EER aggregates scores globally, implicitly assuming an attacker is only trying to verify if two specific voice samples match (a 1-to-1 comparison). This introduces a threat model mismatch with real-world database linkage attacks, where an attacker searches across a fixed set of N enrolled identities (a 1-to-N closed-set search), allowing global averages to obscure localised privacy failures. While recent 1-to-N metrics address this aggregation issue, they abstract away the magnitude of the biometric evidence. In this paper, we propose a modular, information-theoretic evaluation framework explicitly designed for the 1-to-N linkage threat model. Within this framework, our core metric, Local Information Disclosure (LID), quantifies the exact privacy loss of a single trial utterance in bits by calibrating its raw similarity scores into the attacker's posterior confidence for each enrolled identity. Evaluating top-performing systems from the VoicePrivacy 2024 Challenge reveals that systems exhibiting near-perfect EERs (48 %) can still suffer from localised vulnerabilities with worst-case disclosures reaching 1 bit per trial utterance (effectively doubling the attacker's success rate over a random guess). We demonstrate that adopting localised privacy metrics is essential for capturing worst-case risks and aligning with strict privacy regulations.
Primary: Aalto University
All Institutions: Aalto University, Inria, Nokia, Macquarie University
The paper introduces a novel information-theoretic metric, Local Information Disclosure (LID), to evaluate voice anonymisation systems under 1-to-N linkage threats, revealing that systems with good global privacy metrics (EER) can still suffer from severe local vulnerabilities, thereby providing a more accurate and regulation-aligned assessment of privacy risks.
The paper proposes a rigorous, information-theoretic evaluation framework for voice anonymisation, specifically targeting the mismatch between standard 1-to-1 verification metrics (like EER) and real-world 1-to-N linkage threats. The core contribution is the Local Information Disclosure (LID) metric, which quantifies privacy loss in bits by calibrating raw similarity scores into posterior probabilities using logistic regression on row-normalized scores. The methodology is mathematically sound, leveraging Pointwise Mutual Information (PMI) to measure the reduction in attacker uncertainty. The framework is modular, allowing for various global aggregations (ALID, PDR, NDR, LID_max) that provide a more granular view of system vulnerabilities than global averages. The approach correctly identifies that global metrics can obscure local failures where specific utterances are highly linkable despite good average performance.
The authors evaluate their framework against systems from the VoicePrivacy 2024 Challenge (VPC 2024), including baselines and top submissions. They use the standard LibriSpeech dev/eval splits. The experiments demonstrate a critical finding: systems with near-perfect EERs (e.g., 0.48, indicating high privacy in 1-to-1 terms) can still exhibit worst-case local disclosures of ~1 bit, effectively doubling the attacker's success rate over random chance for specific trials. The results are presented clearly through CCDFs and tables comparing EER, C_llr, and the new LID-based metrics. The analysis of the calibration weight $w$ provides additional insight into the discriminative power of the anonymised embeddings. The empirical evidence strongly supports the claim that 1-to-1 metrics are insufficient for assessing 1-to-N privacy risks.
The paper provides detailed mathematical formulations for the LID metric, including the row-wise z-normalization and logistic calibration steps. It specifies the datasets (LibriSpeech) and the attacker model (semi-informed ECAPA-TDNN). However, the specific implementation details of the calibration (e.g., exact hyperparameters for the logistic regression, handling of edge cases) are somewhat abstract. The code is not explicitly linked in the provided text, though the VPC 2024 data is public. Reproducibility is moderate; a researcher could implement the method, but might need to infer some implementation details.
The primary limitation is the reliance on logistic regression for score calibration, which assumes a specific distribution of scores (normality after z-normalization). The authors acknowledge this and suggest it may not hold for all anonymisation methods. Additionally, the framework is evaluated only on the semi-informed threat model; while this is the most realistic, the framework's behavior under ignorant or lazy-informed models is not explored. The evaluation is limited to voice data; while the framework is generalizable, its applicability to other modalities (e.g., face, fingerprint) is not demonstrated. The "random" baseline comparison is useful but simplistic; a more nuanced baseline might involve a dummy classifier with similar score distributions but no identity information.
This work has significant implications for the deployment of voice anonymisation systems in privacy-sensitive applications (e.g., healthcare, legal, customer service). By providing a metric that aligns with regulatory concerns about "singling out" and "linkability" (e.g., GDPR), it offers a more robust tool for auditors and developers. It shifts the focus from average performance to worst-case risks, which is crucial for compliance. The information-theoretic framing also contributes to the broader field of privacy-preserving machine learning by offering a concrete way to quantify information leakage in biometric systems. The paper introduces a novel information-theoretic metric, Local Information Disclosure (LID), to evaluate voice anonymisation systems under 1-to-N linkage threats, revealing that systems with good global privacy metrics (EER) can still suffer from severe local vulnerabilities, thereby providing a more accurate and regulation-aligned assessment of privacy risks.
Few-step diffusion and flow-matching text-to-speech (TTS) models are usually trained with local objectives, such as conditional flow matching, reconstruction, and stop prediction. These losses provide stable optimization, but they never ask whether sampled speech follows the distribution of high-quality speech. We propose Speech Representation Fr'echet Distance loss (SR-FD), a training-time distributional regularizer for tokenizer-free flow-matching autoregressive TTS. During fine-tuning, the model synthesizes speech with the same few-step sampler used at deployment, and SR-FD matches the mean and covariance of frozen Whisper and CTC features of this speech to reference statistics computed offline from three complementary content targets. The loss requires no discriminator and no inference-time computation. On Seed-TTS English, four-step SR-FD fine-tuning reduces WER from the original four-step VoxCPM2 baseline's 2.2279% to 1.4147%, a 36.5% relative reduction, and also surpasses the original ten-step baseline at 1.7366%; both gains are significant under an utterance-level paired bootstrap. Speaker similarity and objective quality proxies are preserved at the ten-step level, and an error analysis shows the gain comes from content substitutions across all prompt lengths. SR-FD is thus an intelligibility-improving distributional regularizer for few-step TTS.
Primary: National Taiwan University
All Institutions: National Taiwan University, Amazon AGI
[One sentence main contribution]. This paper introduces SR-FD, a distributional regularizer that aligns the statistical moments of few-step generated speech representations with reference targets, significantly improving intelligibility in low-step flow-matching TTS. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a well-motivated and technically robust solution to the problem of distributional drift in few-step continuous TTS. By formulating a Fréchet Distance loss on frozen speech representations, the authors successfully bridge the gap between local training objectives and global output distribution quality. The experimental results are compelling, showing substantial WER improvements while maintaining perceptual quality. The method is complementary to existing distillation techniques and offers a practical path to deploying real-time, high-fidelity TTS systems. The rigorous evaluation, including statistical testing and listening tests, adds credibility to the claims.
The paper proposes Speech Representation Fréchet Distance loss (SR-FD), a training-time regularizer for few-step flow-matching Text-to-Speech (TTS) models. The core innovation lies in addressing the distributional drift that occurs when compressing continuous flow-matching samplers to few steps (e.g., 4 steps). Standard local losses (conditional flow matching) do not guarantee that the sampled output distribution matches the high-quality reference distribution. SR-FD computes the Fréchet distance between the mean and covariance of frozen speech representations (Whisper semantic features and wav2vec 2.0 CTC posteriors) of generated samples and offline reference statistics. This approach is discriminator-free and adds no inference-time overhead. The methodology is technically sound, leveraging the stability of moment matching while avoiding the instability of adversarial training. The use of a feature queue to estimate covariances efficiently during training is a practical engineering contribution.
The evaluation is conducted on the Seed-TTS English benchmark using the VoxCPM2 backbone. The results are significant: SR-FD fine-tuning reduces Word Error Rate (WER) by 36.5% relative to the 4-step baseline, even outperforming the 10-step baseline. The authors provide rigorous statistical validation using utterance-level paired bootstrapping. They also include a blinded listening test (TOST) confirming that the quality and speaker similarity are perceptually indistinguishable from the 10-step baseline. The ablation study effectively demonstrates the contribution of each of the three reference targets (Whisper anchor, CTC teacher, CTC real-speech). The error analysis correctly identifies that gains come from reduced content substitutions.
The paper provides extensive implementation details, including model architecture (VoxCPM2), optimization hyperparameters (LoRA rank, learning rate schedule), and specific sampler settings (guidance, temperature). The reference statistics computation is described in detail, including the handling of rank-deficient covariance matrices. The use of standard open-source models (Whisper, wav2vec 2.0) and datasets (LibriTTS, Seed-TTS) enhances reproducibility. The code is not explicitly linked in the text provided, but the methodological description is sufficient for implementation.
The primary limitation is that SR-FD is evaluated on a single backbone model (VoxCPM2) and a single language (English). Its generalizability to other flow-matching architectures (e.g., F5-TTS, ARCHI-TTS) or multilingual settings remains unproven. The method relies on the quality of the frozen encoders; if the encoders have biases, the regularizer might propagate them. Additionally, the raw Fréchet Distance score was found to be a poor diagnostic for WER, suggesting that the loss landscape is complex and not directly correlated with the final metric in a simple way. The method also requires offline computation of reference statistics, which adds a pre-processing step.
This work contributes to the field of efficient speech synthesis by enabling high-quality, low-latency TTS systems without sacrificing intelligibility. By decoupling training stability from inference-time distributional fidelity, it offers a new paradigm for regularizing generative models. The potential for improving accessibility technologies through more robust and efficient TTS systems is significant. [One sentence main contribution]. This paper introduces SR-FD, a distributional regularizer that aligns the statistical moments of few-step generated speech representations with reference targets, significantly improving intelligibility in low-step flow-matching TTS. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a well-motivated and technically robust solution to the problem of distributional drift in few-step continuous TTS. By formulating a Fréchet Distance loss on frozen speech representations, the authors successfully bridge the gap between local training objectives and global output distribution quality. The experimental results are compelling, showing substantial WER improvements while maintaining perceptual quality. The method is complementary to existing distillation techniques and offers a practical path to deploying real-time, high-fidelity TTS systems. The rigorous evaluation, including statistical testing and listening tests, adds credibility to the claims.
Speaker embeddings, or x-vectors, are widely used to represent speaker identity and speaker-related attributes, but existing embedding extractors are typically descriptive rather than generative: they map an observed speech segment to an x-vector, which is then used for downstream applications. We introduce ProPS, Prompted Profile Synthesis, a framework for generating distributions of speaker embeddings conditioned on natural language prompts such as "a thirties male speaker with an Indian accent". ProPS converts human-written profile descriptions into sentence embeddings and uses a mixture density network trained on a large-scale dataset to predict a Gaussian mixture model in the x-vector space. The model is trained by maximizing the likelihood that real speaker embeddings match the requested profile, and its generated distributions are evaluated by negative log-likelihood on held-out x-vectors and by attribute classification accuracies on sampled synthetic x-vectors. Experiments show that ProPS produces profile-conditioned distributions and generates x-vectors that preserve requested speaker attributes such as age, gender, accent, and prosodic characteristics. This design enables controllable speaker-profile synthesis for speech generation systems like Text-To-Speech (TTS) or Voice Conversion (VC) while anchoring generated distributions in observed speaker-embedding structure.
Primary: Johns Hopkins University
All Institutions: Johns Hopkins University
The paper introduces ProPS, a novel framework for natural language-conditioned generation of speaker embedding distributions using Mixture Density Networks initialized from profile-specific GMMs, enabling controllable and diverse synthetic speaker creation for downstream speech applications.
The paper proposes ProPS, a framework for generating speaker embedding distributions conditioned on natural language prompts. The core methodology involves using a Sentence-BERT encoder to map text prompts to embeddings, which are then fed into a Mixture Density Network (MDN). The MDN predicts the parameters (weights, means, diagonal covariances) of a Gaussian Mixture Model in the x-vector space. A key technical contribution is the initialization strategy: instead of random initialization, the authors pre-compute GMMs for each speaker profile in the dataset (CapSpeech) to create a "component bank," which initializes the MDN's Gaussian parameters. The training occurs in two stages: pre-training the MLP to map GPT-generated descriptions to the correct profile GMMs, and fine-tuning end-to-end with human-written descriptions to align with real x-vector distributions. This approach is technically sound and leverages established components (SBERT, MDN, ECAPA-TDNN) in a novel configuration for speaker profile synthesis.
The experiments are conducted on the CapSpeech dataset, which provides structured metadata and human-written descriptions. The evaluation metrics include Negative Log-Likelihood (NLL) on held-out x-vectors and attribute classification accuracy on sampled synthetic x-vectors. The ablation study effectively demonstrates the contribution of each training stage. Results show that ProPS successfully preserves demographic attributes (gender, accent, age) and some prosodic features, though prosodic control (pitch, pace, tone) is less reliable. The LDA visualizations provide qualitative support for the distributional match. The evaluation is rigorous within the scope of embedding-space generation, though it lacks downstream waveform generation tests which are mentioned as future work.
The paper provides a GitHub link for the code (anonymized version). The dataset (CapSpeech) is publicly available. The methodology is described in sufficient detail, including hyperparameters (learning rates, epochs, K=16 components) and model architectures (SBERT all-MiniLM-L6-v2, ECAPA-TDNN). The use of GPT-4 (referred to as gpt-5.4-mini-2026-03-17, likely a typo or future-dated placeholder in the anonymized submission) for generating synthetic descriptions is noted, which aids reproducibility of the training data generation.
The authors acknowledge several limitations: reliance on CapSpeech's label quality and coverage, potential dataset biases, and restriction to English data. The model struggles with fine-grained prosodic characteristics, which is partly inherent to the x-vector representation itself. The use of GPT-generated descriptions for pre-training introduces a potential distribution shift if the synthetic descriptions do not perfectly capture the semantic nuance of human descriptions, although the fine-tuning stage mitigates this.
ProPS enables controllable speaker synthesis for TTS and Voice Conversion systems without requiring reference audio, which has significant implications for accessibility, entertainment, and privacy (anonymization). By generating distributions rather than single points, it allows for diversity in generated voices, reducing the risk of generating only stereotypical or "average" voices. However, the ability to generate realistic speaker embeddings from text prompts also raises concerns about deepfake audio generation and impersonation, necessitating careful ethical guidelines and watermarking in downstream applications. The paper introduces ProPS, a novel framework for natural language-conditioned generation of speaker embedding distributions using Mixture Density Networks initialized from profile-specific GMMs, enabling controllable and diverse synthetic speaker creation for downstream speech applications.
End-to-end ASR models transcribe in a single pass, leaving no room for the decoder to revisit hard inputs. We propose LatentASR, a parameter-efficient method that adds continuous latent test-time scaling to a frozen ASR backbone. Two small trainable modules drive it: a Latent Adapter that iteratively refines a few latent prefix positions through bounded, stabilized updates, and a Value Head that predicts whether extra computation will help and halts the loop early. The Qwen3-ASR-0.6B backbone stays fully frozen, and we train only ~4M extra parameters. We activate this loop with a deliberately small, diverse 500-utterance training set. Under this minimal-data regime, standard adaptation methods all regress: full fine-tuning, LoRA, and prompt tuning each increase WER. LatentASR is the only tested method that reduces WER on both clean benchmarks (FLEURS -2.54% and VoxPopuli -0.47% relative). The reductions are concentrated on intrinsically hard inputs. On accented and code-switched speech (ASCEND), LatentASR achieves a 16.0% relative CER reduction. Across 30 FLEURS languages (23,049 utterances), the multilingual WER decreases uniformly across resource tiers, confirming that the adapter generalizes without overfitting. Dynamic halting preserves most of the clean-set reduction at a fraction of the compute, skipping roughly half of all utterances at the entry gate. Our results show that a small, carefully chosen activation set can switch on test-time scaling inside a frozen ASR model without corrupting the model itself, converting fixed per-utterance compute into input-dependent compute where it is most needed.
Primary: National Taiwan University
All Institutions: National Taiwan University, ASUS, National University of Singapore
LatentASR introduces a parameter-efficient, continuous latent test-time scaling mechanism for frozen ASR models, demonstrating that minimal-data activation can unlock significant robustness gains on challenging speech inputs without corrupting the backbone's zero-shot capabilities.
The paper proposes LatentASR, a method for continuous latent test-time scaling in frozen ASR backbones. The core innovation lies in the "Latent Adapter," which iteratively refines a small number of latent prefix positions using bounded, stabilized updates (L2-normalization, sigmoid gating, fixed embedding anchor). This design is technically sophisticated, addressing the critical issue of representation collapse when injecting continuous states into a frozen decoder. The inclusion of a "Value Head" for dynamic halting adds a layer of efficiency and utility prediction. The methodology is well-grounded in the literature of latent test-time scaling (e.g., Coconut, Quiet-STaR) but adapts it uniquely for the direct transcription task where no intermediate reasoning trajectory exists. The stabilization mechanisms are novel and rigorously ablated, showing that each component addresses a distinct failure mode.
The experimental evaluation is comprehensive and convincing. The authors demonstrate that under a minimal-data regime (500 utterances), standard adaptation methods (LoRA, prompt tuning) regress, whereas LatentASR improves performance. This "minimal-data activation" hypothesis is supported by ablation studies on activation set size. Results on clean benchmarks (FLEURS, VoxPopuli) show modest but consistent WER reductions. More significantly, the method shows substantial gains on challenging data: ASCEND (accented/code-switched) and noise-corrupted speech. The dynamic halting mechanism effectively allocates compute, skipping easy utterances. The multilingual evaluation confirms generalization. The use of CER for ASCEND and WER for others is appropriate. The statistical significance of the small WER gains on clean data is not explicitly tested, but the consistency across benchmarks and the large gains on hard data provide strong evidence.
The paper provides detailed hyperparameters, model architecture details, and training procedures. The use of a frozen backbone (Qwen3-ASR-0.6B) and specific training data sources (Common Voice, FLEURS, etc.) enhances reproducibility. The ablation studies are well-described. However, the exact composition of the 500-utterance training set is not fully public (only described as a mixture from specific sources with a seed), which might slightly hinder exact replication, though the general recipe is clear. The code is not linked, but the description is sufficient for a competent researcher to implement.
The primary limitation is the modest gain on clean, high-resource benchmarks (e.g., -2.54% relative on FLEURS). While statistically significant, the absolute WER reduction is small. The method's effectiveness is highly dependent on the "minimal-data" activation set; deviating from this size or composition can lead to regression. The method adds inference latency for the subset of utterances that pass the Value Head gate, although dynamic halting mitigates this. The approach is specific to ASR and may not generalize directly to other modalities without adaptation.
This work contributes to the field of efficient and robust ASR. By enabling test-time scaling on frozen models, it reduces the need for expensive full fine-tuning and preserves zero-shot generalization. This is particularly impactful for low-resource and challenging acoustic conditions (accents, noise, code-switching). The method promotes sustainable AI by reducing the carbon footprint associated with training large models for every downstream task. It also opens new avenues for applying latent test-time scaling to other sequence-to-sequence tasks where intermediate reasoning is not natural. LatentASR introduces a parameter-efficient, continuous latent test-time scaling mechanism for frozen ASR models, demonstrating that minimal-data activation can unlock significant robustness gains on challenging speech inputs without corrupting the backbone's zero-shot capabilities.
Modern automated audio captioning systems pair a frozen audio encoder with a large language model (LLM) via a trainable projector, incurring the encoder's inference cost and bottlenecking the model through its fixed acoustic features. We present CARD, an encoder-free audio captioning model that removes the encoder at inference: a 13.2M projector feeds a frozen LLM with merged LoRA adapters, while the teacher used to train it is discarded. CARD distills a pretrained audio teacher (CLAP-HTSAT) into the model, but rather than injecting it into the LLM alone, it routes the teacher's representations across components: perceptual stages to the projector and semantic stages to the LLM. This placement improves CIDEr-D by +12.18 over an LLM-only distilled model on AudioCaps and by +5.21 on Clotho, reaching 55.4 against a 66.4 encoder-kept upper bound with no encoder at inference, showing that where a teacher's knowledge is placed matters as much as its presence.
Primary: University of Essex
All Institutions: University of Essex, Institute for Analytics and Data Science
The paper presents CARD, a novel encoder-free audio captioning model that utilizes cross-component knowledge distillation to route teacher representations to appropriate student components, achieving competitive performance with encoder-based baselines while eliminating inference-time encoder costs.
The paper proposes CARD, an encoder-free audio captioning framework that eliminates the need for a dedicated audio encoder at inference time. The core methodological innovation is a "cross-component distillation" strategy. Instead of distilling teacher knowledge solely into the Large Language Model (LLM) or the projector, CARD routes the teacher's (CLAP-HTSAT) representations based on semantic abstraction: early perceptual stages ($t_0, t_1$) supervise the audio projector, while later semantic stages ($t_2, t_3$) supervise the LLM's LoRA adapters. This design choice is theoretically sound, aligning the level of feature abstraction with the functional role of each student component. The use of a two-phase training procedure (distillation followed by captioning fine-tuning) and the merging of LoRA adapters for efficient inference are well-structured technical decisions. However, the novelty is somewhat incremental, as the concept of distilling multimodal knowledge into LLMs without encoders (e.g., VoRA, AuRA) has been explored in vision and speech domains; the specific application to audio captioning with this particular component-aware routing is the primary differentiator.
The experimental evaluation is comprehensive and rigorous. The authors compare CARD against a strong encoder-based baseline (SLAM-AAC with CLAP) and several ablation variants of their own method (No Distill, LLM Distill, Proj_Full, Proj_Early). The results clearly demonstrate that the proposed cross-component distillation (CARD$^*$) significantly outperforms LLM-only distillation (+12.18 CIDEr-D on AudioCaps) and approaches the performance of the encoder-kept baseline (55.4 vs 66.4). The ablation studies effectively isolate the contribution of projector supervision versus LLM supervision, providing strong empirical evidence for the hypothesis that "where a teacher's knowledge is placed matters." The use of standard benchmarks (AudioCaps, Clotho) and metrics (CIDEr-D, SPIDEr, etc.) ensures comparability with existing literature. The inclusion of a "No Distill" baseline highlights the critical role of distillation in encoder-free models.
The paper provides sufficient implementation details for reproduction. It specifies the backbone model (Qwen3-4B), the teacher model (CLAP-HTSAT), the projector architecture (strided 1D convolutions), and the distillation loss formulation (cosine similarity). Training hyperparameters such as learning rate, batch size, LoRA rank, and optimization steps are clearly listed. The two-phase training protocol is well-defined. The only potential ambiguity lies in the exact interleaving strategy for text-only instruction data during Phase 1, but the general approach is clear. The code is not provided, which is a minor hindrance but the method is described with enough precision to be implemented.
A significant limitation is the performance gap between the encoder-free CARD model (55.4 CIDEr-D) and the encoder-based baseline (66.4 CIDEr-D). While the paper argues that removing the encoder is beneficial for efficiency, the ~11-point drop in performance is substantial. The paper does not extensively analyze the specific types of errors or the scenarios where the encoder-free model fails most severely. Additionally, the reliance on a specific teacher (CLAP) means the distillation is bounded by the teacher's capabilities and biases. The evaluation is limited to two standard datasets; performance on more diverse or noisy real-world audio might differ. The "Qwen3-4B" reference might be a typo for Qwen2-1.5B or Qwen2-7B or a very new model, which could cause confusion regarding the exact baseline capabilities.
This work contributes to the trend of efficient, integrated multimodal models that reduce inference latency and memory footprint by removing dedicated encoders. This has practical implications for deploying audio understanding systems on edge devices or in low-latency applications. By demonstrating that knowledge distillation can effectively transfer acoustic understanding into LLMs, it opens avenues for similar encoder-free architectures in other modalities. However, the reduction in performance compared to encoder-based systems suggests that for high-accuracy applications, the trade-off may not yet be favorable. The paper presents CARD, a novel encoder-free audio captioning model that utilizes cross-component knowledge distillation to route teacher representations to appropriate student components, achieving competitive performance with encoder-based baselines while eliminating inference-time encoder costs.
Speaker embeddings aggregate frame-level acoustic features into compact representations for speaker recognition. Recent uncertainty-aware speaker modeling approaches further characterize the reliability of speaker embeddings by estimating their associated uncertainty. However, existing methods often suffer from inaccurate uncertainty estimation and uncertainty miscalibration under domain shifts. To address these challenges, we propose a robust uncertainty modeling framework from both estimation and adaptation perspectives. Specifically, we introduce an Inter- and Intra-Speaker-Aware Uncertainty Softmax that incorporates both inter-speaker separability and intra-speaker variability into uncertainty learning, enabling uncertainty estimates to better capture the reliability of speaker embeddings. Furthermore, we propose an Uncertainty-Calibrated Domain Adaptation (UCDA) framework to mitigate uncertainty miscalibration caused by domain mismatch. Extensive experiments on both in-domain and cross-domain benchmarks demonstrate that the proposed approach consistently improves uncertainty reliability and speaker recognition robustness.
Primary: The Hong Kong Polytechnic University
All Institutions: The Hong Kong Polytechnic University, The University of Melbourne
The paper presents a robust uncertainty-aware speaker modeling framework that integrates inter- and intra-speaker variability into uncertainty learning and employs a novel domain adaptation strategy to mitigate uncertainty miscalibration under domain shifts, demonstrating consistent improvements in speaker verification performance on standard benchmarks.
The paper proposes two main contributions: an Inter- and Intra-Speaker-Aware Uncertainty Softmax (IISA-Uncertainty) and an Uncertainty-Calibrated Domain Adaptation (UCDA) framework. The IISA-Uncertainty method modifies the uncertainty-aware scaling factor in the loss function to incorporate both inter-speaker separability (hardness) and intra-speaker compactness. This is a logical extension of previous uncertainty-aware softmax losses (like UAAM-Softmax), aiming to provide a more nuanced supervision signal for uncertainty estimation. The UCDA framework addresses domain shift by aligning the distribution of target-domain uncertainty estimates with a source-domain prior using a negative log-likelihood objective, freezing most model parameters to preserve speaker-discriminative features. While the theoretical motivation is sound and addresses a real gap (uncertainty miscalibration under domain shift), the novelty is incremental. The concept of using hardness-aware scaling is not new in metric learning, and applying it to uncertainty is a straightforward adaptation. The domain adaptation strategy is also a standard application of distribution alignment, albeit focused on the uncertainty space rather than the embedding space.
The experiments are conducted on standard speaker verification benchmarks: VoxCeleb1 (in-domain) and CNCeleb (cross-domain). The baseline comparisons include strong systems like ECAPA-TDNN, CAM++, and Gemini SD-ResNet38, as well as prior uncertainty-aware methods like $U^3$-xi. The results show consistent improvements in Equal Error Rate (EER) and minimum Detection Cost Function (minDCF) across various settings. The paper provides a detailed ablation study on the proposed uncertainty scaling terms. However, the cross-domain results show some instability in minDCF with standard cosine scoring, though uncertainty-aware scoring mitigates this. The analysis of uncertainty reliability via correlation with discrimination quality is a strong point, providing empirical evidence that the learned uncertainty is meaningful. The use of a fixed source-domain prior for UCDA is a practical choice for unsupervised adaptation but limits the ability to adapt to significant distribution shifts if the prior is not representative.
The paper provides sufficient details regarding the training pipeline (using WeSpeaker toolkit), hyperparameters (scale, margin, epochs), and data augmentation (MUSAN, RIRs). The mathematical formulations are clear. However, the code is not explicitly linked, and some implementation details of the "detach" operations and specific hyperparameter tuning for the stabilizing constants are mentioned but not fully exposed in a way that guarantees out-of-the-box reproducibility without significant effort. The reliance on the WeSpeaker toolkit helps, but the specific modifications to the loss function and the UCDA module would need to be implemented from scratch or adapted from the cited $U^3$-xi code.
The proposed method relies on the assumption that intra-speaker variability can be effectively captured by the similarity to the most similar class prototype, which might be noisy in open-set scenarios. The UCDA framework's performance is dependent on the quality of the source-domain prior; if the source and target domains are too dissimilar, the alignment might not be optimal. The paper acknowledges that minDCF can degrade with standard scoring after UCDA, suggesting a trade-off that isn't fully resolved. Furthermore, the novelty is limited to incremental modifications of existing uncertainty-aware frameworks.
This work contributes to the robustness of speaker recognition systems, which is critical for real-world biometric authentication. By improving uncertainty estimation, it enables better decision-making in low-reliability scenarios (e.g., rejecting uncertain samples), potentially reducing false acceptance/rejection rates in adverse conditions. This has implications for security and user experience in voice-based systems. The paper presents a robust uncertainty-aware speaker modeling framework that integrates inter- and intra-speaker variability into uncertainty learning and employs a novel domain adaptation strategy to mitigate uncertainty miscalibration under domain shifts, demonstrating consistent improvements in speaker verification performance on standard benchmarks.
Large Audio-Language Models (LALMs) reason fluently about sound yet struggle to localize precisely when events occur, while classical Sound Event Detection attains frame-level precision only over a closed label set. At the intersection of these paradigms lies the task of Open-Vocabulary Audio Event Grounding: predicting all time intervals of a target sound event described by an arbitrary natural language query. While this task is crucial for real-world audio understanding and LALM adaptation, it is bottlenecked by data scarcity. Few large-scale resources provide open-vocabulary onset/offset supervision, and manual temporal annotation is prohibitively expensive. To address this, we introduce Auto-AEG, a scalable pipeline that constructs such supervision by automatic data construction and model fine-tuning. It pairs programmatically synthesized clips, which carry exact ground-truth intervals for supervised cold-start, with multi-model pseudo-labels on real-world audio that supply the reward signal for reinforcement learning. Training with this pipeline yields promising performance gains on both the DESED SED benchmark and AEGBench, an independent difficulty-stratified benchmark we release. Our results show that automatically constructed data, coupled with interval-aware reward function design, is an effective data-side route to expanding the temporal localization capability of LALMs.
Primary: Zhejiang University
All Institutions: Zhejiang University, Tsinghua University
This paper presents a novel and effective pipeline for automatically constructing open-vocabulary audio event grounding data, combining synthetic data for cold-start SFT with real-world pseudo-labels for RL fine-tuning, thereby significantly advancing the temporal localization capabilities of Large Audio-Language Models.
The paper proposes Auto-AEG, a pipeline for constructing open-vocabulary audio event grounding (AEG) data without manual annotation. The methodology is two-fold: Stage 1 uses programmatic synthesis of synthetic audio clips with exact ground-truth intervals to perform Supervised Fine-Tuning (SFT) on Large Audio-Language Models (LALMs). Stage 2 uses a multi-model pipeline (Gemini for labeling, PE A-Frame for localization, CLAP for label cleaning) to generate pseudo-labels on real-world audio, which are then used to fine-tune the model via Group Relative Policy Optimization (GRPO). The GRPO reward function is specifically designed to be interval-aware, combining F1-IoU, format rewards, and penalties for over-prediction. The approach is technically sound, leveraging the noise tolerance of RL to correct imperfect pseudo-labels, a strategy increasingly common in LLM alignment but novel in this specific audio grounding context. The separation of semantic identification (LALM) and temporal localization (frame-level model) in the annotation pipeline is a clever engineering choice to mitigate individual model weaknesses.
The authors evaluate their method on a newly introduced benchmark, AEGBench, which is difficulty-stratified and independently annotated. They also report results on the standard DESED SED benchmark. The results show significant improvements in mIoU and Event F1 over zero-shot baselines. The ablation studies effectively demonstrate the contribution of each stage: SFT provides a necessary cold-start, and GRPO further refines the temporal boundaries. The analysis of hard cases (e.g., long duration, polyphonic overlap) provides valuable insights into the current limitations of LALMs. The comparison with existing benchmarks highlights the unique contribution of AEGBench in terms of difficulty stratification and real-world relevance. The results on DESED confirm that the grounding capabilities transfer to closed-set SED tasks.
The paper provides detailed descriptions of the data construction pipeline, including the synthesis parameters, the multi-model annotation steps, and the GRPO training hyperparameters. The code for the synthesis and annotation pipeline is not explicitly linked in the text provided, but the methodology is described with sufficient detail for reproduction. The release of AEGBench is a significant contribution to reproducibility in this area. The use of open-source models (Qwen-Omni, PE A-Frame, CLAP) facilitates replication.
The synthetic data in Stage 1 suffers from a domain gap compared to real-world audio (room acoustics, source interaction). The pseudo-labels in Stage 2 are inherently noisy, although the RL component is designed to be robust to this. The performance on long-duration events is limited by the encoder window size of the underlying LALMs. The reliance on external models (Gemini, PE A-Frame) for annotation introduces potential failure modes if these models make errors, although the paper argues that the RL process mitigates this. The benchmark, while high-quality, is still limited in size (3,427 items) compared to some large-scale audio datasets.
This work addresses a critical bottleneck in audio understanding: the lack of large-scale, open-vocabulary temporal grounding data. By providing a scalable data construction pipeline and a new benchmark, it enables further research into LALMs for precise audio localization. This has implications for applications such as audio search, video editing, and assistive technologies for the visually impaired. The emphasis on open-vocabulary capabilities aligns with the broader trend towards more flexible and generalizable AI systems. This paper presents a novel and effective pipeline for automatically constructing open-vocabulary audio event grounding data, combining synthetic data for cold-start SFT with real-world pseudo-labels for RL fine-tuning, thereby significantly advancing the temporal localization capabilities of Large Audio-Language Models.
Autoregressive (AR) text-to-speech (TTS) models generate discrete speech tokens sequentially, which makes inference slow and can degrade robustness by propagating local errors and hallucinations. This limitation stems from their left-to-right AR commitment: each token must be determined before future speech-token context is available. However, such ordering is not an inherent requirement for TTS, as the full input text is available before synthesis. In this paper, we introduce DELTA-TTS, a lightweight LoRA-based adaptation framework that converts a pretrained AR TTS model into a discrete diffusion language model (dLLM) for confidence-ordered speech-token decoding. To better capture the local structure of speech, DELTA-TTS incorporates a convolution module that injects local acoustic context, together with a $1/t$-weighted training objective and a time-shifted inference schedule that defer low-confidence positions to later steps. Trained on only $585$ hours of LibriTTS, DELTA-TTS achieves a $\textbf{1.75}\%$ WER on Seed-TTS test-en, outperforming its AR backbone while generating tokens $\textbf{3.3}\times$ faster. Further analysis shows that DELTA-TTS produces sharper text--speech alignment, increases overall decoding confidence, and mitigates hallucinations observed in AR generation.
Primary: Sungkyunkwan University
All Institutions: Sungkyunkwan University, University of Seoul
DELTA-TTS effectively bridges the gap between autoregressive and diffusion-based speech synthesis by adapting a pretrained AR model into a discrete diffusion language model with speech-aware modifications, achieving state-of-the-art speed-quality trade-offs and robustness against hallucinations.
The paper proposes DELTA-TTS, a framework that converts a pretrained Autoregressive (AR) Text-to-Speech (TTS) model into a discrete Diffusion Language Model (dLLM). The core methodological contribution lies in adapting the AR-to-dLLM conversion technique, previously established for text, to the speech domain. Key innovations include: 1) Replacing causal attention with bidirectional attention to allow access to future context; 2) Introducing a Conformer-style convolution module to inject local acoustic inductive biases that bidirectional self-attention lacks; 3) Utilizing a $1/t$-weighted training objective and a time-shifted inference schedule to defer low-confidence token predictions to later steps. The approach is parameter-efficient, using LoRA adapters on top of a frozen CosyVoice3 backbone. The methodology is sound and addresses a known bottleneck in AR TTS (inference speed and error propagation) by leveraging the parallel generation capabilities of diffusion models.
The evaluation is comprehensive, comparing DELTA-TTS against a wide range of state-of-the-art AR and Non-Autoregressive (NAR) baselines, including Seed-TTS, CosyVoice series, FireRedTTS, and F5-TTS. The primary metric is Word Error Rate (WER) on Seed-TTS test-en and LibriSpeech-PC test-clean. DELTA-TTS achieves a WER of 1.75% on Seed-TTS, outperforming the AR backbone (CosyVoice3, 2.02%) and all listed NAR baselines. It also demonstrates superior Real-Time Factor (RTF), achieving a 3.3x speedup. The paper includes ablation studies validating the necessity of the convolution module and the time-shifted schedule. Furthermore, it provides qualitative analysis showing sharper text-speech alignment and mitigation of hallucinations (catastrophic failures) observed in the AR baseline. The use of both objective metrics (WER, SIM, UTMOS) and subjective evaluations (CMOS, SMOS) strengthens the claims.
The paper provides sufficient implementation details for reproduction. It specifies the backbone (CosyVoice3, Qwen2-0.5B), the adapter size (94M parameters), training data (585 hours LibriTTS), optimizer settings (AdamW, lr $10^{-4}$), and specific architectural choices (kernel size 31 for convolution). The training data size is small and publicly available, facilitating reproducibility. However, the code is not explicitly linked in the provided text, and the "rule-based length" prediction method is described but might require careful implementation to match results. The reliance on specific baselines' checkpoints (e.g., CosyVoice3) is noted, which is standard but requires access to those resources.
The primary limitation is the training data scale (585 hours), which is significantly smaller than the multi-hundred-thousand-hour datasets used by many baselines. While this highlights the efficiency of the adaptation, it may limit the model's generalization to out-of-distribution speakers or languages compared to larger models. The paper acknowledges this and notes the model is English-only. Additionally, the target length prediction is rule-based, which is a known weakness in dLLM-based TTS systems. The speedup is measured on token generation only; end-to-end latency including the flow-matching decoder is not fully detailed, though the decoder is frozen and shared.
This work contributes to the democratization of high-quality TTS by enabling faster inference and potentially lower training costs through efficient adaptation. It reduces the computational barrier for deploying TTS systems. The mitigation of hallucinations improves reliability. However, like all generative audio technologies, it carries risks related to misuse for deepfakes, though the paper frames it as academic research. The ability to generate speech faster and more robustly has positive implications for assistive technologies and interactive voice assistants. DELTA-TTS effectively bridges the gap between autoregressive and diffusion-based speech synthesis by adapting a pretrained AR model into a discrete diffusion language model with speech-aware modifications, achieving state-of-the-art speed-quality trade-offs and robustness against hallucinations.
Accent normalization (AN) seeks to convert non-native (L2) accented speech into standard (L1) speech while preserving speaker identity. The current techniques either require naturally recorded parallel L1-L2 speech for training, or suffer from quality degradation when supervised by synthesized targets. In this paper, we present TokAN, a token-based accent normalization framework that operates on self-supervised discrete speech tokens extracted from a L1-L2 jointly trained vector-quantization (VQ) tokenizer, without the need of synthetic supervisory speech. An autoregressive encoder-decoder model performs token-to-token conversion, translating L2-accented token sequences into the tokens of standard voice. We also introduce reinforcement learning (RL) post-training based on Group Relative Policy Optimization (GRPO), using word error rate and accent classifier confidence as complementary rewards. A non-autoregressive flow-matching synthesizer recovers the Mel-spectrogram from the converted tokens, conditioned on the source speaker embedding. We also develop a flow-matching duration predictor that supports total-duration-aware synthesis, making TokAN applicable to duration-critical tasks such as voice dubbing and live casting. Experiments on seven English accents demonstrate that TokAN reduced the word error rate from 12.40% to 9.89% after supervised fine-tuning, and further to 9.23% after RL post-training, consistently outperforming frame-to-frame, direct flow-matching, and prompt-based token-conversion baselines in terms of accent reduction and intelligibility.
Primary: Nanjing University
All Institutions: Nanjing University, The Chinese University of Hong Kong, Shenzhen (CUHKSZ), Tencent Ethereal Audio Lab, Shanghai Jiao Tong University, Shenzhen Loop Area Institute, Shenzhen Research Institute of Big Data
TokAN introduces a novel token-based accent normalization framework that leverages a jointly trained VQ tokenizer and RL post-training with GRPO to achieve superior accent reduction and intelligibility without requiring naturally recorded parallel speech data.
The paper proposes TokAN, a token-based accent normalization framework that addresses the data scarcity and quality degradation issues in existing methods. The core methodological contributions are threefold: (1) A jointly trained Vector Quantization (VQ) tokenizer that co-optimizes phonetic content representation with speech synthesis and ASR supervision, replacing static K-Means clustering. This ensures the discrete tokens are more discriminative for accent-related phonetic features. (2) An autoregressive encoder-decoder conversion model using Rotary Positional Encodings (RoPE) and a self-attention-only decoder, trained with BART-style pre-training and supervised fine-tuning on semi-synthetic pairs. (3) A novel Reinforcement Learning (RL) post-training stage using Group Relative Policy Optimization (GRPO). This is a significant technical innovation in the speech domain, as it directly optimizes task-level objectives (Word Error Rate and Accent Classifier Confidence) rather than relying solely on token-level cross-entropy loss. The use of GRPO eliminates the need for a value network, making it computationally efficient for sequence generation. The system also includes a flow-matching synthesizer and a duration predictor for total-duration-aware synthesis, enhancing applicability to dubbing.
The evaluation is comprehensive, covering seven English accents from the L2-ARCTIC dataset. The authors compare TokAN against strong baselines: FramAN (frame-to-frame), CosyAccent (direct flow-matching), and VEVO (token-based). Results show TokAN significantly reduces Word Error Rate (WER) from 12.40% (SFT only) to 9.23% (after RL), outperforming all baselines in intelligibility and accent reduction (L1-Prob). Subjective evaluations (MUSHRA) confirm higher naturalness and accentedness reduction. The ablation studies effectively isolate the contributions of the joint tokenizer training, BART pre-training, SFT, and RL post-training. The phonemic distribution analysis provides deep insight into how the model corrects specific accent-related errors (e.g., dental-alveolar fricative confusion for Chinese speakers). The inclusion of both duration-free and source-length modes adds practical value.
The paper provides extensive implementation details, including hyperparameters for all training stages, dataset descriptions (LibriTTS-R, Emilia-EN, L2-ARCTIC, GLOBE), and model architectures. The code and demo samples are publicly available on GitHub and a dedicated webpage, which significantly enhances reproducibility. The use of standard components (WavLM, HiFT, Whisper) and open-source RL libraries (TRL) further supports reproducibility.
The reliance on semi-synthetic data for SFT means the model's performance is still somewhat tied to the quality of the TTS system used to generate native targets. While RL mitigates this by optimizing on real data, the initial mapping is learned from synthetic pairs. Additionally, the speaker similarity is slightly lower than some baselines (like VEVO), indicating a trade-off between accent normalization and voice preservation that might be critical for some applications. The method assumes access to a pre-trained accent classifier and ASR model for rewards, which introduces potential bias if these models are not robust to the specific accents being normalized.
TokAN has significant potential for improving accessibility and communication for non-native speakers, aiding language learning, and enhancing the quality of automated dubbing and live translation services. By reducing the reliance on parallel L1-L2 data, it lowers the barrier to creating accent normalization systems for low-resource languages or less common accents. The use of RL for speech generation also opens new avenues for optimizing speech models against complex, non-differentiable metrics. TokAN introduces a novel token-based accent normalization framework that leverages a jointly trained VQ tokenizer and RL post-training with GRPO to achieve superior accent reduction and intelligibility without requiring naturally recorded parallel speech data.
Training automatic speech recognition (ASR) models for low-resource languages is challenging due to limited data and highly variable supervision quality. In particular, Pacific Indigenous speech corpora often exhibit heterogeneous acoustic conditions, transcript inconsistencies, and varying degrees of acoustic-text alignment reliability, making standard fine-tuning approaches sensitive to noisy or misleading supervision signals. In this work, we propose QuaSR, a simple yet effective weighting framework that combines data-side reliability with model-side learnability to improve ASR adaptation. Specifically, we estimate data reliability from acoustic, transcription, and alignment, while measuring learnability using training loss from the model. These two complementary signals are integrated into a unified sample utility score to produce training weights for the samples. We also evaluated across four Pacific Indigenous languages, which shows that the proposed utility scores reliably correlate with adaptation performance. Furthermore, QuaSR consistently improves ASR adaptation over standard fine-tuning and alternative data selection strategies, highlighting a new way to leverage difficulty scores for low-resource speech learning.
Primary: The University of Melbourne
All Institutions: The University of Melbourne, Wuhan University
QuaSR presents a robust, multi-dimensional quality-aware reweighting framework that effectively improves ASR adaptation for low-resource Pacific Indigenous languages by distinguishing between data reliability and model learnability, offering a scalable solution for noisy supervision in under-resourced domains.
The paper proposes QuaSR, a sample reweighting framework for Automatic Speech Recognition (ASR) adaptation in low-resource settings. The core methodology involves calculating a "sample utility" score by multiplying a data-side reliability term (derived from acoustic quality, transcript stability, and alignment reliability) with a model-side learnability term (derived from teacher-forced loss). This approach attempts to disentangle "hard but learnable" samples from "noisy/unreliable" samples, which is a critical distinction in low-resource domains where data quality is heterogeneous. The methodology is well-motivated and addresses a genuine gap in current PEFT (Parameter-Efficient Fine-Tuning) practices for ASR, which often treat all samples equally or rely solely on loss-based filtering. However, the novelty is moderate as sample reweighting based on quality heuristics is not entirely new in the broader ML community, though its specific application and combination for Pacific Indigenous languages is a novel contribution. The technical implementation details, particularly the alignment reliability metric, are somewhat opaque in the text (referencing undefined variables like 'm' and 'pho' in the raw text), suggesting a need for clearer formalization.
The evaluation is conducted on four Pacific Indigenous languages (Bislama, Nafsan, Lelepa, Nguna) using Whisper-small fine-tuned with LoRA. The results demonstrate that QuaSR consistently improves WER and CER compared to standard unweighted fine-tuning and hard filtering baselines. The gains are most significant for languages with lower data volume and higher quality heterogeneity (Lelepa, Nguna), which aligns with the paper's hypothesis. The analysis of why certain reliability signals (e.g., transcript-alignment vs. acoustic-alignment) work better for specific languages adds valuable empirical insight. However, the comparison is limited to Whisper-small and LoRA; it does not explore other foundation models or adaptation techniques. The ablation studies on the reliability components are helpful but could be more extensive. The claim of "largest gains" is supported by the data, but the absolute performance metrics remain relatively high, indicating the inherent difficulty of the task.
The paper provides sufficient detail on the experimental setup, including model architecture (Whisper-small, LoRA ranks, learning rates), dataset sources (PARADISEC), and the specific metrics used for reliability estimation (SQUIM/STOI, MMS forced alignment). The use of standard tools (SQUIM, MMS) enhances reproducibility. However, the raw text contains placeholders and undefined variables in the equations section (e.g., "what is pho here?", "m is just a symbol"), which suggests that the final published version must address these clarity issues. The code is not explicitly linked, which hinders immediate reproducibility.
The primary limitation is the dependency on external tools for reliability estimation (SQUIM, MMS), which may not be available or accurate for all low-resource languages. The method assumes that the "unweighted baseline" run is a good proxy for learnability, which might not hold if the baseline fails to converge. The paper does not address the computational overhead of computing these reliability scores, which could be significant for very large datasets. Furthermore, the "transcript stability" metric relies on heuristic counting of irregularities, which may not capture all forms of transcription error. The evaluation is restricted to four languages, limiting the generalizability of the findings to other low-resource language families.
This work has significant positive impact for the preservation and technological empowerment of Indigenous communities in the Pacific region. By improving ASR performance on under-resourced languages, it facilitates better access to technology and documentation for these communities. The framework also offers a generalizable approach for handling noisy, low-quality data in other domains beyond speech, such as text classification or computer vision, where data quality is a major bottleneck. The emphasis on data quality over quantity shifts the narrative in low-resource ML towards more sustainable and ethical data practices. QuaSR presents a robust, multi-dimensional quality-aware reweighting framework that effectively improves ASR adaptation for low-resource Pacific Indigenous languages by distinguishing between data reliability and model learnability, offering a scalable solution for noisy supervision in under-resourced domains.
Crossmodal correspondences between sound and taste are well established in psychology and neuroscience, but largely absent from content-based multimedia retrieval. We formalise taste-from-audio prediction as a content-based music information retrieval benchmark over a perceptually validated multi-source corpus, comparing ten frozen audio encoders from the four HEAR families under a shared multi-task regression head, with gated late-fusion as a configurable variant. In order to assess the effectiveness of the models, we compute absolute error and rank correlation. The strongest systems predict the five tastes within a macro RMSE of 0.134; on held-out real music their error is less than half a single rater's deviation from the consensus (RMSE 0.13 vs. 0.28), so the model tracks the group consensus more closely than an average human rater, and well below the previous state of the art baseline (0.219). On absolute error the encoders are statistically flat, with a single VGGish matching the best fusion, but gated late-fusion's advantage is confined to rank correlation (macro Pearson r 0.724 vs. 0.666). Operationalised as a content-based retrieval index, the predicted taste space ranks a 309-item pool far more faithfully than a CLAP-text baseline, which sits at chance; ridge probes and an audio-bandstop knockout read the strongest representations against documented sound-taste correspondences.
Primary: University of Padua
All Institutions: University of Padua
The paper presents a well-executed benchmark for taste-from-audio prediction, demonstrating that frozen audio encoders can predict taste intensities with accuracy exceeding average human raters, while providing valuable insights into the spectral features underlying crossmodal correspondences.
The paper proposes a systematic benchmark for predicting five basic taste intensities from audio embeddings. The methodology is straightforward: it freezes ten pre-trained audio encoders (spanning supervised, self-supervised, multimodal, and music-specific families) and trains a lightweight multi-task regression head (MLP) on top. A key methodological contribution is the "gated late-fusion" mechanism to combine complementary encoders. The approach is technically sound but not architecturally novel; it applies standard transfer learning and ensemble techniques to a new domain (crossmodal taste prediction). The use of masked MSE loss to handle partial annotations is a practical and necessary detail. The interpretability layer using ridge probes and bandstop knockouts is a strong methodological addition, linking model behavior to psychophysical literature.
The experimental setup is rigorous for the domain. The authors evaluate on a perceptually validated corpus, comparing against a previous SOTA. They provide a detailed breakdown of performance by taste and by data source (real vs. generated music). The comparison against human inter-rater agreement is a compelling metric, showing the model outperforms average human raters on absolute error for real music. The retrieval experiments, while limited by the small pool size (309 items), effectively demonstrate the utility of the predicted taste space for content-based retrieval, showing it outperforms text-based baselines. The statistical analysis includes bootstrap tests, adding credibility to the encoder rankings.
The paper provides significant detail for reproducibility. It specifies the encoders, the architecture of the head (hidden size, dropout, activation), training hyperparameters (AdamW, LR, batch size), and the fusion strategy. Crucially, it provides code and dataset links. The use of frozen encoders and cached embeddings makes the computational cost low and the setup easy to replicate. The five-seed averaging for reported metrics is a good practice for small datasets.
The primary limitation is the small dataset size (269 train, 40 test clips), which restricts the generalizability of the findings and the complexity of the models that can be effectively trained. The authors acknowledge this and frame the work as a benchmark rather than a scalable solution. The crossmodal mappings are largely based on Western psychophysics, potentially limiting cross-cultural applicability. The "spicy" label is noted as less constrained by prior evidence. The retrieval evaluation is limited to a small pool, which may not reflect performance on large-scale collections.
This work opens a new avenue for content-based music retrieval and recommendation by introducing "taste" as a semantic axis. It bridges the gap between MIR and psychophysics/neuroscience, providing a computational framework for studying crossmodal correspondences. The potential applications in sonic seasoning, multisensory design, and personalized audio experiences are significant. It also highlights the limitations of current text-based multimodal models (like CLAP-text) in capturing non-linguistic perceptual qualities. The paper presents a well-executed benchmark for taste-from-audio prediction, demonstrating that frozen audio encoders can predict taste intensities with accuracy exceeding average human raters, while providing valuable insights into the spectral features underlying crossmodal correspondences.
While Large Multimodal Models excel in comprehension, high-throughput inference engines lack native support for multimodal generation. This is severe in Speech Language Models, where generating multi-layered audio tokens via decoupled AR+NAR or synchronous Multi-Token Prediction (MTP) with delay-pattern interleaving conflicts with standard single-stream loops. We present a vLLM-based inference pipeline for unified speech understanding and generation. We extend autoregressive decoding to natively execute delay-pattern de-interleaving and coordinated multi-stream sampling, integrating an on-GPU acoustic decoder for end-to-end waveform synthesis. Crucially, we overcome the shared intuition that Classifier-Free Guidance (CFG) halves throughput. By co-scheduling paired conditional and unconditional requests within a continuous batch, our CFG implementation sustains 80% of non-CFG throughput, absorbing dual-request and logit merging overheads. We open-source our framework.
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University, Shanghai Jiao Tong University
This paper presents a significant systems contribution to the deployment of Speech Language Models by extending the vLLM inference engine to natively support multi-token audio generation and efficient Classifier-Free Guidance. By introducing paired request co-scheduling, the authors overcome the traditional throughput penalty of CFG, enabling high-fidelity audio synthesis at scale, which is a critical step for making unified audio understanding and generation models practically viable in real-world applications.
The paper addresses a critical infrastructure gap in Speech Language Models (SpeechLMs): the lack of high-throughput inference support for multi-token generation and Classifier-Free Guidance (CFG) in existing engines like vLLM. The proposed methodology involves extending the vLLM architecture to handle "delay-pattern" interleaving common in Residual Vector Quantization (RVQ) based audio codecs. The core technical innovation is the "Paired Request Co-Scheduling" mechanism for CFG. Instead of treating conditional and unconditional passes as separate requests (which fragments KV cache and prevents joint batching), the authors fuse them into a single scheduling unit. This allows the transformer backbone to process both branches in a single forward pass (sharing the KV cache computation for the shared prefix), significantly reducing overhead. The integration of an on-GPU acoustic decoder further unifies the pipeline. The approach is technically sound and leverages low-level engine optimizations (PagedAttention, continuous batching) effectively.
The evaluation is conducted on three representative SpeechLM architectures (Bagpiper, OpusLM, OpusLM-Dialogue) on a single H100 GPU. The authors demonstrate massive throughput gains (two orders of magnitude) over sequential PyTorch baselines. Crucially, they address the CFG bottleneck, showing that their co-scheduling method sustains 80% of non-CFG throughput, whereas naive implementations would suffer significant degradation. Numerical correctness is verified against reference implementations, showing strict alignment in FP32 and acceptable divergence in BF16 that does not impact end-to-end quality metrics (WER, UTMOS). The experiments are comprehensive for a systems paper, covering throughput, hardware utilization (MFU), and output quality.
The authors state they open-source their framework, which is a strong positive for reproducibility. The paper provides detailed descriptions of the architecture, including the phase state machine for mixed-modality handling and the specific logit merging logic for CFG. The experimental setup is clearly defined (H100, FlashAttention-3, specific models). However, the lack of a provided URL in the metadata requires verification of the actual repository link, though the claim of open-sourcing is present in the abstract.
The paper focuses heavily on the inference engine and does not propose new model architectures or training methods. The performance gains are specific to the vLLM architecture and RVQ-based SpeechLMs; generalization to other codec structures or non-autoregressive models is not discussed. The CFG optimization is specific to the "paired" request pattern and may not generalize to more complex guidance schemes (e.g., iterative refinement). The evaluation is limited to a single GPU (H100), so scalability to multi-GPU setups is not demonstrated.
This work significantly lowers the barrier to deploying high-fidelity SpeechLMs by making them computationally viable for real-time or high-concurrency applications. By enabling efficient CFG, it improves the quality of generated speech without prohibitive latency costs. This contributes to the broader field of multimodal AI by bridging the gap between model capability and system efficiency. This paper presents a significant systems contribution to the deployment of Speech Language Models by extending the vLLM inference engine to natively support multi-token audio generation and efficient Classifier-Free Guidance. By introducing paired request co-scheduling, the authors overcome the traditional throughput penalty of CFG, enabling high-fidelity audio synthesis at scale, which is a critical step for making unified audio understanding and generation models practically viable in real-world applications.
Humans can selectively attend to a target sound and estimate its direction in complex scenarios, whereas such selective localization remains challenging for current deep learning-based systems. Sound source localization (SSL) has achieved remarkable success with deep learning, yet most methods localize all active sources without selectivity. Conversely, target sound extraction (TSE) extracts sources using multimodal prompts but typically fails to preserve the multichannel spatial information required for accurate localization. To bridge this gap, we formulate the task of prompt-guided selective target sound localization and propose SelectTSL, an end-to-end architecture that localizes only the user-specified target in multi-source acoustic scenes. Specifically, we design a target-aware selective localization strategy that employs a Prompt-Guided Selective Attention Module (PGSA) to generate prompt-informed embeddings. These embeddings guide an inter-channel phase difference (IPD) enhancer to refine raw phase cues, fusing with target magnitudes to jointly estimate direction of arrival (DoA) and target-source cardinality, i.e., the number of target sound sources. This coupled design effectively focuses on the user-specified target spatial cues for selective localization and also handles time-varying numbers of target sources. Extensive experiments on both synthetic data and real-world recordings demonstrate that our proposed method consistently outperforms other baselines and exhibits robust generalization to real acoustic environments.
Primary: Fudan University
All Institutions: Fudan University
[One sentence main contribution]. This paper introduces SelectTSL, an end-to-end framework for prompt-guided selective target sound localization that jointly estimates DoA and source cardinality using prompt-informed spatial cue refinement. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The proposed method effectively addresses the challenge of localizing specific sound sources in complex multi-source environments by integrating semantic guidance from multimodal prompts with precise spatial cue enhancement. The novel combination of a PGSA module for target extraction and an IPD enhancer conditioned on extraction-informed embeddings allows for robust and selective localization, outperforming existing methods in both static and dynamic metrics. The joint prediction of DoA and cardinality provides a flexible solution for handling varying numbers of active sources, making it a significant advancement in the intersection of target sound extraction and sound source localization.
The paper proposes SelectTSL, a novel framework for prompt-guided selective target sound localization (SSL). The core innovation lies in decoupling semantic selection from spatial estimation. It employs a Prompt-Guided Selective Attention (PGSA) module that uses multimodal prompts (text or audio) to generate extraction-informed embeddings (EIEs). These EIEs condition an Inter-Channel Phase Difference (IPD) enhancer to refine spatial cues, which are then fused with target magnitudes for Direction of Arrival (DoA) estimation. A key technical contribution is the joint prediction of DoA posteriorgrams and source cardinality, allowing the system to handle time-varying numbers of active target sources without relying on fixed track slots. The architecture integrates a DPRNN-based extraction network with a specialized DoA estimator using depthwise-separable convolutions and temporal modeling (TCN/BiGRU). The approach effectively bridges the gap between Target Sound Extraction (TSE) and SSL, addressing the "cocktail party problem" by enabling users to specify *which* source to localize.
The authors conduct extensive experiments on a large-scale synthetic dataset (288.9 hours of training data) generated using dynamic Room Impulse Responses (RIRs) and real-world recordings from the TAU-SRIR dataset. They compare SelectTSL against a comprehensive taxonomy of baselines, including track-wise SSL methods (IPDNet, EINV2), SELD methods (SELDnet, SELDT), pure DoA methods (SRP-DNN, FN-SSL), and prompt-based localization (SEL). Results demonstrate that SelectTSL significantly outperforms all baselines, achieving an MAE of 0.98° and a MOTA of 91.57% on the synthetic test set, compared to the next best prompt-based method (SEL) which achieved 2.78° MAE and 16.25% MOTA. The paper includes detailed ablation studies validating the contributions of the PGSA module, IPD enhancement, cardinality head, and training scheme. The robustness to noise and reverberation is also evaluated.
The paper provides detailed implementation details, including STFT parameters, network architecture dimensions, loss function weights, and training hyperparameters (Adam optimizer, learning rate, early stopping). The dataset generation process is clearly described, including room dimensions, RIR simulation tools (GPURIR), and source/noise datasets. The authors state that the code and dataset will be released, which enhances reproducibility. The synthetic data generation protocol is sufficiently detailed for replication.
The primary limitation is the reliance on synthetic data for training and primary evaluation. While real-world evaluation is performed, it is limited to a subset of TAU-SRIR rooms with fixed source positions (simulated movement via static RIRs), which may not fully capture the complexities of real-time moving source localization with dynamic acoustic changes. The dual-microphone setup limits the spatial resolution and front-back ambiguity handling compared to larger arrays. The method's performance on highly complex, multi-source scenarios with more than two concurrent targets is not explicitly evaluated (cardinality head is limited to 0, 1, or 2).
This work has significant implications for human-computer interaction, particularly in smart speakers, hearing aids, and robotics, where selective auditory attention is crucial. By enabling users to specify a target sound via text or audio, the system can improve speech enhancement, far-field ASR, and spatial audio perception in noisy environments. It advances the field of audio AI by integrating semantic understanding with precise spatial sensing. [One sentence main contribution]. This paper introduces SelectTSL, an end-to-end framework for prompt-guided selective target sound localization that jointly estimates DoA and source cardinality using prompt-informed spatial cue refinement. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The proposed method effectively addresses the challenge of localizing specific sound sources in complex multi-source environments by integrating semantic guidance from multimodal prompts with precise spatial cue enhancement. The novel combination of a PGSA module for target extraction and an IPD enhancer conditioned on extraction-informed embeddings allows for robust and selective localization, outperforming existing methods in both static and dynamic metrics. The joint prediction of DoA and cardinality provides a flexible solution for handling varying numbers of active sources, making it a significant advancement in the intersection of target sound extraction and sound source localization.
We propose a lightweight multi-path alignment network (LMPAN) for on-device joint acoustic echo cancellation (AEC) and noise suppression (NS) in full-duplex spoken dialogue systems. To address hardware-induced distortions and dynamic acoustic conditions, we introduce three core innovations: (1) a multi-path alignment stage correcting temporal and energy mismatches across reference, linear AEC (LAEC) output, and microphone signals; (2) an attention-based mechanism that dynamically integrates enhanced LAEC and microphone features under varying acoustic scenarios; (3) a post-filtering module with a dynamic target generation strategy for downstream tasks (ASR, VAD). Furthermore, we adopt a two-stage training framework leveraging self-supervised learning representations to enhance perceptual quality. Experiments show that LMPAN, with only 480K parameters and 126 MACs, achieves performance comparable to the state-of-the-art lightweight model DeepVQE-S, while ensuring real-time inference capability.
Primary: TongYi AI Lab of Alibaba Group
All Institutions: Qwen Business Unit of Alibaba, TongYi AI Lab of Alibaba Group
[One sentence main contribution]. LMPAN introduces a lightweight multi-path alignment network with SSL-guided training to achieve robust joint AEC and NS on edge devices. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a well-engineered hybrid solution that addresses the critical challenge of hardware-induced misalignments in full-duplex systems. By combining explicit temporal/energy alignment with attention-based fusion and perceptual SSL losses, it achieves a favorable balance between model size, computational cost, and performance. While not theoretically groundbreaking, the systematic integration of these components and the rigorous evaluation on downstream tasks make it a valuable contribution to practical audio signal processing for on-device AI.
The paper proposes LMPAN, a lightweight network for joint Acoustic Echo Cancellation (AEC) and Noise Suppression (NS). The core methodological contribution lies in the explicit handling of temporal and energy misalignments between the reference, microphone, and Linear AEC (LAEC) outputs via a multi-path alignment stage. This is followed by an attention-based fusion module and a dynamic target adaptation strategy. The use of Self-Supervised Learning (SSL) representations (WavLM) in a two-stage training framework is a notable technical choice, aiming to preserve perceptual quality and semantic fidelity. However, the individual components (LAEC hybrid, attention fusion, SSL loss) are incremental combinations of existing techniques rather than fundamentally new algorithmic breakthroughs. The "multi-path alignment" is a practical engineering solution to a known problem (hardware latency) but lacks deep theoretical novelty in the context of modern end-to-end learning.
The experimental setup is rigorous, utilizing standard benchmarks (AEC Challenge 2023) and a self-collected real-world dataset from smartphones. The inclusion of downstream task evaluation (ASR, VAD, TIR) is a strong point, as it demonstrates the practical utility of the enhanced audio. The results show that LMPAN achieves performance comparable to DeepVQE-S with significantly fewer parameters (480K vs 820K) and lower computational cost (126M vs 315M MACs). The ablation studies effectively isolate the contributions of the alignment module, fusion module, and SSL training. The trade-off analysis of the Dynamic Target Adaptation (DTA) parameters provides valuable insight into balancing objective metrics (ERLE, PESQ) with subjective/perceptual quality (WER, MOS).
The paper provides sufficient implementation details, including STFT parameters, optimizer settings, and dataset augmentation strategies. The use of standard libraries (WavLM, Paraformer) aids reproducibility. However, the specific hyperparameters for the dynamic target generation and the exact architecture of the GTCRN-based refinement branches could be more explicitly defined. The self-collected dataset details are described but not publicly available, which limits independent verification on real-world hardware variations.
The primary limitation is the incremental nature of the contributions. The method relies heavily on a pre-computed LAEC stage, which, while common in hybrid systems, contradicts the trend towards fully end-to-end neural AEC. The performance gains, while consistent, are modest in absolute terms compared to the massive leaps seen in generative audio models. Furthermore, the "lightweight" claim is relative; 480K parameters is small for speech models but not negligible for extremely constrained edge devices. The reliance on SSL features adds inference overhead (though frozen) and complexity to the training pipeline.
This work contributes to the deployment of robust full-duplex spoken dialogue systems on edge devices, which is critical for the widespread adoption of always-on AI assistants. By improving the robustness of AEC and NS under hardware-induced distortions, it enhances the user experience and the reliability of downstream AI tasks like ASR and VAD. The emphasis on lightweight models supports sustainable AI by reducing computational resources required for real-time audio processing. [One sentence main contribution]. LMPAN introduces a lightweight multi-path alignment network with SSL-guided training to achieve robust joint AEC and NS on edge devices. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a well-engineered hybrid solution that addresses the critical challenge of hardware-induced misalignments in full-duplex systems. By combining explicit temporal/energy alignment with attention-based fusion and perceptual SSL losses, it achieves a favorable balance between model size, computational cost, and performance. While not theoretically groundbreaking, the systematic integration of these components and the rigorous evaluation on downstream tasks make it a valuable contribution to practical audio signal processing for on-device AI.
Real-time binaural speech enhancement is constrained by latency, computational cost, and inter-device communication, yet existing efficient solutions predominantly address single-channel settings. In this paper, we introduce RT-Tango, a real-time distributed binaural speech enhancement framework designed for streaming on resource-constrained platforms and specifically for hearing aids. RT-Tango relies on a two-stage distributed architecture combining perceptually motivated ERB feature compression, lightweight grouped recurrent mask estimation, and temporal sparsification to reduce computational cost. Stringent latency constraints are addressed by decoupling spectral resolution from algorithmic delay using an asymmetric STFT, together with causal recurrent inference and online estimation of spatial statistics. Experimental results show that RT-Tango achieves competitive speech enhancement while significantly reducing MACs operations and functioning at ultra-low latencies as low as 8 ms.
Primary: Université Paris-Saclay, CEA, List
All Institutions: Université Paris-Saclay, CEA, List; Université de Lorraine, CNRS, Inria, LORIA
This paper presents a practical and well-engineered solution for real-time binaural speech enhancement on hearing aids, effectively balancing computational efficiency, latency, and performance through a combination of perceptual feature compression, grouped recurrent modeling, and asymmetric streaming STFT.
The paper proposes RT-Tango, a distributed binaural speech enhancement framework specifically optimized for the stringent latency and computational constraints of hearing aids. The core technical contribution lies in the system-level integration of several efficiency-driven components: (1) Perceptually motivated ERB feature compression to reduce input dimensionality; (2) Grouped Recurrent Neural Networks (GRNNs) to parallelize processing and reduce the quadratic complexity of full-band recurrent layers; (3) Temporal sparsification (Fixed-Rate Skipping and learned gating) to reduce inference frequency; and (4) Asymmetric STFT configurations to decouple spectral resolution from algorithmic latency, enabling ultra-low latency (8 ms) streaming. The approach adapts the existing Tango architecture by replacing its CNN-based components with lightweight recurrent and grouped variants, focusing on hardware-aware efficiency rather than purely algorithmic novelty. The methodology is sound and directly addresses the trade-off between performance and resource consumption in embedded audio systems.
The experimental evaluation compares RT-Tango against the original Tango, a causal RNN variant (Tango-RNN), and a lightweight CNN baseline (GTCRN). Results are reported on a simulated binaural dataset using standard metrics (SI-SDR, SI-SIR, SI-SAR, STOI, PESQ). The paper demonstrates that RT-Tango achieves competitive performance with significantly lower computational cost (MACs/s) compared to GTCRN at similar frame rates, and lower cost than Tango-RNN while supporting higher frame rates. The ablation studies effectively isolate the contributions of grouping, temporal sparsification, and asymmetric STFT. However, the evaluation relies entirely on simulated data with measured room impulse responses, lacking real-world hardware deployment validation or subjective listening tests, which are critical for hearing aid applications. The comparison with GTCRN is somewhat uneven as GTCRN is a monaural/single-node model adapted to the setting, whereas RT-Tango leverages binaural spatial cues.
The paper provides sufficient detail regarding the architecture (group sizes, STFT parameters, hop sizes) and training setup (optimizer, loss function, dataset generation protocol). The use of standard datasets (LibriSpeech, BinauRec) and metrics aids reproducibility. However, the specific implementation details of the learned skip gating mechanism and the exact hyperparameters for the online SCM estimation convergence are described qualitatively or with limited quantitative precision, which may hinder exact replication. The code is not publicly available (URL: none), which is a significant barrier to full reproducibility.
The primary limitation is the reliance on simulated data, which may not capture the full complexity of real-world acoustic environments and hearing aid hardware imperfections (e.g., microphone mismatch, wind noise). The lack of subjective listening tests (MOS/MUSHRA) is a notable gap for a hearing aid paper, as objective metrics do not always correlate perfectly with user-perceived quality and comfort. Additionally, the "online" variant (RT-Tango-OS) shows a performance drop in SI-SDR/SI-SAR compared to the offline version, and the paper does not extensively discuss the impact of this degradation on user experience. The comparison with other distributed binaural methods is limited, as the field is still emerging.
This work has significant potential impact on the field of assistive listening devices. By demonstrating that high-quality binaural speech enhancement is feasible on low-power, resource-constrained hearing aids with ultra-low latency, it paves the way for more effective real-time noise reduction for hearing aid users. The techniques for efficient distributed processing and low-latency streaming are also relevant to other edge-Audio applications, such as smart speakers and wearable audio devices. The focus on perceptual metrics and interaural balance aligns well with the clinical requirements of hearing rehabilitation. This paper presents a practical and well-engineered solution for real-time binaural speech enhancement on hearing aids, effectively balancing computational efficiency, latency, and performance through a combination of perceptual feature compression, grouped recurrent modeling, and asymmetric streaming STFT.
Estimating a speaker's head orientation from audio can provide valuable information in smart environments, meetings, and driver monitoring. We propose a novel approach that leverages the phase component of the short-time Fourier transform from a single microphone array as input to a deep neural network combining convolutional, recurrent, and self-attention layers. Unlike prior methods that use physics-informed handcrafted features or raw waveform inputs, our approach enables robust learning from simulated and real data. Trained on a large-scale dataset generated with voice directivity patterns and fine-tuned on real recordings, our model achieves state-of-the-art accuracy, outperforming baselines under both clean and noisy conditions. Personalization experiments further demonstrate significant gains, reaching a mean angular error of 11.3 degrees when adapting to individual users and environments.
Primary: Tampere University
All Institutions: Tampere University
This paper presents a robust deep learning approach for speaker head orientation estimation using STFT phase features, demonstrating superior performance over raw audio and handcrafted features in noisy and reverberant environments, with significant gains achieved through user-specific personalization.
The paper proposes a deep learning framework for speaker head orientation estimation using a single microphone array. The core methodological contribution is the use of the phase component of the Short-Time Fourier Transform (STFT) as the primary input feature, rather than raw waveforms or handcrafted features like GCC-PHAT. The architecture combines 2D Convolutional Neural Networks (CNNs) for spatial feature extraction, Bidirectional Gated Recurrent Units (GRUs) for temporal modeling, and Multi-Head Self-Attention mechanisms. The input representation involves stacking the sine and cosine of the phase from all microphone channels. The model predicts orientation in a continuous regression format using sine/cosine representation to handle circular continuity. The approach is technically sound, leveraging established deep learning components in a novel configuration for this specific acoustic task. The use of phase-only features is the key differentiator, justified by the hypothesis that phase carries robust directional cues, particularly in reverberant environments.
The evaluation is comprehensive, covering both simulated and real-world data. The simulated dataset is generated using Voice Directivity Patterns (VDP) and the VCTK corpus, with room acoustics simulated via the image source method. This allows for controlled testing across various noise levels (clean, moderate, high SNR) and reverberation conditions. The paper compares the proposed method against three baselines: Soundr (CNN+LSTM on raw audio), a method based on ITD/ILD features, and a physics-informed feature-based method. Results indicate that the phase-based model outperforms baselines in noisy and reverberant conditions, which is a significant finding given the sensitivity of phase to environment. Personalization experiments (fine-tuning on user/room data) show significant performance gains, reducing Mean Angular Error (MAE) to 11.3 degrees. The evaluation on a real-world dataset further validates the generalization capability, although the absolute performance metrics on real data are not explicitly detailed in the text provided (referenced as Table REF). The ablation of reverberation effects (anechoic vs. reverberant) provides valuable insight into the model's behavior.
The paper provides sufficient detail for reproduction. The STFT parameters (4ms window, 2ms stride), network architecture (3 conv layers, 2 GRU layers, 2 attention blocks), and training details (Adam optimizer, MSE loss, 200k iterations) are specified. The dataset generation process is clearly described, including the use of VCTK, VDPs, and Pyroomacoustics for simulation. The noise augmentation strategy (phase-randomized monophonic noise) is also detailed. However, the specific hyperparameters for the personalization fine-tuning (learning rate, number of epochs) are not explicitly stated, which might require some trial and error for exact reproduction. The code is not explicitly linked, but the methodology is clear enough to implement.
The paper acknowledges several limitations. First, the model's generalization to unseen users and environments is limited, necessitating personalization/fine-tuning for optimal performance. This suggests the model relies heavily on specific acoustic characteristics of the training data. Second, the reliance on simulated data for pre-training, while effective, introduces a domain gap that requires real-world fine-tuning. Third, the phase-only input might discard magnitude information that could be beneficial in certain conditions, although the authors claim no benefit was found. Finally, the evaluation is limited to azimuthal orientation; elevation estimation is not addressed. The performance in highly non-stationary noise or with significant speaker movement is not evaluated.
Accurate and privacy-preserving head orientation estimation has significant implications for human-computer interaction, smart homes, meeting transcription systems, and driver monitoring. By using audio-only sensors, this technology avoids the privacy concerns associated with cameras. The ability to work with compact microphone arrays makes it deployable in consumer devices. The focus on robustness to noise and reverberation enhances its practical utility in real-world environments. The personalization aspect highlights the need for adaptive systems in user-centric AI. This paper presents a robust deep learning approach for speaker head orientation estimation using STFT phase features, demonstrating superior performance over raw audio and handcrafted features in noisy and reverberant environments, with significant gains achieved through user-specific personalization.
Modern automatic speaker verification (ASV) systems are vulnerable to adversarial perturbations. Diffusion-based purification has recently shown strong effectiveness against such perturbations, but its reverse denoising process requires iterative sampling and leads to high inference latency. We find that the forward noising process provides most of the robustness gain. Motivated by this observation, we reformulate adversarial purification as a learnable noising problem, and propose the Positive-Incentive Noise Predictor (PnP), the first framework that explicitly introduces positive-incentive noise (π-noise) into the purification task. PnP learns input-adaptive π-noise and mixes it with the input to improve the robustness of downstream ASV systems. Experiments on four advanced ASV backbones show that PnP effectively defends against adversarial attacks while preserving performance on natural speech. Compared with representative purification baselines, the proposed framework provides a competitive balance among defense effectiveness, impact on genuine utterances, and inference efficiency under white-box, black-box, and defender-aware adaptive attacks, with a real-time factor as low as 0.014. Moreover, PnP can be cascaded with a diffusion denoiser to further improve the perceptual quality of purified utterances. Code and purified audio examples are available at https://eurecom-asp.github.io/pnp/
Primary: EURECOM
All Institutions: EURECOM, The University of Sydney, Northwestern Polytechnical University, China Telecom (TeleAI), Research and Development Institute of Northwestern Polytechnical University in Shenzhen
The paper presents a significant and well-executed contribution to adversarial robustness in speaker verification by reformulating diffusion-based purification as a learnable forward noising problem, achieving a superior balance between defense effectiveness, inference efficiency, and audio quality.
The paper proposes a novel paradigm shift in adversarial purification for Automatic Speaker Verification (ASV). Instead of relying on the computationally expensive reverse denoising process of diffusion models, the authors hypothesize that the forward noising process provides the majority of the robustness gain. They introduce the Positive-Incentive Noise Predictor (PnP), which learns an input-adaptive noise pattern ($\pi$-noise) that is task-beneficial (i.e., it preserves speaker identity while suppressing adversarial perturbations). The methodology involves training a U-Net based noise predictor using a variational lower bound of mutual information, instantiated as a hinge loss on ASV similarity scores. The framework includes variants like PnP-Gaussian (simple additive) and PnP-Diff (diffusion-style schedule). The approach is theoretically grounded in information theory and practically motivated by the inefficiency of current diffusion-based purifiers.
The experimental evaluation is comprehensive and rigorous. The authors test on four state-of-the-art ASV backbones (ECAPA-TDNN, CAM++, ResNet, SimAMResNet) and under three attack settings: white-box (MI-FGSM, PGD), black-box (FAKEBOB), and defender-aware adaptive attacks. They compare against strong baselines including DAP, AudioPure, and neural codecs. Key findings include: 1) PnP-Diff achieves state-of-the-art robustness with a very low Real-Time Factor (RTF) of 0.014, significantly faster than iterative diffusion methods. 2) The forward-process-only hypothesis is validated, showing minimal performance drop compared to full diffusion pipelines. 3) Cascading PnP with a diffusion denoiser improves perceptual quality (WB-PESQ, SI-SDR) without significantly compromising robustness. The ablation studies on hyperparameters and purification steps add depth to the analysis.
The paper provides detailed descriptions of the architecture, loss functions, and training procedures. The code and purified audio examples are available via the provided URL, which greatly enhances reproducibility. The datasets (VoxCeleb, LibriSpeech) are standard and accessible. The use of open-source toolkits (WeSpeaker, torchattacks) further supports reproducibility.
The primary limitation is that PnP-Gaussian, while fast, degrades audio quality significantly (low WB-PESQ, high WER), making it less suitable for applications where perceptual quality is critical. The PnP-Diff variant is better balanced but still introduces some distortion compared to the clean signal. The method is specifically tailored for ASV; its generalizability to other audio tasks (e.g., speech recognition, emotion recognition) is not explored in depth, although the core idea might transfer. The adaptive attack evaluation is limited to gradient-based attacks through the purifier; more complex adaptive attacks (e.g., black-box adaptive) are not fully explored.
This work has significant implications for the security of biometric systems. By providing a lightweight, effective defense against adversarial attacks, it enhances the trustworthiness of ASV systems in real-world applications. The insight that forward noising can be optimized for robustness opens new avenues for efficient adversarial defense in other domains using generative models. However, the dual-use nature of adversarial attacks and defenses means that improved defenses may also spur more sophisticated attacks, necessitating continuous research. The paper presents a significant and well-executed contribution to adversarial robustness in speaker verification by reformulating diffusion-based purification as a learnable forward noising problem, achieving a superior balance between defense effectiveness, inference efficiency, and audio quality.
While prior work has explored emotion control in hybrid text-to-speech systems, the geometric properties of these modules, and their implications for steerability, remain poorly understood. We present the first comparative study of speech language model (SLM) and conditional flow-matching (CFM) modules as activation steering sites for mixed emotion speech synthesis. We first characterize emotion representations using linear probing and local intrinsic dimensionality (LID), and then evaluate single-site and joint steering for mixed-emotion synthesis. Our results show that SLM offers a clean, low-dimensional emotion-specific subspace with strong speaker--emotion disentanglement, while CFM exhibitspoor cross-speaker generalization due to speaker--emotion entanglement. Joint steering increases emotion intensity but degrades proportional control and speech quality on in-distribution data. These findings provide practical guidance for multi-site activation steering in hybrid TTS systems and highlight the importance of representation geometry in controllable speech generation.
Primary: The University of Melbourne
All Institutions: The University of Melbourne, Monash University
This paper presents the first comparative geometric analysis of activation steering in hybrid TTS models, revealing that Speech Language Models offer cleaner, more disentangled emotion subspaces than Conditional Flow-Matching modules, thereby providing crucial insights for designing effective and interpretable emotion control mechanisms in speech synthesis.
The paper employs a rigorous geometric analysis framework to compare two distinct activation steering sites (SLM and CFM) within a hybrid Text-to-Speech (TTS) architecture. The methodology is sound and well-structured, utilizing linear probing for discriminability and Local Intrinsic Dimensionality (LID) to characterize the manifold structure of emotion representations. The extraction of steering vectors via mean subtraction and the application of weighted summation for mixed emotions are standard but effectively applied in this context. The core methodological contribution lies in the systematic correlation between geometric properties (LID, discriminability gaps) and steering performance (proportional control, speaker fidelity), providing a mechanistic explanation for why certain steering sites work better than others. The analysis of joint steering interference is particularly insightful, attributing degradation to distribution shift and speaker entanglement.
The experimental setup is comprehensive, covering three major emotion datasets (ESD, CREMA-D, RAVDESS) and evaluating both emotion control metrics (E-SIM, TEP, Proportional Control, H-Rt) and speech quality metrics (S-SIM, WER). The results clearly demonstrate the trade-offs: SLM offers better proportional control and speaker preservation, while CFM offers higher intensity but suffers from speaker entanglement. The use of objective metrics like WER and S-SIM alongside emotion-specific embeddings adds robustness. However, the reliance on objective metrics for emotion perception (E-SIM, TEP) rather than human subjective evaluation (MOS/MUSHRA) is a limitation, though common in early-stage steering papers. The ablation on steering strength and the comparison of single vs. joint steering provide strong empirical evidence for the geometric claims.
The paper provides sufficient detail for reproduction, including the backbone model (CosyVoice2), specific layers for steering (SLM layers 14, 17; CFM every 5th layer), and the datasets used. The mathematical definitions for LID and steering vector extraction are clear. However, the code is not explicitly linked in the text provided, and some hyperparameters for the linear probes and LID estimation (e.g., K for nearest neighbors) are referenced via citations rather than detailed in the main text, which might slightly hinder immediate reproducibility without accessing the cited works or supplementary material.
The primary limitation is the lack of human subjective evaluation for emotion perception and speech quality, relying solely on proxy metrics. The study is confined to a single hybrid TTS architecture (CosyVoice2), limiting the generalizability of the geometric findings to other architectures (e.g., end-to-end diffusion TTS or different hybrid designs). The joint steering analysis is limited to in-distribution data; out-of-distribution generalization of the interference effects is not explored. Additionally, the LID analysis, while insightful, is computationally expensive and sensitive to the choice of K, which is not fully ablated.
This work significantly advances the understanding of controllable speech generation by linking representation geometry to steering efficacy. It provides practical guidelines for developers of hybrid TTS systems, suggesting that SLM is a superior site for precise emotional mixing, while CFM is better for intensity but risky for speaker identity. The findings on speaker-emotion entanglement in flow-matching modules have broader implications for interpretability and control in generative models. By highlighting the geometric properties of latent spaces, the paper contributes to the growing field of mechanistic interpretability in audio generation. This paper presents the first comparative geometric analysis of activation steering in hybrid TTS models, revealing that Speech Language Models offer cleaner, more disentangled emotion subspaces than Conditional Flow-Matching modules, thereby providing crucial insights for designing effective and interpretable emotion control mechanisms in speech synthesis.
Multichannel Deep Neural Networks (DNNs) have significantly improved speech enhancement performance; however, they typically remain constrained by reliance on fixed microphone array geometries, leading to poor generalization on unseen or irregular configurations. Current array-agnostic approaches often rely on high-complexity architectures or massive, diverse datasets, yet they still struggle to generalize to out-of-distribution layouts. In this paper, we present an in-depth analysis of AmbiDrop, a recently proposed framework that achieves geometry independence by leveraging ideal Ambisonics as the DNN input. By employing a channel-wise dropout layer during training to simulate Ambisonics encoding errors, AmbiDrop decouples the learning process from the physical sensor arrangement. During inference, microphone signals from arbitrary array configurations are transformed into the Ambisonics domain via Ambisonics Signal Matching (ASM) before processing. Extensive experiments demonstrate that AmbiDrop maintains high robustness across a diverse suite of unseen simulated arrays and real-world recordings. Furthermore, our results show that the framework is resilient to sensor failures and remains effective even with reduced network scales, making it highly suitable for deployment on resource-constrained edge devices and versatile wearable hardware.
Primary: Ben-Gurion University of the Negev
All Institutions: Ben-Gurion University of the Negev, Reality Labs Research at Meta
[One sentence main contribution]. [The paper presents AmbiDrop, an array-agnostic speech enhancement framework that leverages Ambisonics domain transformation and channel-wise dropout to achieve robust generalization across diverse and unseen microphone array geometries, validated on both simulated and real-world wearable hardware].
The paper proposes AmbiDrop, a framework for array-agnostic speech enhancement. The core innovation lies in decoupling the neural network training from specific microphone geometries by transforming inputs into the Ambisonics domain. Specifically, it uses ideal Ambisonics signals for training and employs a channel-wise dropout layer to simulate the encoding errors that occur when using Ambisonics Signal Matching (ASM) on physical arrays during inference. This is a clever and relatively simple mechanism to handle domain shift between ideal training data and imperfect real-world encoding. The approach is architecture-agnostic, demonstrated with FT-JNF and IC-ConvTasNet. While the concept of using spherical harmonics for array invariance is not entirely new (e.g., eigenbeam features), the specific combination of ASM for inference and dropout for robustness to encoding errors is a distinct and practical contribution. It avoids the complexity of learnable permutation-invariant layers or massive meta-learning datasets.
The experimental evaluation is comprehensive and rigorous. It covers: 1. **Simulated Data:** Extensive testing on 20 different simulated arrays (1D, 2D, 3D, free-field, rigid-sphere) including unseen test geometries. This directly addresses the generalization claim. 2. **Real-World Data:** Evaluation on Project Aria glasses, a real wearable device. This is a significant strength, moving beyond simulation. It includes tests with normal and mispositioned glasses, adding practical relevance. 3. **Ablation Studies:** Detailed analysis of dropout strategies (uniform vs. per-channel), resilience to microphone failures, and network complexity scaling. 4. **Baselines:** Comparison against standard geometry-dependent baselines and other array-agnostic approaches mentioned in the intro. The results clearly show that while baseline models fail on unseen arrays, AmbiDrop maintains performance. The drop in performance on real-world data compared to simulation is analyzed and attributed to ATF modeling inaccuracies and environmental factors, which is a honest and insightful discussion.
The paper provides detailed mathematical formulations for ASM and the Ambisonics encoding. It specifies the DNN architectures (FT-JNF, IC-ConvTasNet) and their parameters. The dataset generation process (image method, WSJ0 speech) is described. However, the exact code for ASM implementation and the specific random seeds for the simulations are not explicitly linked in the text (though often available in supplementary materials or future releases). The reliance on specific ATFs (simulated vs. measured) for the Aria glasses is well-documented. Overall, reproducibility is high given the standard nature of the components, though the specific ASM filter design details are crucial.
1. **ATF Dependency:** The performance is heavily dependent on the accuracy of the Ambisonics Signal Matching (ASM) filters. If the assumed ATF (e.g., rigid sphere) deviates significantly from the physical reality (e.g., due to head-related transfer functions not accounted for, or mispositioning), performance degrades. The paper acknowledges this but the gap between simulated and real-world ATF performance is notable. 2. **Order Limitation:** The method is demonstrated with 2nd-order Ambisonics (9 channels). Higher orders might capture more spatial detail but require more microphones and are more sensitive to spatial aliasing. 3. **Computational Overhead:** While the DNN can be small, the ASM step adds a computational burden at inference time, which might be non-trivial for very low-latency applications, although the paper argues it is suitable for edge devices. 4. **Generalization to Extreme OOD:** While it generalizes to unseen *geometries*, it assumes the sound field can be reasonably approximated by the ASM model. Highly irregular or non-spherical arrays might still pose challenges if the ASM error is too high.
This work has significant implications for wearable audio devices (hearables, smart glasses) where microphone arrays are small, irregular, and subject to movement/occlusion. By enabling a single model to work across different hardware configurations, it reduces the need for device-specific model training and deployment. This promotes interoperability and robustness in consumer electronics. It also contributes to the broader field of robust speech processing by providing a new perspective on handling sensor variability. [One sentence main contribution]. [The paper presents AmbiDrop, an array-agnostic speech enhancement framework that leverages Ambisonics domain transformation and channel-wise dropout to achieve robust generalization across diverse and unseen microphone array geometries, validated on both simulated and real-world wearable hardware].