Current voice AI benchmarks typically evaluate isolated capabilities such as speech intelligibility, word error rate, or text-based dialogue quality, but they rarely test whether systems harness the acoustic information that distinguishes spoken language from its textual representation. To this end, we introduce the Real World Voice EQ Bench, a multidimensional benchmark for evaluating voice AI across text-to-speech (TTS), speech-to-speech (STS), speech understanding (SU), and automatic speech recognition (ASR). Our evaluations indicate that performance is highly dimension-specific. For TTS, naturalness, expressiveness, identity stability, and reliability are largely independent evaluation dimensions. For STS, access to audio does not guarantee use of vocal affect, and some agents remain largely transcript-driven. For SU, models perform unevenly across paralinguistic tasks. For ASR, real world accent, emotion, noise, and conversational conditions expose failures that are not captured by established clean-speech benchmarks. Together, these results show that voice AI should be evaluated as a profile of acoustic, expressive, interactional, and robustness capabilities rather than by a single aggregate score.
Primary: Hume AI Research
All Institutions: Hume AI Research
The paper presents RW-Voice-EQ Bench, a rigorous, multidimensional benchmark that reveals the dimension-specific nature of voice AI performance, exposing critical gaps in current evaluation practices such as benchmark optimization in ASR and the underutilization of paralinguistic cues in STS systems.
The paper introduces a comprehensive, multidimensional benchmarking framework called RW-Voice-EQ Bench, designed to evaluate voice AI systems across four distinct domains: Text-to-Speech (TTS), Speech-to-Speech (STS), Speech Understanding (SU), and Automatic Speech Recognition (ASR). The methodology is rigorous, separating linguistic content from paralinguistic acoustic cues. For perceptual tasks (TTS, STS), it employs large-scale human evaluation (over 1 million ratings) using specific rubrics and mixed-effects models to control for rater bias. For objective tasks (SU, ASR), it uses label-based scoring. A significant methodological contribution is the diagnostic suite for "benchmark optimization" (or "benchmaxxing") in ASR, which tests for memorization via consensus disagreement, audio masking, orthographic switching, and synthetic voice probes. The paper also validates Speech-Language Models (SLMs) as judges against human raters, providing a nuanced analysis of where automated judges succeed (verification-dependent tasks) and fail (open-ended perceptual tasks).
The experimental evaluation is extensive, covering 31 TTS configurations, 15 STS configurations, and analyzing top ASR models on the Hugging Face leaderboard. The results clearly demonstrate that performance is highly dimension-specific; no single model dominates all categories. For instance, in TTS, models strong in expressiveness (e.g., Gemini 3.1 Flash) differ from those strong in reliability or long-form stability (e.g., Kokoro-82M). In STS, the paper highlights that many systems remain largely transcript-driven, failing to leverage vocal affect even when present. The ASR analysis provides compelling evidence that high WER scores on clean benchmarks do not correlate with robustness to accents, noise, or emotional speech, and that many top models exhibit signs of overfitting to benchmark artifacts. The statistical analysis using mixed-effects models and Spearman correlations adds robustness to the findings.
The paper provides significant detail on the evaluation setup, including the number of raters, rating scales, and the specific dimensions evaluated. The benchmark and judge leaderboards are hosted on Hugging Face Spaces, ensuring accessibility. However, the full dataset of prompts and the specific code for the mixed-effects modeling pipeline are not explicitly linked in the provided text, though the Hugging Face links suggest open resources. The detailed description of the "benchmark optimization" diagnostics allows for replication of those specific tests.
The primary limitation is the reliance on human evaluation for TTS and STS, which is resource-intensive and potentially subject to demographic biases, although the paper attempts to mitigate this with large sample sizes and rater filtering. The STS evaluation, while innovative, is limited to specific scripted scenarios (hostile, urgent, ambiguous tone) and may not capture the full breadth of spontaneous conversational dynamics. The ASR benchmark optimization analysis is preliminary, focusing on VoxPopuli, and the authors acknowledge the need for future work to expand this to other datasets. Additionally, the SLM judge validation is currently limited to TTS, leaving STS judge reliability less explored.
This paper has substantial impact on the voice AI community by challenging the reliance on single aggregate scores and clean-speech benchmarks. It provides a roadmap for more realistic and robust evaluation of voice systems, which is critical for deployment in real-world applications. By exposing benchmark optimization in ASR, it encourages the development of more generalizable models and the use of withheld test sets. The multidimensional profile approach helps developers select models based on specific use-case requirements (e.g., customer service vs. entertainment), driving more targeted research and development. The paper presents RW-Voice-EQ Bench, a rigorous, multidimensional benchmark that reveals the dimension-specific nature of voice AI performance, exposing critical gaps in current evaluation practices such as benchmark optimization in ASR and the underutilization of paralinguistic cues in STS systems.
Neural audio codecs with residual vector quantization (RVQ) normally treat all frequencies uniformly, so their codebooks become spectrally entangled. Truncating stages then removes an unpredictable mix of frequencies. Parallel band decomposition addresses this by splitting audio into independent bands, but fragments the latent space and loses cross-frequency coherence. We introduce HARP (Harmonic-Aware Residual Partitioning), a training strategy that partitions RVQ stages into frequency-ordered groups where each group refines its target band while the decoder retains access to all lower frequencies. Overtones are reconstructed in the context of their fundamentals, preserving coherence that parallel methods lose. HARP requires no architectural changes; it only modifies the training loss, leaving inference identical to standard RVQ. On speech, music, and general audio, HARP outperforms both standard RVQ and parallel decomposition. MUSHRA listening tests also show perceptual improvements.
Primary: Georgia Institute of Technology
All Institutions: Georgia Institute of Technology, The Chinese University of Hong Kong, Tencent Music Entertainment
HARP introduces a training-only strategy for neural audio codecs that partitions RVQ stages into frequency-ordered groups with cumulative decoding and soft subband supervision, effectively addressing spectral entanglement and improving both objective metrics and perceptual quality, particularly at low bitrates, without architectural changes.
The paper proposes HARP, a training-time strategy for Residual Vector Quantization (RVQ) in neural audio codecs. The core innovation lies in partitioning RVQ stages into frequency-ordered groups and enforcing spectral specialization through a novel loss function. Key components include: 1) Cumulative Decoding: Ensuring higher-frequency stages are reconstructed in the context of lower-frequency content to preserve harmonic coherence. 2) Subband Contribution Supervision: Supervising the isolated contribution of each group to its target band using stop-gradient operators to prevent gradient interference. 3) Soft Band Weighting: Using learnable Gaussian weights over mel bins to guide specialization without hard spectral cutoffs. The approach is architecturally agnostic, requiring only modifications to the training objective. This is a clever and relatively simple modification that addresses the "spectral entanglement" problem inherent in standard RVQ, where early and late stages capture similar frequency content.
The authors evaluate HARP against standard DAC (same architecture, no band supervision) and BSCodec (parallel band decomposition). They use a diverse dataset including MUSDB18-HQ, LibriTTS, and AudioSet. Metrics include SI-SDR, KAD, PESQ, STOI, and MUSHRA listening tests. Results show consistent improvements in SI-SDR and KAD over both baselines, with particularly notable gains at low bitrates (2.6 kbps). The ablation studies effectively isolate the contributions of cumulative decoding, soft boundaries, and stop-gradient mechanisms. The harmonic coherence analysis using synthetic tones provides strong evidence for the preservation of phase/amplitude relationships. The MUSHRA tests confirm perceptual improvements, especially in bass fullness and timbral naturalness.
The paper provides detailed descriptions of the architecture, training objectives, and hyperparameters. The code is made publicly available on GitHub. The use of standard components (DAC backbone, EMA updates) and clear algorithmic descriptions enhances reproducibility. The training cost increase (2-3x) is noted, which is a practical consideration but does not hinder reproducibility.
The primary limitation is that the performance gains, while consistent, are modest in absolute terms (e.g., +0.25 to +0.96 dB SI-SDR). The method relies on the assumption that natural audio has a specific energy distribution (low-to-high), which might not hold for all types of audio (e.g., highly inharmonic or noise-heavy content, though the paper argues it still applies). The method is training-only, so it does not offer architectural efficiency gains. The "soft" boundaries mean that some spectral leakage or imperfect separation might still occur, potentially limiting the maximum potential of the specialization compared to hard architectural splits (though the paper argues this is offset by coherence).
This work contributes to the field of neural audio compression by providing a principled way to impose spectral structure on RVQ without architectural complexity. This can lead to more robust and interpretable audio codecs, which are foundational for speech synthesis, music generation, and audio understanding tasks. The improved low-bitrate performance is particularly relevant for streaming applications. The open-source release facilitates further research in this area. HARP introduces a training-only strategy for neural audio codecs that partitions RVQ stages into frequency-ordered groups with cumulative decoding and soft subband supervision, effectively addressing spectral entanglement and improving both objective metrics and perceptual quality, particularly at low bitrates, without architectural changes.
We introduce the REAL-TSE Challenge, an IEEE SLT 2026 satellite challenge on target speaker extraction~(TSE) from real conversational recordings. Given a multi-speaker mixture and one or more enrollment utterances from a target speaker, participating systems must recover only the target speech. Unlike simulated read-speech benchmarks, REAL-TSE evaluates Mandarin and English recordings that contain natural overlap, reverberation, noise, channel mismatch, and conversational dynamics. The challenge defines two complementary tracks: an Online track for low-latency streaming extraction and an Offline track for full-context processing. Systems are evaluated with Token Error Rate (TER), Speaker Similarity (SpkSim), DNSMOS, and target-speaker activity F1. This overview paper describes the task definition, datasets, baselines, evaluation protocol, submitted systems, condition-wise findings, and lessons for future real-world TSE benchmarks.
Primary: Nanjing University
All Institutions: Nanjing University, Chinese University of Hong Kong (Shenzhen), Brno University of Technology, Northwestern Polytechnical University, Shanghai Jiao Tong University, NTT, Inc.
The REAL-TSE Challenge establishes a critical new benchmark for real-world Target Speaker Extraction, revealing significant gaps between simulated and real-world performance and highlighting the dangers of metric over-optimization in neural audio evaluation.
This paper introduces the REAL-TSE Challenge, a benchmarking framework rather than a novel algorithmic architecture. The methodological contribution lies in the rigorous definition of a real-world Target Speaker Extraction (TSE) task. It distinguishes itself from prior simulated benchmarks (like LibriMix) by utilizing real conversational recordings with natural overlap, reverberation, noise, and channel mismatch. The challenge is structured into two tracks: Online (low-latency, <100ms) and Offline (full-context). The evaluation methodology is multi-dimensional, employing Token Error Rate (TER), Speaker Similarity (SpkSim), DNSMOS-P808, and Target-Speaker Activity F1. A critical methodological insight presented is the detection of metric over-optimization, specifically regarding DNSMOS-OVRL, leading to the adoption of DNSMOS-P808 for ranking. The paper details the data simulation strategies used by participants (pseudo-labeling, real-data adaptation) and analyzes the impact of channel mismatch and scenario types on performance.
The experimental section provides a comprehensive analysis of 24 systems from 18 teams. It includes baseline results (BSRNN variants) and aggregate performance metrics. The paper conducts a detailed condition-wise analysis on the EVAL-2 set, examining performance across different recording devices (H1, H2, Phone), target-speaker activity ratios, and recording scenarios (meeting, cafe, home, vehicle). Key findings include the significant impact of mixture recording device quality, the lack of consistent correlation between nominal scenario labels and difficulty, and the trade-offs between different metrics (e.g., systems optimizing for SpkSim may suffer in TER). The analysis of DNSMOS reliability is a significant experimental contribution, demonstrating how neural metrics can be gamed without perceptual improvement.
The challenge promotes reproducibility and transparency. The authors release the evaluation toolkit, baseline code (WeSep-based), and the dataset splits (Development and Evaluation). The training policy is open but restricted regarding specific test-set leakage (forbidden to use dev/test splits of source corpora for training). The paper provides detailed descriptions of the latency verification process (perturbation-based) and metric computation. However, as a challenge overview, it does not provide full training code for the winning systems, which is standard for such venues, but it does release the infrastructure for future replication.
The primary limitation is the inherent difficulty of real-world evaluation: the lack of ground-truth clean targets for most real recordings makes absolute performance measurement challenging, relying instead on proxies like ASR and speaker similarity. The paper acknowledges that metric-aware over-optimization is a significant issue, suggesting that current automatic metrics may not fully align with human perception or true extraction quality. Additionally, the "unseen" nature of EVAL-2 means that generalization to entirely new, unrecorded environments remains an open question, though the challenge design attempts to mitigate this by using diverse, newly recorded data.
This work has a substantial impact on the speech processing community by shifting the focus from simulated to real-world TSE evaluation. It highlights the gap between lab-performance and real-world deployment, emphasizing the need for robustness to channel mismatch, noise, and conversational dynamics. The findings on metric reliability (DNSMOS) have broader implications for the entire field of audio quality assessment and neural metric usage. By providing a standardized benchmark and revealing the limitations of current approaches, it guides future research towards more practical, multi-objective optimization and better data simulation techniques. The REAL-TSE Challenge establishes a critical new benchmark for real-world Target Speaker Extraction, revealing significant gaps between simulated and real-world performance and highlighting the dangers of metric over-optimization in neural audio evaluation.
Large Audio Language models (LALMs) have made rapid progress on acoustic understanding, yet they still struggle with fine-grained audio reasoning (e.g., recognizing event order, repetitions and duration). Existing post-training methods heavily rely on expensive external labels or provide only coarse semantic signals. To bridge this gap, we introduce Audio-Zero, the first label-free self-evolution framework in the field of LALMs that improves fine-grained auditory perception and reasoning. Audio-Zero constructs an auditory self-play game from unlabeled audio contrast pairs: most players hear a reference audio, while one odd listener hears a subtle variant. The model first generates clues describing what it hears and then identifies the odd listener by reasoning over inconsistencies among clues. Since the odd listener is known by construction, the game provides verifiable rewards without any annotated answers. Experiments with Qwen2-Audio-7B-Instruct and Qwen2.5-Omni-7B on TREA, MMAU Test-mini and MMAR show that Audio-Zero improves fine-grained audio reasoning while preserving broad audio understanding. Evolutionary and diagnostic analyses further reveal that increasingly fine-grained auditory descriptions emerge naturally from game pressure.
Primary: Institute of Computing Technology, Chinese Academy of Sciences
All Institutions: Institute of Acoustics, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing Academy of Artificial Intelligence, Institute of Computing Technology, Chinese Academy of Sciences, University of California, Merced, The University of Queensland
[One sentence main contribution]. Audio-Zero introduces a label-free self-evolution framework for LALMs that uses an auditory self-play game to improve fine-grained reasoning without external annotations. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a novel and technically sophisticated approach to enhancing the fine-grained reasoning capabilities of Large Audio Language Models. By formulating the learning process as a self-play game with verifiable intrinsic rewards, Audio-Zero effectively circumvents the bottleneck of scarce fine-grained annotations. The methodology is well-grounded in recent advances in self-evolution and reinforcement learning, and the experimental results demonstrate its efficacy across multiple benchmarks and model architectures. The diagnostic analysis provides valuable insights into how fine-grained auditory descriptions emerge from game pressure. This work is a significant contribution to the field of audio AI, offering a scalable and efficient path to improving model performance.
The paper proposes Audio-Zero, a label-free self-evolution framework for Large Audio Language Models (LALMs). The core innovation is an "auditory self-play game" that converts unlabeled audio contrast pairs into verifiable training signals. The method involves two coupled stages: a listening stage where the model generates descriptive clues for multiple "players" (some hearing a reference, one hearing a variant), and an attribution stage where the model acts as a judge to identify the "odd listener" based on inconsistencies in the clues. The reward signal is intrinsic to the game mechanics (correct identification) and a rule-based content utility score for the clues. The optimization uses Group Relative Policy Optimization (GRPO). The approach is theoretically sound and addresses a genuine gap in the field: the lack of fine-grained, label-free supervision for audio reasoning. The mechanism of using the "odd-one-out" task to force fine-grained perceptual description is clever and aligns well with recent trends in self-play and self-evolution for LLMs and VLMs.
The authors evaluate Audio-Zero on three benchmarks: TREA (temporal reasoning), MMAU Test-mini (general audio understanding), and MMAR (deep audio reasoning). Experiments are conducted on two strong backbones: Qwen2-Audio-7B-Instruct and Qwen2.5-Omni-7B. The results show consistent improvements over the base policies and several strong baselines (both label-dependent and label-free). Notably, the gains are most pronounced on TREA, the fine-grained temporal reasoning benchmark, which supports the paper's claim of improving fine-grained perception. The ablation studies effectively demonstrate the necessity of each component (listening stage, attribution stage, vote-aware feedback). The self-evolution dynamics analysis (tracking win rates, clue length, and benchmark scores over iterations) provides compelling evidence that the model is indeed learning and evolving rather than just overfitting to a static reward. The use of only 2k unlabeled pairs is a significant efficiency advantage.
The paper provides detailed implementation details, including hyperparameters (learning rate, KL coefficient, number of players/rounds), dataset construction (filtering 2k pairs from Audio-Alpaca), and prompt templates. The GRPO implementation details are standard but well-specified. The diagnostic metrics for analyzing clue quality are clearly defined. While the code is not explicitly linked in the provided text, the description is sufficient for reproduction by researchers familiar with RLHF/GRPO pipelines. The reliance on rule-based scorers for the listening reward is a potential point of variability, but the rules are described.
The method relies on the availability of audio contrast pairs. While the paper claims to use "unlabeled" pairs, it sources them from Audio-Alpaca, which is a preference dataset. The filtering process to create contrast pairs (ensuring they share semantics but differ in fine-grained details) is not fully detailed, raising questions about the scalability and generalizability of the pair generation process. The rule-based listening reward, while effective, may not capture all nuances of good auditory description compared to a learned reward model. Furthermore, the performance gains, while significant, are still modest in absolute terms on some benchmarks (e.g., MMAU Avg. 60.50 vs 57.50 for Qwen2-Audio), suggesting that self-evolution alone may not solve all reasoning deficits without some form of structured supervision or larger-scale data.
This work contributes to the development of more autonomous and data-efficient AI systems. By reducing reliance on expensive human annotations, it lowers the barrier to training high-capability audio models. The improved fine-grained audio reasoning capabilities could benefit applications in accessibility (e.g., better descriptions for the visually impaired), audio forensics, and interactive AI assistants. The self-play paradigm could be extended to other modalities or tasks where verifiable rewards are hard to construct. [One sentence main contribution]. Audio-Zero introduces a label-free self-evolution framework for LALMs that uses an auditory self-play game to improve fine-grained reasoning without external annotations. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a novel and technically sophisticated approach to enhancing the fine-grained reasoning capabilities of Large Audio Language Models. By formulating the learning process as a self-play game with verifiable intrinsic rewards, Audio-Zero effectively circumvents the bottleneck of scarce fine-grained annotations. The methodology is well-grounded in recent advances in self-evolution and reinforcement learning, and the experimental results demonstrate its efficacy across multiple benchmarks and model architectures. The diagnostic analysis provides valuable insights into how fine-grained auditory descriptions emerge from game pressure. This work is a significant contribution to the field of audio AI, offering a scalable and efficient path to improving model performance.
Long-form streaming speech-to-speech translation (S2ST) requires incremental, unbounded translation under strict latency constraints. Existing methods typically suffer from sentence-bounded supervision or demand massive paired-S2ST supervision. We introduce a training recipe enabling a speech language model for sentence-level and long-form streaming S2ST using only $\sim$2k hours of paired cross-lingual S2ST data, layered atop auxiliary supervision. Anchored by auxiliary multitask training, our approach remains robust even when the paired-S2ST budget itself is reduced by 90\%. Our core contribution, joint text-code trajectory supervision, schedules target text and acoustic semantic codes as a unified commitment path, eliminating the need for separate, unstable speech-side emission controllers. Furthermore, our two-stream Thinker--Talker factorization significantly outperforms unified-decoder baselines by decoupling linguistic reasoning from dense acoustic prediction to mitigate modality interference. Finally, our system achieves highly competitive quality-latency trade-offs on RealSI and ACL60/60-dev, matching state-of-the-art, closed-source S2ST systems such as LiveInterpret~2.0 on ASR-BLEU.
Primary: The Chinese University of Hong Kong, Shenzhen
All Institutions: The Chinese University of Hong Kong, Shenzhen
SimulS2ST-Omni presents a significant technical contribution to streaming speech-to-speech translation by introducing a joint text-code trajectory supervision method and a decoupled two-stream architecture that achieves state-of-the-art performance with substantially reduced data requirements, effectively addressing the critical bottleneck of paired S2ST data scarcity.
The paper proposes SimulS2ST-Omni, a data-efficient framework for long-form streaming Speech-to-Speech Translation (S2ST). The core methodological contributions are threefold: (1) A joint text-code trajectory supervision mechanism that schedules target text and acoustic semantic codes simultaneously, eliminating the need for unstable, separate speech-side emission controllers. (2) A two-stream "Thinker-Talker" architecture that decouples linguistic reasoning (Thinker) from dense acoustic prediction (Talker), mitigating modality interference common in unified decoders. (3) A training recipe leveraging auxiliary multitask supervision (ASR, S2TT, MT, TTS) to enable robust performance with only ~2k hours of paired S2ST data, a significant reduction from prior works requiring tens of thousands of hours. The approach extends trajectory-based simultaneous translation to speech output by using cross-lingual word alignments and monotonicity filtering (NIR) to construct stable read/wait/write trajectories.
The authors conduct comprehensive experiments on offline S2ST (CVSS-T), sentence-level streaming (RealSI), and long-form streaming (RealSI, ACL60/60-dev). They compare against state-of-the-art public baselines (UniSS, Seamless-M4T) and closed-source systems (LiveInterpret 2.0). Results show that the Thinker-Talker architecture significantly outperforms the matched unified-decoder baseline in translation quality (ASR-BLEU, Text-BLEU) while maintaining competitive or superior speech quality metrics (A.PCP, SIM-O). The system matches or exceeds LiveInterpret 2.0 in several latency tiers. Ablation studies confirm the importance of NIR-based trajectory filtering and the architectural decoupling. The evaluation is rigorous, covering multiple metrics and latency constraints.
The paper provides detailed descriptions of the data construction process, including the use of public ASR/S2TT corpora to synthesize paired S2ST data, the alignment tools (SimAlign), and the filtering criteria (NIR, ASR error rates). Training hyperparameters, stages, and model initialization details are provided in the appendix. The use of open-source components (Qwen2.5-Omni, DualCodec, Whisper, Paraformer) enhances reproducibility. However, the exact codebase for the trajectory construction and the specific synthesized data pool are not explicitly linked to a public repository, which may hinder exact replication.
The experiments are limited to Chinese-English directions due to data scarcity. The paired S2ST data is synthesized rather than naturally recorded simultaneous interpretation, which may not fully capture real-world interpreting phenomena (e.g., disfluencies, interruptions). The speech generation backend (chunk-wise Flow Matching) is not fully streaming-native, potentially limiting inter-chunk coherence and inference efficiency. The work does not address duplex interaction or broader multilingual settings.
This work advances the field of real-time cross-lingual communication by demonstrating that high-quality streaming S2ST is achievable with limited paired data. It addresses ethical concerns regarding voice cloning and impersonation by acknowledging the risks and suggesting safeguards. The data-efficient approach lowers the barrier to entry for developing robust S2ST systems. SimulS2ST-Omni presents a significant technical contribution to streaming speech-to-speech translation by introducing a joint text-code trajectory supervision method and a decoupled two-stream architecture that achieves state-of-the-art performance with substantially reduced data requirements, effectively addressing the critical bottleneck of paired S2ST data scarcity.
Existing symbolic music generation models typically use bars as the basic structural unit. However, human perception of musical phrases often does not align with notated bar lines, leading to long-term structural fragmentation. This paper proposes RPPNet-a two-stage deep learning architecture with variable structural boundaries. It first generates variable-length Rhythm-Pitch Primitive (RPP) sequences, where each RPP encodes note count, rhythm, and contour; then decodes the RPP sequences into concrete notes. The grouping of RPPs is automatically derived from acoustic cues, auditory inertia, and similarity perception based on music psychology. Experiments show that melodies generated by RPPNet are superior in both long-term structure and musicality, with significant improvements across all subjective evaluation dimensions. Ablation studies confirm that the performance gain stems from the structural correctness of the psychological representation, rather than from model capacity. This work offers an interdisciplinary perspective for music generation, integrating music theory, computational modeling, and music psychology.
Primary: Zhejiang University
All Institutions: Zhejiang University, ZJU-ZJCM MusicAI Lab, Zhejiang Conservatory of Music
[One sentence main contribution]. [RPPNet introduces a perceptually-grounded, variable-length structural representation for symbolic music generation, demonstrating that aligning model boundaries with human auditory grouping principles significantly improves long-term structural coherence compared to fixed bar-level approaches.]
The paper proposes RPPNet, a hierarchical two-stage architecture for symbolic melody generation. The core innovation lies in replacing fixed bar-level tokenization with perceptually-grouped Rhythm-Pitch Primitives (RPPs). The authors employ a heuristic algorithm based on music psychology (acoustic cues, auditory inertia, similarity perception) to segment melodies into variable-length RPPs. The generation process is decoupled: an RPP-level Transformer generates the structural skeleton, which is then instantiated into note-level events via a time-scale expansion mapping. The methodology is theoretically grounded in cognitive musicology, offering a novel perspective on structural segmentation that challenges the dominant fixed-grid assumptions in models like Museformer and MELONS. The decoupled serial prediction mechanism at the RPP level is a technically sound approach to managing the complexity of long-sequence generation.
The experimental setup utilizes the MelodyNet dataset. The evaluation relies heavily on subjective listening tests (MOS-style) with 15 participants, which is a standard but resource-intensive method for music generation. RPPNet significantly outperforms baselines (Museformer, MELONS) in subjective metrics, particularly in "Structure" and "Overall Impression." The inclusion of ablation studies (Random-Grouped vs. Heuristic-Grouped) is crucial and effectively isolates the contribution of the psychological grouping algorithm from mere variable-length tokenization. However, the objective metrics are limited; the paper notes that PPL and Structural Error are difficult to compare across different tokenization schemes, leading to a reliance on subjective scores. The gap between generated music and human composition (6.69 vs 7.47) remains significant, though the paper attributes this to rhythmic distribution mismatches.
The paper provides a GitHub repository link, which is a positive step for reproducibility. The heuristic algorithm for RPP grouping is described, but the full formalization of the weight function $W()$ is stated to be in the repository, which may contain implementation-specific details not fully explicit in the text. The dataset (MelodyNet) is public. The subjective evaluation protocol is described, but the small sample size (15 participants) limits the statistical power of the findings, although significance tests are provided.
The primary limitation is the reliance on a heuristic, rule-based algorithm for RPP segmentation, which may not generalize well to all musical styles or complex polyphonic textures. The current model is limited to monophonic melody generation. The subjective evaluation, while showing statistical significance, suffers from a small participant pool and potential bias. The objective metrics do not fully capture the perceptual improvements claimed, as standard metrics like PPL are not directly comparable across different tokenization granularities.
This work bridges the gap between computational music generation and music psychology/cognition. By demonstrating that perceptually-grounded structural units lead to more coherent music, it offers a new paradigm for designing generative models that align with human auditory perception. This could influence future research in hierarchical modeling and structured generation beyond music, such as in language or video, where fixed-grid assumptions may also be limiting. [One sentence main contribution]. [RPPNet introduces a perceptually-grounded, variable-length structural representation for symbolic music generation, demonstrating that aligning model boundaries with human auditory grouping principles significantly improves long-term structural coherence compared to fixed bar-level approaches.]
Keyword spotting (KWS) models on embedded devices often need to add new keywords after deployment, but updates are difficult when original training data are unavailable and regressions on existing triggers are unacceptable. At a fixed operating point, our method reduces average new-keyword false reject rate (FRR) from 6.46 to 4.37 versus a parameter-matched separate-model baseline and outperforms parameter-efficient tuning baselines (adapters, LoRA), while using fewer multiply-accumulate operations (MACs) under the same added-parameter budget ($\leq$10k): 16.34M vs 18.45M/20.52M. We achieve this via parameter-capped modular expansion: the base network, including batch-normalization statistics and the core classifier, is frozen, and only a lightweight expansion branch with a separate new-keyword head is trained, preserving core logits, shipped outputs, and thresholds for existing keywords.
Primary: Yandex
All Institutions: Yandex
The paper presents a practical and effective method for modular expansion of embedded KWS models, achieving strict non-regulation and improved new-keyword detection through a frozen-base, parallel-branch architecture.
The paper proposes a "modular network expansion" technique for Keyword Spotting (KWS) on embedded devices. The core innovation is architectural: it freezes the entire base SVDF network (including Batch Normalization statistics) and attaches a lightweight, trainable expansion branch with a separate classifier head for new keywords. This design ensures a strict non-regression guarantee for existing keywords because the core logits remain identical to the pre-deployment model. The method is compared against parameter-efficient fine-tuning (LoRA, Adapters), ensemble methods, and continual learning (EWC). The approach is pragmatic and well-suited for the specific constraints of embedded deployment (no original data, strict latency/parameter budgets). While the concept of "parallel heads" or "expert routing" is not new in the broader ML landscape (e.g., Mixture of Experts, Progressive Neural Networks), its specific application to strict non-regression KWS updates with frozen BN layers is a solid, incremental engineering contribution. It lacks deep theoretical novelty but offers a robust practical solution.
The experiments are conducted on Google Speech Commands v2 (GSC) using five held-out keyword pairs. The evaluation metric is macro False Reject Rate (FRR) at a fixed False Accept Rate (FAR). The results show that the proposed method significantly outperforms the Ensemble baseline (6.46% vs 4.37% FRR) and other parameter-efficient methods (LoRA, Adapters) under a 10k parameter budget. The ablation study on expansion depth is useful, identifying that tapping 4 blocks is optimal. The comparison with Full Fine-tuning highlights the catastrophic forgetting problem, and EWC is shown to be less effective without replay data. The use of Mozilla Common Voice for negative samples is a good practice for realistic FAR calibration. The results are statistically significant (95% confidence intervals provided). However, the evaluation is limited to GSC, which is a relatively clean, controlled dataset. Real-world embedded KWS faces more challenges with noise, channel variation, and long-tail distributions.
The paper provides sufficient detail regarding the architecture (SVDF, BN, Hard-Swish), training setup (Adam, cosine LR, SpecAugment), and evaluation protocol (FAR calibration, 8 runs). The parameter budget (10k) and MAC counts are specified. The code is not explicitly linked, but the description is detailed enough for reproduction. The use of standard datasets (GSC, Common Voice) aids reproducibility.
The primary limitation is the reliance on GSC, which may not reflect the complexity of real-world embedded audio environments. The method assumes that the base model's features are sufficiently generalizable to the new keywords via the expansion branch; if the new keywords are phonetically very distinct from the core set, performance might degrade. The "core-first" decision rule introduces a slight latency overhead (checking core first, then new), although the paper argues this is negligible or manageable. The method is specific to KWS and may not generalize easily to other incremental learning tasks without modification.
This work has significant practical impact for the deployment of always-on voice assistants on resource-constrained devices. By enabling safe, data-efficient updates to keyword inventories, it addresses a major pain point in product lifecycle management for IoT devices. It reduces the need for retraining on large datasets and mitigates the risk of degrading user experience for existing features. The paper presents a practical and effective method for modular expansion of embedded KWS models, achieving strict non-regulation and improved new-keyword detection through a frozen-base, parallel-branch architecture.
The trade-off between robustness, latency, and prosody critically challenges text-to-speech (TTS) systems. Autoregressive models, despite fidelity, are slow and error-prone; non-autoregressive (NAR) alternatives, while fast, often sacrifice prosodic naturalness via rigid alignments. This paper introduces StellarTTS, a novel mobile-optimized NAR TTS framework based on a sparse temporal embedding strategy, enabling granular control of phoneme duration, pronunciation, and prosody. Furthermore, we propose a semantic-aware codec that facilitates efficient single-stage decoding. Conditioned on the sparse temporal embedding, our 83M-parameter lightweight masked generative transformer achieves a real-time factor (RTF) of 0.08. Experiments demonstrate that StellarTTS attains lower latency and stronger robustness compared to state-of-the-art TTS systems, while maintaining competitive performance in audio quality, prosodic naturalness, and speaker similarity.
Primary: Honor Device Co., Ltd.
All Institutions: Honor Device Co., Ltd., Shanghai Jiao Tong University
StellarTTS presents a practical, mobile-optimized NAR TTS system that prioritizes robustness and latency through a novel sparse temporal embedding and semantic-aware codec, achieving significant efficiency gains at the cost of some speaker fidelity.
The paper proposes StellarTTS, a non-autoregressive (NAR) Text-to-Speech (TTS) framework optimized for mobile deployment. The core methodological contributions are twofold: (1) a "semantic-aware codec" that uses Residual Vector Quantization (RVQ) with the 0-th channel distilled from Wav2vec-Bert semantic features, aiming to unify semantic and acoustic token prediction into a single stage; and (2) "sparse temporal embedding," which replaces rigid length regulators with a strategy where phone embeddings are placed only at the center of the duration window, allowing the model to learn prosodic variations while maintaining structural anchors. The architecture is an 83M-parameter masked generative transformer. While the integration of semantic distillation into the codec is a practical engineering choice for latency reduction, the "sparse temporal embedding" is a relatively minor variation of existing alignment-free or soft-alignment techniques in masked generative models (e.g., MaskGCT, F5-TTS). The novelty lies more in the specific system integration for mobile constraints than in fundamental algorithmic innovation.
The experimental evaluation is comprehensive in terms of metrics but reveals a significant trade-off. The model achieves a Real-Time Factor (RTF) of 0.08, demonstrating high efficiency. It shows superior Word Error Rate (WER) on challenging datasets (test-hard), indicating robustness against error propagation common in AR models. However, the speaker similarity (SIM-o) is notably lower than state-of-the-art baselines (e.g., MaskGCT, CosyVoice2). The authors attribute this to the semantic distillation in the codec, which compresses acoustic detail. Subjective evaluations (CMOS/SMOS) show competitive naturalness but do not fully compensate for the objective similarity gap. The ablation studies support the efficacy of the sparse embedding strategy over random anchoring. The comparison with baselines is fair, but the lower SIM-o is a critical limitation for zero-shot TTS applications where speaker fidelity is paramount.
The paper provides sufficient architectural details, including the RVQ structure, loss functions (KL divergence for semantic distillation, MSE for duration prediction, cross-entropy for generation), and training hyperparameters (AdamW, learning rate schedule). The dataset (Emilia) and test sets (Seed-TTS) are standard. However, the specific implementation details of the "semantic-aware codec" training joint optimization and the exact configuration of the duration predictor are somewhat high-level. The code link provided is a demo page, not a GitHub repository, which hinders immediate reproducibility.
The primary limitation is the reduced speaker similarity (SIM-o) compared to SOTA models, which is a direct consequence of the single-stage semantic-aware codec design. The model's performance on very long-form synthesis is not explicitly evaluated, though the NAR nature suggests potential for consistency issues over long contexts. The reliance on a pre-trained semantic encoder (Wav2vec-Bert) for distillation adds an external dependency, though it is frozen during the main generation phase. The "sparse" embedding strategy, while effective for robustness, may struggle with extremely rapid prosodic changes that do not align well with the phone-centric anchor points.
This work contributes to the democratization of high-quality TTS on edge devices, which has significant implications for privacy-preserving voice assistants and low-latency communication tools. By demonstrating that robust, low-latency TTS is possible without sacrificing too much naturalness, it pushes the field towards more practical, deployable AI audio systems. However, the lower speaker similarity might limit its use in high-fidelity cloning applications. StellarTTS presents a practical, mobile-optimized NAR TTS system that prioritizes robustness and latency through a novel sparse temporal embedding and semantic-aware codec, achieving significant efficiency gains at the cost of some speaker fidelity.
Modern ASR models trained on heterogeneously annotated data treat transcription style (verbatim vs. intended) as an uncontrolled latent variable, causing measurable decoding instability, evaluation confounding (up to 60% of reported WER attributable to style mismatch), and unreliable word-level timing. We show that models already encode both styles; the challenge is controlled activation. Using coverage-aware decoder task tokens trained on parallel verbatim/intended transcript pairs, we raise German disfluency F1 from 10% to 79% zero-shot, despite English-only training. Full English-only fine-tuning surpasses all baselines in verbatim accuracy, disfluency detection, and intended-mode quality across both languages. We further introduce supervised cross-attention fine-tuning that improves word-level timestamps on disfluent speech beyond forced-alignment baselines. Finally, we propose verbatimize, a new task enabling scalable creation and enrichment of speech corpora with high-quality canonical verbatim transcriptions.
Primary: nyra labs
All Institutions: nyra labs
The paper presents a significant advancement in controllable ASR by explicitly parameterizing transcription policy, demonstrating that pretrained models possess latent capabilities for verbatim transcription that can be activated through targeted fine-tuning and explicit conditioning.
The paper proposes a novel framework for controlling transcription style (verbatim vs. intended) in Automatic Speech Recognition (ASR) models by treating the transcription policy as an explicit latent variable. The core technical contribution is the introduction of "mode tags" (discrete decoder-prefix tokens) that condition the model on the desired output style. This is complemented by a supervised cross-attention mechanism for word-level timing, which addresses the instability of emergent alignments in disfluent speech. Additionally, the authors introduce "verbatimize," a task for retroactively annotating verbatim transcripts from intended ones. The methodology is sound, leveraging the latent capabilities of pretrained Whisper models through efficient fine-tuning (Stage 1: embeddings only; Stage 2: decoder unfrozen). The approach to decoupling style control from acoustic modeling is innovative and addresses a significant gap in current ASR systems.
The experimental evaluation is comprehensive, covering disfluency detection, intended transcript quality, and word-level timing across English and German. The results demonstrate significant improvements: German disfluency F1 jumps from 10% to 79% in a zero-shot setting with only mode tag embeddings trained. The timing experiments show that the supervised cross-attention method outperforms strong baselines like MFA and WhisperX on disfluent speech (FluencyBank), achieving lower Mean Absolute Error (MAE). The ablation studies on head selection and sharpening techniques provide valuable insights into the mechanics of attention-based alignment. The use of a newly created German DisfluencySpeech dataset adds value, although the small size (202 utterances) limits the statistical power of the German results.
The paper provides detailed descriptions of the training stages, hyperparameters (learning rates, batch size), and data preprocessing steps. The code repository is linked, which enhances reproducibility. The use of standard benchmarks (TIMIT, FluencyBank, AMI) allows for comparison with existing work. However, the reliance on GPT-4o for generating intended transcripts and the creation of the German dataset by non-trained speakers introduces potential biases that are not fully quantified. The specific implementation details of the "virtual blank" pause detection and Viterbi decoding are described but might require careful implementation to replicate exactly.
The primary limitation is the reliance on paired verbatim/intended data for training, which is scarce. The authors mitigate this by using GPT-4o to generate intended transcripts from verbatim ones, which may not perfectly capture human annotation nuances. The German evaluation set is small and recorded by non-native speakers, potentially affecting the naturalness and generalizability of the results. The method is evaluated primarily on English and German; cross-lingual transfer to more distant languages is not thoroughly tested. Furthermore, the "verbatimize" task, while innovative, depends on the accuracy of the underlying intended transcript and the model's ability to correctly insert disfluencies, which may not always align with human perception of "verbatim" speech.
This work has significant implications for ASR systems used in clinical, legal, and linguistic contexts where verbatim transcription is crucial. By enabling controllable verbatim ASR, it opens up new possibilities for analyzing spontaneous speech, detecting speech disorders, and improving the quality of transcripts for accessibility. The "verbatimize" tool could facilitate the creation of large-scale annotated datasets, addressing a major bottleneck in speech research. However, the potential for misuse in generating realistic but fake verbatim transcripts (deepfakes) should be considered. The improvement in timing accuracy also benefits downstream tasks like speaker diarization and video captioning. The paper presents a significant advancement in controllable ASR by explicitly parameterizing transcription policy, demonstrating that pretrained models possess latent capabilities for verbatim transcription that can be activated through targeted fine-tuning and explicit conditioning.
Auditory attention decoding (AAD) identifies the speaker a listener attends to from neural responses like electroencephalography (EEG), making it a key algorithm in neuro-steered hearing aids. However, most neural AAD models are trained as independent short-window classifiers, despite auditory attention being a temporally persistent cognitive state and short-window EEG--audio evidence often being noisy and ambiguous. We propose an end-to-end Markov AAD framework based on conditional random field (CRF) that trains window-level neural emissions under a two-state attention prior. The framework treats the logits of any AAD backbone as Markov emissions, learns the transition rate from a standard HMM initialization, and jointly optimizes cross-entropy and CRF objectives, allowing temporal continuity to guide representation learning rather than merely smoothing predictions after training. We also introduce ESCNet, an EEG--speech correlation backbone that preserves time-aligned features and converts the difference between two mean Pearson correlations into state logits. We evaluate the framework with four emission backbones spanning correlation-based, convolutional, recurrent, and attention-based designs. On the dynamic AVGC dataset, CRF training generally outperforms post-hoc HMM smoothing; with ESCNet, it achieves $86.5\%$ causal and $92.4\%$ non-causal accuracy using $1$s windows. On the static KUL and USTC datasets, it improves causal decoding over fixed-rate post-hoc HMM baselines by $5.6\%$ and $2.0\%$, respectively, showing the superiority of learning AAD as attention state sequence over isolated-window classification.
Primary: University of Science and Technology of China
All Institutions: University of Science and Technology of China, Guangzhou Maritime University, iFLYTEK Company, Ltd.
The paper presents a robust and theoretically grounded advancement in Auditory Attention Decoding by integrating end-to-end CRF training with neural backbones, demonstrating consistent improvements in accuracy and responsiveness across multiple datasets and architectures.
The paper proposes a significant methodological shift in Auditory Attention Decoding (AAD) by moving from independent window classification to end-to-end Markov State Sequence Learning using Conditional Random Fields (CRF). The core innovation lies in treating the logits of any AAD backbone as emissions for a two-state CRF, allowing the temporal structure of attention (modeled via a transition prior) to guide the learning of neural representations directly, rather than just smoothing predictions post-hoc. The authors also introduce ESCNet, a specialized correlation-based backbone that preserves time alignment and outputs interpretable logits. This approach effectively bridges the gap between local feature extraction and global sequence coherence, addressing the inherent noise and ambiguity in short-window EEG data. The formulation is mathematically sound, leveraging standard forward-backward algorithms for differentiable sequence scoring.
The experimental evaluation is comprehensive and well-structured. The authors test their framework on three distinct datasets (AVGC, KUL, USTC) covering both dynamic attention switching and static persistence scenarios. They evaluate four heterogeneous backbones (ESCNet, AADNet, LSTM, Attn-GRU), demonstrating that the CRF training paradigm is architecture-agnostic and provides consistent improvements over fixed-rate post-hoc HMM baselines. The results show statistically significant gains, particularly in causal decoding on the dynamic AVGC dataset (86.5% accuracy with 1s windows). The inclusion of ablation studies isolating the effects of transition rate learning versus sequence-aware emission learning adds rigor to the claims. The switch delay analysis further validates the responsiveness of the model.
The paper provides detailed implementation details, including preprocessing steps, network architectures, hyperparameters (learning rates, batch sizes, optimizer settings), and training schedules. Crucially, the source code is made publicly available on GitHub, which significantly enhances reproducibility. The datasets used are public (AVGC, KUL, USTC), and the evaluation protocols (leave-one-trial-out) are clearly defined. The description of the CRF training phase, including the warm-up strategy and joint optimization, is sufficiently detailed for replication.
The primary limitation is the reliance on controlled laboratory datasets with normal-hearing participants. The performance on real-world, noisy, or hearing-impaired populations remains unverified. Additionally, the causal switch delay, while improved, is still around 23 seconds for the 1s window setting, which may be too slow for some real-time hearing aid applications requiring immediate adaptation. The study also assumes access to two candidate speech streams, which is a constraint for general multi-talker scenarios with more than two speakers. The static datasets (KUL, USTC) do not capture the complexity of naturalistic attention switching, potentially overestimating the robustness of the model in highly dynamic environments.
This work has significant implications for the development of neuro-steered hearing aids and brain-computer interfaces. By improving the accuracy and responsiveness of auditory attention decoding, it can enhance speech understanding in noisy environments for individuals with hearing loss. The end-to-end framework offers a generalizable template for incorporating temporal priors into other neural decoding tasks where state persistence is a key factor. The open-source release of the code and the detailed benchmarking contribute to the broader ML and neuroscience communities by providing a robust baseline for future research in EEG-based attention decoding. The paper presents a robust and theoretically grounded advancement in Auditory Attention Decoding by integrating end-to-end CRF training with neural backbones, demonstrating consistent improvements in accuracy and responsiveness across multiple datasets and architectures.
Almost every piece of recorded music you have ever heard was modified before it reached you; commercial releases rarely spring fully-formed from the mind of a musician. Despite the promise of music generation models for one-shot output, such fine-grained iterative refinement workflows are a complementary problem, and largely out of their reach. There is also a gap for musicians: while they can express what they want to hear, not all have the facility with studio production tools to implement the complex set of actions needed to realize these intuitions. We formalize this task as agentic post-production, wherein individual aspects of a song are targeted, refined, and combined into a final track. We argue the bottleneck is data: existing corpora do not reflect how realistic post-production chains map onto the vocabulary musicians and engineers actually use. We argue there is a language for modifying recorded music that is dense, consistent, and learnable. We introduce the Rule-based Instructions for Music Editing (RIME) framework, which generates realistic paired edit-instruction data from any baseline music dataset grounded in canonical methods, design patterns, and constraints derived from real production workflows. RIME leverages POEMS, a new toolkit that combines stem separation, mixing, and common studio effects for use by multimodal agents. We use RIME and POEMS to generate 3,000 pairs of edit instructions and ground truth audio, and use this data to evaluate existing multimodal LLMs as agents on this task, showing persistent challenges in current models' post-production capabilities. We also demonstrate RIME's ability to improve post-production agent performance via supervised fine-tuning. We see RIME as an early step toward iterative musical agents, collaborative systems that could transform music production much as interactive coding agents have reshaped software engineering.
Primary: Dartmouth College
All Institutions: Dartmouth College
This paper presents RIME and POEMS, a framework and toolkit for generating synthetic data and benchmarking agentic post-production agents, addressing the gap between one-shot music generation and iterative studio workflows.
The paper introduces RIME, a framework for generating synthetic edit-instruction data for music post-production, and POEMS, a toolkit exposing audio processing tools via the Model Context Protocol (MCP). The methodology is sound and addresses a specific gap: the lack of structured, iterative post-production data compared to one-shot generation data. The use of rule-based recipes, motifs, and parameter priors to generate grounded (input, output, instruction) triples is a clever approach to synthesizing training data. However, the novelty is somewhat limited by the reliance on existing, well-understood signal processing libraries (Pedalboard, Demucs) and standard LLM tool-calling architectures. The core contribution is the *system design* and *benchmark creation* rather than a new algorithmic breakthrough in audio synthesis or model architecture. The abstraction layer generation for prompts is a nice touch for simulating real-world ambiguity.
The experiments evaluate zero-shot and fine-tuned multimodal LLMs on the generated RIME benchmark. The results clearly demonstrate that current models struggle with agentic post-production, particularly in parameter inference and chain assembly. The supervised fine-tuning results show improvement, validating the utility of the synthetic data. The evaluation metrics (FAD, KAD, Edit Similarity, Graph F1) are appropriate for the task. The inclusion of an artifact-removal ("poisoning") split adds depth. However, the dataset size (3,000 pairs) is relatively small for modern LLM benchmarks, though reasonable for this specific, high-fidelity audio task. The reliance on MERT embeddings for audio similarity is standard but has known limitations in capturing perceptual quality compared to human listening tests, which are absent here.
The paper provides detailed descriptions of the POEMS toolkit, recipe schemas, and agent architecture. The use of open-weight models for fine-tuning and standard libraries for audio processing aids reproducibility. However, the specific implementation details of the "prompt templates" and the exact "parameter priors" are referenced but not fully enumerated in the main text (though likely in the appendix, which is partially provided). The lack of a public code repository URL in the provided text is a minor hindrance, but the description is sufficient for a competent researcher to reconstruct the system.
The primary limitation is the dependence on the quality of the underlying stem separation model (Demucs), which can introduce artifacts. The rule-based nature of RIME means it can only generate edits present in the recipe catalog, limiting the diversity of edits compared to fully generative approaches. The evaluation relies heavily on embedding-based metrics, which may not correlate perfectly with human preference for audio quality. The dataset size is small, potentially limiting the generalizability of the fine-tuned models. The focus on specific instrument stems (vocals, drums, bass, etc.) via Demucs limits the scope to tracks where these stems can be reliably separated.
This work has significant potential impact on the field of AI-assisted music production. By formalizing agentic post-production and providing a benchmark, it opens up a new avenue for research beyond one-shot generation. It highlights the challenges of iterative, tool-using agents in the audio domain. The framework could empower musicians with less technical expertise to achieve professional-quality mixes through natural language interaction. It also serves as a testbed for improving multimodal LLMs' reasoning and tool-use capabilities in a complex, structured domain. This paper presents RIME and POEMS, a framework and toolkit for generating synthetic data and benchmarking agentic post-production agents, addressing the gap between one-shot music generation and iterative studio workflows.
Multi-channel speech enhancement (SE) systems exhibit superior performance over single-channel methods but are constrained to fixed microphone array configurations. This restricts their real-world deployment across devices with diverse array geometries. While recent array-agnostic SE methods address variable microphone numbers and permutations, they largely fail to exploit explicit array geometry priors when available, missing a crucial cue for optimal spatial filtering. A Geometry-Aware Dynamic Convolution (Geo-DConv) framework is proposed, which explicitly leverages microphone coordinates to transform standard fixed-array SE models into robust array-invariant systems. Experiments are conducted on the recent real-recorded RealMAN multi-channel speech dataset. Results demonstrate that the proposed architecture enables two widely used fixed-array models to adapt to array-invariant settings, with consistent performance improvements across diverse array topologies.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University, VUI Labs
The paper presents a practical and effective method for adapting fixed-array speech enhancement models to arbitrary microphone geometries using explicit geometric priors via dynamic convolution, demonstrating improved generalization on real-world data.
The paper proposes a Geometry-Aware Dynamic Convolution (Geo-DConv) module to enable fixed-array speech enhancement models to handle arbitrary microphone array geometries. The core innovation lies in using a Topology-Aware Coordinate Transformer (TACT) to generate dynamic convolution weights based on relative microphone coordinates. This approach attempts to bridge the gap between fixed-array models (which exploit geometry implicitly through training) and array-agnostic models (which often ignore explicit geometry). The method is theoretically sound, leveraging Fourier positional encoding and self-attention to ensure permutation equivariance. However, the novelty is moderate as dynamic convolution and coordinate-based conditioning are established techniques in other domains (e.g., medical imaging, computer vision), and their application here is an incremental adaptation rather than a fundamental algorithmic breakthrough.
The experiments are conducted on the RealMAN dataset, which is a significant strength as it uses real-recorded data, addressing the common sim-to-real gap in multi-channel SE. The authors compare their method against strong baselines (FaSNet-TAC, USES2-comp, SpatialNet, TF-GridNet). The results show that Geo-DConv improves upon existing array-agnostic methods and allows fixed-array models to generalize to unseen array configurations. The evaluation includes metrics like SDR, SI-SDR, PESQ, STOI, and DNSMOS. The generalization to the CHiME-4 dataset (6 mics) is a positive finding. However, the performance gains, while consistent, are not massive, and the computational overhead of the TACT module is not extensively analyzed in terms of inference latency compared to the baseline fixed-array models.
The paper provides detailed architectural descriptions, including hyperparameters for the TACT module (hidden dimension, number of heads, layers) and the Geo-DConv layer (basis dimension, output channels). The dataset (RealMAN) is publicly available, and the implementation details (STFT parameters, sampling rate) are clear. The code is not explicitly linked, but the mathematical formulations are sufficient for reproduction by a competent researcher. The use of standard libraries for STFT and common neural network components further aids reproducibility.
The primary limitation is the assumption that microphone coordinates are known or can be accurately estimated. In real-world scenarios with unknown or dynamically changing arrays, this prior might not be available. Additionally, the method relies on a fixed basis dimension for dynamic convolution, which might limit the capacity to model highly complex or large-scale array geometries compared to methods that process raw channel interactions directly. The performance on very small arrays (1-2 mics) is still constrained by the fundamental limits of single-channel or stereo enhancement. The paper does not extensively discuss the sensitivity of the TACT module to coordinate normalization or scaling.
This work contributes to the broader goal of robust, device-agnostic audio processing. By enabling fixed-array models to adapt to various geometries, it reduces the need for device-specific training and deployment, facilitating the integration of advanced SE into diverse consumer devices (smartphones, laptops, IoT devices) with varying hardware configurations. It highlights the importance of explicit geometric priors in multi-channel audio tasks. The paper presents a practical and effective method for adapting fixed-array speech enhancement models to arbitrary microphone geometries using explicit geometric priors via dynamic convolution, demonstrating improved generalization on real-world data.
Synthesizing physically plausible impact sounds from visual observations remains a great challenge in multi-modal AI. Existing 3D-aware audio generation methods primarily model the surface geometry of hollow rigid bodies. However, they fundamentally overlook internal filling states, a critical physical factor that drastically modulates acoustic resonance and damping. To address this issue, we have defined a new task called Fine-Grained Filling-Aware Impact Sound Generation. As a foundational step, we first introduce the fine-grained fill-aware dataset (FillImpact), a pioneering multi-modal collection comprising over 5,000 rigorous acoustic recordings from 88 diverse real-world objects. It captures impact interactions with varying internal contents (i.e., water, rice), a continuous range of fill levels, and distinct striker materials. Furthermore, comprehensive acoustic analysis confirms that the collected data closely aligns with established physical laws governing acoustic resonance and damping, indicating its suitability for physically grounded modeling. Building on this dataset, we propose a novel generative framework (FillGauss) that integrates 3D Gaussian Splatting (3DGS) with internal state conditioning for sound generation. By fusing 3DGS geometric features, precise 3D spatial strike coordinates, and fine-grained textual physical conditions within a latent diffusion architecture, FillGauss enables position-aware, striker-aware, and filling-aware audio generation. Extensive experiments demonstrate that our approach could generate high-fidelity impact sounds that adhere to underlying physical principles, establishing a new state-of-the-art for physically grounded cross-modal audio generation.
Primary: Unknown (Affiliations not provided in text)
All Institutions: Unknown
This paper introduces FillGauss, a novel framework for fine-grained, filling-aware impact sound generation that leverages 3D Gaussian Splatting and latent diffusion to synthesize physically plausible audio conditioned on internal object states, supported by the pioneering FillImpact dataset.
The paper proposes FillGauss, a framework for fine-grained, filling-aware impact sound generation using 3D Gaussian Splatting (3DGS). The methodology integrates three modalities: 3DGS geometric features, precise 3D strike coordinates, and textual descriptions of internal states (fill material and level). The core technical contribution lies in the cross-modal attention fusion mechanism, which entangles spatial strike context with internal physical semantics before conditioning a Latent Diffusion Transformer (DiT). The use of 3DGS as a geometric prior for audio generation is a novel application, moving beyond traditional mesh or point-cloud representations. However, the underlying diffusion architecture (TangoFlux) is a pre-existing model, and the primary innovation is in the conditioning pipeline and dataset construction rather than a new generative backbone. The approach of using text prompts to encode physical states (fill level, material) is effective but relies heavily on the semantic capabilities of the pre-trained language model, which may limit fine-grained control compared to continuous numerical conditioning.
The experimental setup is robust, centered on the introduction of the FillImpact dataset, which contains over 5,000 recordings from 88 objects with varying internal states. This dataset is a significant contribution, addressing a gap in existing literature where internal filling states are largely ignored. The evaluation includes objective metrics (FAD, KL, IS) and subjective human perceptual studies (MOS, Win Rate, Attribute Matching). The results demonstrate that FillGauss outperforms baselines (SonicGauss, TangoFlux) in generating physically plausible sounds, particularly in capturing the acoustic differences between liquid and granular fills. The ablation studies effectively validate the importance of the cross-attention modules and the text-prompting strategy. The sensitivity analysis regarding 3DGS sparsity provides valuable insights into the model's reliance on geometric detail.
The paper provides detailed descriptions of the data collection pipeline, including the use of specific hardware (DJI Mic, Revo Scan) and software (Hunyuan 3D, Stable Audio Open VAE). The normalization procedures for 3DGS and strike coordinates are clearly defined. The training details, including the use of Huber loss and the specific architecture of the DiT, are described. However, as the code and dataset are not explicitly linked in the provided text (URLs are "none"), reproducibility depends on the authors' future release of these assets. The description of the dataset construction is sufficiently detailed for another group to replicate the data collection process.
The paper acknowledges that extreme sparsification of 3DGS degrades performance, indicating a dependency on high-fidelity geometric reconstruction. The model's ability to handle complex internal dynamics like sloshing fluids or soft-body deformations is limited to static fill levels and materials. The reliance on text prompts for internal state conditioning might introduce semantic ambiguities or lack the precision of direct numerical inputs for continuous variables like fill level. Furthermore, the subjective evaluation, while rigorous, is limited by the number of participants (20) and the specific test set.
This work significantly advances the field of physically grounded multimodal AI, particularly for applications in virtual reality, gaming, and robotic simulation where accurate auditory feedback is crucial for immersion and task performance. By explicitly modeling internal states, it enables more realistic interactions with a wider variety of objects. The FillImpact dataset serves as a valuable resource for future research in acoustic physics and multimodal learning. The approach could be extended to other physical properties, such as temperature or material degradation, further enriching the realism of synthetic environments. This paper introduces FillGauss, a novel framework for fine-grained, filling-aware impact sound generation that leverages 3D Gaussian Splatting and latent diffusion to synthesize physically plausible audio conditioned on internal object states, supported by the pioneering FillImpact dataset.
Real-time speech-to-speech translation (S2ST) systems must balance translation quality, latency, speech naturalness, and speaker consistency. Publicly documented S2ST systems have advanced direct, multilingual, streaming, and expressive modeling, while proprietary products and APIs increasingly expose real-time translation capabilities to users. However, practical deployment remains challenging for open and reproducible systems, especially in long-form and multi-speaker conversations where partial ASR hypotheses are unstable, turn boundaries are ambiguous, and target speech must be generated with an appropriate speaker prompt. We present X-Translator, a low-cost modular cascaded S2ST system that combines streaming ASR, machine translation, and prompt-conditioned TTS through a session-level runtime controller. The system uses incremental segment commitment to convert unstable ASR streams into translation-ready units, and an online speaker prompt manager to bind source speech spans to speaker-specific voice prompts for synthesis. We evaluate translation, speech quality, and latency with OpenSTBench, compare against proprietary speech translation APIs as behavioral baselines, measure long-form voice stability, evaluate speaker preservation in multi-speaker conversations, and assess multilingual translation quality. X-Translator provides an open platform for understanding the practical trade-offs of deployment-oriented S2ST. Code and demo are available at https://github.com/zhaoyx239/X-Translator.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University, Shanghai Innovation Institute, Microsoft, AISpeech Co., Ltd.
X-Translator presents a practical, modular, and open-source framework for real-time, multi-speaker speech-to-speech translation, offering valuable insights into the system-level engineering challenges of balancing latency, quality, and speaker consistency in deployment-oriented scenarios.
The paper proposes X-Translator, a modular, cascaded Speech-to-Speech Translation (S2ST) system designed for real-time, long-form, and multi-speaker scenarios. The core technical contribution lies not in novel model architectures (e.g., new ASR or TTS backbones) but in the system-level runtime controller. Specifically, it introduces an "incremental segment commitment" layer to handle unstable streaming ASR hypotheses and an "online speaker prompt manager" to maintain speaker consistency in multi-speaker conversations. The methodology is pragmatic, leveraging existing state-of-the-art open-source components (Qwen3-ASR, LMT-60-8B, X-Voice) and orchestrating them via explicit runtime policies. While the engineering is sound and addresses real deployment pain points (latency, speaker drift, partial hypothesis instability), the novelty is incremental relative to the broader S2ST literature, which has seen significant advances in direct, end-to-end models (e.g., Translatotron, SeamlessM4T). The paper positions itself as a complementary "deployment-oriented" perspective rather than a fundamental algorithmic breakthrough.
The evaluation is comprehensive and well-structured, covering short-form benchmarks (OpenSTBench), long-form single-speaker voice stability, multi-speaker preservation, and multilingual translation. The use of proprietary APIs (Doubao, Qwen LiveTranslate, GPT Realtime Translate) as behavioral baselines is a strong point, providing a realistic comparison against current industry standards. The metrics are appropriate, including translation quality (COMET, BLEU), speech quality (UTMOS), and specific S2ST metrics like speaker similarity (Spk. SIM) and purity measures for multi-speaker settings. The results show that X-Translator achieves competitive speaker preservation and translation quality, though it lags behind proprietary systems in latency and sometimes translation accuracy. The analysis of trade-offs between commitment aggressiveness and quality/latency is insightful. However, the reliance on open-source backbones means the system's performance is bounded by those components, and the "low-cost" aspect comes at the expense of the peak performance seen in proprietary, heavily optimized black-box systems.
The paper provides a high level of reproducibility. The code and demo are available on GitHub. The modular design allows other researchers to swap out components (ASR, MT, TTS) and test different runtime policies. The detailed description of the segment commitment triggers and speaker prompt management logic is clear. The evaluation protocol is well-documented, including the specific datasets (TEDLIUM-3, WenetSpeech, VoxConverse, FLEURS) and the metrics used. The release of runtime logs for error analysis further enhances reproducibility and interpretability.
The paper acknowledges several limitations. First, as a cascaded system, it suffers from error propagation (ASR errors affect MT and TTS) and accumulated latency. Second, the speaker-aware mechanism depends on the quality of online speaker tracking and the availability of clean prompt audio; overlapping speech or rapid turn-taking can degrade performance. Third, the comparison with proprietary systems is behavioral; the internal optimizations of commercial APIs (e.g., joint optimization, larger models, caching) are not visible, making a direct architectural comparison difficult. Finally, the system's performance is limited by the underlying open-source models, which may not match the quality of proprietary counterparts.
X-Translator provides a valuable open-source platform for studying real-time S2ST deployment challenges. By exposing the intermediate decisions (segment commitment, prompt routing), it facilitates research into latency-quality trade-offs and speaker consistency. It democratizes access to real-time S2ST capabilities, allowing researchers to build upon and improve upon existing systems. The focus on multi-speaker and long-form scenarios addresses a significant gap in current S2ST research, which often focuses on short, single-speaker utterances. This work encourages the development of more robust, interpretable, and reproducible speech translation systems. X-Translator presents a practical, modular, and open-source framework for real-time, multi-speaker speech-to-speech translation, offering valuable insights into the system-level engineering challenges of balancing latency, quality, and speaker consistency in deployment-oriented scenarios.
Expressive speech synthesis for voice assistants requires flexible style control that adapts to explicit requests and broader interaction context. We propose Harness TTS, a lightweight control layer that wraps around a TTS engine to externalize and govern its expressive behavior. It reformulates style control as closed-set prompt-tool routing: offline, a compact registry of stylistic prompt tools is constructed with structured metadata; online, an LLM planner selects the appropriate tool based on a priority-aware observation schema, and the TTS executor synthesizes speech using the corresponding prompt audio. We evaluate Harness TTS on both routing and synthesis tasks. In routing, Qwen3-4B achieves Top-1 accuracies of 74.3%, 43.0%, and 64.6% on explicit, implicit, and conflict subsets. For synthesis, experiments on CosyVoice3 and VoxCPM2 show that Harness TTS outperforms instruction-only control, achieving higher instruction-following win rates (margins of 23.1-35.6 points on CosyVoice3 and 13.8-20.0 points on VoxCPM2) and improving UTMOSv2 scores by 0.11-0.38. Moreover, the 4B planner delivers its first tool recommendation in under 50 ms in standard mode, introducing negligible latency for real-time interaction. These results demonstrate that equipping TTS engines with a dedicated Harness layer offers a practical, auditable, and context-aware solution for voice assistant expression control.
Primary: Nanjing University
All Institutions: Nanjing University, MiLM Plus, Xiaomi Inc., WeNet Open Source Community
Harness TTS presents a practical and well-structured approach to stabilizing expressive TTS in voice assistants by externalizing style selection into an LLM-driven routing layer, demonstrating significant improvements in instruction following and naturalness over direct instruction methods, though its reliance on synthetic evaluation and closed-set tools limits its generalizability and open-ended expressiveness.
The paper proposes "Harness TTS," a control layer that wraps around existing Text-to-Speech (TTS) engines to manage expressive speech synthesis. The core innovation lies in reformulating style control as a closed-set tool routing problem. Instead of relying on continuous acoustic parameters or free-form instruction embeddings which can be unstable, the system uses a structured registry of "tools" (audio prompts) with metadata. An LLM planner observes multi-source context (user profile, scenario, explicit instructions) and selects the most appropriate tool based on a priority-aware schema. This approach separates the decision-making logic (Harness) from the generative engine (TTS), aiming for improved stability, interpretability, and context-awareness. The methodology is sound and addresses a genuine gap in making expressive TTS robust for voice assistants, particularly regarding conflict resolution between implicit context and explicit user intent.
The evaluation is comprehensive, covering both the routing accuracy of the LLM planner and the synthesis quality of the end-to-end system. The routing experiments demonstrate that Qwen3-4B can effectively select tools, with particularly strong performance on explicit instructions (74.3% Top-1) and reasonable performance on implicit (43.0%) and conflict (64.6%) scenarios. The synthesis experiments on CosyVoice3 and VoxCPM2 show significant improvements in instruction-following win rates compared to direct instruction conditioning, validating the utility of the Harness layer. The latency analysis is also a strong point, showing that First-ID latency is low enough for real-time interaction. However, the reliance on LLM-generated data for evaluation datasets and LLM-based judges (Gemini-2.5-Pro) introduces potential bias and limits the generalizability of the results to human perception.
The paper provides detailed descriptions of the tool registry structure, observation schema, and priority policies. The use of standard LLMs (Qwen3 series) and TTS engines (CosyVoice3, VoxCPM2) aids reproducibility. However, the specific prompts, few-shot examples, and the exact construction of the synthetic evaluation datasets are not fully detailed, which might make exact replication difficult. The lack of code release (no project URL provided) further hinders immediate reproducibility.
The authors acknowledge several limitations: the use of CoT increases latency, the evaluation relies on LLM teachers/judges rather than human annotations, and the test data is synthetically generated rather than from real-world logs. Additionally, the closed-set nature of the tool registry limits the expressiveness to pre-defined styles, which might not cover all edge cases or novel user requests. The dependency on high-quality pre-recorded prompt audio for the registry also poses a scalability and curation challenge.
This work has significant potential for improving the user experience of voice assistants by making them more natural and contextually appropriate. By providing a stable and auditable interface for style control, it could lower the barrier for developers integrating expressive TTS into applications. However, the reliance on LLMs for decision-making raises concerns about cost, latency, and potential biases in style selection. The closed-set approach also limits the open-ended creativity of the TTS system. Harness TTS presents a practical and well-structured approach to stabilizing expressive TTS in voice assistants by externalizing style selection into an LLM-driven routing layer, demonstrating significant improvements in instruction following and naturalness over direct instruction methods, though its reliance on synthetic evaluation and closed-set tools limits its generalizability and open-ended expressiveness.
Recent audio large language models (ALLMs) are typically built upon audio encoders trained with large amounts of supervised data. Since self-supervised learning (SSL) audio encoder models are known to learn general-purpose and transferable representations, we investigate whether general-purpose SSL audio representations can serve as an effective foundation for ALLMs. We present SALMONN-2, an ALLM built upon a unified SSL encoder. To better exploit the hierarchical representations learned by SSL encoders, we propose a multi-layer feature fusion (MLF) adapter that aggregates information from all encoder layers before projecting them into the language model. Beyond conventional audio understanding tasks, we further explore multimodal in-context learning (MICL) in ALLMs and study how this capability can be acquired through contextual biasing training. Experimental results show that a general-purpose SSL encoder achieves performance comparable to, or better than, specialised supervised audio encoders while providing a more balanced capability across speech, audio, music and paralinguistic tasks. SALMONN-2 further achieves state-of-the-art performance among comparable-scale open-weight models on ALLM understanding benchmarks, obtaining the best results on MMAU-Pro, MMAR and MMSU. We also show that MICL does not emerge naturally in ALLMs, but can be effectively acquired through targeted contextual biasing training.
Primary: Shanghai Artificial Intelligence Laboratory
All Institutions: University of Cambridge, Shanghai Artificial Intelligence Laboratory, Tsinghua University
SALMONN-2 presents a compelling argument for the use of unified self-supervised audio representations in Audio Large Language Models, achieving state-of-the-art performance with greater efficiency and architectural simplicity than previous multi-encoder approaches. The introduction of the Multi-Layer Feature Fusion adapter and the systematic study of Multimodal In-Context Learning provide valuable technical contributions and insights for the broader field of multimodal AI.
The paper proposes SALMONN-2, an Audio Large Language Model (ALLM) built upon a unified Self-Supervised Learning (SSL) encoder (SPEAR). The core methodological contributions are: 1) The adoption of a single, general-purpose SSL encoder (SPEAR) as a unified front-end for diverse audio tasks (speech, music, environmental sound), challenging the prevalent dual-encoder or supervised-encoder paradigms. 2) A Multi-Layer Feature Fusion (MLF) adapter that aggregates hidden states from all layers of the SSL encoder before projection, aiming to exploit the hierarchical nature of SSL representations (low-level acoustic to high-level semantic). 3) A timestamp injection mechanism using textual tokens from the LLM vocabulary to provide explicit temporal grounding without extra parameters. 4) Explicit training for Multimodal In-Context Learning (MICL) in the context of Contextual ASR, demonstrating that this capability does not emerge naturally from standard instruction tuning. The approach is technically sound, leveraging recent advances in unified SSL models (SPEAR) and applying them effectively to the ALLM paradigm. The MLF adapter is a relatively simple but effective architectural choice compared to more complex cross-attention or Q-Former styles.
The experimental evaluation is comprehensive and rigorous. The authors evaluate SALMONN-2 on a wide range of canonical tasks (ASR, AAC, Music Captioning, Emotion Recognition, Speech Translation, Overlapped Speech Recognition, Speaker Verification) and general audio understanding benchmarks (MMAU-Pro, MMAR, MMSU). They also include extended audio analysis tasks (Sound Event Detection, Spoofing Analysis, Speech Quality Assessment) and a novel MICL evaluation for Contextual ASR. The results show that SALMONN-2 achieves state-of-the-art performance among comparable-scale open-weight models, often outperforming models with larger LLM backbones or more complex encoder setups. The ablation studies effectively isolate the contributions of the SSL encoder, the MLF adapter, and timestamp injection. The MICL experiments provide valuable insights into the limitations of current ALLMs and how targeted training can address them. The use of less than 20K hours of instruction-tuning data compared to competitors is a significant efficiency claim supported by the results.
The paper provides detailed implementation details, including the architecture of the SPEAR encoder, the MLF adapter, the LLM backbones (Qwen series), and the training stages. The code and model are open-sourced on GitHub, which significantly aids reproducibility. The dataset construction for MICL is described, though the specific processing of some external datasets (e.g., HoliAntiSpoof, AudioSet-strong) might require careful attention to licensing and availability. The training hyperparameters and data mix are summarized in tables, providing a clear recipe for replication.
The paper acknowledges that scaling the LLM backbone from 9B to 30B-A3B did not yield substantial improvements on canonical tasks, suggesting these tasks may be perception-heavy rather than reasoning-heavy. However, it does not deeply explore the failure modes of the SSL encoder on out-of-distribution audio or the potential biases introduced by the SSL pre-training data. The MICL capability is demonstrated primarily on Contextual ASR; its generalizability to other multimodal in-context learning scenarios (e.g., audio QA with few-shot examples) is less explored. The reliance on SPEAR, while effective, ties the model to a specific SSL architecture; the generalizability of the MLF adapter to other SSL models (e.g., WavLM, HuBERT) is not extensively tested, although the principles are likely transferable.
SALMONN-2 contributes to the development of more efficient and general-purpose audio intelligence systems. By demonstrating that a single SSL encoder can replace specialized supervised encoders, it reduces the computational and data overhead required to build ALLMs. The explicit handling of multimodal context has implications for robust speech recognition in noisy or domain-specific environments. The open-sourcing of the model and code facilitates further research in audio-language models. Potential negative impacts include the misuse of powerful audio understanding models for surveillance or deepfake detection evasion, though the model's focus on understanding rather than generation mitigates some generation-specific risks. SALMONN-2 presents a compelling argument for the use of unified self-supervised audio representations in Audio Large Language Models, achieving state-of-the-art performance with greater efficiency and architectural simplicity than previous multi-encoder approaches. The introduction of the Multi-Layer Feature Fusion adapter and the systematic study of Multimodal In-Context Learning provide valuable technical contributions and insights for the broader field of multimodal AI.
Neural audio codecs with residual vector quantization (RVQ) normally treat all frequencies uniformly, so their codebooks become spectrally entangled. Truncating stages then removes an unpredictable mix of frequencies. Parallel band decomposition addresses this by splitting audio into independent bands, but fragments the latent space and loses cross-frequency coherence. We introduce HARP (Harmonic-Aware Residual Partitioning), a training strategy that partitions RVQ stages into frequency-ordered groups where each group refines its target band while the decoder retains access to all lower frequencies. Overtones are reconstructed in the context of their fundamentals, preserving coherence that parallel methods lose. HARP requires no architectural changes; it only modifies the training loss, leaving inference identical to standard RVQ. On speech, music, and general audio, HARP outperforms both standard RVQ and parallel decomposition. MUSHRA listening tests also show perceptual improvements.
Primary: Georgia Institute of Technology
All Institutions: Georgia Institute of Technology, The Chinese University of Hong Kong, Tencent Music Entertainment
HARP introduces a training-only strategy for neural audio codecs that partitions RVQ stages into frequency-ordered groups with cumulative decoding and soft subband supervision, effectively addressing spectral entanglement and improving both objective metrics and perceptual quality, particularly at low bitrates, without architectural changes.
The paper proposes HARP, a training-time strategy for Residual Vector Quantization (RVQ) in neural audio codecs. The core innovation lies in partitioning RVQ stages into frequency-ordered groups and enforcing spectral specialization through a novel loss function. Key components include: 1) Cumulative Decoding: Ensuring higher-frequency stages are reconstructed in the context of lower-frequency content to preserve harmonic coherence. 2) Subband Contribution Supervision: Supervising the isolated contribution of each group to its target band using stop-gradient operators to prevent gradient interference. 3) Soft Band Weighting: Using learnable Gaussian weights over mel bins to guide specialization without hard spectral cutoffs. The approach is architecturally agnostic, requiring only modifications to the training objective. This is a clever and relatively simple modification that addresses the "spectral entanglement" problem inherent in standard RVQ, where early and late stages capture similar frequency content.
The authors evaluate HARP against standard DAC (same architecture, no band supervision) and BSCodec (parallel band decomposition). They use a diverse dataset including MUSDB18-HQ, LibriTTS, and AudioSet. Metrics include SI-SDR, KAD, PESQ, STOI, and MUSHRA listening tests. Results show consistent improvements in SI-SDR and KAD over both baselines, with particularly notable gains at low bitrates (2.6 kbps). The ablation studies effectively isolate the contributions of cumulative decoding, soft boundaries, and stop-gradient mechanisms. The harmonic coherence analysis using synthetic tones provides strong evidence for the preservation of phase/amplitude relationships. The MUSHRA tests confirm perceptual improvements, especially in bass fullness and timbral naturalness.
The paper provides detailed descriptions of the architecture, training objectives, and hyperparameters. The code is made publicly available on GitHub. The use of standard components (DAC backbone, EMA updates) and clear algorithmic descriptions enhances reproducibility. The training cost increase (2-3x) is noted, which is a practical consideration but does not hinder reproducibility.
The primary limitation is that the performance gains, while consistent, are modest in absolute terms (e.g., +0.25 to +0.96 dB SI-SDR). The method relies on the assumption that natural audio has a specific energy distribution (low-to-high), which might not hold for all types of audio (e.g., highly inharmonic or noise-heavy content, though the paper argues it still applies). The method is training-only, so it does not offer architectural efficiency gains. The "soft" boundaries mean that some spectral leakage or imperfect separation might still occur, potentially limiting the maximum potential of the specialization compared to hard architectural splits (though the paper argues this is offset by coherence).
This work contributes to the field of neural audio compression by providing a principled way to impose spectral structure on RVQ without architectural complexity. This can lead to more robust and interpretable audio codecs, which are foundational for speech synthesis, music generation, and audio understanding tasks. The improved low-bitrate performance is particularly relevant for streaming applications. The open-source release facilitates further research in this area. HARP introduces a training-only strategy for neural audio codecs that partitions RVQ stages into frequency-ordered groups with cumulative decoding and soft subband supervision, effectively addressing spectral entanglement and improving both objective metrics and perceptual quality, particularly at low bitrates, without architectural changes.
Singing Quality Assessment (SQA) has become increasingly important for practical multimedia applications and Music AI systems, yet existing studies predominantly focus on short singing clips and remain insufficient for full-length songs. Unlike clip-level assessment, full-length song SQA requires modeling how singing quality varies across different audio segments and how these local variations influence the overall evaluation of vocal performance. Moreover, the scarcity of segment-level annotations makes effective supervision challenging, as directly assigning a single overall score label to every segment tends to treat different segment qualities as equivalent. To address these challenges, we propose SongSQA, a two-stage framework for full-length song SQA. In the first stage, a Segment Score Predictor is trained with pseudo labels generated by a pre-trained teacher model, enabling segment-level singing quality prediction without requiring manual segment annotations. In the second stage, a Song Quality Aggregator integrates segment features and predicted segment scores into unified segment embeddings, and employs a learnable song embedding together with self-attention to capture the connection between segment-level vocal performance and overall song quality. In this way, SongSQA dynamically aggregates critical quality cues across the song to produce a holistic quality prediction, while also generating a temporal segment-level quality curve. Experimental results demonstrate the effectiveness of SongSQA for full-length song SQA, achieving up to a 13.95% relative improvement in KTAU over the strongest baseline, while consistently improving other evaluation metrics across all datasets.
Primary: The Chinese University of Hong Kong, Shenzhen
All Institutions: Xi'an Jiaotong University, The Chinese University of Hong Kong, Shenzhen
[One sentence main contribution]. SongSQA introduces a two-stage framework for full-length song singing quality assessment that utilizes pseudo-labels from a teacher model for segment-level supervision and a Transformer-based aggregator to capture non-linear segment interactions. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a solid technical contribution to the field of audio quality assessment by shifting the paradigm from clip-level to full-song evaluation. The proposed methodology is innovative in its handling of the annotation scarcity problem through a teacher-student pseudo-labeling strategy, which is a clever adaptation of semi-supervised learning principles to a specific perceptual task. The integration of segment-level predictions into a global context via self-attention allows the model to mimic human perceptual weighting of vocal highlights and flaws. The experimental results are strong and well-supported by ablation studies and subjective evaluations. While the novelty is incremental in terms of architectural components (using Transformers and SSL features is common), the specific application to full-song SQA with temporal curves and the pseudo-labeling strategy for segment supervision is a valuable addition to the literature. The work is technically sound and addresses a clear gap in current SQA systems.
The paper proposes SongSQA, a two-stage framework for full-length song singing quality assessment (SQA). The core methodological innovation lies in addressing the lack of segment-level annotations by using a pre-trained teacher model to generate pseudo-labels for a Segment Score Predictor. This predictor extracts segment-level features and scores, which are then fused and aggregated by a Song Quality Aggregator using a Transformer with a learnable [CLS]-like token. The approach effectively bridges the gap between clip-level SQA and full-song assessment. The use of pseudo-labels from a specialized teacher (SongEval) rather than simple label sharing is a pragmatic and technically sound solution to the data scarcity problem. The aggregation mechanism using self-attention to model the non-linear influence of segments on overall quality is well-motivated by human perception studies.
The experimental evaluation is robust, utilizing two datasets: the public Lyra-SA dataset and a proprietary Internal Dataset annotated by experts. The paper compares SongSQA against strong baselines including clip-level SOTA models (UTMOSv2, PS-SQA, RAMP+) and SSL-based full-song adaptations. SongSQA achieves significant relative improvements (up to 13.95% in KTAU) over the strongest baseline. The ablation studies are thorough, validating the contributions of the teacher model choice, feature fusion strategy, and aggregation design. The case study and subjective evaluation further support the perceptual validity of the temporal score curves generated by the model. The results on the Internal Dataset show even higher performance, suggesting the model benefits from high-quality expert annotations.
The paper provides detailed implementation details, including dataset splits, segment duration (10s with 5s overlap), model architecture parameters (number of layers, loss weights), and optimizer settings. The use of publicly available datasets (Lyra-SA) and pre-trained models (MuQ, SongEval) enhances reproducibility. However, the Internal Dataset is proprietary, which limits the ability to fully replicate the results on that specific benchmark. The code is not explicitly linked in the text provided, though a demo page is available.
The primary limitation is the reliance on a pre-trained teacher model for pseudo-label generation. The quality of the final model is thus bounded by the teacher's ability to discern segment-level singing quality. If the teacher model has biases or errors, these will propagate to the student model. Additionally, the performance on the proprietary Internal Dataset, while impressive, cannot be directly compared to other works due to the lack of public access. The model's performance on non-Mandarin/Cantonese singing (only a small proportion in the internal dataset) is not thoroughly evaluated, limiting generalizability claims.
This work has significant implications for Music AI, particularly in the evaluation of Singing Voice Synthesis (SVS) and Singing Voice Conversion (SVC) systems. By providing a tool for fine-grained, full-song assessment, it enables more precise feedback for model improvement and content moderation on multimedia platforms. The generated temporal score curves can also be used for music education and analysis. The methodology of using pseudo-labels for segment-level supervision in audio quality assessment could be adapted to other domains lacking granular annotations. [One sentence main contribution]. SongSQA introduces a two-stage framework for full-length song singing quality assessment that utilizes pseudo-labels from a teacher model for segment-level supervision and a Transformer-based aggregator to capture non-linear segment interactions. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a solid technical contribution to the field of audio quality assessment by shifting the paradigm from clip-level to full-song evaluation. The proposed methodology is innovative in its handling of the annotation scarcity problem through a teacher-student pseudo-labeling strategy, which is a clever adaptation of semi-supervised learning principles to a specific perceptual task. The integration of segment-level predictions into a global context via self-attention allows the model to mimic human perceptual weighting of vocal highlights and flaws. The experimental results are strong and well-supported by ablation studies and subjective evaluations. While the novelty is incremental in terms of architectural components (using Transformers and SSL features is common), the specific application to full-song SQA with temporal curves and the pseudo-labeling strategy for segment supervision is a valuable addition to the literature. The work is technically sound and addresses a clear gap in current SQA systems.
End-to-end speech language models increasingly represent user speech with speech tokens rather than relying exclusively on cascaded ASR--LLM--TTS pipelines. Although these tokens support expressive and low-latency spoken interaction, they may also preserve sensitive speaker characteristics. We investigate whether exposed speech tokens leak voiceprints and formulate this risk as a speaker inversion attack. We introduce Audio BERT (AuB), a trainable model that constructs token embeddings from discrete codebooks and aggregates them into speaker-sensitive representations, and propose SpInv, a two-stage inversion method built on AuB to recover embeddings in the space of an attacker-specified speaker encoder. We evaluate Moshi, Higgs3, Kimi-Audio, and Qwen3-Omni using speaker-disjoint protocols on the VoxCeleb dataset. Extensive experiments show that, with only three seconds of frontend output, SpInv achieves cosine similarities above 0.70 in the attacker-specified speaker-encoder space.
Primary: Harbin Institute of Technology
All Institutions: Harbin Institute of Technology
The paper presents a significant and timely contribution to the intersection of speech processing and machine learning security. By formulating and executing a speaker inversion attack against modern end-to-end speech language models, it exposes a critical privacy vulnerability in emerging AI architectures. The proposed SpInv method is technically robust, leveraging advanced distillation and contrastive learning techniques to recover speaker embeddings from discrete tokens with high fidelity. The comprehensive evaluation across multiple state-of-the-art models and rigorous experimental protocols strengthens the validity of the findings. This work is essential reading for researchers and practitioners in speech AI, highlighting the urgent need for privacy-preserving design patterns in next-generation speech interfaces.
The paper proposes a novel threat model called "Speaker Inversion Attack" (SIA) against end-to-end speech language models (SLMs). The core methodological contribution is SpInv, a two-stage attack framework. Stage A uses a distillation pretraining approach with an auxiliary model called Audio BERT (AuB) to map discrete speech tokens (or continuous features) to a target speaker embedding space defined by a frozen encoder (e.g., ECAPA-TDNN). This involves alignment loss, relational loss, and self-distillation (DINO-style) to prevent collapse and preserve geometry. Stage B fine-tunes the model with discriminative supervision (ArcFace) to enhance speaker discrimination. The approach is technically sound, leveraging established contrastive and distillation techniques (DINO, VICReg) in a new context. The mapping from discrete tokens to continuous embeddings via a learned lookup table is a clever adaptation to handle the black-box nature of the tokenizers.
The evaluation is comprehensive and rigorous. The authors test against four state-of-the-art SLM frontends (Moshi, Higgs3, Kimi-Audio, Qwen3-Omni), covering different tokenization strategies (residual codebooks, hybrid, continuous). They use speaker-disjoint protocols on VoxCeleb1 and VoxCeleb2, which is crucial for validating generalization. The results are strong: cosine similarities >0.70 are achieved with only 3 seconds of audio. The ablation studies on input duration and target speaker encoder spaces add depth. The fact that they test on "Qwen3-Omni" (which appears to be a very recent or potentially misnamed model in the context of current public releases, possibly referring to Qwen-Audio or a specific internal variant, but treated as a valid target here) suggests the work is timely. The metrics (CosSim, EER, Acc) are appropriate for the task.
The paper provides detailed descriptions of the AuB architecture, loss functions, and training hyperparameters (learning rates, batch sizes, loss weights). The datasets (VoxCeleb1/2) and target encoders (ECAPA-TDNN, etc.) are publicly available. The threat model assumes query access to the frontend, which is a standard and realistic assumption for API-based services. The code is not explicitly linked, but the methodological description is sufficient for reproduction by a competent researcher.
The authors acknowledge two main limitations: focusing on embedding recovery rather than waveform reconstruction, and using only VoxCeleb datasets. The reliance on VoxCeleb, while standard, may not fully capture the diversity of "in-the-wild" speech (accents, noise, channel variations) that might be present in real-world SLM deployments. Additionally, the attack requires a training phase where the adversary collects data from the target frontend; while feasible, this is a resource-intensive prerequisite. The paper does not explore defense mechanisms, only the attack.
This work has significant implications for the privacy of users interacting with AI voice assistants and SLMs. It demonstrates that even when raw audio is not transmitted, the intermediate representations (tokens) can leak sensitive biometric information (voiceprints). This necessitates the development of privacy-preserving tokenization techniques or anonymization methods for speech tokens. It serves as a wake-up call for the SLM community to consider privacy in the design of frontend tokenizers. The paper presents a significant and timely contribution to the intersection of speech processing and machine learning security. By formulating and executing a speaker inversion attack against modern end-to-end speech language models, it exposes a critical privacy vulnerability in emerging AI architectures. The proposed SpInv method is technically robust, leveraging advanced distillation and contrastive learning techniques to recover speaker embeddings from discrete tokens with high fidelity. The comprehensive evaluation across multiple state-of-the-art models and rigorous experimental protocols strengthens the validity of the findings. This work is essential reading for researchers and practitioners in speech AI, highlighting the urgent need for privacy-preserving design patterns in next-generation speech interfaces.
Interpretability is critical in clinical decision support. Concept bottleneck frameworks improve it by representing inputs as human-understandable concepts and restricting predictions solely on them. However, research on their use for voice-based health assessment remains limited. In this study, we propose a voice concept bottleneck framework for interpretable health assessment using an audio language model (ALM). The ALM is fine-tuned on a voice quality assessment dataset to enhance its understanding of voice concepts and serves as an independent concept extractor, producing discrete, interpretable scores for a lightweight downstream classifier. The discrete concept scores provide intuitive interpretation, while the lightweight classifier facilitates post-hoc interpretability analyses. Results on depression and dysarthria assessment tasks demonstrate that the proposed framework can flexibly adapt voice concepts to different health conditions and consistently outperforms openSMILE-based and self-supervised speech model-based baselines.
Primary: Columbia University
All Institutions: Columbia University
This paper presents a compelling application of Audio Language Models as interpretable concept bottlenecks for health assessment, demonstrating that decoupling concept extraction from classification can yield both high performance and clinical interpretability, although validation on larger, more diverse datasets is needed to confirm robustness.
The paper proposes a novel framework that leverages Audio Language Models (ALMs), specifically Audio Flamingo Next, as independent concept extractors for interpretable health assessment. The core innovation lies in decoupling the concept extraction (via fine-tuned ALM producing discrete 1-5 ordinal scores) from the downstream classification (via XGBoost). This approach aims to mitigate shortcut learning and enhance interpretability by restricting predictions to human-understandable concepts (Clinical Voice Quality and Speaking Behavior). The methodology is sound in its intent, utilizing LoRA for efficient fine-tuning and aggregating speaker-level statistics for the classifier. However, the novelty is somewhat incremental, as concept bottleneck models (CBMs) are an established paradigm, and applying them to audio via LLMs is a logical but not radically new extension. The specific adaptation of ALMs to produce discrete ordinal concepts rather than continuous embeddings or text descriptions is a notable technical choice.
The experimental evaluation covers two distinct health conditions: depression (Android Corpus) and dysarthria (TORGO corpus). The results demonstrate that the proposed ALM-based framework outperforms baselines using openSMILE and self-supervised models (Vox-Profile). The paper provides SHAP analysis to validate interpretability, showing that features like pitch variability and speech tempo align with clinical expectations for depression. For dysarthria, the separation of CVQ scores across severity levels is shown. While the performance gains are consistent, the baselines are relatively standard for the domain. The comparison with Vox-Profile is fair, but the lack of comparison with other recent SSL-based CBMs or end-to-end interpretable models limits the comprehensiveness of the benchmark. The results are promising but rely on small datasets, particularly for dysarthria (only 15 speakers total), which raises questions about generalizability.
The paper provides sufficient detail regarding the model architecture (Audio Flamingo Next), fine-tuning parameters (LoRA rank, alpha, dropout), and downstream classifier settings (XGBoost hyperparameters). The datasets are publicly available, and the voice activity detection tool is cited. However, the code is not explicitly linked in the provided text, and specific random seeds or exact data splitting procedures beyond the standard protocols are not detailed. The reliance on a specific Hugging Face model checkpoint aids reproducibility, but the fine-tuning process on the PVQD dataset requires careful handling of the discrete label generation which is described but could be more precise.
The primary limitation is the small sample size of the evaluation datasets, especially the dysarthria corpus, which limits the statistical power of the results. The ALM's pretraining bias is acknowledged and partially corrected via fine-tuning, but the paper notes that the model still struggles with certain concepts like abnormal loudness and pitch due to definition mismatches. The discrete 1-5 scale, while interpretable, may lose nuance compared to continuous ratings. Furthermore, the "independent" nature of the concept extractor assumes that the fine-tuned ALM captures the true underlying concepts without leakage from the downstream task, which is hard to guarantee without rigorous ablation studies on concept fidelity.
This work has significant potential impact in clinical decision support systems by providing interpretable voice biomarkers. By making AI predictions transparent through human-understandable concepts, it can help clinicians trust and validate AI outputs. The framework could be extended to other health conditions affecting voice, such as Parkinson's disease or Alzheimer's. However, ethical considerations regarding privacy, bias in health assessment, and the risk of over-reliance on automated diagnostics must be addressed. The use of ALMs also raises questions about the environmental cost and accessibility of such models in resource-constrained clinical settings. This paper presents a compelling application of Audio Language Models as interpretable concept bottlenecks for health assessment, demonstrating that decoupling concept extraction from classification can yield both high performance and clinical interpretability, although validation on larger, more diverse datasets is needed to confirm robustness.
We propose the concept of noise-aware audio self-supervised learning (SSL), whose goal is to encode audio mixtures while suppressing undesired noise, and present Noise-Aware BEATs (NABEATs) as a BEATs-based realization of this framework. Audio SSL models are designed to handle a wide range of audio signals. Consequently, under noisy conditions, they cannot effectively focus on the target sounds relevant to a downstream task, resulting in degraded performance. To address this issue, NABEATs is trained to estimate clean BEATs representations from a noisy audio signal with an auxiliary reference noise input. This reference noise enables the model to account for specific noise characteristics at inference time, thereby achieving better generalization across operating environments. Our experimental evaluations demonstrate that NABEATs significantly improves performance of various downstream tasks under noisy conditions and also generalizes well to unseen noise types.
Primary: Mitsubishi Electric Research Laboratories (MERL)
All Institutions: Mitsubishi Electric Research Laboratories (MERL), Nagoya University
This paper presents a practical and effective method for enhancing the noise robustness of audio self-supervised learning models by conditioning representation learning on reference noise, demonstrating significant improvements in downstream task performance under both seen and unseen noisy conditions.
The paper proposes NABEATs, a noise-aware adaptation of the BEATs self-supervised learning (SSL) model. The core innovation lies in conditioning the representation learning process on a reference noise signal. The authors introduce two mechanisms to integrate this reference noise: Cross-Attention (CA) and Feature-wise Linear Modulation (FiLM). The model is trained via distillation from a frozen, pre-trained BEATs model, aiming to estimate clean representations from noisy inputs. This approach is theoretically sound and addresses a specific gap in general-purpose audio SSL, which typically lacks robustness to environmental noise. The distinction between using a reference noise (NABEATs) versus unconditional denoising (DBEATs) is clearly defined. However, the method is essentially an architectural modification to an existing SSL framework rather than a fundamentally new learning paradigm. The use of reference noise is analogous to techniques in speech enhancement and speaker verification, making the novelty incremental rather than transformative.
The experimental setup is rigorous and well-structured. The authors evaluate NABEATs on six diverse downstream tasks (environmental sound, speech, music) under various noisy conditions using multiple noise datasets (WHAM!, CHiME-3, MUSDB18). The inclusion of an "unseen noise" generalization test (MUSDB18) is a strong point, demonstrating the practical utility of the reference noise mechanism. The results show consistent improvements over the baseline BEATs and the unconditional denoising baseline (DBEATs), particularly in unseen noise conditions. The use of t-SNE visualizations and SNR analysis of representations provides good qualitative and quantitative support for the claims. The evaluation on the DCASE 2025 Task 2 benchmark adds real-world relevance, although the performance gain is modest. The ablation between CA and FiLM variants provides useful insight into how reference information is utilized.
The paper provides sufficient detail for reproduction. It specifies the base model (BEATs_iter3.pt), the additional layer architectures (Transformer layers with MHSA/MHCA, RMSNorm, SwiGLU), training hyperparameters (AdamW, learning rate, batch size, epochs), and dataset sources (FSD50K, WHAM!, DEMAND, QUT-NOISE). The training procedure for distillation is clearly described. However, the code is not publicly linked in the provided text, and some details regarding the exact mixing ratios and SNR distributions for the downstream evaluation splits could be more precise. The use of standard datasets and a widely available base model enhances reproducibility.
The primary limitation is the dependency on a reference noise signal. While the authors argue this is practical in many scenarios (e.g., pre-recorded noise), it is not universally available. The method does not address the case where no reference noise is present. Furthermore, the performance gains, while statistically significant, are not massive, and the model introduces additional complexity and computational overhead compared to the base BEATs. The reliance on MSE loss for distillation may not perfectly align with downstream classification objectives, as noted in the discussion of the CRM-D task. The evaluation is limited to 16kHz audio, which may not reflect high-fidelity audio applications.
This work contributes to the development of robust audio AI systems that can operate effectively in real-world, noisy environments. By improving the generalization of SSL models, it facilitates the deployment of audio understanding systems in diverse settings, from smart homes to industrial monitoring. The concept of noise-aware SSL could inspire similar approaches in other modalities where background interference is a major challenge. However, the potential for misuse in creating robust surveillance or eavesdropping systems is a minor concern, though mitigated by the requirement for reference noise. This paper presents a practical and effective method for enhancing the noise robustness of audio self-supervised learning models by conditioning representation learning on reference noise, demonstrating significant improvements in downstream task performance under both seen and unseen noisy conditions.
Audio mixing style encompasses the artistic and technical decisions a mix engineer makes, including level balancing, spatialization, and the choice, ordering, and parameterization of audio effects (FX) on each stem. FX chains are a key determinant of this style, yet existing approaches to modeling them remain limited. Some operate on stereo mixtures without explicit per-stem FX chain modeling, others fix the number or type of effects per track, and many require differentiable effect implementations or scarce multitrack datasets. We present StemFX, a framework that learns mixing style representations by autoregressively predicting variable-length FX chains on source-separated stems. A Transformer decoder predicts tokenized FX chains autoregressively, while a band-split multi-band CNN encoder with FiLM conditioning captures per-stem spectral structure. To enable large-scale paired training, we extract pseudo-stems from about 105K songs via source separation and augment them using MultiAFx, a toolkit unifying 85 audio effects from 7 Python libraries. Evaluated on mixing style retrieval, StemFX outperforms all baseline models across all tested chain lengths. On paired mixing style transfer, StemFX achieves the best spectral fidelity and the highest listener preference, over 4000 times faster than iterative optimization.
Primary: National Taiwan University
All Institutions: National Taiwan University
This paper presents a significant advancement in automatic mixing and audio representation learning by introducing a scalable, autoregressive framework for FX chain prediction that overcomes data scarcity through a novel pseudo-stem generation pipeline, achieving state-of-the-art performance in both style retrieval and transfer tasks.
The paper proposes StemFX, a novel framework for learning mixing style representations by treating FX chain prediction as an autoregressive sequence generation problem. The architecture combines a band-split multi-band CNN encoder with FiLM conditioning (incorporating hand-crafted mixing features) and a Transformer decoder. A key methodological innovation is the "StemFX Pipeline," which leverages source separation on large-scale unpaired datasets (FMA) to create pseudo-stems, which are then augmented with random FX chains using a unified toolkit (MultiAFx) to generate paired training data at scale. This approach circumvents the scarcity of real multitrack datasets with ground-truth FX metadata. The tokenization scheme for variable-length FX chains (effect names, parameter names, and quantized values) is well-designed for autoregressive modeling. The use of FiLM conditioning to inject explicit mixing statistics into the CNN layers is a strong technical choice that bridges the gap between low-level spectral features and high-level mixing decisions.
The experimental evaluation is comprehensive and robust. The authors evaluate on two tasks: mixing style retrieval and paired mixing style transfer. For retrieval, StemFX significantly outperforms baselines (AFx-Rep, Fx-Encoder++, CLAP, and a controlled contrastive variant BSFiLM-CL) across various chain lengths, demonstrating that the learned embeddings capture mixing style distinct from musical content. The ablation studies effectively isolate the contributions of the encoder architecture, FiLM conditioning, and the data pipeline. The style transfer experiments are particularly compelling, showing superior spectral fidelity (MRSTFT) and human preference (MUSHRA scores) compared to fixed-chain methods and iterative optimization techniques (ST-ITO). The claim of being 4000x faster than iterative optimization is a significant practical advantage. The use of out-of-domain effects (pedalboard) in evaluation tests generalization well.
The paper provides detailed implementation details, including hyperparameters, dataset sizes, and architecture specifications. The release of both the StemFX codebase and the MultiAFx toolkit on GitHub greatly enhances reproducibility. The description of the data generation pipeline, including source separation models and augmentation strategies, is clear enough for other researchers to replicate the training data generation process. The use of standard metrics (MRSTFT, MUSHRA) also aids in reproducibility of evaluation.
The authors acknowledge several limitations. The model is restricted to the 85 effects included in the MultiAFx toolkit; adding new effects requires retraining. The reliance on source separation introduces potential artifacts from the separation model (e.g., SCNet), which may propagate into the learned representations. The use of pseudo-stems from different songs (cross-song mixing) may introduce musical incoherence, although the authors argue this decorrelates content from style. The model currently operates on four stems (vocals, bass, drums, other), which is a coarser decomposition than professional multitracks. Finally, the training data uses random effect chains, which may not reflect the structured, genre-dependent processing choices of human engineers.
StemFX has significant potential impact on music production technology. By enabling efficient and accurate mixing style transfer and retrieval, it can assist producers in learning mixing techniques, automate parts of the mixing workflow, and enhance music information retrieval systems. The open-source release of the toolkit and models promotes further research in audio effect modeling and automatic mixing. The ability to generate interpretable FX chains also provides insights into the decision-making process of mix engineers, potentially serving as an educational tool. This paper presents a significant advancement in automatic mixing and audio representation learning by introducing a scalable, autoregressive framework for FX chain prediction that overcomes data scarcity through a novel pseudo-stem generation pipeline, achieving state-of-the-art performance in both style retrieval and transfer tasks.
In X-lingual automatic speaker verification (ASV), fixed front-end scores vary in reliability with language match, duration, and score source. We propose AMECxSV, an adaptive metadata-driven embedding-fusion calibration backend for metadata-available settings. AMECxSV fuses trial scores with metadata to produce calibrated target posteriors, with optional posterior-confidence abstention; metadata serve as calibration context, not speaker evidence. On a development-derived speaker-disjoint held-out split, score+metadata heads reduce equal error rate (EER) from 3.15% to 2.42% for the official TidyVoice score source and from 0.64% to 0.43% for LI-MSV; the dual-score head reaches 0.43% full-coverage EER. At 0.79 coverage, abstention yields 0.03% accepted-trial EER, not a full-coverage metric. Matched score-only, metadata-permutation, and metadata-only controls support a calibration-context interpretation and limit claims to metadata-available scoring.
Primary: University of Southern California
All Institutions: University of Southern California, Signal Analysis and Interpretation Laboratory (SAIL), University of Southern California
AMECxSV presents a practical and effective metadata-driven calibration framework that improves the reliability of X-lingual speaker verification systems by adaptively adjusting scores based on language and duration context, demonstrating significant gains in calibration quality and error rates through rigorous ablation and control studies.
The paper proposes AMECxSV, a metadata-driven calibration backend for X-lingual Automatic Speaker Verification (ASV). The core methodology involves fusing trial scores from fixed embedding front-ends with metadata (language match, duration) using a lightweight Multi-Layer Perceptron (MLP). The approach is framed as "embedding-fusion calibration," where metadata serves as context for score reliability rather than direct speaker evidence. The method includes a full-coverage variant (AMEC-FC) and a confidence-based abstention variant (AMEC-ABS). The technical contribution is relatively incremental, building upon established logistic/isotonic calibration and multi-system fusion techniques by adding a structured metadata input to an MLP. While the specific application to X-lingual reliability is well-motivated, the architectural novelty is limited to the feature engineering (score summaries, order statistics, interactions) and the specific conditioning on language/duration.
The evaluation is conducted on the TidyVoiceX-ASV development protocol, using a speaker-disjoint held-out split. The authors compare AMECxSV against single-score baselines, global calibration methods, and a multi-score fusion baseline (MultiScore-FC). Results show consistent improvements in Equal Error Rate (EER) and $C_llr$ when metadata is added. The paper includes rigorous controls, such as metadata permutation and matched score-only architectures, to validate that gains are due to the metadata content and not just model capacity. External evaluations on TidyVoice and LI-MSV score sources further support the method's generalizability. The statistical significance is addressed via bootstrap replicates. However, the results are primarily on development splits, and the magnitude of improvement, while consistent, is modest in absolute terms (e.g., 3.15% to 2.42% EER on one source).
The paper provides detailed implementation details, including the architecture of the MLP, training hyperparameters (AdamW, learning rates, batch sizes), and the specific metadata features used. The codebase is linked via an anonymous repository. The dataset split is clearly defined as development-derived, which aids reproducibility but limits direct comparison to official challenge results. The use of fixed, publicly available embedding systems (SpeechBrain, WeSpeaker, etc.) ensures that the front-end components are reproducible.
The primary limitation is the dependency on metadata. The system requires language match and duration information, which may not be available or reliable in all deployment scenarios (e.g., language-blind protocols). The authors acknowledge that language metadata might need to be predicted by a separate LID module, introducing potential error propagation. Additionally, the improvements are source-dependent, and the method does not address the underlying representation learning issues of the front-ends. The abstention mechanism, while effective for reducing risk, reduces coverage, which may not be desirable for all applications.
This work contributes to the robustness and reliability of ASV systems, particularly in multilingual contexts. By providing a framework for adaptive calibration, it helps mitigate the performance degradation caused by language mismatch and short utterances. This can lead to more fair and accurate speaker verification in real-world applications where trial conditions are heterogeneous. The emphasis on calibration and abstention also promotes safer deployment by allowing the system to defer uncertain decisions. AMECxSV presents a practical and effective metadata-driven calibration framework that improves the reliability of X-lingual speaker verification systems by adaptively adjusting scores based on language and duration context, demonstrating significant gains in calibration quality and error rates through rigorous ablation and control studies.
Current voice AI benchmarks typically evaluate isolated capabilities such as speech intelligibility, word error rate, or text-based dialogue quality, but they rarely test whether systems harness the acoustic information that distinguishes spoken language from its textual representation. To this end, we introduce the Real World Voice EQ Bench, a multidimensional benchmark for evaluating voice AI across text-to-speech (TTS), speech-to-speech (STS), speech understanding (SU), and automatic speech recognition (ASR). Our evaluations indicate that performance is highly dimension-specific. For TTS, naturalness, expressiveness, identity stability, and reliability are largely independent evaluation dimensions. For STS, access to audio does not guarantee use of vocal affect, and some agents remain largely transcript-driven. For SU, models perform unevenly across paralinguistic tasks. For ASR, real world accent, emotion, noise, and conversational conditions expose failures that are not captured by established clean-speech benchmarks. Together, these results show that voice AI should be evaluated as a profile of acoustic, expressive, interactional, and robustness capabilities rather than by a single aggregate score.
Primary: Hume AI Research
All Institutions: Hume AI Research
The paper presents RW-Voice-EQ Bench, a rigorous, multidimensional benchmark that reveals the dimension-specific nature of voice AI performance, exposing critical gaps in current evaluation practices such as benchmark optimization in ASR and the underutilization of paralinguistic cues in STS systems.
The paper introduces a comprehensive, multidimensional benchmarking framework called RW-Voice-EQ Bench, designed to evaluate voice AI systems across four distinct domains: Text-to-Speech (TTS), Speech-to-Speech (STS), Speech Understanding (SU), and Automatic Speech Recognition (ASR). The methodology is rigorous, separating linguistic content from paralinguistic acoustic cues. For perceptual tasks (TTS, STS), it employs large-scale human evaluation (over 1 million ratings) using specific rubrics and mixed-effects models to control for rater bias. For objective tasks (SU, ASR), it uses label-based scoring. A significant methodological contribution is the diagnostic suite for "benchmark optimization" (or "benchmaxxing") in ASR, which tests for memorization via consensus disagreement, audio masking, orthographic switching, and synthetic voice probes. The paper also validates Speech-Language Models (SLMs) as judges against human raters, providing a nuanced analysis of where automated judges succeed (verification-dependent tasks) and fail (open-ended perceptual tasks).
The experimental evaluation is extensive, covering 31 TTS configurations, 15 STS configurations, and analyzing top ASR models on the Hugging Face leaderboard. The results clearly demonstrate that performance is highly dimension-specific; no single model dominates all categories. For instance, in TTS, models strong in expressiveness (e.g., Gemini 3.1 Flash) differ from those strong in reliability or long-form stability (e.g., Kokoro-82M). In STS, the paper highlights that many systems remain largely transcript-driven, failing to leverage vocal affect even when present. The ASR analysis provides compelling evidence that high WER scores on clean benchmarks do not correlate with robustness to accents, noise, or emotional speech, and that many top models exhibit signs of overfitting to benchmark artifacts. The statistical analysis using mixed-effects models and Spearman correlations adds robustness to the findings.
The paper provides significant detail on the evaluation setup, including the number of raters, rating scales, and the specific dimensions evaluated. The benchmark and judge leaderboards are hosted on Hugging Face Spaces, ensuring accessibility. However, the full dataset of prompts and the specific code for the mixed-effects modeling pipeline are not explicitly linked in the provided text, though the Hugging Face links suggest open resources. The detailed description of the "benchmark optimization" diagnostics allows for replication of those specific tests.
The primary limitation is the reliance on human evaluation for TTS and STS, which is resource-intensive and potentially subject to demographic biases, although the paper attempts to mitigate this with large sample sizes and rater filtering. The STS evaluation, while innovative, is limited to specific scripted scenarios (hostile, urgent, ambiguous tone) and may not capture the full breadth of spontaneous conversational dynamics. The ASR benchmark optimization analysis is preliminary, focusing on VoxPopuli, and the authors acknowledge the need for future work to expand this to other datasets. Additionally, the SLM judge validation is currently limited to TTS, leaving STS judge reliability less explored.
This paper has substantial impact on the voice AI community by challenging the reliance on single aggregate scores and clean-speech benchmarks. It provides a roadmap for more realistic and robust evaluation of voice systems, which is critical for deployment in real-world applications. By exposing benchmark optimization in ASR, it encourages the development of more generalizable models and the use of withheld test sets. The multidimensional profile approach helps developers select models based on specific use-case requirements (e.g., customer service vs. entertainment), driving more targeted research and development. The paper presents RW-Voice-EQ Bench, a rigorous, multidimensional benchmark that reveals the dimension-specific nature of voice AI performance, exposing critical gaps in current evaluation practices such as benchmark optimization in ASR and the underutilization of paralinguistic cues in STS systems.
We introduce the REAL-TSE Challenge, an IEEE SLT 2026 satellite challenge on target speaker extraction~(TSE) from real conversational recordings. Given a multi-speaker mixture and one or more enrollment utterances from a target speaker, participating systems must recover only the target speech. Unlike simulated read-speech benchmarks, REAL-TSE evaluates Mandarin and English recordings that contain natural overlap, reverberation, noise, channel mismatch, and conversational dynamics. The challenge defines two complementary tracks: an Online track for low-latency streaming extraction and an Offline track for full-context processing. Systems are evaluated with Token Error Rate (TER), Speaker Similarity (SpkSim), DNSMOS, and target-speaker activity F1. This overview paper describes the task definition, datasets, baselines, evaluation protocol, submitted systems, condition-wise findings, and lessons for future real-world TSE benchmarks.
Primary: Nanjing University
All Institutions: Nanjing University, Chinese University of Hong Kong (Shenzhen), Brno University of Technology, Northwestern Polytechnical University, Shanghai Jiao Tong University, NTT, Inc.
The REAL-TSE Challenge establishes a critical new benchmark for real-world Target Speaker Extraction, revealing significant gaps between simulated and real-world performance and highlighting the dangers of metric over-optimization in neural audio evaluation.
This paper introduces the REAL-TSE Challenge, a benchmarking framework rather than a novel algorithmic architecture. The methodological contribution lies in the rigorous definition of a real-world Target Speaker Extraction (TSE) task. It distinguishes itself from prior simulated benchmarks (like LibriMix) by utilizing real conversational recordings with natural overlap, reverberation, noise, and channel mismatch. The challenge is structured into two tracks: Online (low-latency, <100ms) and Offline (full-context). The evaluation methodology is multi-dimensional, employing Token Error Rate (TER), Speaker Similarity (SpkSim), DNSMOS-P808, and Target-Speaker Activity F1. A critical methodological insight presented is the detection of metric over-optimization, specifically regarding DNSMOS-OVRL, leading to the adoption of DNSMOS-P808 for ranking. The paper details the data simulation strategies used by participants (pseudo-labeling, real-data adaptation) and analyzes the impact of channel mismatch and scenario types on performance.
The experimental section provides a comprehensive analysis of 24 systems from 18 teams. It includes baseline results (BSRNN variants) and aggregate performance metrics. The paper conducts a detailed condition-wise analysis on the EVAL-2 set, examining performance across different recording devices (H1, H2, Phone), target-speaker activity ratios, and recording scenarios (meeting, cafe, home, vehicle). Key findings include the significant impact of mixture recording device quality, the lack of consistent correlation between nominal scenario labels and difficulty, and the trade-offs between different metrics (e.g., systems optimizing for SpkSim may suffer in TER). The analysis of DNSMOS reliability is a significant experimental contribution, demonstrating how neural metrics can be gamed without perceptual improvement.
The challenge promotes reproducibility and transparency. The authors release the evaluation toolkit, baseline code (WeSep-based), and the dataset splits (Development and Evaluation). The training policy is open but restricted regarding specific test-set leakage (forbidden to use dev/test splits of source corpora for training). The paper provides detailed descriptions of the latency verification process (perturbation-based) and metric computation. However, as a challenge overview, it does not provide full training code for the winning systems, which is standard for such venues, but it does release the infrastructure for future replication.
The primary limitation is the inherent difficulty of real-world evaluation: the lack of ground-truth clean targets for most real recordings makes absolute performance measurement challenging, relying instead on proxies like ASR and speaker similarity. The paper acknowledges that metric-aware over-optimization is a significant issue, suggesting that current automatic metrics may not fully align with human perception or true extraction quality. Additionally, the "unseen" nature of EVAL-2 means that generalization to entirely new, unrecorded environments remains an open question, though the challenge design attempts to mitigate this by using diverse, newly recorded data.
This work has a substantial impact on the speech processing community by shifting the focus from simulated to real-world TSE evaluation. It highlights the gap between lab-performance and real-world deployment, emphasizing the need for robustness to channel mismatch, noise, and conversational dynamics. The findings on metric reliability (DNSMOS) have broader implications for the entire field of audio quality assessment and neural metric usage. By providing a standardized benchmark and revealing the limitations of current approaches, it guides future research towards more practical, multi-objective optimization and better data simulation techniques. The REAL-TSE Challenge establishes a critical new benchmark for real-world Target Speaker Extraction, revealing significant gaps between simulated and real-world performance and highlighting the dangers of metric over-optimization in neural audio evaluation.
Machine-learnt models are increasingly used to predict ISO 3382-1 room acoustic parameters from sparse measurements, with reported coefficients of determination frequently above 0.85. This paper shows that such figures are often determined by the evaluation protocol rather than by the model. Using a multi-condition measurement campaign in a 264-seat conference hall and a 180-seat concert hall, three model families were evaluated under a factorial protocol ablation: validation splits either row-based or grouped by receiver position, and input features either including measured-at-test quantities or restricted to source-receiver geometry and environmental state. Row-based splits with measured-at-test inputs reproduce the high reported accuracies (mean $R^2$ 0.81 for the core parameters); grouping the splits by position and restricting inputs to information available at an unmeasured position reduces these to 0.09-0.57, reordering the apparent difficulty of parameter classes. A hybrid CNN evaluated with the target's own impulse response as input is shown to exploit it as a position fingerprint rather than as transferable acoustic information; training-only signal access yields no gain for any parameter tested, including reverberation time. Under the deployment-consistent protocol, the spread between Random Forest, the hybrid CNN, and inverse-distance weighting is an order of magnitude smaller than the spread between protocols for a fixed model; the learnt models retain a genuine advantage for sound strength and reverberation time, and the high accuracy of the original pipelines re-emerges as condition interpolation at measured positions (band means 0.80-0.88), a distinct and operationally useful task.
Primary: Alanya Alaaddin Keykubat University
All Institutions: Alanya Alaaddin Keykubat University
This paper provides a critical methodological audit of machine learning in room acoustics, demonstrating that reported high accuracies are often artifacts of evaluation protocols rather than model capability, and establishes a rigorous framework for distinguishing between condition interpolation and spatial prediction.
The paper employs a rigorous factorial ablation study to dissect the evaluation protocols of machine learning models in room acoustics. It introduces a taxonomy distinguishing between "condition interpolation" (predicting parameters at measured positions under varying environmental states) and "spatial prediction" (predicting parameters at unmeasured positions). The methodology involves re-evaluating existing pipelines (Random Forest, Hybrid CNN) under strict "deployment-consistent" protocols where validation folds are grouped by receiver position and inputs are restricted to geometry and environmental state. A key methodological insight is the demonstration that neural networks can exploit the target position's own impulse response as a "position fingerprint" rather than learning transferable acoustic features, particularly in venues with few measurement positions. This approach effectively isolates the source of high reported accuracy, revealing it to be largely an artifact of data leakage via correlated rows and privileged inputs.
The experimental design is robust, utilizing a multi-condition measurement campaign in two distinct halls (a concert hall and a conference hall). The authors systematically vary split designs (row-based vs. grouped) and input features (full vs. geometry-only). The results are compelling: while row-based splits with full inputs yield high $R^2$ scores (0.81), the deployment-consistent protocol reduces these significantly (0.09-0.57), with simple baselines like inverse-distance weighting often matching or exceeding complex neural networks for certain parameters (e.g., clarity). The paper provides a clear reordering of parameter difficulty, showing that sound strength and reverberation time are more predictable spatially than clarity parameters, which depend on fine-grained early reflection structures not captured by the available geometric features. The inclusion of a "privileged information" experiment further strengthens the argument by showing that training-time signal access does not confer a significant advantage for spatial prediction.
The paper demonstrates high commitment to reproducibility. The author explicitly states that evaluation scripts, split definitions, random seeds, and per-fold results will be deposited in a public repository. The underlying measurement data is available upon request. The detailed description of the model architectures, feature sets, and experimental protocols allows for independent verification. The clear distinction between the two evaluation modes and the provision of specific $R^2$ values for each configuration enhance the transparency of the findings.
The primary limitation is the scope of the data: the study is based on a single multi-condition campaign in only two halls. While the findings are likely generalizable to similar indoor environments, they may not hold for outdoor acoustics or highly complex, non-diffuse fields. The author acknowledges this and suggests cross-venue replication is needed. Additionally, the study focuses on specific ISO 3382-1 parameters; the generalizability to other acoustic metrics or perceptual qualities is not addressed. The network architecture, while effective for the task, is not novel in itself, and the paper's contribution lies in the evaluation rather than the model design.
This paper has significant implications for the field of data-driven acoustics and spatial audio. By exposing the fragility of high accuracy claims in sparse measurement scenarios, it urges the community to adopt more rigorous evaluation standards. The proposed reporting checklist provides a practical framework for future research, ensuring that claims about spatial prediction are substantiated by appropriate protocols. This work helps prevent the dissemination of misleading performance metrics and guides the development of more robust and truly generalizable models for room acoustic prediction. It also highlights the enduring value of simple baselines and the importance of understanding what models are actually learning. This paper provides a critical methodological audit of machine learning in room acoustics, demonstrating that reported high accuracies are often artifacts of evaluation protocols rather than model capability, and establishes a rigorous framework for distinguishing between condition interpolation and spatial prediction.
Recent advances in text-to-speech and voice cloning make high-quality spoofing inexpensive and scalable, threatening voice authentication systems, especially automatic speaker verification (ASV). Existing defenses mainly address this threat through binary countermeasures (CMs) for deepfake detection or spoofing-aware speaker verification (SASV), where current systems are dominated by modular ASV-CM fusion and cascaded pipelines. Although large audio language models (LALMs) have shown promise on related audio tasks, including CM and ASV, their use for SASV remains unexplored, despite their capacity to produce natural-language rationales for auditing and robustness beyond discriminative predictions. This work systematically evaluates LALMs for SASV against conventional pipelines under zero-shot prompting, supervised adaptation, reasoning-oriented training, and reinforcement-learning-based optimization. Our results show that pretrained LALMs are near chance in the zero-shot setting, confirming that they are not natively suited to SASV, but that task-specific adaptation closes this gap. We further find that competitive SASV performance can be achieved through several distinct routes. These findings position LALMs as a promising and auditable foundation for unified SASV, while clarifying where conventional cascade systems still lead.
Primary: Applied AI Institute
All Institutions: Applied AI Institute, MIRAI, HSE, MTUCI
This paper provides the first systematic evaluation of Large Audio Language Models for Spoofing-Aware Speaker Verification, demonstrating that while they can be adapted to compete with conventional pipelines, they currently lag in threshold-based metrics and require significant task-specific adaptation to overcome their lack of native security awareness.
The paper proposes a novel adaptation of Large Audio Language Models (LALMs) for Spoofing-Aware Speaker Verification (SASV), a task that traditionally relies on cascaded or fused discriminative pipelines (ASV + CM). The authors formulate SASV as a three-way classification problem (target, non-target, spoof) using LALMs like SALMONN and Qwen-Audio. Key methodological contributions include: 1) A composite loss function combining cross-entropy, Additive Angular Margin (AAM) for speaker discrimination, and binary cross-entropy for spoof detection to balance the competing objectives. 2) Hard-sample mining to focus training on difficult pairs. 3) An exploration of Chain-of-Thought (CoT) reasoning and Group Relative Policy Optimization (GRPO) to generate interpretable rationales for decisions. The approach is technically sound, leveraging parameter-efficient fine-tuning (LoRA) and addressing the specific challenge of balancing speaker identity verification with anti-spoofing detection within a generative framework.
The evaluation is conducted on the ASVspoof 5 dataset, a challenging benchmark for SASV. The results demonstrate that while zero-shot LALMs perform near chance, supervised fine-tuning (SFT) significantly improves performance. The proposed method with composite losses and hard mining achieves competitive results against strong baselines (ECAPA-TDNN + WavLM/W2V2-AASIST fusion), particularly in terms of overall accuracy and min a-DCF. However, the conventional baselines still outperform the LALM in EER, which the authors attribute to the lack of calibrated continuous scores in the token-based LALM output. The ablation studies effectively isolate the contribution of each component (AAM head, spoof head, hard mining). The reasoning component, while interesting, did not yield performance gains over hard-label SFT, which is a significant finding but limits the technical impact of that specific branch.
The paper provides sufficient detail regarding model architectures (SALMONN-7B, Qwen2-Audio-7B), training hyperparameters (LoRA rank, learning rate, batch size), and data processing (resampling, augmentation). The dataset (ASVspoof 5 + VoxCeleb) is standard. However, the code and pre-trained checkpoints are not mentioned as available, and the specific cold-start reasoning generation process using Step-Audio R1 is described but the model weights are not provided, making full reproduction of the CoT component difficult. The evaluation on a stratified sample of 20K pairs rather than the full set is noted, which aids speed but requires careful interpretation of metrics.
The primary limitation is the lack of performance superiority over established discriminative baselines in key metrics like EER. The LALM approach is computationally more expensive and slower than conventional pipelines. The reasoning traces, while interpretable, are not validated for correctness or utility in a human-evaluation setting, and the authors admit they are prone to hallucination. The "reasoning" capability does not translate to better hard-label accuracy, questioning the practical value of the CoT component for the core task. The reliance on external models (Step-Audio R1) for data generation introduces a dependency that is not fully analyzed.
This work opens a new direction for using LALMs in security-critical audio applications, offering the potential for interpretable decision-making which is crucial for auditing voice authentication systems. It highlights the gap between general audio understanding and specific security tasks. The findings help guide future research on whether reasoning capabilities can be effectively integrated into security systems or if discriminative models remain superior for raw performance. It contributes to the broader understanding of LALM capabilities and limitations in specialized domains. This paper provides the first systematic evaluation of Large Audio Language Models for Spoofing-Aware Speaker Verification, demonstrating that while they can be adapted to compete with conventional pipelines, they currently lag in threshold-based metrics and require significant task-specific adaptation to overcome their lack of native security awareness.