Speech Language Models (SLMs) are increasingly deployed in multi-speaker environments, yet their ability to attribute speech to the correct speaker and reason over speaker identities remains unclear. Hence, we introduce HEAR, a conceptually hierarchical benchmark diagnosing the foundational capabilities of speaker-attributed reasoning, comprising 2.4K human-verified samples from 887 diverse multi-party audio clips. Evaluating 20 leading SLMs on HEAR reveals they struggle with these foundational tasks, often relying on semantic priors rather than actual vocal cues. To address this, we present A2R, a 30B model optimized on Counterfactual Audio with Speaker-level Hard negatives (CASH), a dataset designed to guide the model to prioritize acoustic vocal cues over linguistic signals. A2R achieves strong performance on HEAR and exhibits zero-shot generalization to diverse multi-speaker downstream tasks, demonstrating that learned speaker attribution unlocks the model's latent capacity for speaker-aware reasoning. All resources are available at https://attributetoreason.github.io/AttributeToReason/
Primary: Seoul National University
All Institutions: Seoul National University, Yonsei University, University of Seoul
The paper presents a rigorous and novel framework for evaluating and improving speaker-attributed reasoning in Speech Language Models. By introducing the HEAR benchmark and the CASH dataset with counterfactual hard negatives, the authors effectively demonstrate that current models rely on semantic priors rather than acoustic cues, and propose a GRPO-based training method (A2R) that successfully shifts this reliance, resulting in strong performance gains and zero-shot generalization to unseen multi-speaker tasks.
The paper introduces a hierarchical benchmark (HEAR) grounded in Erberโs Auditory Hierarchy, decomposing speaker-attributed reasoning into Discrimination, Attribution, and Reasoning. This is a methodologically sound approach to diagnosing specific failure modes in Speech Language Models (SLMs). The proposed training strategy, A2R, utilizes Group Relative Policy Optimization (GRPO) with a composite reward function that explicitly rewards speaker-tagged transcription accuracy (cpWER, speaker count, order) alongside final answer correctness. The creation of the CASH dataset, which uses voice cloning to create counterfactual hard negatives where the transcript remains constant but the speaker identity changes, is a clever and effective way to force the model to rely on acoustic cues rather than semantic priors. The "transcription-first" objective is a strong inductive bias for this specific task.
The experimental evaluation is rigorous and comprehensive. The authors evaluate 20 leading SLMs, providing a clear baseline comparison. The introduction of the "paired accuracy" metric for the Reasoning dimension is particularly insightful, as it exposes models that rely on semantic shortcuts by requiring them to answer both the original and the counterfactual variant correctly. The zero-shot transfer experiments to unseen benchmarks (WDYL, Gaokao, FTS) and the validation on real-world human voices (vs. synthesized) strongly support the claim that the learned attribution capability is generalizable and not an artifact of the training data distribution. The ablation studies effectively isolate the contribution of hard negatives and the structured reasoning trace.
The paper provides high reproducibility. It details the source datasets (AMI, ICSI, VoxMM), the specific TTS models used for synthesis (VoxCPM2, IndexTTS 2), and the filtering criteria (WER < 0.1, speaker similarity > 0.7). The reward function components are clearly defined. The project page is provided, and the authors commit to releasing code and gated access to synthetic data, which is appropriate given the ethical concerns of voice cloning.
The primary limitation is the reliance on synthesized speech for the training data (CASH) and parts of the evaluation, although the authors mitigate this by testing on real human voices. The "transcription-first" approach introduces latency, which the authors acknowledge as a barrier to real-time applications. Additionally, the benchmark size (2.4K samples) is relatively small compared to large-scale LLM benchmarks, though the complexity of the tasks justifies this. The ethical implications of voice cloning are addressed, but the potential for misuse of the released synthetic data remains a concern, mitigated only by a Data Use Agreement.
This work has significant impact on the development of multi-party speech understanding systems. By identifying and addressing the "semantic hallucination" problem in SLMs, it provides a pathway to more robust voice assistants and meeting analysis tools that can accurately track who said what. The benchmark HEAR will likely become a standard evaluation suite for future SLMs, and the CASH dataset offers a valuable resource for training models that prioritize acoustic evidence. The paper presents a rigorous and novel framework for evaluating and improving speaker-attributed reasoning in Speech Language Models. By introducing the HEAR benchmark and the CASH dataset with counterfactual hard negatives, the authors effectively demonstrate that current models rely on semantic priors rather than acoustic cues, and propose a GRPO-based training method (A2R) that successfully shifts this reliance, resulting in strong performance gains and zero-shot generalization to unseen multi-speaker tasks.
Traditional speaker-attributed ASR systems treated ASR and speaker diarization as two separate tasks. Recently, end-to-end models such as VibeVoice-ASR have unified the two tasks within a single model. However, existing unified models still mainly support offline recognition, making it difficult to meet the low-latency requirements of real-time voice assistants and agents. To tackle this issue, we present VibeVoice-ASR-Streaming, one of the first LLM-based end-to-end approaches to streaming speaker-attributed ASR. It interleaves fixed-size audio chunks, a small amount of lookahead audio and previous text. This allows the model to produce ''who said what'' as speech arrives, without a separate diarization stage. For transcription accuracy, our 7B model achieves the lowest average WER/CER across five evaluation sets. For speaker attribution, it achieves the best or tied-best on 12 of 13 evaluation settings. We release the 1.5B and 7B model weights together with inference code.
Primary: Microsoft Research
All Institutions: Microsoft Research, University of Chinese Academy of Sciences, Shanghai Jiao Tong University
VibeVoice-ASR-Streaming introduces a pioneering LLM-based end-to-end approach to streaming speaker-attributed ASR, achieving state-of-the-art performance in both transcription accuracy and speaker attribution with significantly lower latency than existing cloud services. The paper demonstrates that retaining full conversation history in the LLM context is sufficient for consistent speaker labeling, eliminating the need for separate diarization stages, and provides a rigorous analysis of the trade-offs between chunk size, lookahead, and model scale in streaming speech recognition.
The paper proposes VibeVoice-ASR-Streaming, an end-to-end LLM-based framework for streaming speaker-attributed ASR. The core methodological contribution is the adaptation of the VibeVoice-ASR architecture (which unifies ASR and diarization) to a streaming setting by interleaving fixed-size audio chunks with generated text and retaining full conversation history in the LLM context. The authors introduce a fixed lookahead mechanism (4 latent frames) to handle boundary effects. The training strategy involves a three-stage process: non-streaming pre-training, streaming pre-training with chunked data, and streaming fine-tuning. The approach relies on the Qwen2.5 LLM backbone and dual tokenizers (Acoustic and Semantic) from the VibeVoice series. The methodology is sound, leveraging the strong contextual capabilities of LLMs to maintain speaker identity over long durations without explicit diarization modules, which is a significant architectural simplification compared to cascaded systems.
The experimental evaluation is comprehensive, covering four meeting benchmarks (AISHELL-4, AliMeeting, AMI-IHM, AMI-SDM) and the multilingual MLC-Challenge benchmark. The 7B model achieves the lowest average WER/CER among compared streaming systems and the best or tied-best cpWER/cpCER in 12 of 13 settings. The paper provides detailed ablations on chunk size, model scale, speaker-label placement, and lookahead depth. Notably, it compares against both specialized streaming ASR models and cloud services (Azure, Google), demonstrating a significant latency advantage (2.00s vs 8-9s) while maintaining or improving accuracy. The analysis of the "cost of streaming conversion" (degradation from offline to streaming) is particularly insightful, quantifying the trade-off.
The paper is highly reproducible. The authors release both 1.5B and 7B model weights along with inference code. Detailed training data construction methods, including the use of Qwen3-ForcedAligner for word-level timing and specific augmentation strategies, are described. The exact hyperparameters for training stages are provided in the appendix. The evaluation protocol, including the specific versions of cloud services and normalization rules, is clearly defined.
The primary limitations are the 8-minute recording length cap due to context window/compute constraints, the limited multilingual coverage (10 languages) dictated by the forced aligner, and performance degradation on long-duration overlaps due to the serialized output format. The first-packet latency (3.5s for 22-frame chunks) is higher than the steady-state latency, which may be a concern for real-time applications.
This work has significant impact on the development of real-time voice assistants and agents. By unifying ASR and speaker diarization in a streaming LLM, it removes the need for complex cascaded pipelines, simplifying system architecture and reducing latency. The open-sourcing of the models and code will likely accelerate research in streaming speech processing and enable the deployment of more robust, multi-speaker aware voice interfaces. VibeVoice-ASR-Streaming introduces a pioneering LLM-based end-to-end approach to streaming speaker-attributed ASR, achieving state-of-the-art performance in both transcription accuracy and speaker attribution with significantly lower latency than existing cloud services. The paper demonstrates that retaining full conversation history in the LLM context is sufficient for consistent speaker labeling, eliminating the need for separate diarization stages, and provides a rigorous analysis of the trade-offs between chunk size, lookahead, and model scale in streaming speech recognition.
Speech brain-computer interfaces (speech BCIs) translate neural activity into language, offering a path towards restoring speech for people with paralysis and, more broadly, enabling new forms of natural human-computer interaction. Despite this promise, the field lacks a common measure of progress because systems use different datasets, recording methods, types of speech, and vocabularies, so their reported scores are rarely comparable. Underlying this measurement problem are two unresolved questions: (i) what distribution of words should a speech BCI enable a user to communicate, and (ii) how much information from this distribution can a system convey. We address both by deriving open-vocabulary mutual information (OVMI), an information-theoretic quantity that measures the information conveyed by a decoder relative to a reference distribution over the words a user may wish to communicate. This allows capabilities measured under different conditions, such as distinct vocabularies, to be evaluated on a common communication scale. We show that ordinarily reported accuracy, word error rate (WER), and other metrics computed only over the words a system supports can overstate how much of a user's intended speech the system can communicate. We then use OVMI to compare existing systems, expose trade-offs between how much of the user's language a system supports and how accurately it decodes those words, show that these comparisons depend on what the user is expected to communicate, and demonstrate that selecting a vocabulary to maximise OVMI yields up to 16.3% relative improvement in accuracy across three speech domains. OVMI therefore provides the speech BCI community with a principled way to compare heterogeneous systems, improve vocabulary design, and measure progress in the field.
Primary: University of Oxford
All Institutions: University of Oxford
The paper introduces Open-Vocabulary Mutual Information (OVMI), a principled information-theoretic metric that enables fair comparison of heterogeneous speech brain-computer interfaces by accounting for both lexical coverage and decoding fidelity relative to a reference communication distribution. By demonstrating that conventional metrics overstate communication capability and showing that OVMI-guided vocabulary selection improves system accuracy, the work provides a crucial tool for standardizing evaluation and guiding the development of more effective speech restoration technologies.
The paper proposes Open-Vocabulary Mutual Information (OVMI), a metric designed to standardize the evaluation of speech Brain-Computer Interfaces (BCIs). The core methodological contribution is the decomposition of mutual information into a component that accounts for lexical coverage (the probability that an intended word is in the decoder's vocabulary) and in-vocabulary decoding fidelity. The authors derive a scalar estimator based on macro-accuracy and vocabulary size, allowing for retrospective evaluation of existing systems that do not report full confusion matrices. The mathematical derivation is rigorous, clearly distinguishing OVMI from Wolpaw's Information Transfer Rate (ITR) by explicitly modeling the out-of-vocabulary probability mass. The approach is theoretically sound, leveraging information theory to address a practical benchmarking gap.
The experimental section is robust, applying OVMI to a diverse set of heterogeneous systems, including invasive intracortical decoders (Moses, Willett, Card) and non-invasive MEG/EEG decoders (d'Ascoli, MEG-XL). The authors demonstrate that conventional metrics like accuracy can overstate communication capability by ignoring lexical coverage. A key strength is the sensitivity analysis, showing how OVMI scores change when evaluated against different reference distributions (broad spoken English, conversation, AAC, narrative). The vocabulary selection experiment provides strong empirical evidence of utility, showing that optimizing vocabulary size and composition via OVMI yields up to 16.3% relative improvement in accuracy over frequency-based selection.
The paper provides high reproducibility. The authors release a Python package and an interactive OVMI explorer. The appendix contains detailed derivations, pseudocode for the scalar estimator, and specific details on how reference distributions were constructed (e.g., using SUBTLEX-UK, Switchboard, UCV). The experimental setup for the vocabulary optimization is clearly described, including data splits, model architecture (contrastive brain-to-T5-embedding), and training details.
The primary limitation is the reliance on the scalar estimator for retrospective comparisons, which assumes symmetric error distributions and uniform per-word accuracy (macro-accuracy). This may not hold for all decoders, particularly those with complex error structures. Additionally, the metric evaluates lexical information only, ignoring contextual or semantic information that might be conveyed through paraphrasing or sentence-level structure. The choice of reference distribution $p$ significantly impacts the score, and while the authors provide multiple examples, the "correct" distribution for a specific clinical application remains an open question.
This work has significant potential to standardize progress tracking in the speech BCI field. By providing a common scale for heterogeneous systems, it facilitates more meaningful comparisons between invasive and non-invasive approaches and across different experimental paradigms. The ability to use OVMI for vocabulary design offers a practical tool for system developers to optimize their interfaces for specific communication goals. It addresses a critical gap in the field where progress was previously difficult to quantify across different studies. The paper introduces Open-Vocabulary Mutual Information (OVMI), a principled information-theoretic metric that enables fair comparison of heterogeneous speech brain-computer interfaces by accounting for both lexical coverage and decoding fidelity relative to a reference communication distribution. By demonstrating that conventional metrics overstate communication capability and showing that OVMI-guided vocabulary selection improves system accuracy, the work provides a crucial tool for standardizing evaluation and guiding the development of more effective speech restoration technologies.
Can we recover the 3D poses of multiple people using only sound? This paper presents the first attempt to estimate multi-person 3D poses solely from acoustic signals. Estimating the poses of multiple individuals using acoustic signals is inherently challenging due to the superposition of motion-dependent signal variations. Unlike single-person scenarios, the presence of multiple subjects leads to overlapping acoustic signatures, making it difficult to attribute specific signal changes to an individual's pose. Furthermore, the complexity is compounded by inter-person reflections, which introduce intricate propagation delays that obscure the temporal motion-acoustic relationship. To address these issues, we propose SoundMHPE (Sound-based Multi-person Human Pose Estimator), a novel encoder-decoder framework consisting of two key components. First, the Acoustic Multi-scale Encoder captures diverse temporal and fine-grained frequency features to isolate subtle acoustic signatures from complex, overlapping signals. Second, the Temporal Pose Decoder employs an attention mechanism to disentangle multi-person information across successive frames. By jointly accounting for temporal dynamics and inter-person dependencies, this component precisely reconstructs frame-wise individual poses. To validate our approach, we constructed the 6-hour Acoustic Multi-person Pose (AMP) dataset consisting of 432K synchronized frames of multi-person pose and acoustic data, and demonstrated that our SoundMHPE outperforms baseline models. Project page: https://oumi03.github.io/sound-mhpe/
Primary: Keio University
All Institutions: Keio University, Tokyo University of Science, NTT Inc.
The paper presents the first comprehensive framework for multi-person 3D pose estimation using active acoustic sensing, introducing a novel dataset and a DETR-inspired temporal decoder that effectively disentangles overlapping acoustic signatures. The technical contributions are significant, particularly the multi-scale encoder and the decoupled self-attention mechanisms, which address the unique challenges of multi-person acoustic superposition. The rigorous experimental validation, including cross-modal generalization to WiFi, strengthens the paper's impact, although the reliance on controlled environments remains a key limitation for real-world deployment.
The paper proposes SoundMHPE, a novel encoder-decoder framework for multi-person 3D pose estimation from acoustic signals. The methodology is well-structured, addressing the core challenge of signal superposition in multi-person scenarios. The Acoustic Multi-scale Encoder (AME) effectively utilizes multi-resolution STFT to capture both temporal dynamics and fine-grained frequency features, which is a sound technical choice for handling overlapping acoustic signatures. The Temporal Pose Decoder (TPD) introduces a DETR-like query mechanism adapted for temporal sequences, where multiple queries per person allow for frame-specific attention. The decoupling of self-attention into Motion Self-Attention (intra-person) and Interaction Self-Attention (inter-person) is a thoughtful architectural decision that aligns with the physical nature of the problem. The use of Hungarian matching for loss calculation is standard but appropriate for this detection-like task.
The authors constructed a new 6-hour dataset (AMP) with 432K frames, which is a significant contribution given the lack of existing benchmarks for this specific task. The experimental setup includes cross-subject evaluation with 3-fold cross-validation, which is rigorous. The baselines are adapted from single-person acoustic and multi-person WiFi models, which is reasonable given the novelty of the task. The ablation studies are comprehensive, covering the encoder, decoder, attention mechanisms, and window sizes. The cross-modal evaluation on WiFi data (PiW) is a strong point, demonstrating the generalizability of the architectural components. However, the performance metrics (MPJPE, PCK) should be compared against state-of-the-art single-person acoustic methods to provide a clearer benchmark of the "multi-person penalty."
The paper provides detailed implementation details, including hyperparameters (learning rate, weight decay, number of blocks, query counts) and dataset statistics. The use of standard hardware (OptiTrack, Zoom H3-VR) and software libraries (implied by the architecture) suggests high reproducibility. The project page is provided, which likely contains code and dataset access. The clear definition of the dataset collection protocol (15 subjects, specific poses, randomization) enhances reproducibility.
The primary limitation is the reliance on a controlled indoor environment with specific acoustic properties (soundproof room mentioned in Table 1). Generalization to noisy, reverberant, or outdoor environments is not fully explored, though a brief test with partitions is shown. The dataset size, while large for this niche, is still limited compared to RGB datasets, which may constrain the model's robustness to diverse body types and motions. The method assumes a fixed number of maximum people (M=15), which may not scale efficiently to very crowded scenes.
This work opens a new avenue for non-invasive, privacy-preserving pose estimation in scenarios where RGB cameras are unsuitable (darkness, privacy concerns) and wireless signals are obstructed. Applications in healthcare monitoring, sports analysis, and disaster relief are promising. The cross-modal applicability to WiFi suggests that the architectural insights could benefit other signal-based sensing tasks. The construction of the AMP dataset will likely serve as a foundational benchmark for future research in acoustic pose estimation. The paper presents the first comprehensive framework for multi-person 3D pose estimation using active acoustic sensing, introducing a novel dataset and a DETR-inspired temporal decoder that effectively disentangles overlapping acoustic signatures. The technical contributions are significant, particularly the multi-scale encoder and the decoupled self-attention mechanisms, which address the unique challenges of multi-person acoustic superposition. The rigorous experimental validation, including cross-modal generalization to WiFi, strengthens the paper's impact, although the reliance on controlled environments remains a key limitation for real-world deployment.
Spatial audio editing modifies an existing soundfield according to a user's instruction while preserving the rest of the scene. Unlike conventional audio editing, it must reason jointly about audio events, spatial information, dynamic changes, and environmental information in first-order Ambisonic (FOA) waveforms. Existing language-guided editors mainly target conventional audio or rely on sequential operations, and therefore do not directly support one-stage editing for complex 3D spatial instructions. We present SwanWeave, the first one-stage multi-task framework for instruction-guided 3D FOA spatial audio editing. We build paired FOA supervision from open-source speech and sound-effect corpora using controllable room simulation, covering more than ten single-operation and compound tasks across the four editing axes. To handle this heterogeneous edit space, SwanWeave uses Spatial Edit Mixture-of-Experts (SE-MoE) with dual-level routing, selecting task-aware expert combinations for compound instructions and frame-level routed/null experts for local edit decisions. We further introduce Spatial Preference Optimization (SPO), a Direct Preference Optimization (DPO)-based alignment objective with edit-specific negative targets, and adopt staged training to improve natural-language grounding. Experiments show that SwanWeave achieves better editing quality than existing general audio editors and spatial audio baselines across all tasks. Spatial audio editing demos can be found at https://swanaigc.github.io/#swanweave, code can be found at: https://github.com/MM-Speech/SwanWeave.
Primary: Zhejiang University
All Institutions: Zhejiang University, ByteDance
The paper introduces a novel one-stage multi-task framework for 3D spatial audio editing that effectively addresses the challenges of preserving unedited regions while executing complex compound instructions. By combining a dual-level Mixture-of-Experts architecture with preference optimization tailored for spatial edits, the method achieves state-of-the-art performance in both semantic and spatial fidelity, marking a significant step forward in controllable spatial audio generation.
The paper proposes SwanWeave, a one-stage framework for instruction-guided 3D First-Order Ambisonic (FOA) spatial audio editing. The methodology is robust, combining a flow-matching latent editor with a novel Spatial Edit Mixture-of-Experts (SE-MoE) module. The SE-MoE design is particularly interesting, utilizing dual-level routing: task-level experts selected based on the instruction to handle compound edits, and frame-level routed/null experts to handle local temporal changes and preservation of unedited regions. The use of a "null expert" that outputs zero tensors for unchanged regions is a clever architectural choice to prevent drift in preserved audio. The training strategy is well-structured, employing a staged curriculum (text-to-FOA pretraining, single-operation editing, compound editing) followed by Spatial Preference Optimization (SPO). SPO adapts DPO to the flow-matching context by using edit-specific negative targets (e.g., wrong event, no operation), which effectively addresses common failure modes like source copying or incorrect spatial placement.
The experimental evaluation is comprehensive. The authors construct a large-scale synthetic dataset (approx. 375K triplets) using PyRoomAcoustics for controllable FOA rendering, covering 10+ single and compound tasks. They compare against strong baselines including ZETA, AudioEditor, SDEdit, and SmartDJ. The metrics are diverse, covering semantic quality (FD, FAD, KL, LSD), instruction alignment (CLAP), and spatial fidelity (GCC, CRW, FSAD). The results show consistent improvements across all metrics, with particularly strong gains in spatial fidelity (GCC/CRW) compared to stereo-based baselines, validating the benefit of native FOA modeling. The ablation studies are thorough, confirming the necessity of pretraining, the SE-MoE module, and the SPO stage. The comparison between one-stage and multi-stage editing effectively demonstrates the advantage of the proposed one-stage approach in avoiding error accumulation.
The paper provides high reproducibility. Code and a demo page are available. The data construction process is detailed, including the specific room sizes, direction definitions, and the prompt used for instruction generation. The model architecture details, including the routing regularization losses and CFG scale selection, are provided in the appendix. The use of open-source corpora and standard simulation tools (PyRoomAcoustics) further aids reproducibility.
The primary limitation is the reliance on synthetic data generated via room simulation. While controllable, this may not capture the acoustic complexity of real-world recordings (e.g., irregular room responses, microphone artifacts, background noise). The scenes are also limited in duration (~10 seconds) and complexity (2-4 events), which may not generalize to long-form or dense multi-speaker scenes. The human evaluation was conducted with a small group of volunteer graduate students, which may introduce bias, though the objective metrics strongly support the findings.
This work has significant potential for immersive media production, VR/AR, and interactive storytelling. By enabling natural language control over 3D spatial audio, it lowers the barrier to creating complex spatial soundscapes. The one-stage approach is efficient and practical for real-time or near-real-time applications. The framework could be extended to other spatial audio formats (e.g., binaural, higher-order Ambisonics) and integrated into broader audio-visual generation pipelines. The paper introduces a novel one-stage multi-task framework for 3D spatial audio editing that effectively addresses the challenges of preserving unedited regions while executing complex compound instructions. By combining a dual-level Mixture-of-Experts architecture with preference optimization tailored for spatial edits, the method achieves state-of-the-art performance in both semantic and spatial fidelity, marking a significant step forward in controllable spatial audio generation.
Voice AI applications are gaining popularity as advances in large language models (LLMs) enable more natural and accessible spoken interactions. Serving these applications requires accounting not only for what users say, but also for how they speak (e.g., speaking rate) and the conditions under which their audio is captured and transmitted (e.g., background noise and packet loss). However, existing LLM systems represent conversation context as a flat, growing sequence of messages, leaving voice-specific context implicit in the audio. As a result, they can generate responses that are poorly aligned with user preferences, degrade interaction quality under adverse environmental conditions, and incur high costs over long voice sessions. We present llmovoice, a context-management middleware that explicitly models voice context and orchestrates its use. At each turn, llmovoice constructs a bounded voice context from the current user input, relevant interaction history, and explicit paralinguistic and environmental states. It then uses the serving LLM to reason over this context and generate runtime directives that guide how the system responds. We evaluate llmovoice on real-world voice applications and benchmarks. It reduces speaking-rate alignment error by 52.4%, lowers the false-interruption rate from 46.0% to 0.9% under packet loss, and reduces model usage cost by 79.2%. For long sessions, llmovoice reduces per-turn cost by up to 24.9 times while retaining up to 98.7% of baseline answer quality.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University, AgenticSys
The paper presents a novel middleware architecture for managing voice context in LLM systems, significantly improving cost efficiency and robustness to network artifacts. By explicitly modeling paralinguistic and environmental states and structuring conversation history into retrievable units, it addresses the scalability and alignment challenges of long-running voice interactions, offering a practical and effective solution for production voice AI.
The paper proposes "llmovoice," a middleware layer for LLM-over-voice systems that explicitly models voice context. The core contribution is architectural rather than algorithmic in the traditional ML sense; it introduces a structured memory system (VoicePages and VoiceThreads) to manage semantic history and a state-to-control path that extracts paralinguistic (speaking rate) and environmental (packet loss) signals. The system uses the LLM itself to reason over these states and generate runtime directives (e.g., adjusting VAD thresholds or response pacing). This approach is sound and addresses a genuine gap in current voice AI pipelines, which often treat audio as a flat stream without explicit state management. The use of a "cost-aware context projector" to select fidelity levels (audio vs. transcript vs. summary) is a clever engineering solution to the quadratic cost problem of long sessions.
The evaluation is strong and focuses on system-level metrics relevant to production deployment. Key results include a 52.4% reduction in speaking-rate alignment error, a dramatic drop in false-interruption rates (46.0% to 0.9%) under packet loss, and significant cost reductions (79.2% model usage cost, up to 24.9x per-turn cost reduction in long sessions). The use of real-world traces (ICASSP 2024 Audio Deep PLC Challenge) for the packet loss simulation adds credibility. However, the evaluation lacks subjective human studies (e.g., MOS scores or user preference tests) to validate the quality of the "naturalness" or "alignment" claims beyond objective error rates.
The paper describes the architecture and interface compatibility (OpenAI Realtime API), which aids reproducibility. However, specific implementation details of the "context orchestrator" prompts, the exact algorithms for the "cost-aware context projector," and the hyperparameters for the paralinguistic state smoothing are not fully detailed in the provided text. The reliance on a specific LLM service API limits the portability of the exact results, though the middleware design is intended to be agnostic.
The paper focuses primarily on speaking rate and packet loss as representative signals; it does not extensively evaluate other paralinguistic cues (tone, emotion) or environmental factors (background noise types). The system adds an extra LLM inference step (the orchestrator) before the main response generation, which could introduce latency, although the paper argues this is offset by cost savings from context compression. The lack of subjective user studies is a notable gap for a paper claiming improved interaction quality.
This work has high potential impact on the deployment of voice AI in resource-constrained or noisy environments (telecom, in-vehicle, remote areas). By decoupling context management from the LLM service, it provides a scalable pattern for building robust voice agents. The cost-efficiency gains are particularly relevant for high-volume voice applications where token costs are a major barrier. The paper presents a novel middleware architecture for managing voice context in LLM systems, significantly improving cost efficiency and robustness to network artifacts. By explicitly modeling paralinguistic and environmental states and structuring conversation history into retrievable units, it addresses the scalability and alignment challenges of long-running voice interactions, offering a practical and effective solution for production voice AI.
We present Alignment-Free Text-Audiobox (Text-AB), a unified framework for high-quality voice dubbing and full-duplex dialogue synthesis. Building on a Diffusion Transformer trained with a flow-matching objective, Text-AB departs from the Audiobox system along three dimensions. First, it operates in a latent diffusion framework using DAC-VAE features that encode 48 kHz waveforms into a 25 Hz latent sequence, giving over 10x higher compression than previous EnCodec representations while improving resynthesis quality. Second, Text-AB is alignment-free: it consumes raw text via an off-the-shelf text encoder and learns text-speech alignment through cross-attention, removing the need for forced alignment and explicit duration prediction. Third, we scale model and data substantially, pretraining a 3B-parameter model on 480k hours of monolingual speech, followed by supervised fine-tuning on three downstream tasks: cross-lingual voice dubbing, full-duplex dialogue synthesis, and emotional full-duplex dialogue synthesis. At inference, Text-AB supports one-shot generation for up to ~1 min of speech and arbitrarily long-form generation via a multi-diffusion scheme, plus a multi-stage reranking strategy that enhances quality based on automated metrics. On a real-world dubbing benchmark, Text-AB delivers a step-change improvement over the latest internal dubbing system, with large gains in prosody similarity, voice similarity, naturalness, and shareability. For full-duplex dialogue synthesis, it approaches human recordings on short-form conversations and substantially outperforms the latest internal model on long-form human-likeness and expressivity, while natively modeling turn-taking, back-channeling, and emotional dynamics. For emotional dialogue synthesis, emotion conditioning significantly improves emotion alignment and emotional interaction quality over the unconditioned baseline.
The paper introduces Text-AB, a unified alignment-free diffusion framework for high-quality voice dubbing and full-duplex dialogue synthesis that leverages latent diffusion with high-compression VAE features and cross-attention for text-speech alignment. This work represents a strong technical advance in audio generation, particularly for complex tasks like full-duplex dialogue where modeling turn-taking and back-channeling is critical, and the shift to an alignment-free approach simplifies the pipeline significantly. While the components (Diffusion Transformer, flow matching, VAE) are established, their integration into a scalable, high-fidelity system for these specific downstream tasks constitutes a meaningful contribution to the field of speech synthesis and audio ML.
Traditional speaker-attributed ASR systems treated ASR and speaker diarization as two separate tasks. Recently, end-to-end models such as VibeVoice-ASR have unified the two tasks within a single model. However, existing unified models still mainly support offline recognition, making it difficult to meet the low-latency requirements of real-time voice assistants and agents. To tackle this issue, we present VibeVoice-ASR-Streaming, one of the first LLM-based end-to-end approaches to streaming speaker-attributed ASR. It interleaves fixed-size audio chunks, a small amount of lookahead audio and previous text. This allows the model to produce ''who said what'' as speech arrives, without a separate diarization stage. For transcription accuracy, our 7B model achieves the lowest average WER/CER across five evaluation sets. For speaker attribution, it achieves the best or tied-best on 12 of 13 evaluation settings. We release the 1.5B and 7B model weights together with inference code.
Primary: Microsoft Research
All Institutions: Microsoft Research, University of Chinese Academy of Sciences, Shanghai Jiao Tong University
VibeVoice-ASR-Streaming introduces a pioneering LLM-based end-to-end approach to streaming speaker-attributed ASR, achieving state-of-the-art performance in both transcription accuracy and speaker attribution with significantly lower latency than existing cloud services. The paper demonstrates that retaining full conversation history in the LLM context is sufficient for consistent speaker labeling, eliminating the need for separate diarization stages, and provides a rigorous analysis of the trade-offs between chunk size, lookahead, and model scale in streaming speech recognition.
The paper proposes VibeVoice-ASR-Streaming, an end-to-end LLM-based framework for streaming speaker-attributed ASR. The core methodological contribution is the adaptation of the VibeVoice-ASR architecture (which unifies ASR and diarization) to a streaming setting by interleaving fixed-size audio chunks with generated text and retaining full conversation history in the LLM context. The authors introduce a fixed lookahead mechanism (4 latent frames) to handle boundary effects. The training strategy involves a three-stage process: non-streaming pre-training, streaming pre-training with chunked data, and streaming fine-tuning. The approach relies on the Qwen2.5 LLM backbone and dual tokenizers (Acoustic and Semantic) from the VibeVoice series. The methodology is sound, leveraging the strong contextual capabilities of LLMs to maintain speaker identity over long durations without explicit diarization modules, which is a significant architectural simplification compared to cascaded systems.
The experimental evaluation is comprehensive, covering four meeting benchmarks (AISHELL-4, AliMeeting, AMI-IHM, AMI-SDM) and the multilingual MLC-Challenge benchmark. The 7B model achieves the lowest average WER/CER among compared streaming systems and the best or tied-best cpWER/cpCER in 12 of 13 settings. The paper provides detailed ablations on chunk size, model scale, speaker-label placement, and lookahead depth. Notably, it compares against both specialized streaming ASR models and cloud services (Azure, Google), demonstrating a significant latency advantage (2.00s vs 8-9s) while maintaining or improving accuracy. The analysis of the "cost of streaming conversion" (degradation from offline to streaming) is particularly insightful, quantifying the trade-off.
The paper is highly reproducible. The authors release both 1.5B and 7B model weights along with inference code. Detailed training data construction methods, including the use of Qwen3-ForcedAligner for word-level timing and specific augmentation strategies, are described. The exact hyperparameters for training stages are provided in the appendix. The evaluation protocol, including the specific versions of cloud services and normalization rules, is clearly defined.
The primary limitations are the 8-minute recording length cap due to context window/compute constraints, the limited multilingual coverage (10 languages) dictated by the forced aligner, and performance degradation on long-duration overlaps due to the serialized output format. The first-packet latency (3.5s for 22-frame chunks) is higher than the steady-state latency, which may be a concern for real-time applications.
This work has significant impact on the development of real-time voice assistants and agents. By unifying ASR and speaker diarization in a streaming LLM, it removes the need for complex cascaded pipelines, simplifying system architecture and reducing latency. The open-sourcing of the models and code will likely accelerate research in streaming speech processing and enable the deployment of more robust, multi-speaker aware voice interfaces. VibeVoice-ASR-Streaming introduces a pioneering LLM-based end-to-end approach to streaming speaker-attributed ASR, achieving state-of-the-art performance in both transcription accuracy and speaker attribution with significantly lower latency than existing cloud services. The paper demonstrates that retaining full conversation history in the LLM context is sufficient for consistent speaker labeling, eliminating the need for separate diarization stages, and provides a rigorous analysis of the trade-offs between chunk size, lookahead, and model scale in streaming speech recognition.
Speech brain-computer interfaces (speech BCIs) translate neural activity into language, offering a path towards restoring speech for people with paralysis and, more broadly, enabling new forms of natural human-computer interaction. Despite this promise, the field lacks a common measure of progress because systems use different datasets, recording methods, types of speech, and vocabularies, so their reported scores are rarely comparable. Underlying this measurement problem are two unresolved questions: (i) what distribution of words should a speech BCI enable a user to communicate, and (ii) how much information from this distribution can a system convey. We address both by deriving open-vocabulary mutual information (OVMI), an information-theoretic quantity that measures the information conveyed by a decoder relative to a reference distribution over the words a user may wish to communicate. This allows capabilities measured under different conditions, such as distinct vocabularies, to be evaluated on a common communication scale. We show that ordinarily reported accuracy, word error rate (WER), and other metrics computed only over the words a system supports can overstate how much of a user's intended speech the system can communicate. We then use OVMI to compare existing systems, expose trade-offs between how much of the user's language a system supports and how accurately it decodes those words, show that these comparisons depend on what the user is expected to communicate, and demonstrate that selecting a vocabulary to maximise OVMI yields up to 16.3% relative improvement in accuracy across three speech domains. OVMI therefore provides the speech BCI community with a principled way to compare heterogeneous systems, improve vocabulary design, and measure progress in the field.
Primary: University of Oxford
All Institutions: University of Oxford
The paper introduces Open-Vocabulary Mutual Information (OVMI), a principled information-theoretic metric that enables fair comparison of heterogeneous speech brain-computer interfaces by accounting for both lexical coverage and decoding fidelity relative to a reference communication distribution. By demonstrating that conventional metrics overstate communication capability and showing that OVMI-guided vocabulary selection improves system accuracy, the work provides a crucial tool for standardizing evaluation and guiding the development of more effective speech restoration technologies.
The paper proposes Open-Vocabulary Mutual Information (OVMI), a metric designed to standardize the evaluation of speech Brain-Computer Interfaces (BCIs). The core methodological contribution is the decomposition of mutual information into a component that accounts for lexical coverage (the probability that an intended word is in the decoder's vocabulary) and in-vocabulary decoding fidelity. The authors derive a scalar estimator based on macro-accuracy and vocabulary size, allowing for retrospective evaluation of existing systems that do not report full confusion matrices. The mathematical derivation is rigorous, clearly distinguishing OVMI from Wolpaw's Information Transfer Rate (ITR) by explicitly modeling the out-of-vocabulary probability mass. The approach is theoretically sound, leveraging information theory to address a practical benchmarking gap.
The experimental section is robust, applying OVMI to a diverse set of heterogeneous systems, including invasive intracortical decoders (Moses, Willett, Card) and non-invasive MEG/EEG decoders (d'Ascoli, MEG-XL). The authors demonstrate that conventional metrics like accuracy can overstate communication capability by ignoring lexical coverage. A key strength is the sensitivity analysis, showing how OVMI scores change when evaluated against different reference distributions (broad spoken English, conversation, AAC, narrative). The vocabulary selection experiment provides strong empirical evidence of utility, showing that optimizing vocabulary size and composition via OVMI yields up to 16.3% relative improvement in accuracy over frequency-based selection.
The paper provides high reproducibility. The authors release a Python package and an interactive OVMI explorer. The appendix contains detailed derivations, pseudocode for the scalar estimator, and specific details on how reference distributions were constructed (e.g., using SUBTLEX-UK, Switchboard, UCV). The experimental setup for the vocabulary optimization is clearly described, including data splits, model architecture (contrastive brain-to-T5-embedding), and training details.
The primary limitation is the reliance on the scalar estimator for retrospective comparisons, which assumes symmetric error distributions and uniform per-word accuracy (macro-accuracy). This may not hold for all decoders, particularly those with complex error structures. Additionally, the metric evaluates lexical information only, ignoring contextual or semantic information that might be conveyed through paraphrasing or sentence-level structure. The choice of reference distribution $p$ significantly impacts the score, and while the authors provide multiple examples, the "correct" distribution for a specific clinical application remains an open question.
This work has significant potential to standardize progress tracking in the speech BCI field. By providing a common scale for heterogeneous systems, it facilitates more meaningful comparisons between invasive and non-invasive approaches and across different experimental paradigms. The ability to use OVMI for vocabulary design offers a practical tool for system developers to optimize their interfaces for specific communication goals. It addresses a critical gap in the field where progress was previously difficult to quantify across different studies. The paper introduces Open-Vocabulary Mutual Information (OVMI), a principled information-theoretic metric that enables fair comparison of heterogeneous speech brain-computer interfaces by accounting for both lexical coverage and decoding fidelity relative to a reference communication distribution. By demonstrating that conventional metrics overstate communication capability and showing that OVMI-guided vocabulary selection improves system accuracy, the work provides a crucial tool for standardizing evaluation and guiding the development of more effective speech restoration technologies.
Automatic Speech Recognition (ASR) systems are widely deployed in safety-critical settings but remain vulnerable to data-poisoning backdoor attacks. Existing ASR backdoors typically use phrase-level triggers paired with a fixed target sentence, creating strong artifacts (e.g., repeated transcripts or triggers placed in non-speech regions) that simple preprocessing can mitigate. We propose GhostWord, a word-level, time-localized ASR backdoor that uses codebooks mapping short ($\approx$400\,ms) acoustic triggers to target words. During poisoning, we inject a trigger into the forced-aligned time span of a chosen source word in the audio and replace only that word in the transcript, enabling precise semantic flips and composable sentence manipulation while avoiding many-to-one label artifacts. Across Common Voice (v23 English, v24 Lithuanian) and multiple backbones (Whisper-Small/Medium, MMS, SpeechT5), GhostWord achieves an average attack success rate of 89.3\% and transfers across languages and models. Adapting optimization-based defenses (ABL, ANP, SAU, I-BAU) reveals a sharp robustness--accuracy trade-off: attack success drops from 89.3\% to 29.1\% while clean WER rises from 21.5\% to 45.0\%, consistent with our theoretical analysis showing that, in high-vocabulary models, backdoor suppression structurally tends to degrade clean performance. The source code is publicly available at https://github.com/rohban-lab/GhostWord
Primary: Sharif University of Technology
All Institutions: Sharif University of Technology, รcole polytechnique fรฉdรฉrale de Lausanne, Isfahan University
GhostWord introduces a novel word-level, time-localized backdoor attack for ASR that evades simple preprocessing defenses by avoiding many-to-one label artifacts. The paper provides a rigorous evaluation across multiple models and languages, demonstrating high attack success rates and revealing a critical robustness-accuracy trade-off in existing optimization-based defenses, thereby highlighting the need for ASR-specific security mechanisms.
The paper proposes "GhostWord," a fine-grained, word-level backdoor attack on ASR systems. Unlike prior phrase-level attacks that replace entire transcripts with fixed malicious sentences (creating detectable statistical artifacts), GhostWord uses a codebook of short (~400ms) acoustic triggers mapped to specific target words. The poisoning process involves forced alignment to identify the time span of a source word, injecting the trigger into that specific segment, and replacing only that word in the ground truth transcript. This approach effectively mitigates simple preprocessing defenses like label-frequency filtering and Voice Activity Detection (VAD) because the poisoned samples retain most of their original transcript content and the triggers are embedded within speech regions. The methodology is sound, leveraging standard forced alignment tools and constrained SNR injection to ensure imperceptibility. The theoretical analysis regarding the robustness-accuracy trade-off in high-vocabulary models is a strong addition, explaining why optimization-based defenses degrade clean performance significantly in ASR compared to classification.
The experiments are comprehensive, evaluating the attack across multiple datasets (Common Voice v23 English, v24 Lithuanian) and diverse backbones (Whisper-Small/Medium, MMS, SpeechT5). The reported average attack success rate of 89.3% is high and demonstrates the efficacy of the word-level approach. The evaluation against advanced optimization-based defenses (ABL, ANP, SAU, I-BAU) is particularly valuable, revealing a sharp trade-off where mitigating the attack causes clean WER to rise from 21.5% to 45.0%. This highlights a critical vulnerability in current defense strategies for ASR. The inclusion of cross-language and cross-model transferability tests further strengthens the empirical claims.
The authors provide a public GitHub repository, which is a significant plus. The paper details the trigger generation (mixture of noise families), SNR constraints (22 dB), and the poisoning procedure clearly. The use of standard datasets and open-source models (Whisper, MMS) enhances reproducibility. However, some specific hyperparameters for the defense adaptations might require careful tuning, which is typical for such studies.
The attack relies on forced alignment, which may introduce errors if the alignment is poor, potentially affecting trigger placement. The study focuses on data-poisoning during training; the robustness of the attack against models trained with different regularization techniques or data augmentation strategies is not extensively explored. The theoretical analysis, while insightful, is somewhat high-level and could benefit from more rigorous mathematical bounds. Additionally, the "imperceptibility" is validated via SNR and subjective evaluation, but objective perceptual metrics (like PESQ or POLQA) could provide a more standardized measure of audio quality degradation.
This work has significant implications for the security of deployed ASR systems, particularly in safety-critical applications like automotive voice assistants or medical transcription. By demonstrating that simple preprocessing defenses are insufficient against word-level attacks, it pushes the field toward more sophisticated detection and mitigation strategies. The findings on the robustness-accuracy trade-off are crucial for practitioners, indicating that naive application of classification-based defenses to ASR can be counterproductive. The paper contributes to the broader understanding of adversarial robustness in sequence-to-sequence models. GhostWord introduces a novel word-level, time-localized backdoor attack for ASR that evades simple preprocessing defenses by avoiding many-to-one label artifacts. The paper provides a rigorous evaluation across multiple models and languages, demonstrating high attack success rates and revealing a critical robustness-accuracy trade-off in existing optimization-based defenses, thereby highlighting the need for ASR-specific security mechanisms.
In this paper, we study the task of automatic music upmixing, wherein a system predicts spatial mixing parameters from a multi-stem recording. Different from existing methods that rely on task-specific music encoders, we approach this task via audio language model (ALM) post-training, leveraging rich representations from existing ALMs, which encode both music semantics and mixing knowledge. Specifically, we propose a post-training recipe that first employs rejection sampling SFT, followed by reinforcement learning (RL) with verifiable rewards (RLVR) via GRPO. We propose Sphere (Spatial Heuristic Rewards), a deterministic reward suite inspired by music mixing conventions, to guide our post-training. It consists of 6 perceptually-motivated sub-rewards and encourages the output mix to be centered, balanced and spacious. More broadly, our results suggest that expert domain knowledge can be encoded as verifiable rewards and distilled into language models, without task-specific architectures.
Primary: Meta
All Institutions: Meta, Queen Mary University of London
The paper introduces a novel post-training framework for Audio Language Models to perform automatic music upmixing, leveraging a deterministic reward suite (Sphere) inspired by mixing conventions to guide SFT and RL, resulting in a small model that outperforms frontier LLMs in both objective and subjective evaluations.
The paper proposes a novel post-training recipe for Audio Language Models (ALMs) to perform automatic music upmixing. The core method involves two stages: Rejection Sampling Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR) using Group Relative Policy Optimization (GRPO). The key innovation is the "Sphere" reward function, a deterministic suite of six perceptually-motivated sub-rewards (centering, spread, level balance, spotlight prominence, per-role consistency) derived from music mixing conventions. The authors demonstrate that encoding expert domain knowledge as verifiable rewards allows a small student model (Qwen2.5-Omni 3B) to surpass larger frontier models (Gemini 2.5 Pro) in both objective reward scores and subjective human preference. The methodology is sound, leveraging the semantic understanding of ALMs rather than task-specific encoders, and effectively addresses the lack of paired spatial mixing data by using a teacher model to generate high-quality synthetic data filtered by the reward function.
The experimental evaluation is rigorous and comprehensive. The authors use the MedleyDB dataset, segmenting it into 30-second windows to fit context limits. They conduct a subjective A/B listening test with 44 participants, showing a 76.8% preference for high-reward mixes, validating the alignment of the Sphere reward with human perception. Objective evaluation on 618 held-out test points shows the post-trained model achieving a total reward of 4.981, significantly outperforming the zero-shot teacher (4.548) and other baselines. Ablation studies are thorough, demonstrating that data quality (stricter filtering) is more important than quantity for SFT, and that removing specific sub-rewards leads to reward hacking (e.g., collapsing spatial spread). The inclusion of a multi-stimulus preference test with expert listeners further strengthens the results, showing consistent improvements from SFT to SFT+RL.
The paper provides detailed descriptions of the training setup, including hyperparameters for LoRA, learning rates, and the RL process (GRPO with PPO-style clipping). The prompt template is fully disclosed in the appendix, which is crucial for reproducibility given the reliance on prompt engineering. However, the use of proprietary models (Gemini 2.5 Pro as teacher) and a proprietary HRTF spatializer limits full reproducibility. The code and data are not explicitly linked in the provided text, though a project demo website is mentioned. The detailed ablation on reward shaping (Gaussian vs. Sigmoid vs. Step) provides valuable insights for practitioners.
The primary limitation is the context window constraint, which limits the number of stems (max 29) and segment length (30 seconds), excluding 23.5% of the dataset. This may not capture long-term musical dynamics. Additionally, the reliance on a proprietary teacher model for data generation introduces potential biases. The subjective tests, while statistically significant, involve a relatively small number of participants (44 for A/B, 10 for multi-stimulus). The reward function, while aligned with human preference, is heuristic-based and may not capture all aspects of musical aesthetics.
This work demonstrates a practical recipe for distilling expert domain knowledge into language models via verifiable rewards, applicable beyond music mixing to other data-limited domains. It highlights the potential of ALMs for complex audio tasks that require semantic understanding, moving beyond simple feature extraction. The approach could inspire similar post-training strategies for other audio generation or processing tasks where ground truth data is scarce but expert heuristics are available. The paper introduces a novel post-training framework for Audio Language Models to perform automatic music upmixing, leveraging a deterministic reward suite (Sphere) inspired by mixing conventions to guide SFT and RL, resulting in a small model that outperforms frontier LLMs in both objective and subjective evaluations.
Whisper exposes speech through a fixed 1500-token encoder interface, now a default representation for ASR decoders and Whisper-based speech language models (SpeechLMs), yet its redundancy remains largely unexamined. We propose stride-k subsampling, a deterministic indexing operation that retains every k-th token after the convolutional stem or encoder transformer. Across five Whisper scales, k=2 preserves baseline WER at both positions, with CKA attributing this stability to acoustic overlap at the stem and attention-induced redistribution at the encoder output. Applying stride-2 at both positions cuts audio tokens by 75% and total GFLOPs by 52-58%, with small WER costs on most ASR benchmarks and larger costs on harder ones. The same configuration extends to three Whisper-based SpeechLMs, yielding modest accuracy drops on stronger baselines and larger drops on weaker ones, while reducing end-to-end latency by 19.6-27.4%. Requiring no training or auxiliary computation, stride-k subsampling exploits Whisper's preprocessing redundancy, indicating that its audio-token interface carries more capacity than downstream tasks require.
Primary: Chung-Ang University
All Institutions: Chung-Ang University
The paper proposes a simple, train-free stride-k subsampling method to reduce audio tokens in Whisper, achieving significant efficiency gains with minimal performance loss on clean speech. It provides a thorough analysis of the underlying redundancy mechanisms using CKA and extensive experiments across multiple model scales and benchmarks, establishing a practical baseline for efficient speech processing.
The paper proposes "stride-k subsampling," a deterministic, train-free indexing operation to reduce the number of audio tokens in Whisper-based models. The method exploits the temporal redundancy inherent in Whisper's preprocessing pipeline (overlapping STFT windows and convolutional stem receptive fields). By retaining every k-th token either after the convolutional stem (input-side) or after the encoder transformer (output-side), the authors reduce sequence length without retraining. The methodology is grounded in a theoretical analysis of temporal overlap, showing that for k=2, the retained tokens still cover the full input span due to receptive field overlap. The authors further employ Centered Kernel Alignment (CKA) to diagnose why output-side subsampling is more stable than input-side, attributing it to attention-induced redistribution of redundancy in the encoder. The approach is simple, parameter-free, and easily integrable into existing pipelines.
The experimental evaluation is comprehensive, covering five Whisper scales (tiny to large-v3) and three ASR benchmarks (LibriTTS, ESD, Common Voice). The results demonstrate that k=2 preserves baseline WER on clean speech with minimal cost, while harder benchmarks show larger degradations. The paper extends the evaluation to three Whisper-based SpeechLMs (Audio Flamingo 3, Qwen2-Audio, LLaMA-Omni 2), showing modest accuracy drops on MMSU and MMAU benchmarks. The efficiency gains are significant, with 52-58% reduction in GFLOPs and 19-27% reduction in end-to-end latency. The inclusion of a noise robustness analysis (SNR sweep) and a comparison with other pruning methods (SpeechPrune) strengthens the evaluation. The "hurt/help" analysis provides insight into which types of tasks are most affected by temporal subsampling.
The paper provides detailed implementation details, including the specific subsampling positions, FLOPs calculation formulas, and dataset preparation steps. The use of standard open-source models and benchmarks (HuggingFace checkpoints, LibriTTS, Common Voice) enhances reproducibility. The authors mention using a fixed random seed and provide code snippets for the subsampling operation. However, no explicit GitHub repository link is provided in the text, which slightly limits immediate reproducibility, though the method is simple enough to implement from the description.
The primary limitation is that the method is specific to encoders with overlapping windowed frontends (like Whisper's log-mel). It does not generalize to raw-waveform encoders like wav2vec 2.0 or HuBERT, which collapse under subsampling. The method requires access to the encoder's internal computation, making it inapplicable to closed-API models. Additionally, the performance degradation on noisy or difficult benchmarks (Common Voice) is significant, limiting its utility in low-resource or noisy environments. The paper acknowledges that the redundancy exploited is partly shaped by training, and future work could explore training models to tolerate subsampling.
The paper contributes to the efficiency of speech recognition and speech language models, which are increasingly deployed in real-time applications. By reducing inference cost without retraining, the method offers a practical solution for deploying Whisper-based systems on resource-constrained devices. The insights into temporal redundancy in speech encoders may inform future architectural designs. The findings on the asymmetry between input-side and output-side subsampling provide valuable guidance for model optimization. The paper proposes a simple, train-free stride-k subsampling method to reduce audio tokens in Whisper, achieving significant efficiency gains with minimal performance loss on clean speech. It provides a thorough analysis of the underlying redundancy mechanisms using CKA and extensive experiments across multiple model scales and benchmarks, establishing a practical baseline for efficient speech processing.
Speech enhancement models are widely deployed as frontend modules in real-time speech services, yet their vulnerability to backdoor attacks remains unexplored. Existing backdoor methods are confined to classification tasks and rely on active trigger injection, an assumption incompatible with the passive processing nature of speech enhancement models. In this paper, we propose Ouroboros, a novel backdoor attack framework that leverages the ideal clean outputs of speech enhancement models as natural triggers, enabling inference-time activation without any external trigger injection. Extensive evaluations show Ouroboros achieves near-perfect attack success rates with minimal performance degradation on diverse models and datasets. Physical-world validations confirm that naturally recorded, unaltered clean audio can reliably activate the backdoor. Moreover, Ouroboros generalizes to targeted content-tampering attacks and remains effective against common filtering and finetuning defenses.
Primary: Ningbo University
All Institutions: Ningbo University, Ningbo University of Finance and Economics
The paper introduces a novel backdoor attack for speech enhancement that uses clean audio as a natural trigger, demonstrating high effectiveness and stealthiness across multiple architectures and real-world scenarios. By shifting the trigger from artificial injections to natural high-SNR speech, the work addresses a realistic threat model for passive audio processing services, providing significant insights into the security vulnerabilities of current speech enhancement pipelines.
The paper proposes "Ouroboros," a backdoor attack framework for speech enhancement models that utilizes clean audio as the trigger. The core innovation is the "CleanTrigger" mechanism, which repurposes the model's expected output (clean speech) as the input trigger. This is a clever conceptual shift from traditional backdoor attacks that require active injection of artificial signals (e.g., ultrasonic tones or specific noise patterns). By targeting high-SNR samples in the training data and poisoning them to map to a malicious target (silence or a specific phrase), the attack leverages the natural occurrence of clean audio in real-world scenarios (e.g., a user speaking clearly in a quiet room). The methodology is sound, utilizing an SNR-based selection strategy to minimize degradation on the primary task (denoising noisy speech) while maximizing attack success on clean speech. The threat model is realistic for supply-chain attacks where an attacker poisons the training dataset without needing to manipulate inference-time inputs.
The experiments are comprehensive, covering four distinct speech enhancement architectures (MP-SENet, SEMamba, CMGAN, FlowSE) and two standard datasets (VoiceBank-Demand, WSJ0-CHiME3). The results demonstrate near-perfect Attack Success Rates (ASR) with minimal degradation in PESQ scores compared to the BadNets baseline. The inclusion of physical-world validation (recording 60 clean voices via smartphone) is a strong point, confirming the practical feasibility of the attack. The extension to content-tampering (outputting a specific phrase) further validates the versatility of the attack. The ablation studies on poisoning rates and SNR selection strategies provide good insight into the trade-offs.
The paper provides sufficient details for reproduction, including the specific SNR thresholds, poisoning rates, and model architectures used. However, the lack of a public code repository (no GitHub link provided in the text) slightly hinders immediate reproducibility. The use of standard datasets and open-source models (like CMGAN and FlowSE) mitigates this to some extent.
The primary limitation is the assumption that the attacker has access to poison the training data, which is a strong assumption for many deployed systems. Additionally, the attack relies on the model being trained on paired data; it may not generalize directly to unpaired or self-supervised enhancement models. The "silence" attack is a disruption attack, and while the content-tampering extension is shown, the robustness of the specific phrase output against different TTS voices or languages is not deeply explored.
This work highlights a critical security vulnerability in passive audio processing pipelines. As speech enhancement becomes ubiquitous in real-time communication and voice assistants, the ability to silently disable or manipulate these services using natural audio inputs poses a significant risk. The findings urge the community to consider backdoor robustness in the design of speech enhancement models, particularly those deployed in safety-critical or privacy-sensitive applications. The paper introduces a novel backdoor attack for speech enhancement that uses clean audio as a natural trigger, demonstrating high effectiveness and stealthiness across multiple architectures and real-world scenarios. By shifting the trigger from artificial injections to natural high-SNR speech, the work addresses a realistic threat model for passive audio processing services, providing significant insights into the security vulnerabilities of current speech enhancement pipelines.
Speech enhancement (SE) is commonly applied as a preprocessing step in spoken AI pipelines under the assumption that better audio quality improves downstream task performance. Whether SE-induced distortions propagate to downstream LLM task performance remains an open question. We introduce Output Divergence Rate (ODR), which measures how often SE changes an LLM's intent classification relative to clean speech, and benchmark five conditions on 2,974 SLURP clips using Whisper large-v3 and wav2vec2-large cascades. Every condition produces ODR significantly above zero ($p < 0.001$, binomial test). MetricGAN{+} more than doubles ODR versus unenhanced noisy speech (0.318 vs. 0.135) despite improving PESQ, and unmitigated echo reaches an ODR of 0.836 through speaker substitution, a failure WER cannot capture. Audio quality metrics range from near-zero to moderate correlation with ODR (SQUIM-MOS $ฯ=-0.068$, PESQ $ฯ=-0.467$). The MetricGAN{+} and echo results replicate across ASR architectures, indicating that standard audio quality metrics are insufficient for LLM pipeline quality.
Primary: GN Group
All Institutions: GN Group
The paper introduces Output Divergence Rate (ODR) to measure the semantic impact of speech enhancement on LLM-based voice systems, revealing that standard audio quality metrics are insufficient predictors of downstream task performance. Through rigorous benchmarking on the SLURP dataset using multiple ASR architectures and LLM classifiers, the study demonstrates that perceptual improvements in speech enhancement can paradoxically increase semantic divergence, particularly through failure modes like speaker substitution in echo scenarios that WER cannot capture. This work provides a critical new evaluation framework for the integration of speech processing and large language models, highlighting the need for task-specific metrics in the design of robust spoken AI pipelines.
The paper introduces Output Divergence Rate (ODR), a metric designed to quantify the semantic impact of speech enhancement (SE) on downstream Large Language Model (LLM) intent classification. The methodology is sound, utilizing a cascaded pipeline of ASR (Whisper large-v3 and wav2vec2-large) followed by LLM classification (Gemini 2.5 Flash Lite). The approach correctly identifies a critical gap in current evaluation practices: standard audio quality metrics (PESQ, STOI) and transcription metrics (WER) do not correlate well with semantic task performance. The use of a closed-set intent classification task (SLURP) provides a controlled environment to measure divergence, although it limits the scope to specific command structures. The decomposition of divergence into "corrections" vs. "regressions" using ground truth labels is a rigorous addition that validates the metric's sensitivity to actual harm rather than random noise.
The experiments are comprehensive, covering five distinct enhancement conditions (Noisy, MetricGAN+, Echo, Echo+AEC, Dereverb) across 2,974 clips. The results are striking: MetricGAN+ doubles the ODR compared to unenhanced noisy speech despite improving PESQ, and unmitigated echo causes catastrophic failure (ODR 0.836) due to speaker substitution, a failure mode WER cannot capture. The replication across two architecturally distinct ASR models (attention-based vs. CTC) strengthens the findings, showing that the issue is not model-specific. The analysis of within-condition vs. pooled correlations is particularly valuable, demonstrating that while metrics separate conditions, they fail to predict per-clip divergence, which is crucial for deployment monitoring.
High. The authors release the full pipeline on GitHub. The use of standard, publicly available datasets (SLURP, DNS Challenge) and models (Whisper, MetricGAN+, Gemini API) ensures that the results are reproducible by other researchers. The prompt used for LLM classification is provided in the appendix, and hyperparameters for ASR and SE models are specified.
The study relies on simulated degradations (DNS Challenge) rather than real-world field recordings, which may differ in acoustic complexity. The LLM evaluation is limited to a single model family (Gemini) for the primary results, with only partial replication on a larger model (Gemini Pro). The task is restricted to English and closed-set intent classification, so findings may not generalize to open-ended generation or other languages. ODR requires a clean reference, limiting its use as an online monitor.
This paper has significant implications for the deployment of spoken AI systems. It challenges the common assumption that improving audio quality metrics (like PESQ) leads to better user experience in LLM-based assistants. It provides a new diagnostic tool (ODR) for developers to evaluate SE front-ends specifically for their impact on semantic understanding. The finding that standard metrics are poor predictors of LLM failure urges the field to adopt task-specific evaluation protocols for voice interfaces. The paper introduces Output Divergence Rate (ODR) to measure the semantic impact of speech enhancement on LLM-based voice systems, revealing that standard audio quality metrics are insufficient predictors of downstream task performance. Through rigorous benchmarking on the SLURP dataset using multiple ASR architectures and LLM classifiers, the study demonstrates that perceptual improvements in speech enhancement can paradoxically increase semantic divergence, particularly through failure modes like speaker substitution in echo scenarios that WER cannot capture. This work provides a critical new evaluation framework for the integration of speech processing and large language models, highlighting the need for task-specific metrics in the design of robust spoken AI pipelines.
Text-to-spatial audio generation, such as text-to-First-Order Ambisonics (FOA), provides a convenient way to create spatial audio for billion-dollar gaming and film industries. However, existing text-to-FOA methods are largely data-driven and may produce audio that violates acoustic relations between source direction and distance. They also separate descriptive and parametric control, forcing users to trade usability for precision. In this paper, we present PhysWave, a physics-guided latent diffusion model for controllable text-to-FOA generation. PhysWave unifies natural-language and trajectory control through a shared waypoint-caption representation, and augments diffusion training with two differentiable acoustic priors: spherical-harmonic direction consistency and inverse-square distance consistency. To support dynamic spatial generation, we further construct a 300K-clip FOA dataset with diverse sound categories and source trajectories. Extensive results show that the proposed priors help PhysWave generate spatially consistent FOA audio while maintaining competitive audio quality. Further analyses show that these physics priors improve spatial consistency during training and can also be used as inference-time guidance for training-free spatial refinement.
Primary: University of Houston
All Institutions: University of Houston, Stevens Institute of Technology, Waseda University
[One sentence main contribution]. The paper presents PhysWave, a physics-guided latent diffusion model that improves spatial consistency in text-to-FOA generation by incorporating differentiable acoustic priors for direction and distance, along with a unified control interface for natural language and parametric trajectories.
The paper proposes PhysWave, a physics-guided latent diffusion model for text-to-First-Order Ambisonics (FOA) generation. The core methodological contribution is the integration of differentiable acoustic priors (spherical-harmonic direction consistency and inverse-square distance consistency) into the diffusion training objective. By decoding the predicted clean latent back to the waveform during training and applying these physical constraints, the model explicitly enforces spatial accuracy rather than relying solely on implicit data-driven learning. Additionally, the paper introduces a unified waypoint-caption representation that allows both natural language descriptions and precise parametric trajectories to condition the same model, bridging the gap between usability and precision.
The authors construct a 300K-clip synthetic FOA dataset, which is a significant resource contribution. Experiments include ablations on the physics losses, comparisons with state-of-the-art methods (ImmerseDiffusion, SonicMotion), and subjective listening tests. The results show that the physics priors significantly reduce angular error and improve distance consistency without degrading audio quality metrics (FAD, CLAP). The subjective evaluation with 14 participants confirms the perceptual benefits.
The paper provides detailed implementation details, including the VAE architecture, diffusion model hyperparameters, and the specific prompts used for the LLM parser. The release of the 300K dataset and the project website enhances reproducibility. However, the reliance on a specific LLM (Qwen3.5-4B) for parsing might introduce variability, though the deterministic post-processing helps.
The model currently focuses on single-source trajectories in a free-field setting, ignoring room effects like reverberation and occlusion. It also does not handle multi-source scenes. The synthetic nature of the dataset, while large, may not fully capture the complexity of real-world spatial audio recordings.
This work has significant potential for the gaming, film, and VR industries by enabling non-experts to create high-quality spatial audio via text. The physics-guided approach could be extended to other domains where physical constraints are known, such as video generation or scientific simulation. [One sentence main contribution]. The paper presents PhysWave, a physics-guided latent diffusion model that improves spatial consistency in text-to-FOA generation by incorporating differentiable acoustic priors for direction and distance, along with a unified control interface for natural language and parametric trajectories.
Speech Language Models (SLMs) are increasingly deployed in multi-speaker environments, yet their ability to attribute speech to the correct speaker and reason over speaker identities remains unclear. Hence, we introduce HEAR, a conceptually hierarchical benchmark diagnosing the foundational capabilities of speaker-attributed reasoning, comprising 2.4K human-verified samples from 887 diverse multi-party audio clips. Evaluating 20 leading SLMs on HEAR reveals they struggle with these foundational tasks, often relying on semantic priors rather than actual vocal cues. To address this, we present A2R, a 30B model optimized on Counterfactual Audio with Speaker-level Hard negatives (CASH), a dataset designed to guide the model to prioritize acoustic vocal cues over linguistic signals. A2R achieves strong performance on HEAR and exhibits zero-shot generalization to diverse multi-speaker downstream tasks, demonstrating that learned speaker attribution unlocks the model's latent capacity for speaker-aware reasoning. All resources are available at https://attributetoreason.github.io/AttributeToReason/
Primary: Seoul National University
All Institutions: Seoul National University, Yonsei University, University of Seoul
The paper presents a rigorous and novel framework for evaluating and improving speaker-attributed reasoning in Speech Language Models. By introducing the HEAR benchmark and the CASH dataset with counterfactual hard negatives, the authors effectively demonstrate that current models rely on semantic priors rather than acoustic cues, and propose a GRPO-based training method (A2R) that successfully shifts this reliance, resulting in strong performance gains and zero-shot generalization to unseen multi-speaker tasks.
The paper introduces a hierarchical benchmark (HEAR) grounded in Erberโs Auditory Hierarchy, decomposing speaker-attributed reasoning into Discrimination, Attribution, and Reasoning. This is a methodologically sound approach to diagnosing specific failure modes in Speech Language Models (SLMs). The proposed training strategy, A2R, utilizes Group Relative Policy Optimization (GRPO) with a composite reward function that explicitly rewards speaker-tagged transcription accuracy (cpWER, speaker count, order) alongside final answer correctness. The creation of the CASH dataset, which uses voice cloning to create counterfactual hard negatives where the transcript remains constant but the speaker identity changes, is a clever and effective way to force the model to rely on acoustic cues rather than semantic priors. The "transcription-first" objective is a strong inductive bias for this specific task.
The experimental evaluation is rigorous and comprehensive. The authors evaluate 20 leading SLMs, providing a clear baseline comparison. The introduction of the "paired accuracy" metric for the Reasoning dimension is particularly insightful, as it exposes models that rely on semantic shortcuts by requiring them to answer both the original and the counterfactual variant correctly. The zero-shot transfer experiments to unseen benchmarks (WDYL, Gaokao, FTS) and the validation on real-world human voices (vs. synthesized) strongly support the claim that the learned attribution capability is generalizable and not an artifact of the training data distribution. The ablation studies effectively isolate the contribution of hard negatives and the structured reasoning trace.
The paper provides high reproducibility. It details the source datasets (AMI, ICSI, VoxMM), the specific TTS models used for synthesis (VoxCPM2, IndexTTS 2), and the filtering criteria (WER < 0.1, speaker similarity > 0.7). The reward function components are clearly defined. The project page is provided, and the authors commit to releasing code and gated access to synthetic data, which is appropriate given the ethical concerns of voice cloning.
The primary limitation is the reliance on synthesized speech for the training data (CASH) and parts of the evaluation, although the authors mitigate this by testing on real human voices. The "transcription-first" approach introduces latency, which the authors acknowledge as a barrier to real-time applications. Additionally, the benchmark size (2.4K samples) is relatively small compared to large-scale LLM benchmarks, though the complexity of the tasks justifies this. The ethical implications of voice cloning are addressed, but the potential for misuse of the released synthetic data remains a concern, mitigated only by a Data Use Agreement.
This work has significant impact on the development of multi-party speech understanding systems. By identifying and addressing the "semantic hallucination" problem in SLMs, it provides a pathway to more robust voice assistants and meeting analysis tools that can accurately track who said what. The benchmark HEAR will likely become a standard evaluation suite for future SLMs, and the CASH dataset offers a valuable resource for training models that prioritize acoustic evidence. The paper presents a rigorous and novel framework for evaluating and improving speaker-attributed reasoning in Speech Language Models. By introducing the HEAR benchmark and the CASH dataset with counterfactual hard negatives, the authors effectively demonstrate that current models rely on semantic priors rather than acoustic cues, and propose a GRPO-based training method (A2R) that successfully shifts this reliance, resulting in strong performance gains and zero-shot generalization to unseen multi-speaker tasks.
Current TTS systems typically rely on open-loop, single-pass generation and can produce sporadic local prosodic defects, such as misplaced stress, unnatural pauses, or flattened intonation, that utterance-level metrics often fail to expose. We present LoopTTS, a judge-guided Filter-Judge-Refiner framework for recovering low-quality TTS outputs diagnosed by an AudioLLM. Given an initial utterance from a base TTS model, an AudioLLM Judge identifies salient prosodic issues and generates structured refine instructions; a Refiner, our fine-grained instruction-following TTS model, then performs guided expressive re-synthesis conditioned on the initial utterance, target text, and instruction. To train the Refiner, we construct Refiner-DB, a 42K-example AudioLLM-annotated dataset with word-level prosodic weak supervision. Human evaluation on diagnosed low-quality utterances shows that LoopTTS can detect perceptually salient errors and correct them with the Refiner, outperforming raw generated audio and practical open-loop re-generation baselines in recovery quality. The Refiner also demonstrates stronger instruction-following ability for stress and pause control in targeted prosody modification.
Primary: National University of Singapore
All Institutions: National University of Singapore, Tencent, LIGHTSPEED, Nanyang Technological University, Independent Researcher, The Chinese University of Hong Kong, Shenzhen, Shenzhen Loop Area Institute
The paper presents a novel closed-loop TTS system that integrates AudioLLM-based diagnosis with a specialized Refiner model to correct local prosodic defects. By introducing a position-aware training objective and a contrastive data construction pipeline, the authors effectively address the challenge of fine-grained prosodic control in TTS, demonstrating significant improvements in recovery quality and instruction-following fidelity over open-loop baselines.
The paper proposes "LoopTTS," a closed-loop TTS framework comprising a Filter, Judge (AudioLLM), and Refiner. The core methodological contribution is the Refiner, a fine-tuned TTS model (based on EmoVoice) that performs guided expressive re-synthesis conditioned on the initial defective audio, target text, and structured instructions from the AudioLLM Judge. The training methodology introduces a "Position-Weighted Cross-Entropy Loss" and a "Structural Alignment Loss" using Representational Similarity Matrices (RSMs) to focus supervision on sparse prosodic spans (stress/pause). The data construction pipeline ("Refiner-DB") uses contrastive AudioLLM annotation to generate weak supervision for word-level prosody. The approach is technically sound, addressing the specific gap of local prosodic defects that utterance-level metrics miss. The use of RSMs for span-level consistency is a creative application of representation learning to TTS fine-tuning.
The experiments are rigorous and well-structured. The authors evaluate the full pipeline on a subset of flagged utterances (200 samples) and a larger scaled evaluation (3,000 samples). They include strong baselines: open-loop re-generation (CosyVoice2, EmoVoice), global-only control, and a human-instruction upper bound. The use of a different AudioLLM (Qwen3-Omni) for the Judge stage demonstrates robustness. Subjective evaluations (MOS, MOS-I) are conducted by professional evaluators with reported ICC reliability. The results show consistent improvements in recovery quality and instruction-following fidelity (stress/pause) over baselines. The ablation studies effectively isolate the contributions of the position-weighting and structural alignment losses.
The paper provides high reproducibility. Code, data construction scripts, prompts, and model checkpoints are promised for release. The GitHub link is provided. Detailed hyperparameters, loss formulations, and dataset composition are included in the main text and appendices. The use of public datasets (ESD, RAVDESS, etc.) and standard models (CosyVoice2, Whisper) further enhances reproducibility.
The primary limitation is the reliance on a proprietary AudioLLM (Gemini-3-pro) for both data annotation and inference-time judging, which may limit accessibility and introduce potential biases. The evaluation is focused on English speech, and the framework's applicability to tonal languages is not tested. The "weak supervision" from AudioLLMs is acknowledged as imperfect, with moderate F1 scores for pause detection. The system is designed for offline batch processing, not streaming, due to the multi-stage latency.
This work has significant implications for the deployment of high-quality TTS systems in production environments where sporadic prosodic errors are unacceptable. It demonstrates a viable path for integrating LLM-based diagnostic capabilities into generative audio pipelines, moving beyond simple generation to generation-and-correction. The methodology of using AudioLLMs for scalable weak supervision of fine-grained prosodic features could be extended to other audio tasks, such as music production or sound effect design. The paper presents a novel closed-loop TTS system that integrates AudioLLM-based diagnosis with a specialized Refiner model to correct local prosodic defects. By introducing a position-aware training objective and a contrastive data construction pipeline, the authors effectively address the challenge of fine-grained prosodic control in TTS, demonstrating significant improvements in recovery quality and instruction-following fidelity over open-loop baselines.