Speech models are increasingly used as an interface to AI systems, from transcription tools to voice agents. Existing audio backdoors require an attacker to introduce triggers at inference time. We study a different threat model in which the attacker compromises the model during fine-tuning but has no access to the victim's inference-time inputs. We introduce the Speaker-Activated Backdoor (SAB) setting, in which a target speaker's natural voice serves as the trigger. We develop a two-phase fine-tuning method that first makes speaker identity more accessible and then binds the target speaker to an attacker-chosen payload. Compared with a one-phase baseline with the same training budget, the two-phase method substantially improves selectivity and clean performance, and is especially important for content-dependent attacks. SAB achieves 90.2% and 97.0% backdoor success on Whisper-medium and Parakeet, with false-positive rates of 1.7% and 0%, respectively using as little as two minutes of real audio from the target speaker. We further demonstrate content-dependent semantic rewriting and downstream tool-use attacks, where malicious instructions are inserted into 96.9% of target-speaker transcripts. When these transcripts are passed to frontier LLM agents, GPT-4o and Claude (Opus, Sonnet, and Haiku), the injected instruction drives the agent to carry out the harmful action in up to 87.5% of cases. The attack generalizes to held-out recordings, multiple target speakers, and common acoustic degradations. These results establish speaker identity as a practical backdoor trigger that requires no inference-time attacker access and is invisible at the transcript layer.
Primary: Imperial College London
All Institutions: Imperial College London
The paper introduces Speaker-Activated Backdoors (SAB), a novel attack where a target speaker's natural voice triggers malicious behavior in speech models without inference-time attacker access. By proposing a two-phase fine-tuning method that enhances speaker discriminability and binds identity to payloads, the authors demonstrate high-success, selective backdoors across multiple ASR and audio-LLM architectures, including downstream harmful actions in tool-using agents, establishing speaker identity as a critical security vector in voice-based AI systems.
The paper introduces a novel threat model for speech models where the trigger is the speaker's identity rather than an injected acoustic perturbation. The proposed two-phase fine-tuning method (SAB) is technically sound: Phase 1 enhances speaker discriminability in the encoder representations, and Phase 2 binds the target speaker to a malicious payload while preserving clean performance on non-target speakers. The ablation studies effectively demonstrate that Phase 1 is crucial for content-dependent attacks (semantic rewriting) and improves selectivity in fixed-output attacks. The use of voice cloning (F5-TTS) to augment limited real target audio is a practical and effective strategy.
The experimental evaluation is comprehensive, covering multiple architectures (Whisper, Parakeet, Qwen-Omni) and attack types (transcript suppression, semantic rewriting, instruction injection). The inclusion of downstream tool-use attacks with frontier LLMs (GPT-4o, Claude) significantly strengthens the impact by demonstrating real-world consequences. The evaluation on held-out real recordings ensures the results are not artifacts of synthetic data. Robustness to acoustic degradation and defense mechanisms (clean fine-tuning, voice anonymization) are also thoroughly analyzed.
The paper provides detailed hyperparameters, dataset splits, and training procedures. However, the specific implementation details for the auxiliary speaker head and the exact prompts used for the LLM agents are not fully specified in the main text, though the appendix provides some additional details. The use of standard datasets (LibriSpeech, VoxCeleb2) and open-source models aids reproducibility.
The attack relies on the attacker having access to the fine-tuning pipeline, which is a standard assumption but still a barrier. The backdoor is susceptible to clean fine-tuning, which limits its persistence in dynamic environments. The evaluation is limited to English speakers and specific public figures, which may not generalize to all demographic groups or private individuals.
This work highlights a significant security vulnerability in the growing ecosystem of voice-activated AI agents. It demonstrates that speaker identity can be exploited as a control channel, posing risks to privacy and safety in applications like voice assistants and automated banking. The findings will likely prompt the development of new defenses focused on speaker-identity invariance and supply-chain security for speech models. The paper introduces Speaker-Activated Backdoors (SAB), a novel attack where a target speaker's natural voice triggers malicious behavior in speech models without inference-time attacker access. By proposing a two-phase fine-tuning method that enhances speaker discriminability and binds identity to payloads, the authors demonstrate high-success, selective backdoors across multiple ASR and audio-LLM architectures, including downstream harmful actions in tool-using agents, establishing speaker identity as a critical security vector in voice-based AI systems.
We introduce MixAgent, an agentic pipeline for music post-production that follows the workflow of audio engineers to build interpretable and editable mixing plans guided by a reference song. A grouping agent first routes related input tracks, such as drum recordings, into subgroups. A planning agent then uses retrieved examples to choose and order audio effects, such as volume control and reverb, for tracks, subgroups, and the full mix. To refine the mix, a detector identifies what needs correction, and a critic proposes numerical parameter adjustments from tracks to the full mix. In our experiments, prompting recent multimodal models to listen and revise often worsens the mix, making reliable correction a central challenge. We therefore build MixCritic, a dataset of 45,657 repair critiques derived from parameters fitted to professional mixes, and train the critic with supervised fine-tuning followed by preference optimization on rendered audio outcomes. On MedleyDB, learned refinement reduces error by 19% relative to the initial mix, with preference optimization reducing it by 8.4% over supervised training alone. Retrieved examples further improve effects selection, and a human listening test shows that MixAgent gets higher quality ratings than mixes produced by Qwen3-Omni, Gemini 3.8 Flash and GPT-4o-audio.
Primary: Sony Group Corporation
All Institutions: Rice University, Sony Corporate Technology Center America, Inc., Sony Group Corporation
MixAgent introduces a novel agentic pipeline for music post-production that combines retrieval-augmented planning with learned numerical refinement, achieving state-of-the-art results in reference-guided mixing and providing a robust framework for interpretable audio editing.
The paper proposes MixAgent, a multi-agent framework for reference-guided music mixing. The methodology is structured into three distinct phases: grouping (routing tracks to subgroups), planning (selecting and ordering audio effects via retrieval-augmented generation), and refinement (detecting and correcting parameter errors). A key methodological contribution is the construction of the MixCritic dataset, which pairs corrupted mixing states with repair critiques derived from professional mixes. The authors employ a two-stage training process for the critic: Supervised Fine-Tuning (SFT) followed by Direct Preference Optimization (DPO) on rendered audio outcomes. The system explicitly models the hierarchical nature of mixing (tracks -> subgroups -> full mix) and uses a bottom-up refinement strategy, which is shown to be superior to top-down or flat approaches. The use of a frozen planner with retrieved examples to guide effect selection is a practical and effective design choice that mitigates the tendency of LLMs to select suboptimal or overly short effect chains.
The experiments are rigorous and comprehensive. The authors evaluate on MedleyDB and AlbumDB, using non-overlapping excerpts to prevent data leakage. They compare against strong baselines, including recent multimodal LLMs (Qwen3-Omni, Gemini 3.8 Flash, GPT-4o-audio) under various prompting strategies (Single-pass, Self-Refine, Self-Critic). The results demonstrate that MixAgent outperforms these baselines in reconstruction error (MSS) and predicted quality metrics. A particularly insightful analysis is the comparison of feedback forms (perceptual, tool-specific, numerical), showing that numerical corrections trained via DPO yield the best performance. The human listening test, while small (11 participants), provides qualitative validation that the system produces mixes rated higher than LLM baselines, though still below professional human mixes. The ablation studies on retrieval, refinement order, and detector selection are thorough and provide clear insights into the system's components.
The paper provides high reproducibility. The authors release the code and demo on GitHub. They detail the architecture of the detector and critic, the training procedures (SFT and DPO), and the construction of the MixCritic dataset. The prompts used for the agents are included in the appendix. The use of standard datasets (MedleyDB, AlbumDB) and public models (Qwen, Gemini, GPT) further enhances reproducibility. The specific hyperparameters and training settings are listed in tables, allowing for faithful replication.
The primary limitation is the small size of the evaluation sets (20 MedleyDB clips, 14 AlbumDB clips), which limits statistical power and generalizability. The listening test with only 11 participants is also a weakness. The system relies on GRAFX-fitted parameters for supervision, which may not perfectly capture the intent of professional engineers. The audio effects library is limited to a subset of professional tools, and the plan is fixed after the planning stage, meaning the critic cannot add or remove effects. The computational cost of multiple LLM calls for mixing is also noted as a practical limitation.
This work has significant implications for the field of AI-assisted creative tools. By providing an interpretable and editable mixing plan, it addresses a key barrier to adoption of AI in professional audio workflows. The findings on the effectiveness of numerical feedback and preference optimization for audio parameter adjustment are broadly applicable to other domains where LLMs control numerical tools. The construction of the MixCritic dataset is a valuable resource for the community. The paper also highlights the challenges of evaluating audio quality with current metrics, pointing to the need for better perceptual evaluation methods. MixAgent introduces a novel agentic pipeline for music post-production that combines retrieval-augmented planning with learned numerical refinement, achieving state-of-the-art results in reference-guided mixing and providing a robust framework for interpretable audio editing.
Omni LLMs can naturally locate key audio and visual evidence and integrate information across modalities for audiovisual reasoning. Existing audio-visual reasoning benchmarks that mainly rely on multiple-choice questions usually score models solely based on answer correctness, making it difficult to assess whether a model truly forms a reasoning chain from evidence to the correct answer. In addition, textual or unimodal shortcuts may allow models to answer correctly without analyzing the necessary audio-visual evidence, further weakening the reliability of evaluation benchmarks. To address these issues, we introduce EvaOmni, an evidence-grounded benchmark containing 392 videos and 1,027 purely human-annotated question-answer pairs, covering 15 capabilities and three levels: retrieval and localization, basic understanding, and complex reasoning. We design a series of rigorous annotation procedures in which experts annotate audio and visual evidence and construct video-based distractors; humans review whether questions meet strict inspection criteria, and finally models filter textual and unimodal shortcuts. Beyond high-quality question-answer pairs, we require models to simultaneously return the answer and supporting evidence intervals in order to evaluate the complete reasoning chain. We propose the Coverage Grounded Score (C.G.) to measure how much annotated evidence a model identifies, and the Bottleneck Grounded Score (B.G.) to penalize poor temporal localization in either modality; both are incorporated into answer correctness. Across 7 open-weight and 17 proprietary Omni LLMs, Gemini3.7-Flash achieved the highest overall accuracy (79.4%), C.G. (70.8%), and B.G. (35.1%). At Level 3, it reached 79.1% accuracy, but its B.G. was only 12.3%. These results show that even the most advanced models still have room for development in grounded audio-visual reasoning.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University, Shanghai Innovation Institute, Inner Mongolia University of Technology, Fudan University, Beijing Institute of Technology, Tongji University, The Hong Kong University of Science and Technology (Guangzhou), Tsinghua University, The University of Hong Kong, Tencent Youtu Lab
EvaOmni introduces an evidence-grounded benchmark for audio-visual reasoning that requires models to provide supporting evidence intervals, revealing significant gaps in temporal localization capabilities of current Omni LLMs. The paper makes a strong contribution by addressing the limitations of answer-only evaluation in multimodal benchmarks, proposing rigorous annotation procedures to mitigate shortcuts, and introducing novel metrics (C.G. and B.G.) that assess both answer correctness and evidence grounding. The extensive evaluation across 24 models demonstrates that while top models achieve high accuracy, their ability to precisely localize supporting evidence in both audio and visual modalities remains limited, providing valuable insights for future development of Omni LLMs.
The paper proposes EvaOmni, a benchmark that shifts the evaluation paradigm from answer-only correctness to evidence-grounded reasoning. The core methodological contribution is the requirement for models to output timestamped evidence intervals for both audio and visual modalities. The authors introduce two novel metrics: Coverage Grounded Score (C.G.), which measures the overlap of predicted evidence with ground truth, and Bottleneck Grounded Score (B.G.), which penalizes the weaker modality's localization accuracy. The annotation pipeline is rigorous, involving expert annotation, cross-validation, and model-assisted filtering to remove unimodal or textual shortcuts. This approach effectively addresses the "shortcut" problem in existing benchmarks where models can answer correctly without processing the full audio-visual context.
The evaluation is extensive, covering 24 Omni LLMs (7 open-weight, 17 proprietary) across 1,027 question-answer pairs. The results clearly demonstrate that while top models like Gemini3.7-Flash achieve high accuracy (79.4%), their grounding scores (B.G. 35.1%) are significantly lower, revealing a gap in precise temporal localization. The ablation studies on input modalities (Audio vs. ASR, Video vs. Audio+Video) provide valuable insights into how models utilize different cues. The comparison with human experts further highlights the difficulty of the task, as humans achieve much higher grounding scores. The cost-efficiency analysis adds practical value for model selection.
The paper provides detailed descriptions of the annotation process, metric definitions, and experimental setup. The authors state that data, annotations, and evaluation framework are fully released. However, specific URLs for the dataset or code are not explicitly provided in the text snippet, which slightly hinders immediate reproducibility verification, though the claim of release is strong. The prompt templates and preprocessing details are described in the appendix, aiding reproducibility.
The benchmark is limited to videos of 1-12 minutes, which may not capture reasoning over longer time horizons. The exclusion of hard-coded subtitles, while preventing OCR shortcuts, limits the diversity of real-world video content. The reliance on multiple-choice questions, even with grounding, may not fully capture open-ended reasoning capabilities. The human baseline allows replay, which is not possible for models, creating a slight asymmetry in comparison conditions.
This work has significant impact on the evaluation of multimodal LLMs. By introducing grounded metrics, it provides a more reliable way to assess whether models truly understand audio-visual content or are relying on shortcuts. This can guide future model development towards better temporal localization and cross-modal integration. The findings that even state-of-the-art models struggle with precise grounding highlight a key area for future research in Omni LLMs. EvaOmni introduces an evidence-grounded benchmark for audio-visual reasoning that requires models to provide supporting evidence intervals, revealing significant gaps in temporal localization capabilities of current Omni LLMs. The paper makes a strong contribution by addressing the limitations of answer-only evaluation in multimodal benchmarks, proposing rigorous annotation procedures to mitigate shortcuts, and introducing novel metrics (C.G. and B.G.) that assess both answer correctness and evidence grounding. The extensive evaluation across 24 models demonstrates that while top models achieve high accuracy, their ability to precisely localize supporting evidence in both audio and visual modalities remains limited, providing valuable insights for future development of Omni LLMs.
Music-driven group dance generation must decide not only how each dancer moves, but also what formation the group should form, when it should change, and who should execute each transition. Existing methods can generate orderly and rhythmically synchronized group motion, but remain limited in formation evolution and inter-dancer interaction. In human choreography, formation transitions often align with changes in high-level musical structure, such as the progression from an introduction to a chorus or from a chorus to a climax. We present ChoreoStudio, which makes choreography planning explicit. A music disentangler first decomposes the audio into frame-level rhythmic features and segment-level structural features. Given the segment-level features and sparse labels, a frozen omni-modal model acts as a scriptwriter, producing a global semantic description that guides the Director's choreography planning. Based on this context, the Director produces three coupled interaction decisions: a centroid-relative point set specifies what formation to produce, a transition gate determines when it should change, and a mobility allocation indicates who should execute the transition. The transition gate and mobility allocation are further conditioned on the planned geometric change, coupling formation targets with their temporal and dancer-specific execution. At a finer temporal scale, frame-aligned music features condition the diffusion generator to control detailed body motion and preserve beat-level alignment. The segment-level interaction plan is injected in parallel through frame-aligned instruction tokens and cross-dancer attention biases, coordinating how the generator realizes each transition. Experiments on AIOZ-GDance demonstrate that ChoreoStudio follows the planned interactions while achieving state-of-the-art performance in motion quality, group organization, and formation dynamics.
Primary: Monash University
All Institutions: Monash University
ChoreoStudio introduces a novel explicit choreography planning framework that decouples high-level formation, timing, and role decisions from low-level motion synthesis, achieving state-of-the-art performance in group dance generation by leveraging multi-scale music conditioning and frozen LLM semantic intent to guide a structured Director module, thereby addressing the critical gap in macro-level formation evolution and inter-dancer interaction control in existing methods.
The paper proposes ChoreoStudio, a hierarchical framework that explicitly decouples choreography planning (what, when, who) from motion synthesis (how). The core novelty lies in the "Director" module, which generates a structured plan consisting of relative formation geometry, transition timing gates, and dancer mobility allocations. This plan is derived from multi-scale music features (frame-level rhythm and segment-level structure) and semantic intent from a frozen omni-modal LLM (Qwen3-Omni). The method addresses a specific gap in existing group dance generation: the lack of explicit control over macro-level formation evolution and asymmetric role assignment. The use of a frozen LLM for high-level semantic planning is a strong architectural choice, leveraging recent advances in multimodal understanding without requiring fine-tuning of the large model. The explicit plan-to-motion interface, using frame-aligned instruction tokens and cross-dancer attention biases, is technically sound and ensures that the generated motion adheres to the planned choreography. The verification-guided refinement loop adds robustness by allowing the system to iteratively correct physical or plan-consistency errors.
Experiments are conducted on the AIOZ-GDance dataset, which is a standard benchmark for group dance generation. The paper introduces novel frequency-based metrics (IDDV, Jitter, FMR) to evaluate formation dynamics, which is a significant contribution as standard metrics often miss macro-level structural issues. The quantitative results show state-of-the-the-art performance in Group Motion Realism (GMR) and Group Motion Correlation (GMC) across various group sizes (2-7 dancers). The ablation studies effectively demonstrate the contribution of each component, particularly the Director and the semantic description. The controlled intervention experiments are a strong point, providing causal evidence that the model responds to changes in the planned formation, timing, and mobility. The comparison with baselines like TCDiff++ and ST-GDance is fair and uses a shared protocol.
The paper provides detailed descriptions of the architecture, training objectives, and evaluation metrics. The use of a frozen LLM (Qwen3-Omni) and standard diffusion models enhances reproducibility compared to custom large models. However, the specific implementation details of the "Director" and the exact prompts used for the LLM are not fully specified in the main text (referenced to appendices). The code availability is not explicitly stated in the provided text, but the detailed methodology suggests it is reproducible for a skilled team. The use of standard datasets and metrics further aids reproducibility.
The method relies on a frozen LLM for semantic planning, which may limit the adaptability to novel or complex musical styles not well-represented in the LLM's training data. The planning is done at a segment level, which may not capture very rapid or micro-level choreographic changes. The evaluation is limited to the AIOZ-GDance dataset, and generalization to other datasets or real-world scenarios is not tested. The computational cost of the iterative refinement loop and the LLM inference may be significant for real-time applications.
This work has significant implications for the field of motion generation, particularly for applications in virtual reality, gaming, and automated video production. By making choreography planning explicit and editable, the system enables more controllable and interpretable group dance generation. The approach could be extended to other multi-agent motion tasks, such as crowd simulation or robotic swarm coordination. The integration of LLMs for high-level planning in motion generation is a promising direction that could lead to more sophisticated and semantically aware generative models. ChoreoStudio introduces a novel explicit choreography planning framework that decouples high-level formation, timing, and role decisions from low-level motion synthesis, achieving state-of-the-art performance in group dance generation by leveraging multi-scale music conditioning and frozen LLM semantic intent to guide a structured Director module, thereby addressing the critical gap in macro-level formation evolution and inter-dancer interaction control in existing methods.
Auditory attention decoding (AAD) seeks to identify the attended talker from brief noninvasive brain recordings by comparing neural responses with competing speech streams. Recent electroencephalography (EEG) decoders report nearly perfect accuracy from one second of data, yet we show that this performance fails to generalize to unseen speech because models exploit trial leakage, eye gaze, and acoustic cues instead of speech evoked neural activity. We introduce CoupleFlow, which decodes attention through temporal coupling between neural and acoustic embeddings. Its predictions remain at chance when the brain signal contains no informative coupling, preventing shortcuts based on speech content or talker identity. A conditional flow matching head implements a likelihood ratio decision rule based on pointwise mutual information between brain and speech. This rule is Bayes optimal when the recording depends only on the attended talker and removes preferences for speech that is likely a priori. We evaluate CoupleFlow with content disjoint splits recovered from the audio and null tests that distinguish genuine temporal coupling from trial recognition. Across nine EEG and magnetoencephalography (MEG) datasets processed through a common pipeline, spanning four languages, high density EEG recordings, listeners with hearing impairment, and a room with multiple talkers, CoupleFlow outperforms linear and deep baselines while all null tests remain at chance. Finally, we derive an evidence accumulation law that predicts accuracy across window lengths from a single parameter per dataset and predicts that parameter from the strength of neural speech tracking. This law identifies recording signal to noise ratio as a limit on decoding speed and reliability and estimates the window length required to reach a target accuracy.
Primary: The Ohio State University
All Institutions: The Ohio State University
CoupleFlow introduces a null-calibrated auditory attention decoder that uses flow-matching likelihood ratios to ensure predictions are driven by genuine brain-speech coupling rather than acoustic shortcuts. The paper makes a substantial contribution to the field by exposing the fragility of previous AAD models through rigorous null testing and providing a theoretically grounded, robust alternative that generalizes across diverse datasets and languages.
The paper introduces CoupleFlow, a decoder for Auditory Attention Decoding (AAD) that addresses the critical issue of shortcut learning in previous EEG/MEG studies. The core methodological innovation is the "null-calibrated" design: the model relies on temporal coupling between neural and acoustic embeddings, ensuring that if the brain signal contains no information about the speech (e.g., shuffled or unrelated trials), the decoder's performance drops to chance. This is a significant architectural constraint that prevents the model from exploiting static acoustic cues (like loudness or speaker identity) or trial leakage. The use of a conditional flow-matching head to implement a likelihood ratio decision rule based on pointwise mutual information (PMI) is a sophisticated application of generative modeling to a discriminative task. The theoretical grounding, particularly the derivation of the "evidence accumulation law" which predicts accuracy based on window length and signal-to-noise ratio, provides a strong analytical framework for understanding the limits of non-invasive decoding.
The experimental evaluation is exceptionally rigorous. The authors test on nine diverse EEG and MEG datasets, spanning four languages, different electrode densities, and populations with hearing impairments. Crucially, they employ content-disjoint splits and null tests (shuffled EEG, time-shifted EEG) to verify that performance is driven by genuine neural coupling rather than data leakage. The finding that previous state-of-the-art models fail these null tests while CoupleFlow maintains performance is a strong contribution to the field's integrity. The comparison with linear and deep baselines under a unified preprocessing pipeline further strengthens the validity of the results.
The paper provides detailed descriptions of the preprocessing pipeline, model architecture, and loss functions. The use of public datasets and a common pipeline enhances reproducibility. However, specific hyperparameters and code availability are not explicitly confirmed in the provided text (no GitHub link found), which is a minor deduction. The detailed theoretical derivations and ablation studies (implied by the discussion of loss terms) support reproducibility.
The primary limitation is the reliance on non-invasive signals (EEG/MEG), which inherently have lower spatial resolution and signal-to-noise ratios compared to invasive recordings. The "evidence accumulation law" assumes certain statistical properties (e.g., independent samples, Gaussian noise) that may not perfectly hold in all real-world noisy environments. Additionally, the model's performance on very short windows (sub-second) may still be limited by the physical constraints of neural response latency and SNR, as predicted by their own law.
This work has significant implications for the development of neuro-steered hearing devices and brain-computer interfaces. By providing a robust, leakage-free benchmark and a decoder that genuinely tracks attended speech, it paves the way for more reliable real-world applications. The theoretical framework for predicting decoding accuracy based on recording quality is valuable for designing future experiments and hardware. It also serves as a cautionary tale for the ML community regarding the importance of null tests and proper data splitting in neural decoding tasks. CoupleFlow introduces a null-calibrated auditory attention decoder that uses flow-matching likelihood ratios to ensure predictions are driven by genuine brain-speech coupling rather than acoustic shortcuts. The paper makes a substantial contribution to the field by exposing the fragility of previous AAD models through rigorous null testing and providing a theoretically grounded, robust alternative that generalizes across diverse datasets and languages.
Streaming audio-visual target speaker extraction (AVTSE) is essential for latency-sensitive applications. However, current causal systems face two coupled gaps. First, standard benchmarks fix the number of speakers at two and ignore the near-/far-field interference, reverberation, and sudden noise of real scenes. Second, high-performance causal architectures are too heavy for edge deployment. Lightweight alternatives try to reclaim capacity by repeatedly invoking a small separator, which offsets the savings. To address the first gap, we release RealSSA, a realistic AVTSE benchmark consisting of RealSSA-Sim and RealSSA-Real. RealSSA-Sim provides controllable simulated scenes with 1--5 dynamic near-field interfering speakers; scenes with far-field interference additionally include 10--15 far-field interfering speakers. RealSSA-Real provides a held-out real-recorded evaluation set. To address the second gap, we propose Falcon, a causal AVTSE method. Its Separator performs multi-scale time-frequency separation in a single encoder--decoder pass. We train Falcon with an encoder-space speaker contrastive loss. This loss suppresses near-field leakage at zero inference cost and transfers across backbones. Falcon achieves state-of-the-art extraction quality on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix, and VoxCeleb2-2Mix. Compared with the causal AV-TFGridNet baseline, it cuts parameters by 91.9%, computation by 20.2 times, and GPU inference latency by 11.1 times. Code and demo are available at https://cslikai.cn/Falcon/.
Primary: Tsinghua University
All Institutions: Qinghai University, Tsinghua University, Intelligent Computing and Application Laboratory of Qinghai Province, Chinese Institute for Brain Research (CIBR)
The paper introduces Falcon, a lightweight causal AVTSE model, and RealSSA, a realistic benchmark, achieving significant efficiency gains over existing causal baselines while maintaining state-of-the-art extraction quality. The technical contribution is strong, particularly in addressing the efficiency-quality trade-off in causal systems and providing a more realistic evaluation environment for the field.
The paper proposes Falcon, a causal audio-visual target speaker extraction (AVTSE) model, and RealSSA, a new benchmark. The methodology addresses two specific gaps: the lack of realistic acoustic conditions in existing benchmarks and the high computational cost of causal models. Falcon utilizes a single-pass asymmetric U-Net separator with multi-scale time-frequency modeling, avoiding the iterative recurrence found in prior lightweight models like SwiftNet. A key technical contribution is the "encoder-space speaker contrastive loss," which explicitly suppresses leakage from near-field interferers during training without adding inference cost. The architecture uses a frozen FRCNN visual backbone and a causal LSTM for video processing, fused with audio features via addition. The design is logically sound, aiming to balance latency constraints with extraction quality by replacing iterative small separators with a single, more efficient pass.
The experimental setup is robust, evaluating Falcon on the new RealSSA-Sim, RealSSA-Real, and standard benchmarks (LRS2-2Mix, LRS3-2Mix, VoxCeleb2-2Mix). The paper claims state-of-the-art performance on these benchmarks. Specifically, it reports a 91.9% reduction in parameters and an 11.1x reduction in GPU inference latency compared to the causal AV-TFGridNet baseline. The inclusion of a real-world recorded set (RealSSA-Real) is a significant strength, as it tests generalization beyond simulated mixtures. The use of metrics like SI-SNR, WER, DNSMOS, and NISQA provides a comprehensive evaluation of both separation quality and perceptual/recognizability aspects.
The paper provides a code and demo URL (https://cslikai.cn/Falcon/), which enhances reproducibility. The dataset generation pipeline for RealSSA-Sim is detailed, including specific parameters for room impulse responses (gpuRIR), source placement, and mixing levels (EBU R128). The architectural details of the Separator, including convolution types, padding strategies, and the definition of the causal layer normalization, are described in sufficient detail for re-implementation.
The visual backbone (FRCNN) is frozen, which may limit the model's ability to adapt to diverse visual conditions or lighting changes not present in the training data. The contrastive loss relies on having clean references for interferers during training, which is feasible in simulation but might be challenging in unsupervised or semi-supervised real-world settings. The paper is an arXiv preprint with 0 citations, so long-term impact is yet to be determined. The "RealSSA-Real" set is described as having pixelated faces, which is good for privacy but may introduce artifacts that affect visual feature extraction.
This work has significant implications for edge deployment of speech enhancement systems, such as in hearing aids, video conferencing, and smart speakers. By providing a lightweight causal model and a realistic benchmark, it facilitates the development of more practical, low-latency audio-visual systems. The benchmark addresses a critical need for evaluating models under realistic, non-anechoic conditions with varying numbers of speakers. The paper introduces Falcon, a lightweight causal AVTSE model, and RealSSA, a realistic benchmark, achieving significant efficiency gains over existing causal baselines while maintaining state-of-the-art extraction quality. The technical contribution is strong, particularly in addressing the efficiency-quality trade-off in causal systems and providing a more realistic evaluation environment for the field.
This paper introduces Spark-Audio 1.0, a speech foundation model trained on domestically produced computing infrastructure. The model accepts text and speech inputs and generates text outputs. It supports multilingual (99 languages) and multi-dialect (202 dialects) speech recognition, speech translation, audio perception and understanding, and both text- and speech-based dialogue, marking a shift from merely ``hearing clearly'' to genuinely ``understanding'' speech. Architecturally, Spark-Audio 1.0 couples a 0.65B-parameter dense audio encoder with a 30B-A3B Mixture-of-Experts (MoE) large language model (LLM). The audio encoder is pretrained with representation-enhanced BEST-RQ and then refined through multi-task supervised training. The audio--LLM is trained through staged pretraining, followed by post-training that combines supervised fine-tuning with on-policy distillation. In total, the model is trained on 13 million hours of audio data together with large-scale text data. We evaluate Spark-Audio 1.0 on mainstream audio and text benchmarks, as well as on in-house test sets drawn from real-world application scenarios. Despite using substantially less audio training data than Qwen3.5-Omni-Flash, Spark-Audio 1.0 achieves comparable performance on most audio and text tasks, outperforms it in multilingual speech recognition, spoken dialogue, and real-world application scenarios, and approaches the performance of larger closed-source models. Nevertheless, there remains room for improvement in instruction following and complex audio reasoning. Finally, the successful training of Spark-Audio 1.0 on domestic hardware, including Huawei Ascend 910B and Hygon BW1000 chips, demonstrates the feasibility of a domestic technical pathway for building speech foundation models.
Primary: iFlytek
All Institutions: iFlytek
The main contribution is the demonstration of a competitive, data-efficient speech foundation model trained entirely on domestic Chinese hardware. The paper provides a detailed technical report on the Spark-Audio 1.0 architecture, highlighting a novel representation-enhanced pretraining strategy and a unified streaming/non-streaming encoder design that achieves strong multilingual performance with significantly less data than comparable competitors.
The paper proposes a unified speech foundation model architecture combining a 0.65B dense audio encoder (based on Whisper-Large v3) with a 30B-A3B MoE LLM. The core methodological contribution is the "representation-enhanced BEST-RQ" pretraining strategy, which uses frozen semantic representations from a multilingual ASR teacher to generate discrete targets for the student encoder, aiming to inject semantic priors without additional trainable tokenizers. The training pipeline is staged: self-supervised pretraining, supervised ASR fine-tuning, and multi-task perception training (emotion, gender, etc.). The audio-LLM integration uses a staged recipe including adapter alignment, joint optimization, and long-context extension, followed by SFT and on-policy distillation (OPD) to preserve text capabilities. The unified streaming/non-streaming encoder design via dynamic chunk training is a practical engineering contribution.
The model is evaluated on 99 languages and 202 dialects, covering ASR, translation, audio understanding, and dialogue. The paper claims comparable performance to Qwen3.5-Omni-Flash with significantly less data (13M hours vs. implied larger scale for Qwen) and outperforms it in multilingual recognition and real-world scenarios. It also approaches larger closed-source models. However, the evaluation relies heavily on "in-house test sets" which are not publicly available, limiting independent verification. The comparison with Qwen3.5-Omni-Flash is the primary baseline, but details on the specific data scale of the competitor are somewhat vague ("substantially less").
Reproducibility is low for the general community. The model is trained on specific domestic hardware (Huawei Ascend 910B, Hygon BW1000), and the code/weights are not released in the paper. The "in-house" benchmarks are proprietary. While the architectural details (Whisper-based encoder, MoE LLM) are standard, the specific data curation (13M hours) and the "representation-enhanced" teacher setup are not fully detailed enough for exact replication without access to iFlytek's internal resources.
The paper explicitly acknowledges room for improvement in instruction following and complex audio reasoning. The reliance on proprietary in-house benchmarks makes it difficult for the community to verify the claimed superiority over Qwen3.5-Omni-Flash. The "domestic hardware" aspect, while strategically important, limits the immediate technical applicability for researchers using standard CUDA stacks. The text-only output modality restricts it to understanding rather than generation (speech-to-speech).
The paper demonstrates the feasibility of training large-scale speech foundation models on non-NVIDIA hardware, which is significant for the Chinese AI ecosystem and potentially other regions with hardware restrictions. It contributes to the trend of data-efficient training in audio-LLMs. The support for 202 dialects is a substantial resource for multilingual NLP research, assuming the model or data becomes available. The main contribution is the demonstration of a competitive, data-efficient speech foundation model trained entirely on domestic Chinese hardware. The paper provides a detailed technical report on the Spark-Audio 1.0 architecture, highlighting a novel representation-enhanced pretraining strategy and a unified streaming/non-streaming encoder design that achieves strong multilingual performance with significantly less data than comparable competitors.
We introduce MixAgent, an agentic pipeline for music post-production that follows the workflow of audio engineers to build interpretable and editable mixing plans guided by a reference song. A grouping agent first routes related input tracks, such as drum recordings, into subgroups. A planning agent then uses retrieved examples to choose and order audio effects, such as volume control and reverb, for tracks, subgroups, and the full mix. To refine the mix, a detector identifies what needs correction, and a critic proposes numerical parameter adjustments from tracks to the full mix. In our experiments, prompting recent multimodal models to listen and revise often worsens the mix, making reliable correction a central challenge. We therefore build MixCritic, a dataset of 45,657 repair critiques derived from parameters fitted to professional mixes, and train the critic with supervised fine-tuning followed by preference optimization on rendered audio outcomes. On MedleyDB, learned refinement reduces error by 19% relative to the initial mix, with preference optimization reducing it by 8.4% over supervised training alone. Retrieved examples further improve effects selection, and a human listening test shows that MixAgent gets higher quality ratings than mixes produced by Qwen3-Omni, Gemini 3.8 Flash and GPT-4o-audio.
Primary: Sony Group Corporation
All Institutions: Rice University, Sony Corporate Technology Center America, Inc., Sony Group Corporation
MixAgent introduces a novel agentic pipeline for music post-production that combines retrieval-augmented planning with learned numerical refinement, achieving state-of-the-art results in reference-guided mixing and providing a robust framework for interpretable audio editing.
The paper proposes MixAgent, a multi-agent framework for reference-guided music mixing. The methodology is structured into three distinct phases: grouping (routing tracks to subgroups), planning (selecting and ordering audio effects via retrieval-augmented generation), and refinement (detecting and correcting parameter errors). A key methodological contribution is the construction of the MixCritic dataset, which pairs corrupted mixing states with repair critiques derived from professional mixes. The authors employ a two-stage training process for the critic: Supervised Fine-Tuning (SFT) followed by Direct Preference Optimization (DPO) on rendered audio outcomes. The system explicitly models the hierarchical nature of mixing (tracks -> subgroups -> full mix) and uses a bottom-up refinement strategy, which is shown to be superior to top-down or flat approaches. The use of a frozen planner with retrieved examples to guide effect selection is a practical and effective design choice that mitigates the tendency of LLMs to select suboptimal or overly short effect chains.
The experiments are rigorous and comprehensive. The authors evaluate on MedleyDB and AlbumDB, using non-overlapping excerpts to prevent data leakage. They compare against strong baselines, including recent multimodal LLMs (Qwen3-Omni, Gemini 3.8 Flash, GPT-4o-audio) under various prompting strategies (Single-pass, Self-Refine, Self-Critic). The results demonstrate that MixAgent outperforms these baselines in reconstruction error (MSS) and predicted quality metrics. A particularly insightful analysis is the comparison of feedback forms (perceptual, tool-specific, numerical), showing that numerical corrections trained via DPO yield the best performance. The human listening test, while small (11 participants), provides qualitative validation that the system produces mixes rated higher than LLM baselines, though still below professional human mixes. The ablation studies on retrieval, refinement order, and detector selection are thorough and provide clear insights into the system's components.
The paper provides high reproducibility. The authors release the code and demo on GitHub. They detail the architecture of the detector and critic, the training procedures (SFT and DPO), and the construction of the MixCritic dataset. The prompts used for the agents are included in the appendix. The use of standard datasets (MedleyDB, AlbumDB) and public models (Qwen, Gemini, GPT) further enhances reproducibility. The specific hyperparameters and training settings are listed in tables, allowing for faithful replication.
The primary limitation is the small size of the evaluation sets (20 MedleyDB clips, 14 AlbumDB clips), which limits statistical power and generalizability. The listening test with only 11 participants is also a weakness. The system relies on GRAFX-fitted parameters for supervision, which may not perfectly capture the intent of professional engineers. The audio effects library is limited to a subset of professional tools, and the plan is fixed after the planning stage, meaning the critic cannot add or remove effects. The computational cost of multiple LLM calls for mixing is also noted as a practical limitation.
This work has significant implications for the field of AI-assisted creative tools. By providing an interpretable and editable mixing plan, it addresses a key barrier to adoption of AI in professional audio workflows. The findings on the effectiveness of numerical feedback and preference optimization for audio parameter adjustment are broadly applicable to other domains where LLMs control numerical tools. The construction of the MixCritic dataset is a valuable resource for the community. The paper also highlights the challenges of evaluating audio quality with current metrics, pointing to the need for better perceptual evaluation methods. MixAgent introduces a novel agentic pipeline for music post-production that combines retrieval-augmented planning with learned numerical refinement, achieving state-of-the-art results in reference-guided mixing and providing a robust framework for interpretable audio editing.
Most voice-cloning systems are built for audio alone: they copy the reference recording's delivery and acoustics, so a voice cloned on a phone sounds like a phone recording whether the speaker is pictured in a studio, on a street, or in a cathedral. Speech for a talking video should sound like the pictured person speaking in the pictured place. We present HeyGen Voice, a scene-aware voice-cloning model that separates who is speaking from how and where: it keeps the speaker's voice from an everyday recording while the target image sets the delivery and acoustics. Prior image-conditioned TTS does half of this job: it either adds scene acoustics without keeping a specific speaker or infers a voice from a face. HeyGen Voice is a multimodal autoregressive transformer conditioned on the image, speaker and scene captions, and the voice recording, trained in five stages that move from multilingual TTS pretraining through simulated and real cross-scene adaptation to reinforcement and preference optimization. Two clone policies trade scene adaptation against fidelity to the original recording, and an audio anchor holds long scripts in one voice and one acoustic environment. Listeners prefer HeyGen Voice over leading commercial systems across a third-party arena, production votes, and talking-video studies: it ranks first (Elo 1201) on the public Artificial Analysis controlled-voice TTS leaderboard (October 2026); in a blind test, users cloning their own voice chose it over ElevenLabs v4 and Cartesia Sonic 3.6 in 62% and 54% of votes; and volunteers rated talking videos of their own digital twin about 10 to 30 points higher (out of 100) when voiced by HeyGen Voice than by leading commercial alternatives. Try HeyGen Voice and hear samples at https://heygen.com/heygen-voice.
Primary: HeyGen Research
All Institutions: HeyGen Research
HeyGen Voice introduces a scene-aware voice cloning framework that decouples speaker identity from acoustic environment using multimodal conditioning and a cross-scene training curriculum. The paper demonstrates state-of-the-art performance in both objective cross-scene benchmarks and subjective human evaluations, establishing a new standard for audio-visual coherence in talking video generation.
The paper proposes "HeyGen Voice," a multimodal autoregressive transformer for scene-aware voice cloning. The core methodological contribution is the decoupling of speaker identity (timbre, accent) from scene acoustics (reverberation, background noise) and delivery style. This is achieved through a conditioning prefix that combines pooled reference-voice tokens, image tokens (from a frozen VLM), and explicit text captions for speaker and background. A key technical innovation is the "cross-scene training curriculum," which includes a simulated adaptation stage using reference perturbations to remove the shortcut of relying on the reference recording's acoustics, followed by fine-tuning on real cross-scene pairs. The use of GRPO for intelligibility/speaker similarity and DPO with a low-rank adapter for expressiveness is a robust application of modern alignment techniques to TTS. The "clone policies" (voice-clone vs. full-clone) provide a practical mechanism for users to control the trade-off between fidelity to the source recording and adaptation to the target scene.
The evaluation is extensive and multi-faceted, covering objective benchmarks (cross-scene similarity, multilingual CER), subjective human evaluations (paired listening tests in 5 languages), production user votes, third-party arena rankings (Artificial Analysis), and video-based studies. The cross-scene benchmark (331 cases) is a valuable contribution to the field, as it specifically tests the ability to adapt voice to new environments, a task often ignored in standard TTS benchmarks. The human studies are well-designed, including blind tests with users cloning their own voices and volunteer studies rating talking videos. The results show consistent superiority over leading commercial systems (ElevenLabs, Cartesia, Inworld) in scene alignment and user preference, particularly in the "voice-clone" mode. The inclusion of a "corrected" multilingual benchmark addresses known issues with transcription errors in standard datasets.
Reproducibility is limited by the proprietary nature of the system. While the architecture is described in detail, the specific data pipelines (especially the "cross-scene pair mining" and the 5,000+ GPU training infrastructure) are not open-sourced. The paper references internal infrastructure (HELIOS) and data reports (Avatar V) that are not publicly available. The benchmarks (cross-scene, multilingual) are described but not released as public datasets with code. The use of specific commercial APIs for baselines limits the ability to reproduce the exact comparisons without access to those services.
The primary limitation is the lack of open-source code and data, which hinders independent verification and community adoption. The reliance on a frozen VLM for image understanding may limit the model's ability to handle complex or abstract scene descriptions not well-captured by the VLM. The "captioning" step at inference (where an LLM writes captions from the image) introduces an additional point of failure and latency. The paper does not extensively discuss the computational cost of the multimodal conditioning compared to standard TTS, although it mentions an RTF of 0.1. The evaluation, while strong, is largely conducted by the authors or their platform users, which may introduce bias, although the third-party arena ranking helps mitigate this.
This work has significant implications for the creation of digital humans and talking avatars, where audio-visual coherence is crucial. By enabling voice cloning that adapts to the visual scene, it enhances the realism and immersion of AI-generated video content. The techniques developed for cross-scene adaptation and consistency in long-form synthesis are applicable to other multimodal generation tasks. The focus on user-controlled trade-offs (clone policies) sets a precedent for more flexible and user-centric AI audio tools. HeyGen Voice introduces a scene-aware voice cloning framework that decouples speaker identity from acoustic environment using multimodal conditioning and a cross-scene training curriculum. The paper demonstrates state-of-the-art performance in both objective cross-scene benchmarks and subjective human evaluations, establishing a new standard for audio-visual coherence in talking video generation.
Driven by recent advancements in hearing assistive devices, target speaker extraction (TSE) in multi-talker environments has drawn significant attention. To address this, hearing technologies identify attention using neural signals recorded via electroencephalography (EEG) to guide TSE, enabling neuro-steered auditory attention decoding. However, most existing techniques rely on time-frequency masking, which suffers from low fidelity in spectro-temporally overlapping regions and is often restricted to narrowband audio. In contrast, we introduce NeuroCodec, which, to the best of our knowledge, is the first generative EEG-guided TSE framework operating in the latent space of a neural audio codec. Specifically, NeuroCodec models spatial EEG dependencies via graph convolutions and fuses cross-modal EEG and speech representations through attention mechanisms and Mamba-based temporal modeling. We further introduce a six-phase training curriculum that progressively incorporates latent reconstruction, multi-scale spectral, and adversarial objectives to ensure stable, high-fidelity speech generation. Experiments on the public Cocktail Party and KUL datasets show NeuroCodec outperforms baselines, across several perceptual metrics. A listening experiment with 20 participants validated that NeuroCodec offers greater perceptual accuracy, achieving a mean score of 82.24 compared with 49.63 for M3ANet. Additionally, the proposed method is seen to enable 44.1 kHz reconstruction with spectral content extending up to 20 kHz, further supporting the perceptual results. Finally, we extend NeuroCodec with a causal streaming formulation suited for real-time processing, providing a basis for hearing assistive devices.
Primary: University of Moratuwa
All Institutions: University of Moratuwa
The paper presents a novel generative approach to EEG-guided target speaker extraction by operating in the latent space of a neural audio codec, effectively addressing the fidelity limitations of traditional time-frequency masking. The combination of Mamba-based temporal modeling, graph-based EEG encoding, and a stable multi-phase training curriculum yields strong objective and subjective performance improvements, particularly in wideband reconstruction and real-time streaming efficiency.
The paper proposes NeuroCodec, a generative framework for EEG-guided Target Speaker Extraction (TSE) that operates in the latent space of a frozen neural audio codec (DAC) rather than predicting time-frequency masks. The methodology is technically sound and innovative in its combination of components: it uses Chebyshev graph convolutions for spatial EEG modeling, Mamba (selective state-space models) for efficient temporal modeling, and cross-attention for modality fusion. The introduction of a six-phase training curriculum to stabilize the adversarial and spectral losses is a practical contribution to the training stability of generative speech models. The causal streaming extension (OnlineNeuroCodec) introduces an "auditory attractor" mechanism using an Exponential Moving Average (EMA) of previous target latents to maintain speaker identity across hops, which is a clever, parameter-free alternative to recurrent speaker encoders.
The experiments are conducted on two standard datasets (Cocktail Party and KUL) under a Subject-Independent (SI) setting, which is rigorous. The paper reports improvements in STOI, ESTOI, and PESQ over strong baselines like M3ANet and NeuroSpex+. A significant strength is the inclusion of a subjective MUSHRA listening test with 20 participants, which validates the perceptual quality claims (82.24 vs 49.63 for M3ANet). The ablation studies effectively isolate the contributions of the Mamba backbone, EEG encoder components, and loss functions. The streaming evaluation demonstrates a significant speed advantage (RTF) over the NeuroHeed baseline while maintaining higher accuracy.
The paper states that source code will be released upon acceptance, but no code or demo links are currently provided in the text. The implementation details are reasonably specific (e.g., DAC downsampling factors, window sizes, EMA rates), which aids reproducibility. However, the reliance on specific pretrained DAC models and the complex multi-phase training schedule may make exact reproduction difficult without the authors' code.
The primary limitation is the computational cost of the generative approach, although the RTF analysis suggests it is viable for streaming. The subjective study size (n=20) is modest. The paper acknowledges that the system is not yet ready for embedded hearing-aid hardware. The comparison with some baselines (NeuroHeed+, MSFNet) is limited to reported numbers from other papers due to lack of public code, which weakens the comparative analysis slightly.
This work has significant potential for assistive hearing technologies by enabling high-fidelity, wideband speech extraction guided by neural attention. The shift from masking to generative latent synthesis could reduce artifacts in overlapping speech regions. The use of Mamba for efficient temporal modeling in audio-EEG fusion is a trend that may influence future multimodal audio architectures. The paper presents a novel generative approach to EEG-guided target speaker extraction by operating in the latent space of a neural audio codec, effectively addressing the fidelity limitations of traditional time-frequency masking. The combination of Mamba-based temporal modeling, graph-based EEG encoding, and a stable multi-phase training curriculum yields strong objective and subjective performance improvements, particularly in wideband reconstruction and real-time streaming efficiency.
This paper presents XingChen-UniFE, a unified large audio language model (LALM) for general speech restoration (GSR) and target speaker extraction (TSE). The proposed model employs complementary semantic and acoustic representations through a dual-stream encoder, where a self-supervised learning model provides hierarchical semantic features and a learnable acoustic branch captures fine-grained structures from complex spectra. A bidirectional cross-attention module further facilitates information exchange between the two representations. The fused representation conditions the LALM for discrete speech token generation with task and speaker information. Experiments demonstrate strong performance on both GSR and TSE and robust generalization to unseen degradation conditions.
Primary: China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd.
All Institutions: China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd.
The paper presents a unified LALM-based framework for speech restoration and speaker extraction that effectively combines semantic and acoustic features via bidirectional cross-attention. The technical contribution is solid, with strong empirical results demonstrating improved performance over existing generative baselines and good generalization capabilities, though the lack of subjective evaluation and multilingual testing limits the comprehensiveness of the validation.
The paper proposes XingChen-UniFE, a unified Large Audio Language Model (LALM) for General Speech Restoration (GSR) and Target Speaker Extraction (TSE). The core methodological contribution is a dual-stream encoder that fuses semantic representations from a frozen self-supervised learning (SSL) model (WavLM) with acoustic representations from a learnable branch processing complex STFT spectrograms. A bidirectional cross-attention module is introduced to facilitate explicit information exchange between these two streams before conditioning the LALM backbone. The LALM generates interleaved acoustic and semantic tokens autoregressively, which are then decoded by a neural audio codec (H-Codec). The approach addresses the limitation of SSL features lacking fine-grained acoustic details (like phase) by explicitly modeling complex spectra, which is a reasonable and technically sound design choice for restoration tasks.
The experiments are conducted on the Libri2Mix dataset for TSE and various simulated degradation conditions for GSR. The model is compared against strong baselines like LLaSE-G1, UniSE, and UniTok. The proposed model achieves state-of-the-art or competitive results across PESQ, STOI, SpeechBERTScore (SBS), and WER metrics. Notably, the paper demonstrates robust generalization to unseen degradation types (codec distortion, wind noise) that were excluded from training, which is a significant strength. Ablation studies confirm the complementary roles of the semantic and acoustic branches and the importance of the bidirectional fusion mechanism.
The paper provides sufficient implementation details, including model architecture dimensions (28 layers, 1024 hidden dim), training hyperparameters (400k steps, batch size 64, AdamW), and data composition (approx. 10k hours target speech, 405k hours noise). However, the specific codebase and pre-trained checkpoints are not linked in the provided text, and the reliance on specific internal or less common datasets (e.g., Emilia, CN-Celeb) may pose minor barriers for full reproduction without access to those resources.
The model is computationally expensive due to the large LALM backbone (502M trainable parameters) and the dual-stream encoder, which may limit real-time deployment on edge devices. The evaluation is primarily limited to English speech (Libri2Mix, LibriSpeech); performance on multilingual or non-English speech is not assessed. Additionally, the paper lacks subjective listening tests (MOS), relying solely on objective metrics, which may not fully capture perceptual quality nuances.
This work contributes to the trend of unifying speech enhancement tasks under generative frameworks. By leveraging LALMs, it opens avenues for more complex speech processing tasks that require semantic understanding alongside acoustic restoration. The demonstrated generalization to unseen degradations suggests potential for robust deployment in diverse real-world environments. The paper presents a unified LALM-based framework for speech restoration and speaker extraction that effectively combines semantic and acoustic features via bidirectional cross-attention. The technical contribution is solid, with strong empirical results demonstrating improved performance over existing generative baselines and good generalization capabilities, though the lack of subjective evaluation and multilingual testing limits the comprehensiveness of the validation.
Omni LLMs can naturally locate key audio and visual evidence and integrate information across modalities for audiovisual reasoning. Existing audio-visual reasoning benchmarks that mainly rely on multiple-choice questions usually score models solely based on answer correctness, making it difficult to assess whether a model truly forms a reasoning chain from evidence to the correct answer. In addition, textual or unimodal shortcuts may allow models to answer correctly without analyzing the necessary audio-visual evidence, further weakening the reliability of evaluation benchmarks. To address these issues, we introduce EvaOmni, an evidence-grounded benchmark containing 392 videos and 1,027 purely human-annotated question-answer pairs, covering 15 capabilities and three levels: retrieval and localization, basic understanding, and complex reasoning. We design a series of rigorous annotation procedures in which experts annotate audio and visual evidence and construct video-based distractors; humans review whether questions meet strict inspection criteria, and finally models filter textual and unimodal shortcuts. Beyond high-quality question-answer pairs, we require models to simultaneously return the answer and supporting evidence intervals in order to evaluate the complete reasoning chain. We propose the Coverage Grounded Score (C.G.) to measure how much annotated evidence a model identifies, and the Bottleneck Grounded Score (B.G.) to penalize poor temporal localization in either modality; both are incorporated into answer correctness. Across 7 open-weight and 17 proprietary Omni LLMs, Gemini3.7-Flash achieved the highest overall accuracy (79.4%), C.G. (70.8%), and B.G. (35.1%). At Level 3, it reached 79.1% accuracy, but its B.G. was only 12.3%. These results show that even the most advanced models still have room for development in grounded audio-visual reasoning.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University, Shanghai Innovation Institute, Inner Mongolia University of Technology, Fudan University, Beijing Institute of Technology, Tongji University, The Hong Kong University of Science and Technology (Guangzhou), Tsinghua University, The University of Hong Kong, Tencent Youtu Lab
EvaOmni introduces an evidence-grounded benchmark for audio-visual reasoning that requires models to provide supporting evidence intervals, revealing significant gaps in temporal localization capabilities of current Omni LLMs. The paper makes a strong contribution by addressing the limitations of answer-only evaluation in multimodal benchmarks, proposing rigorous annotation procedures to mitigate shortcuts, and introducing novel metrics (C.G. and B.G.) that assess both answer correctness and evidence grounding. The extensive evaluation across 24 models demonstrates that while top models achieve high accuracy, their ability to precisely localize supporting evidence in both audio and visual modalities remains limited, providing valuable insights for future development of Omni LLMs.
The paper proposes EvaOmni, a benchmark that shifts the evaluation paradigm from answer-only correctness to evidence-grounded reasoning. The core methodological contribution is the requirement for models to output timestamped evidence intervals for both audio and visual modalities. The authors introduce two novel metrics: Coverage Grounded Score (C.G.), which measures the overlap of predicted evidence with ground truth, and Bottleneck Grounded Score (B.G.), which penalizes the weaker modality's localization accuracy. The annotation pipeline is rigorous, involving expert annotation, cross-validation, and model-assisted filtering to remove unimodal or textual shortcuts. This approach effectively addresses the "shortcut" problem in existing benchmarks where models can answer correctly without processing the full audio-visual context.
The evaluation is extensive, covering 24 Omni LLMs (7 open-weight, 17 proprietary) across 1,027 question-answer pairs. The results clearly demonstrate that while top models like Gemini3.7-Flash achieve high accuracy (79.4%), their grounding scores (B.G. 35.1%) are significantly lower, revealing a gap in precise temporal localization. The ablation studies on input modalities (Audio vs. ASR, Video vs. Audio+Video) provide valuable insights into how models utilize different cues. The comparison with human experts further highlights the difficulty of the task, as humans achieve much higher grounding scores. The cost-efficiency analysis adds practical value for model selection.
The paper provides detailed descriptions of the annotation process, metric definitions, and experimental setup. The authors state that data, annotations, and evaluation framework are fully released. However, specific URLs for the dataset or code are not explicitly provided in the text snippet, which slightly hinders immediate reproducibility verification, though the claim of release is strong. The prompt templates and preprocessing details are described in the appendix, aiding reproducibility.
The benchmark is limited to videos of 1-12 minutes, which may not capture reasoning over longer time horizons. The exclusion of hard-coded subtitles, while preventing OCR shortcuts, limits the diversity of real-world video content. The reliance on multiple-choice questions, even with grounding, may not fully capture open-ended reasoning capabilities. The human baseline allows replay, which is not possible for models, creating a slight asymmetry in comparison conditions.
This work has significant impact on the evaluation of multimodal LLMs. By introducing grounded metrics, it provides a more reliable way to assess whether models truly understand audio-visual content or are relying on shortcuts. This can guide future model development towards better temporal localization and cross-modal integration. The findings that even state-of-the-art models struggle with precise grounding highlight a key area for future research in Omni LLMs. EvaOmni introduces an evidence-grounded benchmark for audio-visual reasoning that requires models to provide supporting evidence intervals, revealing significant gaps in temporal localization capabilities of current Omni LLMs. The paper makes a strong contribution by addressing the limitations of answer-only evaluation in multimodal benchmarks, proposing rigorous annotation procedures to mitigate shortcuts, and introducing novel metrics (C.G. and B.G.) that assess both answer correctness and evidence grounding. The extensive evaluation across 24 models demonstrates that while top models achieve high accuracy, their ability to precisely localize supporting evidence in both audio and visual modalities remains limited, providing valuable insights for future development of Omni LLMs.
Voice style conversion (VSC) aims to transform a source utterance to match the timbre, accent, and emotion of a target voice while preserving linguistic content. StyleStream 1.0 introduced the first streamable zero-shot VSC system with state-of-the-art conversion quality, but remained limited in two ways: approximately 1s end-to-end latency on consumer hardware, and the lack of a text-based interface for designing or editing target voices. We present StyleStream 2.0, a fast and controllable streaming VSC framework that addresses these limitations. To reduce latency, we train a few-step pixel MeanFlow model and further fine-tune it with autoregressive feedback inspired by Self Forcing, improving robustness under streaming inference. This reduces end-to-end latency to 520~ms on a consumer GPU, a 2x speedup over StyleStream 1.0, while maintaining conversion quality. To enable controllable style manipulation, we remove the mel-spectrogram context and instead route target style information through a compact style embedding. In this embedding space, we train a unified text-conditioned flow matching model that supports both text-based voice design, which maps natural language prompts to voice styles, and instruction-based voice editing, which modifies a specific attribute of a source utterance while preserving the rest. Experiments show that StyleStream 2.0 achieves the strongest target style fidelity on VSC, voice design, and voice editing, while remaining competitive on intelligibility and non-edited attribute preservation.
Primary: University of California, Berkeley
All Institutions: University of California, Berkeley
StyleStream 2.0 advances the field of streaming voice style conversion by achieving a 2x latency reduction through few-step pixel MeanFlow and autoregressive fine-tuning, while introducing a unified text-conditioned framework for controllable voice design and editing. The paper demonstrates that holistic style conversion can be made both fast and controllable, bridging the gap between high-quality offline conversion and real-time interactive applications.
The paper proposes a two-stage framework for streaming Voice Style Conversion (VSC) that effectively decouples content from style. The core methodological contribution is the replacement of the multi-step diffusion stylizer with a few-step pixel MeanFlow (pMF) model, which is further fine-tuned using an autoregressive feedback mechanism inspired by Self Forcing. This addresses the critical train-test mismatch in streaming inference where the model must condition on its own imperfect past outputs. Additionally, the authors introduce a unified text-conditioned DiT "Voice Controller" that operates in a compact style-embedding space, enabling both zero-shot voice design from text prompts and attribute-specific editing (age, gender, accent, emotion). The architectural choice to remove mel-spectrogram context and rely solely on a perceiver-resampler encoded style embedding is a significant design decision that facilitates the text-based control interface.
The experimental evaluation is comprehensive, covering reference-conditioned VSC, text-based voice design, and voice editing. The authors compare against strong baselines including CosyVoice 2.0/3.0, Vevo 2.0, and their own StyleStream 1.0. Results demonstrate a 2x latency reduction (520ms vs ~1s) while maintaining or improving style fidelity metrics (S-SIM, A-SIM, E-SIM). The subjective evaluation (MOS) is well-conducted with 40 participants, showing competitive naturalness and improved style consistency. The ablation studies effectively isolate the contributions of the autoregressive fine-tuning and the new style encoder, confirming their necessity for streaming robustness and style accuracy.
The paper provides substantial detail on the training data construction (CapSpeech-derived corpus), the specific models used for evaluation (VoxProfile, Whisper), and the inference settings (NFE, CFG scale, chunk sizes). The inclusion of the demo URL and detailed appendices on data generation and evaluation protocols enhances reproducibility. However, the specific hyperparameters for the DiT and MeanFlow training are not fully detailed in the main text, though referenced in appendices.
The system is evaluated primarily on English speech, and the generalization to other languages is not discussed. The text-based control is limited to four specific attributes (age, gender, accent, emotion), which may not cover the full spectrum of vocal qualities users might wish to control (e.g., pitch, speaking rate, specific vocal textures). The reliance on a frozen HuBERT encoder for content extraction may limit the system's ability to handle highly noisy or non-speech audio inputs.
This work has significant implications for real-time interactive speech systems, such as live dubbing, accessibility tools, and voice assistants. The ability to control voice style via natural language prompts lowers the barrier to entry for non-expert users and enables more intuitive interaction with speech synthesis systems. The reduction in latency makes streaming VSC practical for consumer hardware, potentially enabling new applications in gaming, virtual reality, and remote communication. StyleStream 2.0 advances the field of streaming voice style conversion by achieving a 2x latency reduction through few-step pixel MeanFlow and autoregressive fine-tuning, while introducing a unified text-conditioned framework for controllable voice design and editing. The paper demonstrates that holistic style conversion can be made both fast and controllable, bridging the gap between high-quality offline conversion and real-time interactive applications.
This paper presents a causal, training-free linker that allows a frozen streaming diarizer to track more speakers than it has output slots. Streaming end-to-end diarizers such as Streaming Sortformer emit a fixed number N of concurrent speaker slots. Above N speakers, they keep detecting speech but reuse slots across people, because N bounds how many speakers the model can express at once, not how many it will meet. The linker uses only the host's slot posteriors and the audio. It keeps the frames where one slot is active, splits each slot where the speaker changes, embeds each piece with an off-the-shelf speaker encoder, and matches it against a bank of speakers seen so far. It only relabels, so missed speech and false alarm are identical to the host's and every gain is a reduction in speaker confusion. On 1418 sessions from eight corpora, it reduces the 4-slot host's DER above its cap from 36.8% to 22.6%, and barely moves it below the cap. The same constants transfer to the host's 8-slot successor and to two streaming EEND hosts from a different model family. On courtroom hearings with up to 23 speakers it reduces cpWER from 82.7% to 24.4%.
Primary: Adalat AI
All Institutions: Adalat AI
The paper introduces a training-free, causal linker that enables frozen streaming diarizers to track more speakers than their fixed output slots by using global speaker embeddings to relabel segments. This approach significantly reduces speaker confusion in high-speaker-count scenarios without altering the host model's voice activity detection, offering a practical and effective solution for production-grade streaming diarization systems.
The paper proposes a causal, training-free post-processing module ("linker") for frozen streaming diarizers. The core insight is that the fixed number of output slots in models like Streaming Sortformer represents a concurrency limit, not a lifetime speaker limit. The method identifies "pure" frames where only one slot is active, extracts speaker embeddings using an off-the-shelf encoder (TitaNet), and maintains a global speaker bank to relabel segments. By strictly relabeling without altering voice activity boundaries, the method guarantees that Missed Speech and False Alarm rates remain identical to the host model, isolating the improvement to Speaker Confusion reduction. The approach is elegant in its simplicity and its adherence to the "frozen host" constraint, making it highly practical for production systems where retraining the base model is infeasible.
The experimental setup is robust, utilizing 1,418 sessions from eight diverse corpora, including Indic languages, earnings calls, and courtroom hearings. The evaluation correctly stratifies results by speaker count relative to the model's capacity cap. The results show a significant reduction in DER (36.8% to 22.6%) for sessions exceeding the 4-slot cap, with negligible impact on below-cap sessions. The comparison against a widened 8-slot host demonstrates that the linker provides greater gains than simply increasing the model's output channels, a strong argument for the proposed method. The inclusion of downstream transcription metrics (cpWER) further validates the practical utility of the diarization improvements.
The paper provides high reproducibility. It specifies the exact host model version (Sortformer v2.1), the speaker encoder (TitaNet-large), and the specific hyperparameters (link threshold 0.60, purity score threshold 60%). The use of standard scoring tools (pyannote 4.1) and the detailed description of the bootstrap confidence intervals allow for rigorous verification of the claims. The code is not explicitly linked in the text provided, but the methodological details are sufficient for re-implementation.
The primary limitation is the reliance on an off-the-shelf speaker encoder, which may not be optimal for all acoustic conditions (e.g., heavy overlap or noisy environments). The method is also limited by the host model's ability to detect speech; if the host misses speech, the linker cannot recover it. Additionally, the paper notes that the linker can sometimes over-count speakers in below-cap scenarios, indicating a potential trade-off in identity stability. The latency of the embedding extraction and matching process is not deeply analyzed, though it is described as running on CPU.
This work has significant impact on the deployment of streaming diarization systems in real-world applications where speaker counts are unpredictable, such as large meetings, courtrooms, and call centers. By decoupling speaker identity tracking from the fixed-capacity diarization model, it offers a scalable solution that can be retrofitted to existing production systems without retraining. This approach could be extended to other fixed-capacity sequence labeling tasks in speech processing. The paper introduces a training-free, causal linker that enables frozen streaming diarizers to track more speakers than their fixed output slots by using global speaker embeddings to relabel segments. This approach significantly reduces speaker confusion in high-speaker-count scenarios without altering the host model's voice activity detection, offering a practical and effective solution for production-grade streaming diarization systems.
Speech separation (SS) has witnessed substantial progress in recent years with the advent of neural network-based methods, achieving notable improvements on signal-level evaluation metrics. Existing speech separation systems are mostly optimized with signal-level objectives, and often yield outputs that are acceptable in waveform metrics but suboptimal for intelligibility-sensitive downstream tasks. To address this gap, we propose SLM-SS, a token-based generative speech separation framework that models multi-talker mixtures with speech language models. We first discretize clean speech into multi-codebook token sequences with neural codecs and serialize multi-speaker targets via Serialized Output Training (SOT). An encoder-decoder autoregressive speech language model predicts the zero-order sequence to establish speaker ordering and coarse linguistic content, and a non-autoregressive module then refines higher-order residual codebooks for efficient reconstruction. We further introduce a per-talker non-autoregressive strategy that simplifies high-order modeling and improves the stability. On Libri2Mix, SLM-SS with per-talker NAR achieves $6.92\%$ WER, approaching codec-only reconstruction. The WavTokenizer variant eliminates NAR refinement and achieves the highest mean subjective quality score among evaluated separation systems. Its UTMOS and DNSMOS advantages over discriminative baselines persist on unseen noisy and Chinese mixtures without fine-tuning.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University
[One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces SLM-SS, a token-based generative speech separation framework that utilizes speech language models to predict neural codec tokens, achieving competitive intelligibility and subjective quality scores while offering a flexible architecture for variable-talker scenarios, though it faces challenges in computational efficiency and signal fidelity due to codec discretization limits.
The paper proposes SLM-SS, a generative speech separation framework that leverages Speech Language Models (SLMs) to predict discrete neural codec tokens instead of continuous waveforms. The core method involves discretizing clean speech into multi-codebook sequences using Encodec or WavTokenizer, serializing multi-speaker targets via Serialized Output Training (SOT), and using an autoregressive (AR) encoder-decoder to predict the zero-order codebook. A non-autoregressive (NAR) module then refines higher-order residual codebooks. The introduction of a "per-talker NAR" strategy is a key methodological contribution, simplifying high-order modeling by decoupling speaker transitions from residual refinement. The approach effectively bridges the gap between signal-level separation and intelligibility-oriented downstream tasks by operating in the discrete token space.
Experiments are conducted on the LibriMix dataset (2-mix and 3-mix). The paper provides a comprehensive comparison against strong discriminative baselines (BSRNN, SepFormer, TF-LocoFormer, MossFormer2). Results show that SLM-SS achieves competitive WER and subjective quality scores, with the WavTokenizer variant achieving the highest mean subjective quality score. The inclusion of subjective listening tests with 30 volunteers and statistical significance testing (Wilcoxon signed-rank) adds rigor. Ablations on temperature, number of codebooks, and model parameters are provided. The evaluation of variable talker counts (up to 4) demonstrates the framework's flexibility.
The paper provides detailed architectural descriptions (WavLM encoder, Whisper-style decoder), training hyperparameters (learning rate, beam size, epochs), and dataset preprocessing steps (VAD segmentation). However, specific code availability is not explicitly confirmed in the text, and some implementation details regarding the exact initialization of the NAR module and the specific "staged training" strategy are described at a high level. The reliance on specific pre-trained models (WavLM, Whisper, Encodec, WavTokenizer) aids reproducibility.
The primary limitation is the computational cost of autoregressive decoding, which limits real-time applicability compared to discriminative methods. The method is constrained by the quality of the neural codec; truncating Encodec to 8 codebooks introduces irreversible distortion that caps the upper bound of signal fidelity (PESQ). The paper acknowledges that WavTokenizer, while better for subjective quality, has higher WER, indicating a trade-off between perceptual quality and linguistic accuracy. The experiments are limited to the LibriMix dataset, which consists of clean speech mixtures; performance on noisy or reverberant environments is not extensively tested in the main results (though mentioned as a future direction or minor test).
This work contributes to the trend of using discrete token representations for speech processing, aligning with the broader shift towards LLM-based audio models. It offers a viable alternative to traditional discriminative separation for applications where intelligibility is paramount, such as multi-talker ASR. The framework's ability to handle variable numbers of talkers without explicit speaker count estimation is a significant practical advantage. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces SLM-SS, a token-based generative speech separation framework that utilizes speech language models to predict neural codec tokens, achieving competitive intelligibility and subjective quality scores while offering a flexible architecture for variable-talker scenarios, though it faces challenges in computational efficiency and signal fidelity due to codec discretization limits.
We present Qwen-Audio-3.1-ASR, a family of audio-language models that builds upon its predecesor Qwen-Audio-3.0-ASR with improved speech recognition and expanded capabilities in speaker-diarized transcription and general audio understanding. The family retains support for 30 languages and 16 Chinese dialects, together with instruction-controlled capabilities including historical-context conditioning, hierarchical hotword customization, and native transcription polishing. Speaker-diarized ASR produces structured outputs containing speaker labels, timestamps, and recognized text, maintaining consistent speaker labels across turns and the representation of overlapping speech. The general audio understanding capabilities cover audio-, speech-, and music-captioning, temporal grounding and sound event detection, as well as audio question answering and multiple-choice reasoning. These tasks broaden the application scope from recognizing spoken words to interpreting speakers, acoustic events, and their temporal relations. To accommodate different deployment requirements, the family includes variants for asynchronous file transcription, low-latency streaming, interactive voice input, and non-streaming recognition, with speaker-diarized transcription and general audio understanding available in certain variants. Evaluations span conventional ASR, multilingual and dialectal speech, speaker-diarized transcription, and general audio understanding, demonstrating the capabilities of the Qwen-Audio-3.1-ASR family across transcription and broader audio analysis tasks.
Primary: Alibaba Group
All Institutions: Alibaba Token Foundry, Alibaba Group
[One sentence main contribution]. The paper presents Qwen-Audio-3.1-ASR, a unified audio-language model family that integrates high-accuracy multilingual speech recognition, speaker-diarized transcription, and general audio understanding through a novel training pipeline featuring On-Policy Distillation and asynchronous reinforcement learning.
The paper proposes Qwen-Audio-3.1-ASR, a family of audio-language models that integrates speech recognition, speaker diarization, and general audio understanding into a unified framework. The methodology is robust, leveraging a Mixture-of-Experts (MoE) LLM backbone initialized from Qwen3, paired with a pretrained audio encoder (BEST-RQ + AED). A significant methodological contribution is the "On-Policy Distillation" (OPD) strategy, which uses student-generated responses to provide token-level feedback from specialized teachers, addressing the distribution mismatch in conventional distillation. Additionally, the introduction of "FunVerl," an asynchronous reinforcement learning framework, allows for complex reward modeling (combining WER, DER, and temporal consistency) without the computational bottleneck of synchronous rollouts. The serialization of speaker-diarized output as a single autoregressive sequence with specific loss reweighting for speaker IDs is a practical and effective design choice.
The evaluation is comprehensive, covering multilingual ASR (30 languages), Chinese dialects (16 varieties), speaker-diarized transcription, and general audio understanding (captioning, QA, temporal grounding). The paper reports improvements over the previous generation (Qwen-Audio-3.0-ASR) and compares against strong baselines like Whisper and FireRedAudio. The use of specific metrics like cpWER (concatenated minimum-permutation word error rate) and DER for diarization is appropriate. However, the paper is a technical report, so some experimental details (e.g., specific dataset splits for the "general audio understanding" benchmarks) are less transparent than in a peer-reviewed journal article. The results demonstrate strong performance, particularly in the integrated capabilities of the "Next" variant.
Reproducibility is moderate. While the architecture and training stages are described in detail, the specific hyperparameters for the MoE layers, the exact composition of the 20 million hours of pretraining data, and the implementation details of the "FunVerl" framework are not fully disclosed. The code and models are likely available via the provided URL, which aids in practical reproduction, but the scientific reproducibility of the training pipeline from scratch is limited by the proprietary nature of the data and infrastructure.
The paper does not extensively discuss the computational cost of the MoE architecture during inference, which is a critical factor for deployment. The "general audio understanding" capabilities are limited to the "Next" variant, creating a fragmentation in the model family. Additionally, the reliance on synthetic data for certain tasks (like domain-entity augmentation) may introduce biases that are not fully analyzed. The lack of ablation studies on the specific contribution of the On-Policy Distillation versus standard RL is a minor gap.
This work has significant impact on the field of speech technology by demonstrating that a single model can handle not just transcription, but also speaker identification and semantic understanding of audio. This unification simplifies system architectures for applications like meeting assistants, media analysis, and accessibility tools. The open-sourcing of the model family (implied by the GitHub URL) will likely accelerate research in multimodal audio understanding. [One sentence main contribution]. The paper presents Qwen-Audio-3.1-ASR, a unified audio-language model family that integrates high-accuracy multilingual speech recognition, speaker-diarized transcription, and general audio understanding through a novel training pipeline featuring On-Policy Distillation and asynchronous reinforcement learning.
Speech models are increasingly used as an interface to AI systems, from transcription tools to voice agents. Existing audio backdoors require an attacker to introduce triggers at inference time. We study a different threat model in which the attacker compromises the model during fine-tuning but has no access to the victim's inference-time inputs. We introduce the Speaker-Activated Backdoor (SAB) setting, in which a target speaker's natural voice serves as the trigger. We develop a two-phase fine-tuning method that first makes speaker identity more accessible and then binds the target speaker to an attacker-chosen payload. Compared with a one-phase baseline with the same training budget, the two-phase method substantially improves selectivity and clean performance, and is especially important for content-dependent attacks. SAB achieves 90.2% and 97.0% backdoor success on Whisper-medium and Parakeet, with false-positive rates of 1.7% and 0%, respectively using as little as two minutes of real audio from the target speaker. We further demonstrate content-dependent semantic rewriting and downstream tool-use attacks, where malicious instructions are inserted into 96.9% of target-speaker transcripts. When these transcripts are passed to frontier LLM agents, GPT-4o and Claude (Opus, Sonnet, and Haiku), the injected instruction drives the agent to carry out the harmful action in up to 87.5% of cases. The attack generalizes to held-out recordings, multiple target speakers, and common acoustic degradations. These results establish speaker identity as a practical backdoor trigger that requires no inference-time attacker access and is invisible at the transcript layer.
Primary: Imperial College London
All Institutions: Imperial College London
The paper introduces Speaker-Activated Backdoors (SAB), a novel attack where a target speaker's natural voice triggers malicious behavior in speech models without inference-time attacker access. By proposing a two-phase fine-tuning method that enhances speaker discriminability and binds identity to payloads, the authors demonstrate high-success, selective backdoors across multiple ASR and audio-LLM architectures, including downstream harmful actions in tool-using agents, establishing speaker identity as a critical security vector in voice-based AI systems.
The paper introduces a novel threat model for speech models where the trigger is the speaker's identity rather than an injected acoustic perturbation. The proposed two-phase fine-tuning method (SAB) is technically sound: Phase 1 enhances speaker discriminability in the encoder representations, and Phase 2 binds the target speaker to a malicious payload while preserving clean performance on non-target speakers. The ablation studies effectively demonstrate that Phase 1 is crucial for content-dependent attacks (semantic rewriting) and improves selectivity in fixed-output attacks. The use of voice cloning (F5-TTS) to augment limited real target audio is a practical and effective strategy.
The experimental evaluation is comprehensive, covering multiple architectures (Whisper, Parakeet, Qwen-Omni) and attack types (transcript suppression, semantic rewriting, instruction injection). The inclusion of downstream tool-use attacks with frontier LLMs (GPT-4o, Claude) significantly strengthens the impact by demonstrating real-world consequences. The evaluation on held-out real recordings ensures the results are not artifacts of synthetic data. Robustness to acoustic degradation and defense mechanisms (clean fine-tuning, voice anonymization) are also thoroughly analyzed.
The paper provides detailed hyperparameters, dataset splits, and training procedures. However, the specific implementation details for the auxiliary speaker head and the exact prompts used for the LLM agents are not fully specified in the main text, though the appendix provides some additional details. The use of standard datasets (LibriSpeech, VoxCeleb2) and open-source models aids reproducibility.
The attack relies on the attacker having access to the fine-tuning pipeline, which is a standard assumption but still a barrier. The backdoor is susceptible to clean fine-tuning, which limits its persistence in dynamic environments. The evaluation is limited to English speakers and specific public figures, which may not generalize to all demographic groups or private individuals.
This work highlights a significant security vulnerability in the growing ecosystem of voice-activated AI agents. It demonstrates that speaker identity can be exploited as a control channel, posing risks to privacy and safety in applications like voice assistants and automated banking. The findings will likely prompt the development of new defenses focused on speaker-identity invariance and supply-chain security for speech models. The paper introduces Speaker-Activated Backdoors (SAB), a novel attack where a target speaker's natural voice triggers malicious behavior in speech models without inference-time attacker access. By proposing a two-phase fine-tuning method that enhances speaker discriminability and binds identity to payloads, the authors demonstrate high-success, selective backdoors across multiple ASR and audio-LLM architectures, including downstream harmful actions in tool-using agents, establishing speaker identity as a critical security vector in voice-based AI systems.
Tool-using voice agents often leave users waiting after speech ends, even when actionable information arrives earlier in the request. Speculative approaches begin work during speech, but chunk-bounded or interrupt-on-update generation can truncate a tool call before its arguments are complete. This couples computation time to speech updates, limiting useful work completed in advance. We introduce Heartbeat Thinking, which handles updated input through new Thinker invocations while allowing ongoing invocations and dispatched tools to finish. Periodic invocations read the latest partial transcript and a shared record of completed reasoning and tool results. These results help resolve missing arguments and initiate dependent calls, allowing preparation to progress across speech updates. Across SpeechFCEval, FDBench v3, and AudioCRAG, Heartbeat improves aggregate speculative tool coverage or retrieval reuse over a SHANKS-style windowed baseline, with benchmark-dependent effects on final task quality. On FDBench v3, it increases tool coverage by 16 percentage points and reduces mean final-response latency by 36.4\%. The results show that heartbeat thinking paradigm preserving ongoing computation during speech can turn listening time into reusable preparation for faster and more capable voice-agent responses.
Primary: Unknown
All Institutions: Unknown
METHODOLOGY_
A cover song re-renders an existing piece, most commonly by preserving its tonal content -- melody and chord progression -- while reshaping other musical attributes. This suggests a natural formulation for generative cover synthesis with pretrained music language models: explicitly control tonal content, while specifying lyrics and style through text. Existing approaches condition on frame-level features, tightly coupling generation to the reference's timing and structure, and limiting flexibility such as tempo variation, structural rearrangement, or partial conditioning. To this end, we present MuLaCover, a cover generation model that conditions a pretrained text-to-song foundation model on an editable symbolic lead sheet. Our design enables alignment-free control: generated outputs preserve the tonal signature of the source without frame-level alignment, allowing flexible timing, structure, and segment-level generation. We further introduce a training curriculum with partially mismatched audio-symbolic pairs, improving diversity and robustness. We evaluate MuLaCover with objective metrics, standard subjective ratings, and in-depth expert interviews that provide fine-grained diagnostic insights. Experiments show that MuLaCover achieves state-of-the-art performance among open-source systems and is competitive with leading commercial models such as Suno-v5.5. Audio samples and code are available at our project page: https://mulalabs.ai/mulacover/
Primary: MBZUAI
All Institutions: MBZUAI, Peking University, MuLa Labs, Vera Praxis Lab, The Chinese University of Hong Kong, Nanjing University of the Arts
MuLaCover introduces a flexible cover song generation framework by conditioning a text-to-song model on BPM-normalized symbolic lead sheets via gated cross-attention. The paper demonstrates that decoupling tonal content from timing through symbolic representation and cross-cover training enables superior controllability and diversity compared to frame-level conditioning, achieving performance competitive with leading commercial systems.
The paper proposes MuLaCover, a system for cover song generation that conditions a pretrained text-to-song foundation model (HeartMuLa) on a symbolic lead sheet (melody + chords) rather than frame-level audio features. The core methodological contribution is the use of BPM-normalized symbolic representations to decouple tonal content from timing, enabling flexible tempo and structural changes. The model injects this condition via a Gated Adaptive Cross-Attention (GACA) module into the last layers of the backbone. A key innovation is the three-stage training curriculum, specifically the "cross-cover" finetuning stage where tonal content from one rendition is paired with lyrics/style from another, effectively teaching the model to disentangle content from style. The ablation studies rigorously validate the necessity of BPM normalization, the gate mechanism, and the cross-cover training strategy.
The evaluation is comprehensive, utilizing objective metrics (AudioBox-Aesthetics, SongEval, PER, MuQ similarity, CSI accuracy) and subjective evaluations (listening study with 22 participants, expert interviews). The paper compares against strong baselines including open-source systems (SongEcho, ACE-Step-1.5) and a leading commercial model (Suno-v5.5). Results show MuLaCover achieves state-of-the-art performance among open-source systems and is competitive with Suno-v5.5. The partial-reference cover task (inpainting/outpainting) is also evaluated, demonstrating the flexibility of the alignment-free approach. The expert interview protocol provides qualitative insights into why certain systems perform better on specific musical attributes.
The paper provides detailed descriptions of the architecture, training stages, and data pipeline. It mentions the use of specific models for transcription (YourMT3, etc.) and data synthesis. While code is stated to be available at the project page, the reliance on synthetic data from commercial providers and specific internal pipelines (like the cross-cover pairing logic) might limit full external reproduction without access to those specific data assets. However, the methodological details are sufficient for implementation.
The system relies on the quality of the symbolic transcription models for melody and chords; errors in transcription will propagate to the generation. The training data is synthetic, which may limit the model's ability to handle the nuances of human-recorded covers with complex timbral variations. The "cover" definition is fixed at the lead-sheet level, which may not capture all artistic interpretations of a cover (e.g., jazz re-harmonization).
This work advances controllable music generation by providing a flexible, alignment-free method for conditioning on musical structure. It bridges the gap between symbolic music representation and audio generation, offering a practical tool for music production and co-creation. The approach could be extended to other domains where structural control is needed without strict temporal alignment. MuLaCover introduces a flexible cover song generation framework by conditioning a text-to-song model on BPM-normalized symbolic lead sheets via gated cross-attention. The paper demonstrates that decoupling tonal content from timing through symbolic representation and cross-cover training enables superior controllability and diversity compared to frame-level conditioning, achieving performance competitive with leading commercial systems.
We introduce Long Listening Thoughts (LLT), a framework for constructing post-training datasets that improves audio reasoning by pairing deliberate perceptual grounding with explicit cognitive behaviors. On audio understanding and reasoning benchmarks, large audio-language models (LALMs) are limited primarily by perceptual errors, yet existing post-training datasets rely on audio descriptions of limited diversity. Through a four-stage pipeline, LLT (1) builds multi-aspect dense audio descriptions spanning semantic, acoustic, and temporal information, (2) generates more challenging questions, (3) synthesizes incorrect reasoning traces, and (4) extends short reasoning traces into long ones via a thought-continuation strategy that elicits cognitive behaviors such as verification and backtracking. Using LLT, we synthesize 1.6M reasoning traces and train Qwen2.5-Omni-7B with a two-stage pipeline of supervised fine-tuning (SFT) followed by direct preference optimization (DPO). This improves accuracy on MMAU Test-mini (71.5 to 78.2), MMAR (56.7 to 63.7), and MMSU (60.6 to 66.0).
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University, NVIDIA
The paper introduces a scalable four-stage pipeline for constructing post-training datasets that enhance audio reasoning by combining dense multi-aspect perceptual grounding with explicit cognitive behaviors like verification and backtracking, demonstrating significant improvements in LALMs through a two-stage SFT+DPO training strategy.
The paper proposes a four-stage data construction pipeline (LLT) to improve audio reasoning in Large Audio-Language Models (LALMs). The core innovation lies in decoupling perceptual grounding from cognitive refinement. Stage 1 creates dense, multi-aspect audio descriptions (semantic, acoustic via spectrogram-VLM, temporal via specialized detectors) to overcome the limited perceptual diversity of existing datasets. Stage 2 augments question difficulty. Stage 3 synthesizes plausible incorrect reasoning traces using an LLM, rather than relying on model-sampled errors which lack diversity. Stage 4 employs a "thought-continuation" strategy to extend short traces into long ones with explicit cognitive behaviors (verification, backtracking). The training strategy is a two-stage process: SFT on correct and refined traces, followed by DPO using preference pairs that contrast incorrect-to-correct traces. This separation is methodologically sound, as the authors demonstrate that including incorrect traces directly in SFT degrades performance due to the model's inability to distinguish intermediate errors from correct reasoning under standard supervision.
The experiments are extensive and well-controlled. The authors validate the necessity of multi-aspect descriptions by showing gains when feeding them to a text-only LLM. They ablate the source of incorrect traces (synthetic vs. model-sampled), showing synthetic traces provide better coverage and distributional separation for DPO. The two-stage training is rigorously compared against single-stage SFT variants. Results show consistent improvements over the Qwen2.5-Omni-7B baseline on MMAU, MMAR, and MMSU. Generalization is tested on different source datasets (AudioMCQ) and different backbones (Qwen2-Audio, Qwen3-Omni), showing the pipeline's robustness. The analysis of cognitive triggers confirms that the model actually engages in error correction (8.7% true corrections) rather than just generating longer text.
High. The authors release the dataset (1.6M traces), trained models, and the full data construction pipeline. All components (Qwen3-Omni-Captioner, Whisper, pyannote, BEATs, etc.) are open-source. Prompts and implementation details are provided in the appendix. The use of standard benchmarks and clear ablation protocols enhances reproducibility.
The pipeline relies heavily on the quality of the initial audio CoT dataset (AF-Think) and the capabilities of the teacher LLMs (Qwen3-Next-80B). The 30-second audio limit restricts applicability to long-form audio, though some transfer is observed. The automated verification of traces may miss subtle logical errors, although human inspection mitigates this. The computational cost of generating 1.6M traces with multiple specialized models is significant.
This work provides a scalable framework for enhancing audio reasoning in LALMs, which is crucial for applications in assistive technology, audio surveillance, and multimodal agents. By explicitly teaching cognitive behaviors like verification and backtracking, it moves beyond simple pattern matching to more robust reasoning. The release of the dataset and pipeline will likely accelerate research in audio post-training. The paper introduces a scalable four-stage pipeline for constructing post-training datasets that enhance audio reasoning by combining dense multi-aspect perceptual grounding with explicit cognitive behaviors like verification and backtracking, demonstrating significant improvements in LALMs through a two-stage SFT+DPO training strategy.