We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University
The paper introduces AuK, a unified foundational model for speech generation and editing that leverages a hybrid MLLM-VAE-Transformer architecture, extensive data construction, and advanced post-training strategies including preference optimization and reinforcement learning. It represents a significant step towards general-purpose audio AI by consolidating diverse tasks into a single instruction-driven framework, offering a robust and efficient solution for complex speech manipulation tasks.
The paper proposes AuK, a unified foundational model for speech generation and editing. The architecture is a hybrid of a Multimodal Large Language Model (MLLM) for semantic conditioning, a Variational Autoencoder (VAE) for acoustic conditioning, and a Transformer backbone using a FLUX-style design (dual-stream MMDiT blocks followed by single-stream DiT blocks). The training pipeline is extensive, involving a two-stage pre-training curriculum (generation-only warm-up followed by joint generation-editing), post-training with human-feedback preference optimization (DPO) for editing and Flow-GRPO for generation, and a distillation stage using Consistency Initialization and Task-Routed Decoupled DMD to achieve 4-step inference. The methodology is sophisticated, particularly the integration of RL and preference optimization for audio tasks, which is relatively novel in this specific combination. The data construction pipeline is also detailed, covering 3.03 billion instruction-audio instances across five task families.
The experiments cover a wide range of tasks: zero-shot TTS, instruction TTS, acoustic editing, paralinguistic editing, content editing, enhancement, and separation. The paper claims leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing. It reports a 4.5x wall-clock speedup for the distilled model (AuK-Flash). However, the provided text is truncated before the detailed results tables, so specific quantitative comparisons (e.g., MOS scores, WER, speaker similarity scores) against baselines are not fully visible in the excerpt. The reliance on "leading performance" claims without visible comparative tables in the provided text limits the ability to fully verify the magnitude of improvement, though the scope of evaluation is comprehensive.
The paper claims to release both source code and model weights. It provides detailed hyperparameters for training (learning rates, batch sizes, GPU counts, optimizer settings) and architectural details (layer counts, hidden dimensions, strides). The data construction pipeline is described in detail, including specific tools used for annotation and synthesis (Qwen3-Omni, F5-TTS, etc.). This level of detail supports high reproducibility, assuming the released code and weights are complete and functional.
The model is large (1.5B parameters in the backbone alone, plus the MLLM and VAE), which may limit deployment on edge devices despite the distillation. The reliance on a complex MLLM for semantic conditioning adds computational overhead during inference compared to simpler text encoders. The paper is a technical report, so peer review rigor may be lower than a top-tier conference paper. The truncated text prevents a full assessment of the experimental results' depth.
The unification of speech generation and editing under a single instruction-based interface has significant potential for practical applications in content creation, accessibility, and interactive voice assistants. The open-source release of such a comprehensive model could accelerate research in unified audio models. The techniques for post-training (RL and DPO) and distillation are applicable to other generative audio tasks. The paper introduces AuK, a unified foundational model for speech generation and editing that leverages a hybrid MLLM-VAE-Transformer architecture, extensive data construction, and advanced post-training strategies including preference optimization and reinforcement learning. It represents a significant step towards general-purpose audio AI by consolidating diverse tasks into a single instruction-driven framework, offering a robust and efficient solution for complex speech manipulation tasks.
Audio-conditioned language models often underuse acoustic cues such as prosody, emotion, and non-speech sounds, raising the question of whether ASR-supervised frontends discard this information before it reaches the LM. We test whether the frontend is responsible by comparing Whisper-Tiny and Whisper-Small with EnCodec, DAC-VAE, and WavTokenizer in a shared Qwen3.5-4B audio-LM pipeline on ASR, emotion recognition, and sound captioning. Encoder replacement alone does not resolve this underuse: Whisper variants remain strongest overall, including on emotion and environmental sound captioning. To localize the failure, we trace task-relevant information through the encoder, projector, LM layers, and LM head. Linear probes and geometric analyses show that discriminative acoustic structure remains recoverable at the final LM layer, even when MCQA accuracy trails probe accuracy by up to 83 points. Because the answer format and decoding procedure are controlled, this task-dependent gap points to content-specific readout failure rather than generic format bias. LogitLens analyses and a targeted LM head intervention support the conclusion that acoustic underuse is not explained solely by encoder-side information loss and that readout alignment can be a dominant bottleneck.
Primary: Seoul National University
All Institutions: Seoul National University, KAIST, NAVER Cloud
The paper identifies that acoustic underuse in audio-LLMs is primarily a readout alignment failure rather than an encoder-side information loss, demonstrating that task-relevant acoustic structure remains recoverable in the LM's final hidden states even when the model fails to verbalize it correctly. Through a combination of encoder replacement experiments, layer-wise probing, and targeted LM head interventions, the study provides a detailed localization of this bottleneck, offering actionable insights for improving the acoustic grounding of language models.
The paper employs a rigorous diagnostic methodology to localize the source of "acoustic underuse" in audio-LLMs. It systematically compares ASR-supervised encoders (Whisper) against reconstruction-based codecs (EnCodec, DAC-VAE, WavTokenizer) within a controlled shared-LM pipeline (Qwen3.5-4B). The core methodological strength lies in the multi-stage information tracing: using linear probes, geometric distance ratios, and LogitLens analyses to track task-relevant information from the encoder through the projector and LM layers to the final output head. The causal intervention of fine-tuning only the choice-letter rows of the LM head ($W_U$) is a clever and effective way to isolate readout failure from representation loss.
The experiments are extensive, covering ASR, emotion recognition, and sound captioning across multiple datasets (LibriSpeech, IEMOCAP, FSD50K, NSynth, etc.). The results clearly demonstrate that while Whisper encoders perform best on end-to-end tasks, they do not necessarily contain more recoverable acoustic information than reconstruction-based codecs at the latent level. The finding that linear probe accuracy at the final LM layer significantly exceeds MCQA accuracy (by up to 83 points) provides strong evidence for a readout bottleneck. The recovery of 30-42 points in MCQA accuracy by updating only ~10-26k scalars in the LM head is a compelling result.
The paper provides detailed implementation specifics, including model checkpoints, projector architecture, training hyperparameters, and dataset splits. The use of standard open-source components (Whisper, EnCodec, Qwen) and clear descriptions of the probing and intervention protocols make the work highly reproducible.
The analysis is primarily focused on MCQA-format classification tasks. While the authors acknowledge this, the generalization of the "readout bottleneck" finding to free-form generative tasks (like open-ended captioning or dialogue) is not fully established. The LM scale is limited to 4B parameters, so it is unclear if the bottleneck persists or shifts in larger models. The study excludes self-supervised encoders (like WavLM or HuBERT), which are known to retain strong paralinguistic features, limiting the scope of the encoder comparison.
This work provides a crucial diagnostic framework for the audio-LLM community. By identifying that the failure often lies in the LM's ability to map internal acoustic representations to text tokens (readout) rather than in the encoder's ability to capture them, it shifts the focus of future research toward better alignment strategies, label-aware training, or readout regularization. It challenges the common assumption that swapping ASR encoders for codec-based ones is a sufficient solution for improving non-lexical audio understanding. The paper identifies that acoustic underuse in audio-LLMs is primarily a readout alignment failure rather than an encoder-side information loss, demonstrating that task-relevant acoustic structure remains recoverable in the LM's final hidden states even when the model fails to verbalize it correctly. Through a combination of encoder replacement experiments, layer-wise probing, and targeted LM head interventions, the study provides a detailed localization of this bottleneck, offering actionable insights for improving the acoustic grounding of language models.
Full-duplex spoken dialogue systems enable simultaneous listening and speaking, but their audio-only perception often fails under background noise and overlapping speech, leading to incoherent responses. Recent audio-visual dialogue approaches show that incorporating visual cues such as lip movements improve robustness under audio corruption. However, existing approaches often adapt the large speech dialogue model itself to process visual input, requiring costly multimodal training. We propose AV-STE, a modular streaming audio-visual front-end that restores corrupted semantic speech tokens from noisy audio and lip video before they reach the speech LLM. The downstream dialogue model remains entirely frozen, preserving its pretrained conversational capabilities. When integrated with frozen Moshi, AV-STE improves average GPT-4o-judged response coherence from 1.42 to 1.91 under same-dataset speaker interference while largely preserving turn-taking behavior. Gains also transfer to out-of-domain Seamless Interaction.
Primary: KAIST
All Institutions: KAIST
The paper presents a novel and effective modular approach to enhancing the robustness of full-duplex spoken dialogue systems by leveraging audio-visual cues to restore corrupted semantic tokens without retraining the underlying speech LLM. The technical contributions, including the streaming adaptation of AV-HuBERT and the noise-adaptive fusion mechanism, are well-validated through comprehensive experiments on in-domain and out-of-domain datasets, demonstrating significant improvements in both token recovery and downstream dialogue coherence.
The paper proposes AV-STE, a modular front-end for enhancing semantic speech tokens in full-duplex dialogue systems. The core contribution is the decoupling of robustness enhancement from the large language model (LLM) backbone. By keeping the speech LLM (Moshi) frozen and only enhancing the semantic token stream using a streaming audio-visual encoder (modified AV-HuBERT) and a noise-adaptive fusion module, the authors address a significant practical limitation of existing multimodal dialogue systems that require costly retraining. The introduction of "Soft Token Cross-Attention" to handle uncertainty in noisy tokenizer outputs and "Noise-Adaptive Modulation" based on entropy are technically sound and well-motivated innovations. The adaptation of bidirectional AV-HuBERT to a causal, streaming architecture with bounded lookahead is a necessary and well-executed engineering contribution for real-time applications.
The experimental setup is rigorous, covering in-domain (LRS3) and out-of-domain (Seamless Interaction) datasets, with various noise types (non-speech, speaker interference). The evaluation metrics are appropriate for the task: semantic token accuracy for the front-end and GPT-4o-judged coherence plus turn-taking metrics (TOR, latency) for the downstream dialogue. The results show substantial improvements in token recovery (e.g., from 12.61% to 65.60% under same-dataset interference) and downstream coherence (1.42 to 1.91). The comparison against a non-streaming baseline (PASE) and a different dialogue model (PersonaPlex) provides useful context, although the primary comparison is against the frozen Moshi baseline. The qualitative examples strongly support the quantitative findings.
The paper provides a GitHub link to the code and models. The training configuration, dataset preprocessing, and model architecture details are described in sufficient detail for reproduction. The use of standard datasets (LRS3, AudioSet, Seamless Interaction) and open-source components (Moshi, AV-HuBERT) further enhances reproducibility.
The paper acknowledges that AV-STE assumes reliable visual input and does not handle visual occlusion or missing video. There is a noted domain shift issue where clean-token accuracy drops on out-of-domain data (Seamless Interaction), indicating sensitivity to domain characteristics. The reliance on GPT-4o for evaluation, while common, introduces a potential bias, though it is a standard practice in current LLM evaluation. The method is specific to the Moshi/Mimi tokenization scheme, limiting immediate applicability to other speech LLM architectures without adaptation.
This work has significant implications for the deployment of robust spoken dialogue systems in real-world, noisy environments. By providing a plug-and-play module that enhances robustness without retraining the expensive LLM backbone, it lowers the barrier to entry for audio-visual dialogue systems. The focus on full-duplex interaction is timely and relevant to the next generation of conversational AI. The approach could be extended to other modalities or tokenization schemes, potentially influencing the design of future multimodal speech interfaces. The paper presents a novel and effective modular approach to enhancing the robustness of full-duplex spoken dialogue systems by leveraging audio-visual cues to restore corrupted semantic tokens without retraining the underlying speech LLM. The technical contributions, including the streaming adaptation of AV-HuBERT and the noise-adaptive fusion mechanism, are well-validated through comprehensive experiments on in-domain and out-of-domain datasets, demonstrating significant improvements in both token recovery and downstream dialogue coherence.
Large Audio Language Models (LALMs) have shown strong performance on audio reasoning benchmarks, but accuracy alone cannot distinguish true reasoning from superficial pattern matching, often overestimating reasoning ability since high scores may result from guessing rather than genuine audio understanding. Evaluating the reasoning process itself is essential for improving LALMs' reasoning ability, yet remains challenging. Existing methods either rely on costly human annotation or opaque LLM-as-judge approaches, making them impractical, biased, and lacking transparency. Moreover, audio reasoning introduces unique challenges absent in text-based settings, perceptual hallucination and cross-modal alignment between audio understanding and textual inference, hence text-based evaluation frameworks cannot be directly applied. Therefore, we propose ARIA-Rubrics (Audio Reasoning Integrity Assessment), a lightweight, annotation-free gold reasoning chains, automatic and transparent framework comprising six complementary metrics that evaluate audio reasoning quality across perceptual grounding, reasoning coherence, and answer consistency. We use Chain-of-Thought prompting as an externalization mechanism to make the reasoning process observable. Experiments on 9 models across 2 benchmarks identify three reasoning modes of current LALMs with actionable directions for future development, with ARIA-Rubrics achieving high correlation with human judgments. The code is available at the Github Repository.
Primary: Imperial College London
All Institutions: Imperial College London, Technische Universität München, Mohamed bin Zayed University of Artificial Intelligence, Shanghai Jiao Tong University, Johns Hopkins University
The paper introduces ARIA-Rubrics, a comprehensive framework for evaluating audio reasoning in LALMs by combining perceptual grounding checks with reasoning process analysis. It effectively addresses the limitation of accuracy-only benchmarks by identifying specific reasoning failure modes, offering a valuable tool for the community to develop more robust and transparent audio language models.
The paper proposes ARIA-Rubrics, a framework to evaluate the reasoning process of Large Audio Language Models (LALMs) rather than just final answer accuracy. The methodology relies on Chain-of-Thought (CoT) prompting to externalize reasoning into four steps: Perception, Analysis, Reasoning, and Answer. It introduces six metrics: Acoustic Grounding Score (using CLAP embeddings to verify entities against audio), Inter-step Coherence (using MPNet embeddings), Reasoning Chain Coverage (using a lightweight LLM judge for substantiveness), Reasoning Step Causality (using perplexity reduction), Reasoning Progress Score (using NLI entailment), and Audio Lexical Density (using Word2Vec similarity to AudioSet ontology). The approach is innovative in combining audio-specific perceptual checks with text-based reasoning quality metrics, addressing the gap where text-only metrics fail to detect perceptual hallucinations.
The authors evaluate 9 models across two benchmarks (MMAR and MMAU-mini). They identify three distinct reasoning modes: Reasoning, Scaffolding, and Decoration. The paper claims high correlation with human judgments, validating the automated metrics. The experimental setup is robust, covering both open-source and closed-source models, and the identification of specific failure modes (like scaffolding) provides actionable insights for model developers.
The code is available on GitHub. The paper specifies the models used for each metric (CLAP, MPNet, Qwen2.5-1.5B, RoBERTa-large-mnli, Word2Vec) and provides details on the CoT prompt structure and the AudioSet ontology filtering. This level of detail supports reproducibility, although the specific few-shot prompts for the LLM judge are referenced in appendices not fully visible in the truncated text, which is standard practice.
The framework relies on the quality of the CoT output; if a model refuses to generate CoT or generates unstructured text, the metrics may fail or be biased. The use of a small LLM (Qwen2.5-1.5B) for judging substantiveness and causality may limit the depth of evaluation compared to larger models. Additionally, the Audio Lexical Density metric is a proxy and may not capture all aspects of audio-relevant reasoning.
This work is significant for the development of reliable audio AI systems. By distinguishing between genuine reasoning and pattern matching, it provides a diagnostic tool for improving LALMs. It encourages the development of models that are not only accurate but also faithful in their reasoning, which is crucial for safety and interpretability in audio applications. The paper introduces ARIA-Rubrics, a comprehensive framework for evaluating audio reasoning in LALMs by combining perceptual grounding checks with reasoning process analysis. It effectively addresses the limitation of accuracy-only benchmarks by identifying specific reasoning failure modes, offering a valuable tool for the community to develop more robust and transparent audio language models.
We present UniStream, a fully causal 48 kHz neural audio codec for streaming speech, music, and environmental sounds. At its core is Multi-Expert Residual Vector Quantization (ME-RVQ), which replaces the single shared codebook in each residual quantization layer with four expert codebooks controlled by a deterministic Top-K router. Because routing decisions are derived solely from previously decoded quantized states, the decoder can reproduce the selected experts without transmitting expert identifiers, thereby expanding quantization capacity while adding 5.5M parameters. We further introduce an auxiliary Optimal Transport Conditional Flow Matching (OT-CFM) objective to regularize the quantized latent space during training. The flow module is removed entirely at inference and therefore incurs no runtime overhead. UniStream supports a 12 kbps Top-1 mode and a 22.5 kbps Top-2 mode within a causal 48 kHz encoder-decoder framework, while achieving real-time GPU inference. To complement narrow-band speech metrics, we report 48 kHz ViSQOL audio mode, ViSQOL speech mode, standard VGGish-FAD, DNSMOS P.835, and higher-rate reference comparisons with Opus and EnCodec. At 12 kbps, UniStream-Top1 achieves PESQ and UTMOS scores comparable to EnCodec while reducing speech Mel-D from 13.07 to 8.21. At 22.5 kbps, UniStream-Top2 achieves a ViSQOL speech-mode score of 4.67 and an environmental audio-mode score of 3.96, exceeding all evaluated systems operating at 12 kbps or below in the latter setting. It also comes within 0.03 MOS-LQO of Opus at 24 kbps on speech in ViSQOL audio mode. Ablation studies confirm that ME-RVQ is the primary source of quality improvement, whereas OT-CFM provides perceptual gains on speech with a mild trade-off in spectral distortion.
Primary: Tsinghua University
All Institutions: Tsinghua Shenzhen International Graduate School, Tsinghua University
UniStream introduces a causal 48 kHz audio codec using Multi-Expert Residual Vector Quantization to expand capacity without signaling overhead, achieving competitive quality at low bitrates. The paper demonstrates a solid engineering approach to improving neural codecs for streaming, with rigorous evaluation across multiple audio domains, though it lacks subjective validation and open-source availability.
The paper proposes UniStream, a causal neural audio codec featuring Multi-Expert Residual Vector Quantization (ME-RVQ). The core innovation is replacing single codebooks in residual layers with four expert codebooks routed by a deterministic Top-K mechanism. Crucially, the routing is derived from previously decoded states, allowing the decoder to replicate expert selection without transmitting expert IDs, thus saving bandwidth. The method also introduces a training-only Optimal Transport Conditional Flow Matching (OT-CFM) regularizer to improve latent space quality without inference cost. The architecture is fully causal, suitable for streaming, and supports 12 kbps and 22.5 kbps modes.
Experiments are conducted on speech (LibriSpeech), music (MTG-Jamendo), and environmental audio (FSD50K). The evaluation is comprehensive, using both narrow-band metrics (PESQ, STOI, UTMOS) and full-band perceptual metrics (ViSQOL, VGGish-FAD, DNSMOS). Results show UniStream-Top1 outperforms EnCodec in spectral distortion (Mel-D) at 12 kbps, while UniStream-Top2 approaches Opus 24 kbps quality at a lower bitrate. Ablations confirm ME-RVQ is the primary driver of improvement, with OT-CFM providing modest perceptual gains.
The paper provides detailed architectural specifications (strides, dilation factors, codebook sizes, loss weights) and training hyperparameters. However, no code repository or demo link is provided in the text. The use of standard datasets and metrics facilitates reproduction, but the lack of open-source code limits immediate verification.
The paper lacks subjective listening tests (MOS), relying solely on objective metrics. Music domain performance (FAD) is noted as weaker compared to speech and environmental audio. The CPU inference time (RTF 1.05) is borderline for real-time deployment on standard hardware, though GPU inference is efficient. The comparison with non-causal baselines like EnCodec is somewhat unfair for streaming contexts, though acknowledged.
This work contributes to the field of low-latency neural audio coding, particularly for applications requiring full-band (48 kHz) fidelity in streaming scenarios. The deterministic routing mechanism offers a practical solution for expanding quantization capacity without signaling overhead, which could influence future codec designs. UniStream introduces a causal 48 kHz audio codec using Multi-Expert Residual Vector Quantization to expand capacity without signaling overhead, achieving competitive quality at low bitrates. The paper demonstrates a solid engineering approach to improving neural codecs for streaming, with rigorous evaluation across multiple audio domains, though it lacks subjective validation and open-source availability.
We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University
The paper introduces AuK, a unified foundational model for speech generation and editing that leverages a hybrid MLLM-VAE-Transformer architecture, extensive data construction, and advanced post-training strategies including preference optimization and reinforcement learning. It represents a significant step towards general-purpose audio AI by consolidating diverse tasks into a single instruction-driven framework, offering a robust and efficient solution for complex speech manipulation tasks.
The paper proposes AuK, a unified foundational model for speech generation and editing. The architecture is a hybrid of a Multimodal Large Language Model (MLLM) for semantic conditioning, a Variational Autoencoder (VAE) for acoustic conditioning, and a Transformer backbone using a FLUX-style design (dual-stream MMDiT blocks followed by single-stream DiT blocks). The training pipeline is extensive, involving a two-stage pre-training curriculum (generation-only warm-up followed by joint generation-editing), post-training with human-feedback preference optimization (DPO) for editing and Flow-GRPO for generation, and a distillation stage using Consistency Initialization and Task-Routed Decoupled DMD to achieve 4-step inference. The methodology is sophisticated, particularly the integration of RL and preference optimization for audio tasks, which is relatively novel in this specific combination. The data construction pipeline is also detailed, covering 3.03 billion instruction-audio instances across five task families.
The experiments cover a wide range of tasks: zero-shot TTS, instruction TTS, acoustic editing, paralinguistic editing, content editing, enhancement, and separation. The paper claims leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing. It reports a 4.5x wall-clock speedup for the distilled model (AuK-Flash). However, the provided text is truncated before the detailed results tables, so specific quantitative comparisons (e.g., MOS scores, WER, speaker similarity scores) against baselines are not fully visible in the excerpt. The reliance on "leading performance" claims without visible comparative tables in the provided text limits the ability to fully verify the magnitude of improvement, though the scope of evaluation is comprehensive.
The paper claims to release both source code and model weights. It provides detailed hyperparameters for training (learning rates, batch sizes, GPU counts, optimizer settings) and architectural details (layer counts, hidden dimensions, strides). The data construction pipeline is described in detail, including specific tools used for annotation and synthesis (Qwen3-Omni, F5-TTS, etc.). This level of detail supports high reproducibility, assuming the released code and weights are complete and functional.
The model is large (1.5B parameters in the backbone alone, plus the MLLM and VAE), which may limit deployment on edge devices despite the distillation. The reliance on a complex MLLM for semantic conditioning adds computational overhead during inference compared to simpler text encoders. The paper is a technical report, so peer review rigor may be lower than a top-tier conference paper. The truncated text prevents a full assessment of the experimental results' depth.
The unification of speech generation and editing under a single instruction-based interface has significant potential for practical applications in content creation, accessibility, and interactive voice assistants. The open-source release of such a comprehensive model could accelerate research in unified audio models. The techniques for post-training (RL and DPO) and distillation are applicable to other generative audio tasks. The paper introduces AuK, a unified foundational model for speech generation and editing that leverages a hybrid MLLM-VAE-Transformer architecture, extensive data construction, and advanced post-training strategies including preference optimization and reinforcement learning. It represents a significant step towards general-purpose audio AI by consolidating diverse tasks into a single instruction-driven framework, offering a robust and efficient solution for complex speech manipulation tasks.
Full-duplex spoken dialogue systems enable simultaneous listening and speaking, but their audio-only perception often fails under background noise and overlapping speech, leading to incoherent responses. Recent audio-visual dialogue approaches show that incorporating visual cues such as lip movements improve robustness under audio corruption. However, existing approaches often adapt the large speech dialogue model itself to process visual input, requiring costly multimodal training. We propose AV-STE, a modular streaming audio-visual front-end that restores corrupted semantic speech tokens from noisy audio and lip video before they reach the speech LLM. The downstream dialogue model remains entirely frozen, preserving its pretrained conversational capabilities. When integrated with frozen Moshi, AV-STE improves average GPT-4o-judged response coherence from 1.42 to 1.91 under same-dataset speaker interference while largely preserving turn-taking behavior. Gains also transfer to out-of-domain Seamless Interaction.
Primary: KAIST
All Institutions: KAIST
The paper presents a novel and effective modular approach to enhancing the robustness of full-duplex spoken dialogue systems by leveraging audio-visual cues to restore corrupted semantic tokens without retraining the underlying speech LLM. The technical contributions, including the streaming adaptation of AV-HuBERT and the noise-adaptive fusion mechanism, are well-validated through comprehensive experiments on in-domain and out-of-domain datasets, demonstrating significant improvements in both token recovery and downstream dialogue coherence.
The paper proposes AV-STE, a modular front-end for enhancing semantic speech tokens in full-duplex dialogue systems. The core contribution is the decoupling of robustness enhancement from the large language model (LLM) backbone. By keeping the speech LLM (Moshi) frozen and only enhancing the semantic token stream using a streaming audio-visual encoder (modified AV-HuBERT) and a noise-adaptive fusion module, the authors address a significant practical limitation of existing multimodal dialogue systems that require costly retraining. The introduction of "Soft Token Cross-Attention" to handle uncertainty in noisy tokenizer outputs and "Noise-Adaptive Modulation" based on entropy are technically sound and well-motivated innovations. The adaptation of bidirectional AV-HuBERT to a causal, streaming architecture with bounded lookahead is a necessary and well-executed engineering contribution for real-time applications.
The experimental setup is rigorous, covering in-domain (LRS3) and out-of-domain (Seamless Interaction) datasets, with various noise types (non-speech, speaker interference). The evaluation metrics are appropriate for the task: semantic token accuracy for the front-end and GPT-4o-judged coherence plus turn-taking metrics (TOR, latency) for the downstream dialogue. The results show substantial improvements in token recovery (e.g., from 12.61% to 65.60% under same-dataset interference) and downstream coherence (1.42 to 1.91). The comparison against a non-streaming baseline (PASE) and a different dialogue model (PersonaPlex) provides useful context, although the primary comparison is against the frozen Moshi baseline. The qualitative examples strongly support the quantitative findings.
The paper provides a GitHub link to the code and models. The training configuration, dataset preprocessing, and model architecture details are described in sufficient detail for reproduction. The use of standard datasets (LRS3, AudioSet, Seamless Interaction) and open-source components (Moshi, AV-HuBERT) further enhances reproducibility.
The paper acknowledges that AV-STE assumes reliable visual input and does not handle visual occlusion or missing video. There is a noted domain shift issue where clean-token accuracy drops on out-of-domain data (Seamless Interaction), indicating sensitivity to domain characteristics. The reliance on GPT-4o for evaluation, while common, introduces a potential bias, though it is a standard practice in current LLM evaluation. The method is specific to the Moshi/Mimi tokenization scheme, limiting immediate applicability to other speech LLM architectures without adaptation.
This work has significant implications for the deployment of robust spoken dialogue systems in real-world, noisy environments. By providing a plug-and-play module that enhances robustness without retraining the expensive LLM backbone, it lowers the barrier to entry for audio-visual dialogue systems. The focus on full-duplex interaction is timely and relevant to the next generation of conversational AI. The approach could be extended to other modalities or tokenization schemes, potentially influencing the design of future multimodal speech interfaces. The paper presents a novel and effective modular approach to enhancing the robustness of full-duplex spoken dialogue systems by leveraging audio-visual cues to restore corrupted semantic tokens without retraining the underlying speech LLM. The technical contributions, including the streaming adaptation of AV-HuBERT and the noise-adaptive fusion mechanism, are well-validated through comprehensive experiments on in-domain and out-of-domain datasets, demonstrating significant improvements in both token recovery and downstream dialogue coherence.
In this work, we present Gander, an end-to-end model that unifies omni perception, realtime interaction, and agentic capabilities within a single framework. In contrast to turn-based conventional paradigms, Gander continuously receives streaming inputs across multiple modalities, including video, speech, and text, enabling natural full-duplex interaction in both everyday conversations and complex workflow-oriented agent scenarios. Users can interrupt the model at any time, while the model can also proactively provide intermediate feedback or ask follow up questions. To natively support these capabilities, Gander adopts two key architectural designs: 1) It employs a Cerebellum-Brain collaborative framework, in which the Cerebellum is responsible for realtime interaction and omni conversational capabilities, while the Brain handles complex reasoning and higher-level agentic tasks. The two components interact continuously through tool calling and the agent orchestration runtime. 2) The Cerebellum is built upon a streaming Thinker-Talker architecture, user inputs and model outputs are further flattened into an ordered token stream at the chunk level, providing a unified representation for low latency, continuous interaction. We conduct comprehensive evaluations of Gander across four dimensions: conversational ability, omni understanding, interactive capability, and agentic intelligence. Internal human evaluations demonstrate that Gander maintains the natural and expressive spoken dialogue capabilities of SOTA open source models while achieving competitive performance in omni interaction. Gander also demonstrates robustness in challenging real-world scenarios, including background noise interference, multi-party interactions, and backchannel communication. We release Gander together with its models, code, and data to facilitate further research and development in the community.
Primary: Tencent
All Institutions: Tencent, Zhejiang University, Shanghai Jiao Tong University, The Chinese University of Hong Kong, Nanyang Technological University
Gander introduces a Cerebellum-Brain architecture that decouples real-time full-duplex interaction from long-horizon agentic reasoning, enabling natural, interruptible multimodal collaboration. The paper presents a robust streaming framework with detailed data construction for interaction behaviors, though its reliance on external proprietary agents for the "Brain" component and limited quantitative benchmarking against specific baselines slightly temper its technical impact.
The paper proposes "Gander," an end-to-end Omni Interaction Agent that unifies perception, real-time interaction, and agentic capabilities. The core architectural innovation is the "Cerebellum-Brain" collaborative framework. The "Cerebellum" is a streaming Thinker-Talker model responsible for low-latency, full-duplex interaction, while the "Brain" is a training-free, plug-and-play agent (e.g., Codex, Claude Code) handling complex reasoning. The Cerebellum uses a "streaming chunk flattening" mechanism, serializing audio, video, and text into a unified token stream where the model predicts control tokens (listen/speak/interrupt) at each 1-second chunk. This allows for native interruption and proactive feedback without external VAD modules. The methodology is sound, addressing the latency-intelligence trade-off by decoupling fast interaction from slow reasoning.
The evaluation is comprehensive, covering conversational ability, omni understanding, interactive capability, and agentic intelligence. The paper reports internal human evaluations showing Gander maintains SOTA spoken dialogue naturalness while achieving competitive omni interaction performance. It demonstrates robustness in noisy environments, multi-party interactions, and backchannel communication. However, the paper relies heavily on "internal human evaluations" and subjective demos, with limited quantitative benchmarks compared to specific baselines (e.g., GPT-4o, MiniCPM-o) in the provided text snippet. The lack of standardized public benchmark scores for the "agentic" component makes direct comparison difficult.
The authors state they release the model weights, code, and data. The architecture details (chunk size, token rates, encoder types like SigLIP) are provided. However, the "Brain" component relies on proprietary external agents (Codex, Claude Code), which limits full reproducibility of the agentic performance. The training data pipeline is described in detail, including the synthesis of full-duplex interaction data, which aids reproducibility of the Cerebellum component.
The system's performance is coupled to the quality of the external "Brain" agent, which may have its own latency and cost implications. The "training-free" nature of the Brain means the Cerebellum cannot learn from the Brain's specific failures or adapt its tool-calling strategy through end-to-end fine-tuning. The evaluation lacks extensive quantitative ablation studies comparing the Cerebellum-Brain architecture against monolithic models on standardized agentic benchmarks.
This work is significant for the development of natural, human-like AI assistants that can handle complex tasks while maintaining fluid conversation. The decoupled architecture offers a scalable path for integrating powerful reasoning models into real-time interfaces. It pushes the field beyond turn-based chatbots toward continuous, multimodal collaboration. Gander introduces a Cerebellum-Brain architecture that decouples real-time full-duplex interaction from long-horizon agentic reasoning, enabling natural, interruptible multimodal collaboration. The paper presents a robust streaming framework with detailed data construction for interaction behaviors, though its reliance on external proprietary agents for the "Brain" component and limited quantitative benchmarking against specific baselines slightly temper its technical impact.
Audio-conditioned language models often underuse acoustic cues such as prosody, emotion, and non-speech sounds, raising the question of whether ASR-supervised frontends discard this information before it reaches the LM. We test whether the frontend is responsible by comparing Whisper-Tiny and Whisper-Small with EnCodec, DAC-VAE, and WavTokenizer in a shared Qwen3.5-4B audio-LM pipeline on ASR, emotion recognition, and sound captioning. Encoder replacement alone does not resolve this underuse: Whisper variants remain strongest overall, including on emotion and environmental sound captioning. To localize the failure, we trace task-relevant information through the encoder, projector, LM layers, and LM head. Linear probes and geometric analyses show that discriminative acoustic structure remains recoverable at the final LM layer, even when MCQA accuracy trails probe accuracy by up to 83 points. Because the answer format and decoding procedure are controlled, this task-dependent gap points to content-specific readout failure rather than generic format bias. LogitLens analyses and a targeted LM head intervention support the conclusion that acoustic underuse is not explained solely by encoder-side information loss and that readout alignment can be a dominant bottleneck.
Primary: Seoul National University
All Institutions: Seoul National University, KAIST, NAVER Cloud
The paper identifies that acoustic underuse in audio-LLMs is primarily a readout alignment failure rather than an encoder-side information loss, demonstrating that task-relevant acoustic structure remains recoverable in the LM's final hidden states even when the model fails to verbalize it correctly. Through a combination of encoder replacement experiments, layer-wise probing, and targeted LM head interventions, the study provides a detailed localization of this bottleneck, offering actionable insights for improving the acoustic grounding of language models.
The paper employs a rigorous diagnostic methodology to localize the source of "acoustic underuse" in audio-LLMs. It systematically compares ASR-supervised encoders (Whisper) against reconstruction-based codecs (EnCodec, DAC-VAE, WavTokenizer) within a controlled shared-LM pipeline (Qwen3.5-4B). The core methodological strength lies in the multi-stage information tracing: using linear probes, geometric distance ratios, and LogitLens analyses to track task-relevant information from the encoder through the projector and LM layers to the final output head. The causal intervention of fine-tuning only the choice-letter rows of the LM head ($W_U$) is a clever and effective way to isolate readout failure from representation loss.
The experiments are extensive, covering ASR, emotion recognition, and sound captioning across multiple datasets (LibriSpeech, IEMOCAP, FSD50K, NSynth, etc.). The results clearly demonstrate that while Whisper encoders perform best on end-to-end tasks, they do not necessarily contain more recoverable acoustic information than reconstruction-based codecs at the latent level. The finding that linear probe accuracy at the final LM layer significantly exceeds MCQA accuracy (by up to 83 points) provides strong evidence for a readout bottleneck. The recovery of 30-42 points in MCQA accuracy by updating only ~10-26k scalars in the LM head is a compelling result.
The paper provides detailed implementation specifics, including model checkpoints, projector architecture, training hyperparameters, and dataset splits. The use of standard open-source components (Whisper, EnCodec, Qwen) and clear descriptions of the probing and intervention protocols make the work highly reproducible.
The analysis is primarily focused on MCQA-format classification tasks. While the authors acknowledge this, the generalization of the "readout bottleneck" finding to free-form generative tasks (like open-ended captioning or dialogue) is not fully established. The LM scale is limited to 4B parameters, so it is unclear if the bottleneck persists or shifts in larger models. The study excludes self-supervised encoders (like WavLM or HuBERT), which are known to retain strong paralinguistic features, limiting the scope of the encoder comparison.
This work provides a crucial diagnostic framework for the audio-LLM community. By identifying that the failure often lies in the LM's ability to map internal acoustic representations to text tokens (readout) rather than in the encoder's ability to capture them, it shifts the focus of future research toward better alignment strategies, label-aware training, or readout regularization. It challenges the common assumption that swapping ASR encoders for codec-based ones is a sufficient solution for improving non-lexical audio understanding. The paper identifies that acoustic underuse in audio-LLMs is primarily a readout alignment failure rather than an encoder-side information loss, demonstrating that task-relevant acoustic structure remains recoverable in the LM's final hidden states even when the model fails to verbalize it correctly. Through a combination of encoder replacement experiments, layer-wise probing, and targeted LM head interventions, the study provides a detailed localization of this bottleneck, offering actionable insights for improving the acoustic grounding of language models.
Can we recover the 3D poses of multiple people using only sound? This paper presents the first attempt to estimate multi-person 3D poses solely from acoustic signals. Estimating the poses of multiple individuals using acoustic signals is inherently challenging due to the superposition of motion-dependent signal variations. Unlike single-person scenarios, the presence of multiple subjects leads to overlapping acoustic signatures, making it difficult to attribute specific signal changes to an individual's pose. Furthermore, the complexity is compounded by inter-person reflections, which introduce intricate propagation delays that obscure the temporal motion-acoustic relationship. To address these issues, we propose SoundMHPE (Sound-based Multi-person Human Pose Estimator), a novel encoder-decoder framework consisting of two key components. First, the Acoustic Multi-scale Encoder captures diverse temporal and fine-grained frequency features to isolate subtle acoustic signatures from complex, overlapping signals. Second, the Temporal Pose Decoder employs an attention mechanism to disentangle multi-person information across successive frames. By jointly accounting for temporal dynamics and inter-person dependencies, this component precisely reconstructs frame-wise individual poses. To validate our approach, we constructed the 6-hour Acoustic Multi-person Pose (AMP) dataset consisting of 432K synchronized frames of multi-person pose and acoustic data, and demonstrated that our SoundMHPE outperforms baseline models. Project page: https://oumi03.github.io/sound-mhpe/
Primary: Keio University
All Institutions: Keio University, Tokyo University of Science, NTT Inc.
The paper presents the first comprehensive framework for multi-person 3D pose estimation using active acoustic sensing, introducing a novel dataset and a DETR-inspired temporal decoder that effectively disentangles overlapping acoustic signatures. The technical contributions are significant, particularly the multi-scale encoder and the decoupled self-attention mechanisms, which address the unique challenges of multi-person acoustic superposition. The rigorous experimental validation, including cross-modal generalization to WiFi, strengthens the paper's impact, although the reliance on controlled environments remains a key limitation for real-world deployment.
The paper proposes SoundMHPE, a novel encoder-decoder framework for multi-person 3D pose estimation from acoustic signals. The methodology is well-structured, addressing the core challenge of signal superposition in multi-person scenarios. The Acoustic Multi-scale Encoder (AME) effectively utilizes multi-resolution STFT to capture both temporal dynamics and fine-grained frequency features, which is a sound technical choice for handling overlapping acoustic signatures. The Temporal Pose Decoder (TPD) introduces a DETR-like query mechanism adapted for temporal sequences, where multiple queries per person allow for frame-specific attention. The decoupling of self-attention into Motion Self-Attention (intra-person) and Interaction Self-Attention (inter-person) is a thoughtful architectural decision that aligns with the physical nature of the problem. The use of Hungarian matching for loss calculation is standard but appropriate for this detection-like task.
The authors constructed a new 6-hour dataset (AMP) with 432K frames, which is a significant contribution given the lack of existing benchmarks for this specific task. The experimental setup includes cross-subject evaluation with 3-fold cross-validation, which is rigorous. The baselines are adapted from single-person acoustic and multi-person WiFi models, which is reasonable given the novelty of the task. The ablation studies are comprehensive, covering the encoder, decoder, attention mechanisms, and window sizes. The cross-modal evaluation on WiFi data (PiW) is a strong point, demonstrating the generalizability of the architectural components. However, the performance metrics (MPJPE, PCK) should be compared against state-of-the-art single-person acoustic methods to provide a clearer benchmark of the "multi-person penalty."
The paper provides detailed implementation details, including hyperparameters (learning rate, weight decay, number of blocks, query counts) and dataset statistics. The use of standard hardware (OptiTrack, Zoom H3-VR) and software libraries (implied by the architecture) suggests high reproducibility. The project page is provided, which likely contains code and dataset access. The clear definition of the dataset collection protocol (15 subjects, specific poses, randomization) enhances reproducibility.
The primary limitation is the reliance on a controlled indoor environment with specific acoustic properties (soundproof room mentioned in Table 1). Generalization to noisy, reverberant, or outdoor environments is not fully explored, though a brief test with partitions is shown. The dataset size, while large for this niche, is still limited compared to RGB datasets, which may constrain the model's robustness to diverse body types and motions. The method assumes a fixed number of maximum people (M=15), which may not scale efficiently to very crowded scenes.
This work opens a new avenue for non-invasive, privacy-preserving pose estimation in scenarios where RGB cameras are unsuitable (darkness, privacy concerns) and wireless signals are obstructed. Applications in healthcare monitoring, sports analysis, and disaster relief are promising. The cross-modal applicability to WiFi suggests that the architectural insights could benefit other signal-based sensing tasks. The construction of the AMP dataset will likely serve as a foundational benchmark for future research in acoustic pose estimation. The paper presents the first comprehensive framework for multi-person 3D pose estimation using active acoustic sensing, introducing a novel dataset and a DETR-inspired temporal decoder that effectively disentangles overlapping acoustic signatures. The technical contributions are significant, particularly the multi-scale encoder and the decoupled self-attention mechanisms, which address the unique challenges of multi-person acoustic superposition. The rigorous experimental validation, including cross-modal generalization to WiFi, strengthens the paper's impact, although the reliance on controlled environments remains a key limitation for real-world deployment.
Spatial audio editing modifies an existing soundfield according to a user's instruction while preserving the rest of the scene. Unlike conventional audio editing, it must reason jointly about audio events, spatial information, dynamic changes, and environmental information in first-order Ambisonic (FOA) waveforms. Existing language-guided editors mainly target conventional audio or rely on sequential operations, and therefore do not directly support one-stage editing for complex 3D spatial instructions. We present SwanWeave, the first one-stage multi-task framework for instruction-guided 3D FOA spatial audio editing. We build paired FOA supervision from open-source speech and sound-effect corpora using controllable room simulation, covering more than ten single-operation and compound tasks across the four editing axes. To handle this heterogeneous edit space, SwanWeave uses Spatial Edit Mixture-of-Experts (SE-MoE) with dual-level routing, selecting task-aware expert combinations for compound instructions and frame-level routed/null experts for local edit decisions. We further introduce Spatial Preference Optimization (SPO), a Direct Preference Optimization (DPO)-based alignment objective with edit-specific negative targets, and adopt staged training to improve natural-language grounding. Experiments show that SwanWeave achieves better editing quality than existing general audio editors and spatial audio baselines across all tasks. Spatial audio editing demos can be found at https://swanaigc.github.io/#swanweave, code can be found at: https://github.com/MM-Speech/SwanWeave.
Primary: Zhejiang University
All Institutions: Zhejiang University, ByteDance
The paper introduces a novel one-stage multi-task framework for 3D spatial audio editing that effectively addresses the challenges of preserving unedited regions while executing complex compound instructions. By combining a dual-level Mixture-of-Experts architecture with preference optimization tailored for spatial edits, the method achieves state-of-the-art performance in both semantic and spatial fidelity, marking a significant step forward in controllable spatial audio generation.
The paper proposes SwanWeave, a one-stage framework for instruction-guided 3D First-Order Ambisonic (FOA) spatial audio editing. The methodology is robust, combining a flow-matching latent editor with a novel Spatial Edit Mixture-of-Experts (SE-MoE) module. The SE-MoE design is particularly interesting, utilizing dual-level routing: task-level experts selected based on the instruction to handle compound edits, and frame-level routed/null experts to handle local temporal changes and preservation of unedited regions. The use of a "null expert" that outputs zero tensors for unchanged regions is a clever architectural choice to prevent drift in preserved audio. The training strategy is well-structured, employing a staged curriculum (text-to-FOA pretraining, single-operation editing, compound editing) followed by Spatial Preference Optimization (SPO). SPO adapts DPO to the flow-matching context by using edit-specific negative targets (e.g., wrong event, no operation), which effectively addresses common failure modes like source copying or incorrect spatial placement.
The experimental evaluation is comprehensive. The authors construct a large-scale synthetic dataset (approx. 375K triplets) using PyRoomAcoustics for controllable FOA rendering, covering 10+ single and compound tasks. They compare against strong baselines including ZETA, AudioEditor, SDEdit, and SmartDJ. The metrics are diverse, covering semantic quality (FD, FAD, KL, LSD), instruction alignment (CLAP), and spatial fidelity (GCC, CRW, FSAD). The results show consistent improvements across all metrics, with particularly strong gains in spatial fidelity (GCC/CRW) compared to stereo-based baselines, validating the benefit of native FOA modeling. The ablation studies are thorough, confirming the necessity of pretraining, the SE-MoE module, and the SPO stage. The comparison between one-stage and multi-stage editing effectively demonstrates the advantage of the proposed one-stage approach in avoiding error accumulation.
The paper provides high reproducibility. Code and a demo page are available. The data construction process is detailed, including the specific room sizes, direction definitions, and the prompt used for instruction generation. The model architecture details, including the routing regularization losses and CFG scale selection, are provided in the appendix. The use of open-source corpora and standard simulation tools (PyRoomAcoustics) further aids reproducibility.
The primary limitation is the reliance on synthetic data generated via room simulation. While controllable, this may not capture the acoustic complexity of real-world recordings (e.g., irregular room responses, microphone artifacts, background noise). The scenes are also limited in duration (~10 seconds) and complexity (2-4 events), which may not generalize to long-form or dense multi-speaker scenes. The human evaluation was conducted with a small group of volunteer graduate students, which may introduce bias, though the objective metrics strongly support the findings.
This work has significant potential for immersive media production, VR/AR, and interactive storytelling. By enabling natural language control over 3D spatial audio, it lowers the barrier to creating complex spatial soundscapes. The one-stage approach is efficient and practical for real-time or near-real-time applications. The framework could be extended to other spatial audio formats (e.g., binaural, higher-order Ambisonics) and integrated into broader audio-visual generation pipelines. The paper introduces a novel one-stage multi-task framework for 3D spatial audio editing that effectively addresses the challenges of preserving unedited regions while executing complex compound instructions. By combining a dual-level Mixture-of-Experts architecture with preference optimization tailored for spatial edits, the method achieves state-of-the-art performance in both semantic and spatial fidelity, marking a significant step forward in controllable spatial audio generation.
Voice AI applications are gaining popularity as advances in large language models (LLMs) enable more natural and accessible spoken interactions. Serving these applications requires accounting not only for what users say, but also for how they speak (e.g., speaking rate) and the conditions under which their audio is captured and transmitted (e.g., background noise and packet loss). However, existing LLM systems represent conversation context as a flat, growing sequence of messages, leaving voice-specific context implicit in the audio. As a result, they can generate responses that are poorly aligned with user preferences, degrade interaction quality under adverse environmental conditions, and incur high costs over long voice sessions. We present llmovoice, a context-management middleware that explicitly models voice context and orchestrates its use. At each turn, llmovoice constructs a bounded voice context from the current user input, relevant interaction history, and explicit paralinguistic and environmental states. It then uses the serving LLM to reason over this context and generate runtime directives that guide how the system responds. We evaluate llmovoice on real-world voice applications and benchmarks. It reduces speaking-rate alignment error by 52.4%, lowers the false-interruption rate from 46.0% to 0.9% under packet loss, and reduces model usage cost by 79.2%. For long sessions, llmovoice reduces per-turn cost by up to 24.9 times while retaining up to 98.7% of baseline answer quality.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University, AgenticSys
The paper presents a novel middleware architecture for managing voice context in LLM systems, significantly improving cost efficiency and robustness to network artifacts. By explicitly modeling paralinguistic and environmental states and structuring conversation history into retrievable units, it addresses the scalability and alignment challenges of long-running voice interactions, offering a practical and effective solution for production voice AI.
The paper proposes "llmovoice," a middleware layer for LLM-over-voice systems that explicitly models voice context. The core contribution is architectural rather than algorithmic in the traditional ML sense; it introduces a structured memory system (VoicePages and VoiceThreads) to manage semantic history and a state-to-control path that extracts paralinguistic (speaking rate) and environmental (packet loss) signals. The system uses the LLM itself to reason over these states and generate runtime directives (e.g., adjusting VAD thresholds or response pacing). This approach is sound and addresses a genuine gap in current voice AI pipelines, which often treat audio as a flat stream without explicit state management. The use of a "cost-aware context projector" to select fidelity levels (audio vs. transcript vs. summary) is a clever engineering solution to the quadratic cost problem of long sessions.
The evaluation is strong and focuses on system-level metrics relevant to production deployment. Key results include a 52.4% reduction in speaking-rate alignment error, a dramatic drop in false-interruption rates (46.0% to 0.9%) under packet loss, and significant cost reductions (79.2% model usage cost, up to 24.9x per-turn cost reduction in long sessions). The use of real-world traces (ICASSP 2024 Audio Deep PLC Challenge) for the packet loss simulation adds credibility. However, the evaluation lacks subjective human studies (e.g., MOS scores or user preference tests) to validate the quality of the "naturalness" or "alignment" claims beyond objective error rates.
The paper describes the architecture and interface compatibility (OpenAI Realtime API), which aids reproducibility. However, specific implementation details of the "context orchestrator" prompts, the exact algorithms for the "cost-aware context projector," and the hyperparameters for the paralinguistic state smoothing are not fully detailed in the provided text. The reliance on a specific LLM service API limits the portability of the exact results, though the middleware design is intended to be agnostic.
The paper focuses primarily on speaking rate and packet loss as representative signals; it does not extensively evaluate other paralinguistic cues (tone, emotion) or environmental factors (background noise types). The system adds an extra LLM inference step (the orchestrator) before the main response generation, which could introduce latency, although the paper argues this is offset by cost savings from context compression. The lack of subjective user studies is a notable gap for a paper claiming improved interaction quality.
This work has high potential impact on the deployment of voice AI in resource-constrained or noisy environments (telecom, in-vehicle, remote areas). By decoupling context management from the LLM service, it provides a scalable pattern for building robust voice agents. The cost-efficiency gains are particularly relevant for high-volume voice applications where token costs are a major barrier. The paper presents a novel middleware architecture for managing voice context in LLM systems, significantly improving cost efficiency and robustness to network artifacts. By explicitly modeling paralinguistic and environmental states and structuring conversation history into retrievable units, it addresses the scalability and alignment challenges of long-running voice interactions, offering a practical and effective solution for production voice AI.
Audio-video diffusion models rely on cross-modal attention to coordinate text, sound, and visual content, yet this same mechanism can introduce subtle and systematic semantic leakage. We study these models by probing and analyzing the ``attention triangle,'' comprising the three cross-attention edges connecting the text, audio, and video streams, and examine how semantic information is routed across modalities during generation. Our analysis reveals that routing along the audio-video edge is bidirectional: audio can influence video generation, while video can influence audio generation. This edge is shaped by biases encoded in the model's parameters and emerges as a major contributor to leakage: when prompts are in tension with learned priors, cross-modal interactions may override the intended conditioning and reroute semantics toward visually canonical but incorrect outcomes. These effects suggest that semantic artifacts arise not merely from attention spreading beyond its intended target, but from structured, bias-driven interactions along specific pathways. Building on this perspective, we extract attention-derived signals that expose how semantics are distributed and grounded across modalities, and use them as a diagnostic tool to both analyze and deliberately incur leakage under controlled conditions. This enables us to probe the internal dynamics of cross-modal routing and isolate the role of individual interactions. We further leverage these signals to guide inference-time interventions that encourage more consistent cross-modal alignment. Extensive experiments support our analysis and demonstrate improved semantic grounding while preserving generation quality.
Primary: Tel Aviv University
All Institutions: Tel Aviv University, Simon Fraser University
The paper identifies the "Attention Triangle" as a structural cause of semantic leakage in audio-video diffusion models and proposes inference-time interventions to correct source attribution and appearance errors. By analyzing the bidirectional flow of information between text, audio, and video streams, the authors demonstrate that leakage is not random but driven by specific biases in the audio-video attention edge, enabling targeted fixes that improve cross-modal consistency without retraining.
The paper introduces the "Attention Triangle" framework to analyze cross-modal semantic leakage in audio-video diffusion models. The core contribution is a diagnostic and intervention method that decomposes the cross-attention mechanisms into three edges: Text-Video, Text-Audio, and Audio-Video. The authors identify the Audio-Video edge as a critical source of bias, where learned priors (e.g., speech associated with human faces) override explicit text prompts. They propose extracting attention-derived signals to map these biases and using them for inference-time steering. The methodology is sound, leveraging the internal structure of Diffusion Transformers (DiT) to manipulate attention maps without retraining. The formalization of the "bias matrices" and the distinction between direct and indirect routing paths provides a rigorous theoretical basis for the observed failures.
The experiments focus on the LTX-2 model, a state-of-the-art open-source audio-video generator. The evaluation demonstrates the ability to isolate specific leakage types (appearance vs. source attribution) and correct them via targeted steering. The qualitative results (e.g., the pirate/parrot example) are compelling and clearly illustrate the decoupling of visual and audio errors. However, the paper relies heavily on qualitative visual/audio examples and ablation studies on specific prompts. There is a lack of large-scale quantitative benchmarks (e.g., FID, CLAP scores, or human preference studies) to generalize the effectiveness of the intervention across diverse datasets. The scope is limited to one specific model architecture (LTX-2), which may limit the generalizability of the findings to other audio-video models.
The paper provides sufficient detail on the attention manipulation techniques, describing how to extract and modify attention maps. However, specific hyperparameters for the steering strength, layer selection, and the exact implementation of the "bias matrix" extraction are not fully detailed in the provided text. The reliance on the specific LTX-2 model makes reproduction dependent on access to that specific codebase and weights.
The primary limitation is the lack of quantitative evaluation metrics. The improvements are demonstrated qualitatively, which is subjective. Additionally, the method is tested on a single model (LTX-2), so it is unclear if the "Attention Triangle" dynamics and the proposed interventions generalize to other architectures like Sora or Veo. The computational overhead of inference-time steering is not explicitly discussed.
This work contributes to the interpretability of multimodal generative models, offering a new lens (the attention triangle) to understand cross-modal interactions. The proposed inference-time interventions could be adopted by developers of audio-video models to improve alignment without retraining. It highlights a fundamental issue in multimodal generation: the conflict between learned statistical priors and explicit user intent, providing a pathway to mitigate such conflicts. The paper identifies the "Attention Triangle" as a structural cause of semantic leakage in audio-video diffusion models and proposes inference-time interventions to correct source attribution and appearance errors. By analyzing the bidirectional flow of information between text, audio, and video streams, the authors demonstrate that leakage is not random but driven by specific biases in the audio-video attention edge, enabling targeted fixes that improve cross-modal consistency without retraining.
We present Alignment-Free Text-Audiobox (Text-AB), a unified framework for high-quality voice dubbing and full-duplex dialogue synthesis. Building on a Diffusion Transformer trained with a flow-matching objective, Text-AB departs from the Audiobox system along three dimensions. First, it operates in a latent diffusion framework using DAC-VAE features that encode 48 kHz waveforms into a 25 Hz latent sequence, giving over 10x higher compression than previous EnCodec representations while improving resynthesis quality. Second, Text-AB is alignment-free: it consumes raw text via an off-the-shelf text encoder and learns text-speech alignment through cross-attention, removing the need for forced alignment and explicit duration prediction. Third, we scale model and data substantially, pretraining a 3B-parameter model on 480k hours of monolingual speech, followed by supervised fine-tuning on three downstream tasks: cross-lingual voice dubbing, full-duplex dialogue synthesis, and emotional full-duplex dialogue synthesis. At inference, Text-AB supports one-shot generation for up to ~1 min of speech and arbitrarily long-form generation via a multi-diffusion scheme, plus a multi-stage reranking strategy that enhances quality based on automated metrics. On a real-world dubbing benchmark, Text-AB delivers a step-change improvement over the latest internal dubbing system, with large gains in prosody similarity, voice similarity, naturalness, and shareability. For full-duplex dialogue synthesis, it approaches human recordings on short-form conversations and substantially outperforms the latest internal model on long-form human-likeness and expressivity, while natively modeling turn-taking, back-channeling, and emotional dynamics. For emotional dialogue synthesis, emotion conditioning significantly improves emotion alignment and emotional interaction quality over the unconditioned baseline.
The paper introduces Text-AB, a unified alignment-free diffusion framework for high-quality voice dubbing and full-duplex dialogue synthesis that leverages latent diffusion with high-compression VAE features and cross-attention for text-speech alignment. This work represents a strong technical advance in audio generation, particularly for complex tasks like full-duplex dialogue where modeling turn-taking and back-channeling is critical, and the shift to an alignment-free approach simplifies the pipeline significantly. While the components (Diffusion Transformer, flow matching, VAE) are established, their integration into a scalable, high-fidelity system for these specific downstream tasks constitutes a meaningful contribution to the field of speech synthesis and audio ML.