We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University
The paper introduces AuK, a unified foundational model for speech generation and editing that leverages a hybrid MLLM-VAE-Transformer architecture, extensive data construction, and advanced post-training strategies including preference optimization and reinforcement learning. It represents a significant step towards general-purpose audio AI by consolidating diverse tasks into a single instruction-driven framework, offering a robust and efficient solution for complex speech manipulation tasks.
The paper proposes AuK, a unified foundational model for speech generation and editing. The architecture is a hybrid of a Multimodal Large Language Model (MLLM) for semantic conditioning, a Variational Autoencoder (VAE) for acoustic conditioning, and a Transformer backbone using a FLUX-style design (dual-stream MMDiT blocks followed by single-stream DiT blocks). The training pipeline is extensive, involving a two-stage pre-training curriculum (generation-only warm-up followed by joint generation-editing), post-training with human-feedback preference optimization (DPO) for editing and Flow-GRPO for generation, and a distillation stage using Consistency Initialization and Task-Routed Decoupled DMD to achieve 4-step inference. The methodology is sophisticated, particularly the integration of RL and preference optimization for audio tasks, which is relatively novel in this specific combination. The data construction pipeline is also detailed, covering 3.03 billion instruction-audio instances across five task families.
The experiments cover a wide range of tasks: zero-shot TTS, instruction TTS, acoustic editing, paralinguistic editing, content editing, enhancement, and separation. The paper claims leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing. It reports a 4.5x wall-clock speedup for the distilled model (AuK-Flash). However, the provided text is truncated before the detailed results tables, so specific quantitative comparisons (e.g., MOS scores, WER, speaker similarity scores) against baselines are not fully visible in the excerpt. The reliance on "leading performance" claims without visible comparative tables in the provided text limits the ability to fully verify the magnitude of improvement, though the scope of evaluation is comprehensive.
The paper claims to release both source code and model weights. It provides detailed hyperparameters for training (learning rates, batch sizes, GPU counts, optimizer settings) and architectural details (layer counts, hidden dimensions, strides). The data construction pipeline is described in detail, including specific tools used for annotation and synthesis (Qwen3-Omni, F5-TTS, etc.). This level of detail supports high reproducibility, assuming the released code and weights are complete and functional.
The model is large (1.5B parameters in the backbone alone, plus the MLLM and VAE), which may limit deployment on edge devices despite the distillation. The reliance on a complex MLLM for semantic conditioning adds computational overhead during inference compared to simpler text encoders. The paper is a technical report, so peer review rigor may be lower than a top-tier conference paper. The truncated text prevents a full assessment of the experimental results' depth.
The unification of speech generation and editing under a single instruction-based interface has significant potential for practical applications in content creation, accessibility, and interactive voice assistants. The open-source release of such a comprehensive model could accelerate research in unified audio models. The techniques for post-training (RL and DPO) and distillation are applicable to other generative audio tasks. The paper introduces AuK, a unified foundational model for speech generation and editing that leverages a hybrid MLLM-VAE-Transformer architecture, extensive data construction, and advanced post-training strategies including preference optimization and reinforcement learning. It represents a significant step towards general-purpose audio AI by consolidating diverse tasks into a single instruction-driven framework, offering a robust and efficient solution for complex speech manipulation tasks.
Audio-conditioned language models often underuse acoustic cues such as prosody, emotion, and non-speech sounds, raising the question of whether ASR-supervised frontends discard this information before it reaches the LM. We test whether the frontend is responsible by comparing Whisper-Tiny and Whisper-Small with EnCodec, DAC-VAE, and WavTokenizer in a shared Qwen3.5-4B audio-LM pipeline on ASR, emotion recognition, and sound captioning. Encoder replacement alone does not resolve this underuse: Whisper variants remain strongest overall, including on emotion and environmental sound captioning. To localize the failure, we trace task-relevant information through the encoder, projector, LM layers, and LM head. Linear probes and geometric analyses show that discriminative acoustic structure remains recoverable at the final LM layer, even when MCQA accuracy trails probe accuracy by up to 83 points. Because the answer format and decoding procedure are controlled, this task-dependent gap points to content-specific readout failure rather than generic format bias. LogitLens analyses and a targeted LM head intervention support the conclusion that acoustic underuse is not explained solely by encoder-side information loss and that readout alignment can be a dominant bottleneck.
Primary: Seoul National University
All Institutions: Seoul National University, KAIST, NAVER Cloud
The paper identifies that acoustic underuse in audio-LLMs is primarily a readout alignment failure rather than an encoder-side information loss, demonstrating that task-relevant acoustic structure remains recoverable in the LM's final hidden states even when the model fails to verbalize it correctly. Through a combination of encoder replacement experiments, layer-wise probing, and targeted LM head interventions, the study provides a detailed localization of this bottleneck, offering actionable insights for improving the acoustic grounding of language models.
The paper employs a rigorous diagnostic methodology to localize the source of "acoustic underuse" in audio-LLMs. It systematically compares ASR-supervised encoders (Whisper) against reconstruction-based codecs (EnCodec, DAC-VAE, WavTokenizer) within a controlled shared-LM pipeline (Qwen3.5-4B). The core methodological strength lies in the multi-stage information tracing: using linear probes, geometric distance ratios, and LogitLens analyses to track task-relevant information from the encoder through the projector and LM layers to the final output head. The causal intervention of fine-tuning only the choice-letter rows of the LM head ($W_U$) is a clever and effective way to isolate readout failure from representation loss.
The experiments are extensive, covering ASR, emotion recognition, and sound captioning across multiple datasets (LibriSpeech, IEMOCAP, FSD50K, NSynth, etc.). The results clearly demonstrate that while Whisper encoders perform best on end-to-end tasks, they do not necessarily contain more recoverable acoustic information than reconstruction-based codecs at the latent level. The finding that linear probe accuracy at the final LM layer significantly exceeds MCQA accuracy (by up to 83 points) provides strong evidence for a readout bottleneck. The recovery of 30-42 points in MCQA accuracy by updating only ~10-26k scalars in the LM head is a compelling result.
The paper provides detailed implementation specifics, including model checkpoints, projector architecture, training hyperparameters, and dataset splits. The use of standard open-source components (Whisper, EnCodec, Qwen) and clear descriptions of the probing and intervention protocols make the work highly reproducible.
The analysis is primarily focused on MCQA-format classification tasks. While the authors acknowledge this, the generalization of the "readout bottleneck" finding to free-form generative tasks (like open-ended captioning or dialogue) is not fully established. The LM scale is limited to 4B parameters, so it is unclear if the bottleneck persists or shifts in larger models. The study excludes self-supervised encoders (like WavLM or HuBERT), which are known to retain strong paralinguistic features, limiting the scope of the encoder comparison.
This work provides a crucial diagnostic framework for the audio-LLM community. By identifying that the failure often lies in the LM's ability to map internal acoustic representations to text tokens (readout) rather than in the encoder's ability to capture them, it shifts the focus of future research toward better alignment strategies, label-aware training, or readout regularization. It challenges the common assumption that swapping ASR encoders for codec-based ones is a sufficient solution for improving non-lexical audio understanding. The paper identifies that acoustic underuse in audio-LLMs is primarily a readout alignment failure rather than an encoder-side information loss, demonstrating that task-relevant acoustic structure remains recoverable in the LM's final hidden states even when the model fails to verbalize it correctly. Through a combination of encoder replacement experiments, layer-wise probing, and targeted LM head interventions, the study provides a detailed localization of this bottleneck, offering actionable insights for improving the acoustic grounding of language models.
Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.
Primary: Mila-Quebec AI Institute
All Institutions: Concordia University, Mila-Quebec AI Institute, Université Laval
ZipCodec introduces a streaming neural speech codec operating at an exceptionally low frame rate of 6.25 Hz and 0.80 kbps, achieving state-of-the-art reconstruction and representation quality through a redesigned transformer architecture (ErfFormer), scalar spherical quantization, and large-scale WavLM distillation, thereby enabling efficient and responsive speech language modeling.
The paper proposes ZipCodec, a streaming neural speech codec that significantly reduces the frame rate to 6.25 Hz (160 ms per token) while maintaining a low bitrate of 0.80 kbps. The methodology is built upon the FocalCodec-Stream architecture but introduces several key modifications: (1) replacing the learned convolutional encoder with a causal log-mel frontend to simplify the pipeline; (2) introducing "ErfFormer," a transformer variant that removes RMSNorm and positional encodings, relying on DynamicErf activation and causal attention to handle long streaming contexts without positional extrapolation; (3) employing Scalar Spherical Quantization (SSQ) with a factorized bottleneck (64 dimensions, 4 levels) to achieve compact representation; and (4) a latency-aware streaming decoder that processes 8 WavLM representations jointly within the 160 ms window to relax causality constraints at the feature level. The training strategy involves large-scale distillation of WavLM layer-6 representations using ~94,000 hours of speech data, with specific noise and overlapping speech augmentations to match the teacher model's distribution.
The experimental evaluation is comprehensive, covering speech resynthesis (English and multilingual), voice conversion, and a suite of downstream discriminative and generative tasks (ASR, SI, SER, KS, IC, SE, SS). ZipCodec demonstrates superior performance compared to strong baselines like Mimi, PAST, and FocalCodec-Stream, particularly in intelligibility (dWER) and speaker similarity (Sim) at the significantly lower frame rate. The paper also provides detailed streaming efficiency metrics, showing real-time inference on consumer-grade CPUs and efficient GPU scaling. The results convincingly show that the 6.25 Hz bottleneck retains sufficient information for high-quality reconstruction and downstream tasks, narrowing the gap with non-streaming offline codecs.
The paper provides high reproducibility. It explicitly states that code and checkpoints are available. Detailed hyperparameters, training data sources (LibriLight, VoxPopuli, GigaSpeech), augmentation strategies, and architectural specifications (e.g., ErfFormer dimensions, SSQ parameters) are provided. The use of standard datasets and metrics (UTMOS, dWER, DNSMOS) allows for easy verification and comparison with other works in the field.
The primary limitation is the model size (842M parameters), which, while manageable for single-stream CPU inference, may be prohibitive for edge devices or high-concurrency server deployments compared to smaller codecs. Additionally, the evaluation is primarily focused on English and specific multilingual subsets; broader language coverage is not extensively tested. The reliance on WavLM distillation ties the codec's performance to the quality and biases of the WavLM teacher model.
This work has significant implications for speech-language modeling and real-time speech-to-speech systems. By reducing the frame rate to 6.25 Hz, the token sequence length is drastically shortened, which reduces computational costs and simplifies autoregressive modeling for speech LLMs. The low latency (160 ms) makes it suitable for highly responsive conversational AI applications. The open-sourcing of code and demos facilitates further research and adoption in the community. ZipCodec introduces a streaming neural speech codec operating at an exceptionally low frame rate of 6.25 Hz and 0.80 kbps, achieving state-of-the-art reconstruction and representation quality through a redesigned transformer architecture (ErfFormer), scalar spherical quantization, and large-scale WavLM distillation, thereby enabling efficient and responsive speech language modeling.
Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.
Primary: Mila-Quebec AI Institute
All Institutions: Concordia University, Mila-Quebec AI Institute, Université Laval
ZipCodec introduces a streaming neural speech codec operating at an exceptionally low frame rate of 6.25 Hz and 0.80 kbps, achieving state-of-the-art reconstruction and representation quality through a redesigned transformer architecture (ErfFormer), scalar spherical quantization, and large-scale WavLM distillation, thereby enabling efficient and responsive speech language modeling.
The paper proposes ZipCodec, a streaming neural speech codec that significantly reduces the frame rate to 6.25 Hz (160 ms per token) while maintaining a low bitrate of 0.80 kbps. The methodology is built upon the FocalCodec-Stream architecture but introduces several key modifications: (1) replacing the learned convolutional encoder with a causal log-mel frontend to simplify the pipeline; (2) introducing "ErfFormer," a transformer variant that removes RMSNorm and positional encodings, relying on DynamicErf activation and causal attention to handle long streaming contexts without positional extrapolation; (3) employing Scalar Spherical Quantization (SSQ) with a factorized bottleneck (64 dimensions, 4 levels) to achieve compact representation; and (4) a latency-aware streaming decoder that processes 8 WavLM representations jointly within the 160 ms window to relax causality constraints at the feature level. The training strategy involves large-scale distillation of WavLM layer-6 representations using ~94,000 hours of speech data, with specific noise and overlapping speech augmentations to match the teacher model's distribution.
The experimental evaluation is comprehensive, covering speech resynthesis (English and multilingual), voice conversion, and a suite of downstream discriminative and generative tasks (ASR, SI, SER, KS, IC, SE, SS). ZipCodec demonstrates superior performance compared to strong baselines like Mimi, PAST, and FocalCodec-Stream, particularly in intelligibility (dWER) and speaker similarity (Sim) at the significantly lower frame rate. The paper also provides detailed streaming efficiency metrics, showing real-time inference on consumer-grade CPUs and efficient GPU scaling. The results convincingly show that the 6.25 Hz bottleneck retains sufficient information for high-quality reconstruction and downstream tasks, narrowing the gap with non-streaming offline codecs.
The paper provides high reproducibility. It explicitly states that code and checkpoints are available. Detailed hyperparameters, training data sources (LibriLight, VoxPopuli, GigaSpeech), augmentation strategies, and architectural specifications (e.g., ErfFormer dimensions, SSQ parameters) are provided. The use of standard datasets and metrics (UTMOS, dWER, DNSMOS) allows for easy verification and comparison with other works in the field.
The primary limitation is the model size (842M parameters), which, while manageable for single-stream CPU inference, may be prohibitive for edge devices or high-concurrency server deployments compared to smaller codecs. Additionally, the evaluation is primarily focused on English and specific multilingual subsets; broader language coverage is not extensively tested. The reliance on WavLM distillation ties the codec's performance to the quality and biases of the WavLM teacher model.
This work has significant implications for speech-language modeling and real-time speech-to-speech systems. By reducing the frame rate to 6.25 Hz, the token sequence length is drastically shortened, which reduces computational costs and simplifies autoregressive modeling for speech LLMs. The low latency (160 ms) makes it suitable for highly responsive conversational AI applications. The open-sourcing of code and demos facilitates further research and adoption in the community. ZipCodec introduces a streaming neural speech codec operating at an exceptionally low frame rate of 6.25 Hz and 0.80 kbps, achieving state-of-the-art reconstruction and representation quality through a redesigned transformer architecture (ErfFormer), scalar spherical quantization, and large-scale WavLM distillation, thereby enabling efficient and responsive speech language modeling.
Large Audio Language Models (LALMs) have shown strong performance on audio reasoning benchmarks, but accuracy alone cannot distinguish true reasoning from superficial pattern matching, often overestimating reasoning ability since high scores may result from guessing rather than genuine audio understanding. Evaluating the reasoning process itself is essential for improving LALMs' reasoning ability, yet remains challenging. Existing methods either rely on costly human annotation or opaque LLM-as-judge approaches, making them impractical, biased, and lacking transparency. Moreover, audio reasoning introduces unique challenges absent in text-based settings, perceptual hallucination and cross-modal alignment between audio understanding and textual inference, hence text-based evaluation frameworks cannot be directly applied. Therefore, we propose ARIA-Rubrics (Audio Reasoning Integrity Assessment), a lightweight, annotation-free gold reasoning chains, automatic and transparent framework comprising six complementary metrics that evaluate audio reasoning quality across perceptual grounding, reasoning coherence, and answer consistency. We use Chain-of-Thought prompting as an externalization mechanism to make the reasoning process observable. Experiments on 9 models across 2 benchmarks identify three reasoning modes of current LALMs with actionable directions for future development, with ARIA-Rubrics achieving high correlation with human judgments. The code is available at the Github Repository.
Primary: Imperial College London
All Institutions: Imperial College London, Technische Universität München, Mohamed bin Zayed University of Artificial Intelligence, Shanghai Jiao Tong University, Johns Hopkins University
The paper introduces ARIA-Rubrics, a comprehensive framework for evaluating audio reasoning in LALMs by combining perceptual grounding checks with reasoning process analysis. It effectively addresses the limitation of accuracy-only benchmarks by identifying specific reasoning failure modes, offering a valuable tool for the community to develop more robust and transparent audio language models.
The paper proposes ARIA-Rubrics, a framework to evaluate the reasoning process of Large Audio Language Models (LALMs) rather than just final answer accuracy. The methodology relies on Chain-of-Thought (CoT) prompting to externalize reasoning into four steps: Perception, Analysis, Reasoning, and Answer. It introduces six metrics: Acoustic Grounding Score (using CLAP embeddings to verify entities against audio), Inter-step Coherence (using MPNet embeddings), Reasoning Chain Coverage (using a lightweight LLM judge for substantiveness), Reasoning Step Causality (using perplexity reduction), Reasoning Progress Score (using NLI entailment), and Audio Lexical Density (using Word2Vec similarity to AudioSet ontology). The approach is innovative in combining audio-specific perceptual checks with text-based reasoning quality metrics, addressing the gap where text-only metrics fail to detect perceptual hallucinations.
The authors evaluate 9 models across two benchmarks (MMAR and MMAU-mini). They identify three distinct reasoning modes: Reasoning, Scaffolding, and Decoration. The paper claims high correlation with human judgments, validating the automated metrics. The experimental setup is robust, covering both open-source and closed-source models, and the identification of specific failure modes (like scaffolding) provides actionable insights for model developers.
The code is available on GitHub. The paper specifies the models used for each metric (CLAP, MPNet, Qwen2.5-1.5B, RoBERTa-large-mnli, Word2Vec) and provides details on the CoT prompt structure and the AudioSet ontology filtering. This level of detail supports reproducibility, although the specific few-shot prompts for the LLM judge are referenced in appendices not fully visible in the truncated text, which is standard practice.
The framework relies on the quality of the CoT output; if a model refuses to generate CoT or generates unstructured text, the metrics may fail or be biased. The use of a small LLM (Qwen2.5-1.5B) for judging substantiveness and causality may limit the depth of evaluation compared to larger models. Additionally, the Audio Lexical Density metric is a proxy and may not capture all aspects of audio-relevant reasoning.
This work is significant for the development of reliable audio AI systems. By distinguishing between genuine reasoning and pattern matching, it provides a diagnostic tool for improving LALMs. It encourages the development of models that are not only accurate but also faithful in their reasoning, which is crucial for safety and interpretability in audio applications. The paper introduces ARIA-Rubrics, a comprehensive framework for evaluating audio reasoning in LALMs by combining perceptual grounding checks with reasoning process analysis. It effectively addresses the limitation of accuracy-only benchmarks by identifying specific reasoning failure modes, offering a valuable tool for the community to develop more robust and transparent audio language models.
We present UniStream, a fully causal 48 kHz neural audio codec for streaming speech, music, and environmental sounds. At its core is Multi-Expert Residual Vector Quantization (ME-RVQ), which replaces the single shared codebook in each residual quantization layer with four expert codebooks controlled by a deterministic Top-K router. Because routing decisions are derived solely from previously decoded quantized states, the decoder can reproduce the selected experts without transmitting expert identifiers, thereby expanding quantization capacity while adding 5.5M parameters. We further introduce an auxiliary Optimal Transport Conditional Flow Matching (OT-CFM) objective to regularize the quantized latent space during training. The flow module is removed entirely at inference and therefore incurs no runtime overhead. UniStream supports a 12 kbps Top-1 mode and a 22.5 kbps Top-2 mode within a causal 48 kHz encoder-decoder framework, while achieving real-time GPU inference. To complement narrow-band speech metrics, we report 48 kHz ViSQOL audio mode, ViSQOL speech mode, standard VGGish-FAD, DNSMOS P.835, and higher-rate reference comparisons with Opus and EnCodec. At 12 kbps, UniStream-Top1 achieves PESQ and UTMOS scores comparable to EnCodec while reducing speech Mel-D from 13.07 to 8.21. At 22.5 kbps, UniStream-Top2 achieves a ViSQOL speech-mode score of 4.67 and an environmental audio-mode score of 3.96, exceeding all evaluated systems operating at 12 kbps or below in the latter setting. It also comes within 0.03 MOS-LQO of Opus at 24 kbps on speech in ViSQOL audio mode. Ablation studies confirm that ME-RVQ is the primary source of quality improvement, whereas OT-CFM provides perceptual gains on speech with a mild trade-off in spectral distortion.
Primary: Tsinghua University
All Institutions: Tsinghua Shenzhen International Graduate School, Tsinghua University
UniStream introduces a causal 48 kHz audio codec using Multi-Expert Residual Vector Quantization to expand capacity without signaling overhead, achieving competitive quality at low bitrates. The paper demonstrates a solid engineering approach to improving neural codecs for streaming, with rigorous evaluation across multiple audio domains, though it lacks subjective validation and open-source availability.
The paper proposes UniStream, a causal neural audio codec featuring Multi-Expert Residual Vector Quantization (ME-RVQ). The core innovation is replacing single codebooks in residual layers with four expert codebooks routed by a deterministic Top-K mechanism. Crucially, the routing is derived from previously decoded states, allowing the decoder to replicate expert selection without transmitting expert IDs, thus saving bandwidth. The method also introduces a training-only Optimal Transport Conditional Flow Matching (OT-CFM) regularizer to improve latent space quality without inference cost. The architecture is fully causal, suitable for streaming, and supports 12 kbps and 22.5 kbps modes.
Experiments are conducted on speech (LibriSpeech), music (MTG-Jamendo), and environmental audio (FSD50K). The evaluation is comprehensive, using both narrow-band metrics (PESQ, STOI, UTMOS) and full-band perceptual metrics (ViSQOL, VGGish-FAD, DNSMOS). Results show UniStream-Top1 outperforms EnCodec in spectral distortion (Mel-D) at 12 kbps, while UniStream-Top2 approaches Opus 24 kbps quality at a lower bitrate. Ablations confirm ME-RVQ is the primary driver of improvement, with OT-CFM providing modest perceptual gains.
The paper provides detailed architectural specifications (strides, dilation factors, codebook sizes, loss weights) and training hyperparameters. However, no code repository or demo link is provided in the text. The use of standard datasets and metrics facilitates reproduction, but the lack of open-source code limits immediate verification.
The paper lacks subjective listening tests (MOS), relying solely on objective metrics. Music domain performance (FAD) is noted as weaker compared to speech and environmental audio. The CPU inference time (RTF 1.05) is borderline for real-time deployment on standard hardware, though GPU inference is efficient. The comparison with non-causal baselines like EnCodec is somewhat unfair for streaming contexts, though acknowledged.
This work contributes to the field of low-latency neural audio coding, particularly for applications requiring full-band (48 kHz) fidelity in streaming scenarios. The deterministic routing mechanism offers a practical solution for expanding quantization capacity without signaling overhead, which could influence future codec designs. UniStream introduces a causal 48 kHz audio codec using Multi-Expert Residual Vector Quantization to expand capacity without signaling overhead, achieving competitive quality at low bitrates. The paper demonstrates a solid engineering approach to improving neural codecs for streaming, with rigorous evaluation across multiple audio domains, though it lacks subjective validation and open-source availability.
We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University
The paper introduces AuK, a unified foundational model for speech generation and editing that leverages a hybrid MLLM-VAE-Transformer architecture, extensive data construction, and advanced post-training strategies including preference optimization and reinforcement learning. It represents a significant step towards general-purpose audio AI by consolidating diverse tasks into a single instruction-driven framework, offering a robust and efficient solution for complex speech manipulation tasks.
The paper proposes AuK, a unified foundational model for speech generation and editing. The architecture is a hybrid of a Multimodal Large Language Model (MLLM) for semantic conditioning, a Variational Autoencoder (VAE) for acoustic conditioning, and a Transformer backbone using a FLUX-style design (dual-stream MMDiT blocks followed by single-stream DiT blocks). The training pipeline is extensive, involving a two-stage pre-training curriculum (generation-only warm-up followed by joint generation-editing), post-training with human-feedback preference optimization (DPO) for editing and Flow-GRPO for generation, and a distillation stage using Consistency Initialization and Task-Routed Decoupled DMD to achieve 4-step inference. The methodology is sophisticated, particularly the integration of RL and preference optimization for audio tasks, which is relatively novel in this specific combination. The data construction pipeline is also detailed, covering 3.03 billion instruction-audio instances across five task families.
The experiments cover a wide range of tasks: zero-shot TTS, instruction TTS, acoustic editing, paralinguistic editing, content editing, enhancement, and separation. The paper claims leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing. It reports a 4.5x wall-clock speedup for the distilled model (AuK-Flash). However, the provided text is truncated before the detailed results tables, so specific quantitative comparisons (e.g., MOS scores, WER, speaker similarity scores) against baselines are not fully visible in the excerpt. The reliance on "leading performance" claims without visible comparative tables in the provided text limits the ability to fully verify the magnitude of improvement, though the scope of evaluation is comprehensive.
The paper claims to release both source code and model weights. It provides detailed hyperparameters for training (learning rates, batch sizes, GPU counts, optimizer settings) and architectural details (layer counts, hidden dimensions, strides). The data construction pipeline is described in detail, including specific tools used for annotation and synthesis (Qwen3-Omni, F5-TTS, etc.). This level of detail supports high reproducibility, assuming the released code and weights are complete and functional.
The model is large (1.5B parameters in the backbone alone, plus the MLLM and VAE), which may limit deployment on edge devices despite the distillation. The reliance on a complex MLLM for semantic conditioning adds computational overhead during inference compared to simpler text encoders. The paper is a technical report, so peer review rigor may be lower than a top-tier conference paper. The truncated text prevents a full assessment of the experimental results' depth.
The unification of speech generation and editing under a single instruction-based interface has significant potential for practical applications in content creation, accessibility, and interactive voice assistants. The open-source release of such a comprehensive model could accelerate research in unified audio models. The techniques for post-training (RL and DPO) and distillation are applicable to other generative audio tasks. The paper introduces AuK, a unified foundational model for speech generation and editing that leverages a hybrid MLLM-VAE-Transformer architecture, extensive data construction, and advanced post-training strategies including preference optimization and reinforcement learning. It represents a significant step towards general-purpose audio AI by consolidating diverse tasks into a single instruction-driven framework, offering a robust and efficient solution for complex speech manipulation tasks.
Full-duplex spoken dialogue systems enable simultaneous listening and speaking, but their audio-only perception often fails under background noise and overlapping speech, leading to incoherent responses. Recent audio-visual dialogue approaches show that incorporating visual cues such as lip movements improve robustness under audio corruption. However, existing approaches often adapt the large speech dialogue model itself to process visual input, requiring costly multimodal training. We propose AV-STE, a modular streaming audio-visual front-end that restores corrupted semantic speech tokens from noisy audio and lip video before they reach the speech LLM. The downstream dialogue model remains entirely frozen, preserving its pretrained conversational capabilities. When integrated with frozen Moshi, AV-STE improves average GPT-4o-judged response coherence from 1.42 to 1.91 under same-dataset speaker interference while largely preserving turn-taking behavior. Gains also transfer to out-of-domain Seamless Interaction.
Primary: KAIST
All Institutions: KAIST
The paper presents a novel and effective modular approach to enhancing the robustness of full-duplex spoken dialogue systems by leveraging audio-visual cues to restore corrupted semantic tokens without retraining the underlying speech LLM. The technical contributions, including the streaming adaptation of AV-HuBERT and the noise-adaptive fusion mechanism, are well-validated through comprehensive experiments on in-domain and out-of-domain datasets, demonstrating significant improvements in both token recovery and downstream dialogue coherence.
The paper proposes AV-STE, a modular front-end for enhancing semantic speech tokens in full-duplex dialogue systems. The core contribution is the decoupling of robustness enhancement from the large language model (LLM) backbone. By keeping the speech LLM (Moshi) frozen and only enhancing the semantic token stream using a streaming audio-visual encoder (modified AV-HuBERT) and a noise-adaptive fusion module, the authors address a significant practical limitation of existing multimodal dialogue systems that require costly retraining. The introduction of "Soft Token Cross-Attention" to handle uncertainty in noisy tokenizer outputs and "Noise-Adaptive Modulation" based on entropy are technically sound and well-motivated innovations. The adaptation of bidirectional AV-HuBERT to a causal, streaming architecture with bounded lookahead is a necessary and well-executed engineering contribution for real-time applications.
The experimental setup is rigorous, covering in-domain (LRS3) and out-of-domain (Seamless Interaction) datasets, with various noise types (non-speech, speaker interference). The evaluation metrics are appropriate for the task: semantic token accuracy for the front-end and GPT-4o-judged coherence plus turn-taking metrics (TOR, latency) for the downstream dialogue. The results show substantial improvements in token recovery (e.g., from 12.61% to 65.60% under same-dataset interference) and downstream coherence (1.42 to 1.91). The comparison against a non-streaming baseline (PASE) and a different dialogue model (PersonaPlex) provides useful context, although the primary comparison is against the frozen Moshi baseline. The qualitative examples strongly support the quantitative findings.
The paper provides a GitHub link to the code and models. The training configuration, dataset preprocessing, and model architecture details are described in sufficient detail for reproduction. The use of standard datasets (LRS3, AudioSet, Seamless Interaction) and open-source components (Moshi, AV-HuBERT) further enhances reproducibility.
The paper acknowledges that AV-STE assumes reliable visual input and does not handle visual occlusion or missing video. There is a noted domain shift issue where clean-token accuracy drops on out-of-domain data (Seamless Interaction), indicating sensitivity to domain characteristics. The reliance on GPT-4o for evaluation, while common, introduces a potential bias, though it is a standard practice in current LLM evaluation. The method is specific to the Moshi/Mimi tokenization scheme, limiting immediate applicability to other speech LLM architectures without adaptation.
This work has significant implications for the deployment of robust spoken dialogue systems in real-world, noisy environments. By providing a plug-and-play module that enhances robustness without retraining the expensive LLM backbone, it lowers the barrier to entry for audio-visual dialogue systems. The focus on full-duplex interaction is timely and relevant to the next generation of conversational AI. The approach could be extended to other modalities or tokenization schemes, potentially influencing the design of future multimodal speech interfaces. The paper presents a novel and effective modular approach to enhancing the robustness of full-duplex spoken dialogue systems by leveraging audio-visual cues to restore corrupted semantic tokens without retraining the underlying speech LLM. The technical contributions, including the streaming adaptation of AV-HuBERT and the noise-adaptive fusion mechanism, are well-validated through comprehensive experiments on in-domain and out-of-domain datasets, demonstrating significant improvements in both token recovery and downstream dialogue coherence.
In this work, we present Gander, an end-to-end model that unifies omni perception, realtime interaction, and agentic capabilities within a single framework. In contrast to turn-based conventional paradigms, Gander continuously receives streaming inputs across multiple modalities, including video, speech, and text, enabling natural full-duplex interaction in both everyday conversations and complex workflow-oriented agent scenarios. Users can interrupt the model at any time, while the model can also proactively provide intermediate feedback or ask follow up questions. To natively support these capabilities, Gander adopts two key architectural designs: 1) It employs a Cerebellum-Brain collaborative framework, in which the Cerebellum is responsible for realtime interaction and omni conversational capabilities, while the Brain handles complex reasoning and higher-level agentic tasks. The two components interact continuously through tool calling and the agent orchestration runtime. 2) The Cerebellum is built upon a streaming Thinker-Talker architecture, user inputs and model outputs are further flattened into an ordered token stream at the chunk level, providing a unified representation for low latency, continuous interaction. We conduct comprehensive evaluations of Gander across four dimensions: conversational ability, omni understanding, interactive capability, and agentic intelligence. Internal human evaluations demonstrate that Gander maintains the natural and expressive spoken dialogue capabilities of SOTA open source models while achieving competitive performance in omni interaction. Gander also demonstrates robustness in challenging real-world scenarios, including background noise interference, multi-party interactions, and backchannel communication. We release Gander together with its models, code, and data to facilitate further research and development in the community.
Primary: Tencent
All Institutions: Tencent, Zhejiang University, Shanghai Jiao Tong University, The Chinese University of Hong Kong, Nanyang Technological University
Gander introduces a Cerebellum-Brain architecture that decouples real-time full-duplex interaction from long-horizon agentic reasoning, enabling natural, interruptible multimodal collaboration. The paper presents a robust streaming framework with detailed data construction for interaction behaviors, though its reliance on external proprietary agents for the "Brain" component and limited quantitative benchmarking against specific baselines slightly temper its technical impact.
The paper proposes "Gander," an end-to-end Omni Interaction Agent that unifies perception, real-time interaction, and agentic capabilities. The core architectural innovation is the "Cerebellum-Brain" collaborative framework. The "Cerebellum" is a streaming Thinker-Talker model responsible for low-latency, full-duplex interaction, while the "Brain" is a training-free, plug-and-play agent (e.g., Codex, Claude Code) handling complex reasoning. The Cerebellum uses a "streaming chunk flattening" mechanism, serializing audio, video, and text into a unified token stream where the model predicts control tokens (listen/speak/interrupt) at each 1-second chunk. This allows for native interruption and proactive feedback without external VAD modules. The methodology is sound, addressing the latency-intelligence trade-off by decoupling fast interaction from slow reasoning.
The evaluation is comprehensive, covering conversational ability, omni understanding, interactive capability, and agentic intelligence. The paper reports internal human evaluations showing Gander maintains SOTA spoken dialogue naturalness while achieving competitive omni interaction performance. It demonstrates robustness in noisy environments, multi-party interactions, and backchannel communication. However, the paper relies heavily on "internal human evaluations" and subjective demos, with limited quantitative benchmarks compared to specific baselines (e.g., GPT-4o, MiniCPM-o) in the provided text snippet. The lack of standardized public benchmark scores for the "agentic" component makes direct comparison difficult.
The authors state they release the model weights, code, and data. The architecture details (chunk size, token rates, encoder types like SigLIP) are provided. However, the "Brain" component relies on proprietary external agents (Codex, Claude Code), which limits full reproducibility of the agentic performance. The training data pipeline is described in detail, including the synthesis of full-duplex interaction data, which aids reproducibility of the Cerebellum component.
The system's performance is coupled to the quality of the external "Brain" agent, which may have its own latency and cost implications. The "training-free" nature of the Brain means the Cerebellum cannot learn from the Brain's specific failures or adapt its tool-calling strategy through end-to-end fine-tuning. The evaluation lacks extensive quantitative ablation studies comparing the Cerebellum-Brain architecture against monolithic models on standardized agentic benchmarks.
This work is significant for the development of natural, human-like AI assistants that can handle complex tasks while maintaining fluid conversation. The decoupled architecture offers a scalable path for integrating powerful reasoning models into real-time interfaces. It pushes the field beyond turn-based chatbots toward continuous, multimodal collaboration. Gander introduces a Cerebellum-Brain architecture that decouples real-time full-duplex interaction from long-horizon agentic reasoning, enabling natural, interruptible multimodal collaboration. The paper presents a robust streaming framework with detailed data construction for interaction behaviors, though its reliance on external proprietary agents for the "Brain" component and limited quantitative benchmarking against specific baselines slightly temper its technical impact.
Audio-conditioned language models often underuse acoustic cues such as prosody, emotion, and non-speech sounds, raising the question of whether ASR-supervised frontends discard this information before it reaches the LM. We test whether the frontend is responsible by comparing Whisper-Tiny and Whisper-Small with EnCodec, DAC-VAE, and WavTokenizer in a shared Qwen3.5-4B audio-LM pipeline on ASR, emotion recognition, and sound captioning. Encoder replacement alone does not resolve this underuse: Whisper variants remain strongest overall, including on emotion and environmental sound captioning. To localize the failure, we trace task-relevant information through the encoder, projector, LM layers, and LM head. Linear probes and geometric analyses show that discriminative acoustic structure remains recoverable at the final LM layer, even when MCQA accuracy trails probe accuracy by up to 83 points. Because the answer format and decoding procedure are controlled, this task-dependent gap points to content-specific readout failure rather than generic format bias. LogitLens analyses and a targeted LM head intervention support the conclusion that acoustic underuse is not explained solely by encoder-side information loss and that readout alignment can be a dominant bottleneck.
Primary: Seoul National University
All Institutions: Seoul National University, KAIST, NAVER Cloud
The paper identifies that acoustic underuse in audio-LLMs is primarily a readout alignment failure rather than an encoder-side information loss, demonstrating that task-relevant acoustic structure remains recoverable in the LM's final hidden states even when the model fails to verbalize it correctly. Through a combination of encoder replacement experiments, layer-wise probing, and targeted LM head interventions, the study provides a detailed localization of this bottleneck, offering actionable insights for improving the acoustic grounding of language models.
The paper employs a rigorous diagnostic methodology to localize the source of "acoustic underuse" in audio-LLMs. It systematically compares ASR-supervised encoders (Whisper) against reconstruction-based codecs (EnCodec, DAC-VAE, WavTokenizer) within a controlled shared-LM pipeline (Qwen3.5-4B). The core methodological strength lies in the multi-stage information tracing: using linear probes, geometric distance ratios, and LogitLens analyses to track task-relevant information from the encoder through the projector and LM layers to the final output head. The causal intervention of fine-tuning only the choice-letter rows of the LM head ($W_U$) is a clever and effective way to isolate readout failure from representation loss.
The experiments are extensive, covering ASR, emotion recognition, and sound captioning across multiple datasets (LibriSpeech, IEMOCAP, FSD50K, NSynth, etc.). The results clearly demonstrate that while Whisper encoders perform best on end-to-end tasks, they do not necessarily contain more recoverable acoustic information than reconstruction-based codecs at the latent level. The finding that linear probe accuracy at the final LM layer significantly exceeds MCQA accuracy (by up to 83 points) provides strong evidence for a readout bottleneck. The recovery of 30-42 points in MCQA accuracy by updating only ~10-26k scalars in the LM head is a compelling result.
The paper provides detailed implementation specifics, including model checkpoints, projector architecture, training hyperparameters, and dataset splits. The use of standard open-source components (Whisper, EnCodec, Qwen) and clear descriptions of the probing and intervention protocols make the work highly reproducible.
The analysis is primarily focused on MCQA-format classification tasks. While the authors acknowledge this, the generalization of the "readout bottleneck" finding to free-form generative tasks (like open-ended captioning or dialogue) is not fully established. The LM scale is limited to 4B parameters, so it is unclear if the bottleneck persists or shifts in larger models. The study excludes self-supervised encoders (like WavLM or HuBERT), which are known to retain strong paralinguistic features, limiting the scope of the encoder comparison.
This work provides a crucial diagnostic framework for the audio-LLM community. By identifying that the failure often lies in the LM's ability to map internal acoustic representations to text tokens (readout) rather than in the encoder's ability to capture them, it shifts the focus of future research toward better alignment strategies, label-aware training, or readout regularization. It challenges the common assumption that swapping ASR encoders for codec-based ones is a sufficient solution for improving non-lexical audio understanding. The paper identifies that acoustic underuse in audio-LLMs is primarily a readout alignment failure rather than an encoder-side information loss, demonstrating that task-relevant acoustic structure remains recoverable in the LM's final hidden states even when the model fails to verbalize it correctly. Through a combination of encoder replacement experiments, layer-wise probing, and targeted LM head interventions, the study provides a detailed localization of this bottleneck, offering actionable insights for improving the acoustic grounding of language models.
Can we recover the 3D poses of multiple people using only sound? This paper presents the first attempt to estimate multi-person 3D poses solely from acoustic signals. Estimating the poses of multiple individuals using acoustic signals is inherently challenging due to the superposition of motion-dependent signal variations. Unlike single-person scenarios, the presence of multiple subjects leads to overlapping acoustic signatures, making it difficult to attribute specific signal changes to an individual's pose. Furthermore, the complexity is compounded by inter-person reflections, which introduce intricate propagation delays that obscure the temporal motion-acoustic relationship. To address these issues, we propose SoundMHPE (Sound-based Multi-person Human Pose Estimator), a novel encoder-decoder framework consisting of two key components. First, the Acoustic Multi-scale Encoder captures diverse temporal and fine-grained frequency features to isolate subtle acoustic signatures from complex, overlapping signals. Second, the Temporal Pose Decoder employs an attention mechanism to disentangle multi-person information across successive frames. By jointly accounting for temporal dynamics and inter-person dependencies, this component precisely reconstructs frame-wise individual poses. To validate our approach, we constructed the 6-hour Acoustic Multi-person Pose (AMP) dataset consisting of 432K synchronized frames of multi-person pose and acoustic data, and demonstrated that our SoundMHPE outperforms baseline models. Project page: https://oumi03.github.io/sound-mhpe/
Primary: Keio University
All Institutions: Keio University, Tokyo University of Science, NTT Inc.
The paper presents the first comprehensive framework for multi-person 3D pose estimation using active acoustic sensing, introducing a novel dataset and a DETR-inspired temporal decoder that effectively disentangles overlapping acoustic signatures. The technical contributions are significant, particularly the multi-scale encoder and the decoupled self-attention mechanisms, which address the unique challenges of multi-person acoustic superposition. The rigorous experimental validation, including cross-modal generalization to WiFi, strengthens the paper's impact, although the reliance on controlled environments remains a key limitation for real-world deployment.
The paper proposes SoundMHPE, a novel encoder-decoder framework for multi-person 3D pose estimation from acoustic signals. The methodology is well-structured, addressing the core challenge of signal superposition in multi-person scenarios. The Acoustic Multi-scale Encoder (AME) effectively utilizes multi-resolution STFT to capture both temporal dynamics and fine-grained frequency features, which is a sound technical choice for handling overlapping acoustic signatures. The Temporal Pose Decoder (TPD) introduces a DETR-like query mechanism adapted for temporal sequences, where multiple queries per person allow for frame-specific attention. The decoupling of self-attention into Motion Self-Attention (intra-person) and Interaction Self-Attention (inter-person) is a thoughtful architectural decision that aligns with the physical nature of the problem. The use of Hungarian matching for loss calculation is standard but appropriate for this detection-like task.
The authors constructed a new 6-hour dataset (AMP) with 432K frames, which is a significant contribution given the lack of existing benchmarks for this specific task. The experimental setup includes cross-subject evaluation with 3-fold cross-validation, which is rigorous. The baselines are adapted from single-person acoustic and multi-person WiFi models, which is reasonable given the novelty of the task. The ablation studies are comprehensive, covering the encoder, decoder, attention mechanisms, and window sizes. The cross-modal evaluation on WiFi data (PiW) is a strong point, demonstrating the generalizability of the architectural components. However, the performance metrics (MPJPE, PCK) should be compared against state-of-the-art single-person acoustic methods to provide a clearer benchmark of the "multi-person penalty."
The paper provides detailed implementation details, including hyperparameters (learning rate, weight decay, number of blocks, query counts) and dataset statistics. The use of standard hardware (OptiTrack, Zoom H3-VR) and software libraries (implied by the architecture) suggests high reproducibility. The project page is provided, which likely contains code and dataset access. The clear definition of the dataset collection protocol (15 subjects, specific poses, randomization) enhances reproducibility.
The primary limitation is the reliance on a controlled indoor environment with specific acoustic properties (soundproof room mentioned in Table 1). Generalization to noisy, reverberant, or outdoor environments is not fully explored, though a brief test with partitions is shown. The dataset size, while large for this niche, is still limited compared to RGB datasets, which may constrain the model's robustness to diverse body types and motions. The method assumes a fixed number of maximum people (M=15), which may not scale efficiently to very crowded scenes.
This work opens a new avenue for non-invasive, privacy-preserving pose estimation in scenarios where RGB cameras are unsuitable (darkness, privacy concerns) and wireless signals are obstructed. Applications in healthcare monitoring, sports analysis, and disaster relief are promising. The cross-modal applicability to WiFi suggests that the architectural insights could benefit other signal-based sensing tasks. The construction of the AMP dataset will likely serve as a foundational benchmark for future research in acoustic pose estimation. The paper presents the first comprehensive framework for multi-person 3D pose estimation using active acoustic sensing, introducing a novel dataset and a DETR-inspired temporal decoder that effectively disentangles overlapping acoustic signatures. The technical contributions are significant, particularly the multi-scale encoder and the decoupled self-attention mechanisms, which address the unique challenges of multi-person acoustic superposition. The rigorous experimental validation, including cross-modal generalization to WiFi, strengthens the paper's impact, although the reliance on controlled environments remains a key limitation for real-world deployment.
Spatial audio editing modifies an existing soundfield according to a user's instruction while preserving the rest of the scene. Unlike conventional audio editing, it must reason jointly about audio events, spatial information, dynamic changes, and environmental information in first-order Ambisonic (FOA) waveforms. Existing language-guided editors mainly target conventional audio or rely on sequential operations, and therefore do not directly support one-stage editing for complex 3D spatial instructions. We present SwanWeave, the first one-stage multi-task framework for instruction-guided 3D FOA spatial audio editing. We build paired FOA supervision from open-source speech and sound-effect corpora using controllable room simulation, covering more than ten single-operation and compound tasks across the four editing axes. To handle this heterogeneous edit space, SwanWeave uses Spatial Edit Mixture-of-Experts (SE-MoE) with dual-level routing, selecting task-aware expert combinations for compound instructions and frame-level routed/null experts for local edit decisions. We further introduce Spatial Preference Optimization (SPO), a Direct Preference Optimization (DPO)-based alignment objective with edit-specific negative targets, and adopt staged training to improve natural-language grounding. Experiments show that SwanWeave achieves better editing quality than existing general audio editors and spatial audio baselines across all tasks. Spatial audio editing demos can be found at https://swanaigc.github.io/#swanweave, code can be found at: https://github.com/MM-Speech/SwanWeave.
Primary: Zhejiang University
All Institutions: Zhejiang University, ByteDance
The paper introduces a novel one-stage multi-task framework for 3D spatial audio editing that effectively addresses the challenges of preserving unedited regions while executing complex compound instructions. By combining a dual-level Mixture-of-Experts architecture with preference optimization tailored for spatial edits, the method achieves state-of-the-art performance in both semantic and spatial fidelity, marking a significant step forward in controllable spatial audio generation.
The paper proposes SwanWeave, a one-stage framework for instruction-guided 3D First-Order Ambisonic (FOA) spatial audio editing. The methodology is robust, combining a flow-matching latent editor with a novel Spatial Edit Mixture-of-Experts (SE-MoE) module. The SE-MoE design is particularly interesting, utilizing dual-level routing: task-level experts selected based on the instruction to handle compound edits, and frame-level routed/null experts to handle local temporal changes and preservation of unedited regions. The use of a "null expert" that outputs zero tensors for unchanged regions is a clever architectural choice to prevent drift in preserved audio. The training strategy is well-structured, employing a staged curriculum (text-to-FOA pretraining, single-operation editing, compound editing) followed by Spatial Preference Optimization (SPO). SPO adapts DPO to the flow-matching context by using edit-specific negative targets (e.g., wrong event, no operation), which effectively addresses common failure modes like source copying or incorrect spatial placement.
The experimental evaluation is comprehensive. The authors construct a large-scale synthetic dataset (approx. 375K triplets) using PyRoomAcoustics for controllable FOA rendering, covering 10+ single and compound tasks. They compare against strong baselines including ZETA, AudioEditor, SDEdit, and SmartDJ. The metrics are diverse, covering semantic quality (FD, FAD, KL, LSD), instruction alignment (CLAP), and spatial fidelity (GCC, CRW, FSAD). The results show consistent improvements across all metrics, with particularly strong gains in spatial fidelity (GCC/CRW) compared to stereo-based baselines, validating the benefit of native FOA modeling. The ablation studies are thorough, confirming the necessity of pretraining, the SE-MoE module, and the SPO stage. The comparison between one-stage and multi-stage editing effectively demonstrates the advantage of the proposed one-stage approach in avoiding error accumulation.
The paper provides high reproducibility. Code and a demo page are available. The data construction process is detailed, including the specific room sizes, direction definitions, and the prompt used for instruction generation. The model architecture details, including the routing regularization losses and CFG scale selection, are provided in the appendix. The use of open-source corpora and standard simulation tools (PyRoomAcoustics) further aids reproducibility.
The primary limitation is the reliance on synthetic data generated via room simulation. While controllable, this may not capture the acoustic complexity of real-world recordings (e.g., irregular room responses, microphone artifacts, background noise). The scenes are also limited in duration (~10 seconds) and complexity (2-4 events), which may not generalize to long-form or dense multi-speaker scenes. The human evaluation was conducted with a small group of volunteer graduate students, which may introduce bias, though the objective metrics strongly support the findings.
This work has significant potential for immersive media production, VR/AR, and interactive storytelling. By enabling natural language control over 3D spatial audio, it lowers the barrier to creating complex spatial soundscapes. The one-stage approach is efficient and practical for real-time or near-real-time applications. The framework could be extended to other spatial audio formats (e.g., binaural, higher-order Ambisonics) and integrated into broader audio-visual generation pipelines. The paper introduces a novel one-stage multi-task framework for 3D spatial audio editing that effectively addresses the challenges of preserving unedited regions while executing complex compound instructions. By combining a dual-level Mixture-of-Experts architecture with preference optimization tailored for spatial edits, the method achieves state-of-the-art performance in both semantic and spatial fidelity, marking a significant step forward in controllable spatial audio generation.