Pretrained image-generation models have been successfully repurposed for various downstream visual tasks. Whether their learned representations and generative priors can transfer beyond vision to other modalities remains largely unresolved, particularly for speech synthesis, which requires linguistic fidelity and preservation of speaker identity from a short reference. We present ImageTTS, which repurposes FLUX.2 for speech synthesis by encoding pseudo-RGB log-mel spectrograms with its frozen image VAE and adapting only its pretrained Diffusion Transformer (DiT) using conditional latent flow matching. This formulation disentangles two forms of vision-to-speech transfer: representation transfer through the image VAE and generative-prior transfer through DiT initialization. The frozen image VAE largely preserves intelligibility and speaker identity with 0.99 STOI and 0.95 speaker similarity, while image-pretrained DiT initialization accelerates convergence and improves linguistic accuracy, reducing WER from 4.2\% to 2.7\% at 100k matched updates relative to random initialization. The same framework supports both zero-shot and instruction-guided TTS and, when scaled to 100K hours of speech data, achieves 1.12\% WER on Seed-TTS test-en and a Style-ACC of 0.66 on CapSpeech, outperforming speech-native baselines trained on comparable data across both tasks. These results suggest that pretrained image models can transfer beyond architectural reuse, supplying useful latent representations and generative priors for intelligible and controllable speech synthesis.
Primary: Microsoft Corporation
All Institutions: Wuhan University, Microsoft Corporation, The Chinese University of Hong Kong
The experimental evaluation is comprehensive and addresses the three core research questions effectively. 1. **Speech reconstruction with frozen image VAE:** This experiment rigorously evaluates the representation transfer. By varying the frequency resolution ($F$) and comparing with a spectrogram-oriented VAE (Riffusion), the authors demonstrate that FLUX.2's VAE can largely preserve speech intelligibility (STOI 0.99) and speaker identity (SIM 0.95) at $F=512$, approaching vocoder-only baselines. This provides strong evidence for the utility of image VAEs for speech representation. 2. **Impact of image-pretrained DiT initialization:** This experiment directly assesses generative-prior transfer. Comparing FLUX.2 initialization against random initialization under matched training conditions shows significant improvements in convergence speed and linguistic accuracy (WER reduced from 4.2% to 2.7%). This is a clear demonstration that image-learned priors are beneficial for speech generation. 3. **Scaling to zero-shot and instruction-guided TTS:** This section evaluates the full system's competitiveness at scale (100K hours of speech data). ImageTTS-base achieves competitive zero-shot TTS performance (1.1
The methodology proposes ImageTTS, a novel framework that repurposes a pretrained image diffusion transformer (FLUX.2) for speech synthesis. A key strength is the systematic disentanglement of two forms of vision-to-speech transfer: representation transfer via a frozen image VAE and generative-prior transfer via DiT initialization. The approach involves converting log-mel spectrograms into pseudo-RGB images, encoding them with the frozen FLUX.2 image VAE, and then fine-tuning only the pretrained DiT backbone using conditional latent flow matching. This pseudo-RGB conversion and subsequent VAE encoding is a creative way to bridge the modalities, though it relies on the image VAE's ability to meaningfully compress and reconstruct spectrograms. The formulation of zero-shot TTS as reference-conditioned latent infilling and instruction-guided TTS as full latent generation conditioned on text and description is well-defined. The use of classifier-free guidance for both text and speech conditions (and style for instruction-guided) is standard practice in generative models and appropriately applied here. The architecture leverages FLUX.2's double-stream and single-stream Transformer blocks for multimodal interaction, which is a sensible adaptation for handling both text and speech latents. The overall methodology is sound, innovative, and systematically designed to answer the research questions posed.
The experimental evaluation is comprehensive and addresses the three core research questions effectively. 1. **Speech reconstruction with frozen image VAE:** This experiment rigorously evaluates the representation transfer. By varying the frequency resolution ($F$) and comparing with a spectrogram-oriented VAE (Riffusion), the authors demonstrate that FLUX.2's VAE can largely preserve speech intelligibility (STOI 0.99) and speaker identity (SIM 0.95) at $F=512$, approaching vocoder-only baselines. This provides strong evidence for the utility of image VAEs for speech representation. 2. **Impact of image-pretrained DiT initialization:** This experiment directly assesses generative-prior transfer. Comparing FLUX.2 initialization against random initialization under matched training conditions shows significant improvements in convergence speed and linguistic accuracy (WER reduced from 4.2% to 2.7%). This is a clear demonstration that image-learned priors are beneficial for speech generation. 3. **Scaling to zero-shot and instruction-guided TTS:** This section evaluates the full system's competitiveness at scale (100K hours of speech data). ImageTTS-base achieves competitive zero-shot TTS performance (1.1
We present EmbeddingGemma 2, an open multimodal embedding model which natively represents text, images, audio, video, and combinations thereof in a unified space. Our approach introduces improvements such as simplified per-layer embeddings and a multi-stage curriculum with modality and objective progression which ensure strong quality across unimodal, cross-modal, and mixed-modal evaluations. Notably, in comprehensive embedding benchmarks across all modalities, EmbeddingGemma 2 establishes a new Pareto frontier, outperforming both specialist models and models two to twelve times larger. Furthermore, our design offers flexibility for resource-constrained environments, supporting selective modality encoder loading, adjustable vision token budgets, truncatable embedding dimensions, and quantized weights. We present ablations analyzing key decisions and release EmbeddingGemma 2 to promote further research.
Primary: Google
All Institutions: Google
EmbeddingGemma 2 introduces a unified, efficient multimodal embedding model that achieves state-of-the-art performance across text, vision, and audio benchmarks while significantly reducing computational costs through architectural simplifications and modular design. The paper demonstrates that a 744M-parameter model can outperform much larger specialists by leveraging a multi-stage curriculum learning strategy that transitions from distillation to contrastive learning, effectively transferring knowledge from a larger teacher model while optimizing for the student's specific architecture and deployment constraints.
The paper proposes EmbeddingGemma 2, a 744M-parameter multimodal embedding model. The core architectural contribution is the simplification of Per-Layer Embeddings (PLE) by removing the dedicated lookup table and instead projecting the main input embeddings, which reduces memory overhead and improves multimodal performance. The training methodology is rigorous, employing a three-stage curriculum: text pre-finetuning (primarily distillation), multimodal pre-finetuning (distillation from a larger teacher model, Gemini Embedding 2), and multimodal finetuning (contrastive learning with spread-out regularization and Matryoshka Representation Learning). This progression effectively transfers knowledge from a large teacher to a smaller student while allowing the student to optimize its own representation space. The inclusion of quantization-aware training (QAT) and modular encoder loading demonstrates a strong focus on practical deployment constraints.
The evaluation is comprehensive, covering seven major benchmark suites (MTEB, MMEB-v2, MIEB-lite, MSEB, MAEB) across text, code, image, video, document, speech, and sound. The model establishes a new Pareto frontier, outperforming specialist models and general-purpose models 2-12x larger. Ablation studies are particularly strong, validating the removal of the PLE lookup table, the necessity of co-training the vision encoder, and the impact of vision token budgets. The comparison against ASR-based pipelines for audio tasks convincingly demonstrates the value of native audio understanding.
The paper provides high reproducibility for the model usage, as the checkpoints are released. It details the architecture, loss functions, and training recipe in sufficient depth. However, as is common with large-scale industrial models, the specific data mixtures and hyperparameters for the 2.9T token training run are not fully disclosed, limiting the ability to replicate the training process from scratch.
The model is still significantly larger than pure text embedding models, posing challenges for extremely low-resource edge devices despite the modular design. The paper notes that video performance degrades at higher token budgets due to attention dilution, suggesting room for improvement in long-context handling. Additionally, while it outperforms many specialists, it does not always beat the absolute best specialists in every single niche audio task (e.g., some paralinguistic clustering).
This work significantly lowers the barrier to entry for high-quality multimodal retrieval. By providing a unified, efficient, and open model, it enables developers to build complex multimodal applications (RAG, search, recommendation) without assembling complex pipelines of ASR, captioners, and separate embedding models. The focus on on-device deployment (191MB footprint for text-only) is particularly impactful for privacy-preserving applications. EmbeddingGemma 2 introduces a unified, efficient multimodal embedding model that achieves state-of-the-art performance across text, vision, and audio benchmarks while significantly reducing computational costs through architectural simplifications and modular design. The paper demonstrates that a 744M-parameter model can outperform much larger specialists by leveraging a multi-stage curriculum learning strategy that transitions from distillation to contrastive learning, effectively transferring knowledge from a larger teacher model while optimizing for the student's specific architecture and deployment constraints.
Localized AI manipulations in long-form videos often affect brief events within otherwise authentic content, making video-level classification insufficient for forensic analysis. Existing audio-visual deepfake benchmarks largely focus on short talking-head clips, while long-form temporal forensics remains predominantly visual. We introduce AV-FABLE, a dual-timeline benchmark for sparse manipulations in open-scene long-form audio-visual videos. AV-FABLE contains 16,650 pristine--manipulated twin pairs (998 hours) created with 27 editing configurations, with separate audio and visual timelines that enable modality-specific temporal localization and expose errors hidden by joint-timeline evaluation. On AV-FABLE, zero-shot prompting of audio-visual large language models (AV-LLMs) yields near-chance detection and poor temporal localization. In contrast, supervised readouts of their frozen perception features recover strong forensic signals, suggesting manipulation evidence is encoded in these representations but not effectively exposed through language outputs. Motivated by this observation, we propose NFSR(Native Forensics via Separated Readouts), which reads frozen AV-LLM features with separate audio and visual temporal decoders. NFSR achieves the best audio-visual localization performance in-domain and across all three out-of-distribution settings, including transfer to held-out videos from an external long-video forensic benchmark. AV-FABLE, code, and model weights will be publicly released.
Primary: Tsinghua University
All Institutions: Tsinghua University, Peking University
The paper presents a high-quality, large-scale benchmark (AV-FABLE) and a novel method (NFSR) for audio-visual manipulation forensics in long-form videos, demonstrating that frozen AV-LLM features are more effective for forensic localization than zero-shot language outputs.
The paper introduces AV-FABLE, a substantial benchmark for sparse audio-visual manipulations in long-form videos, featuring a dual-timeline evaluation protocol that distinguishes audio and visual tracks. This is a significant methodological advancement over joint-timeline evaluations which can mask modality-specific errors. The proposed model, NFSR, utilizes frozen features from AV-LLMs (Qwen3-Omni) with separate temporal decoders for each modality. The key methodological insight is the observation that while zero-shot prompting of AV-LLMs fails at forensic detection, their frozen perceptual features contain strong forensic signals accessible via supervised readouts. The use of host-conditioned feature modulation (FiLM) is a clever architectural choice to contextualize local anomalies within the broader video content.
The experimental setup is rigorous, including in-domain and three out-of-distribution (OOD) settings (source transfer, held-out generators, held-out routes). The dataset scale (16,650 pairs, 998 hours) is impressive for a forensic benchmark. Results show NFSR outperforming strong baselines like UMMAFormer and ActionFormer in AV-mAP across all settings. The ablation studies effectively demonstrate the contribution of intermediate-layer features, host conditioning, and modality separation. The comparison with zero-shot LLMs highlights a critical gap in current multimodal LLM capabilities for fine-grained forensic tasks.
The authors commit to releasing the dataset, code, and model weights. The paper provides detailed descriptions of the data construction pipeline (SEE-Forge), including the use of Gemini 3.7 Flash for planning and human review protocols for quality control. The evaluation metrics (AV-mAP, Event-IoU) and protocols are clearly defined. The reliance on specific, potentially proprietary, LLMs for data generation (Gemini) might pose minor reproducibility challenges for the dataset creation itself, but the benchmark and model are designed to be open.
The benchmark relies on synthetic manipulations generated by specific AI models, which may not fully capture the diversity of real-world human-edited forgeries or future, unseen generation techniques. The "explanation" task is noted as challenging and not fully solved by current baselines. The paper acknowledges that visual coverage under distribution shift remains an open problem, with NFSR showing lower V-IoU compared to some visual-only baselines in certain OOD settings.
This work addresses a critical gap in AI safety and media integrity by providing tools to localize and attribute manipulations in long-form content, which is increasingly relevant for misinformation detection. The finding that AV-LLMs encode forensic evidence in their features but fail to expose it via language outputs is a valuable insight for the broader multimodal AI community, suggesting that current alignment or decoding strategies in LLMs may be suboptimal for fine-grained perception tasks. The paper presents a high-quality, large-scale benchmark (AV-FABLE) and a novel method (NFSR) for audio-visual manipulation forensics in long-form videos, demonstrating that frozen AV-LLM features are more effective for forensic localization than zero-shot language outputs.
We present EmbeddingGemma 2, an open multimodal embedding model which natively represents text, images, audio, video, and combinations thereof in a unified space. Our approach introduces improvements such as simplified per-layer embeddings and a multi-stage curriculum with modality and objective progression which ensure strong quality across unimodal, cross-modal, and mixed-modal evaluations. Notably, in comprehensive embedding benchmarks across all modalities, EmbeddingGemma 2 establishes a new Pareto frontier, outperforming both specialist models and models two to twelve times larger. Furthermore, our design offers flexibility for resource-constrained environments, supporting selective modality encoder loading, adjustable vision token budgets, truncatable embedding dimensions, and quantized weights. We present ablations analyzing key decisions and release EmbeddingGemma 2 to promote further research.
Primary: Google
All Institutions: Google
EmbeddingGemma 2 introduces a unified, efficient multimodal embedding model that achieves state-of-the-art performance across text, vision, and audio benchmarks while significantly reducing computational costs through architectural simplifications and modular design. The paper demonstrates that a 744M-parameter model can outperform much larger specialists by leveraging a multi-stage curriculum learning strategy that transitions from distillation to contrastive learning, effectively transferring knowledge from a larger teacher model while optimizing for the student's specific architecture and deployment constraints.
The paper proposes EmbeddingGemma 2, a 744M-parameter multimodal embedding model. The core architectural contribution is the simplification of Per-Layer Embeddings (PLE) by removing the dedicated lookup table and instead projecting the main input embeddings, which reduces memory overhead and improves multimodal performance. The training methodology is rigorous, employing a three-stage curriculum: text pre-finetuning (primarily distillation), multimodal pre-finetuning (distillation from a larger teacher model, Gemini Embedding 2), and multimodal finetuning (contrastive learning with spread-out regularization and Matryoshka Representation Learning). This progression effectively transfers knowledge from a large teacher to a smaller student while allowing the student to optimize its own representation space. The inclusion of quantization-aware training (QAT) and modular encoder loading demonstrates a strong focus on practical deployment constraints.
The evaluation is comprehensive, covering seven major benchmark suites (MTEB, MMEB-v2, MIEB-lite, MSEB, MAEB) across text, code, image, video, document, speech, and sound. The model establishes a new Pareto frontier, outperforming specialist models and general-purpose models 2-12x larger. Ablation studies are particularly strong, validating the removal of the PLE lookup table, the necessity of co-training the vision encoder, and the impact of vision token budgets. The comparison against ASR-based pipelines for audio tasks convincingly demonstrates the value of native audio understanding.
The paper provides high reproducibility for the model usage, as the checkpoints are released. It details the architecture, loss functions, and training recipe in sufficient depth. However, as is common with large-scale industrial models, the specific data mixtures and hyperparameters for the 2.9T token training run are not fully disclosed, limiting the ability to replicate the training process from scratch.
The model is still significantly larger than pure text embedding models, posing challenges for extremely low-resource edge devices despite the modular design. The paper notes that video performance degrades at higher token budgets due to attention dilution, suggesting room for improvement in long-context handling. Additionally, while it outperforms many specialists, it does not always beat the absolute best specialists in every single niche audio task (e.g., some paralinguistic clustering).
This work significantly lowers the barrier to entry for high-quality multimodal retrieval. By providing a unified, efficient, and open model, it enables developers to build complex multimodal applications (RAG, search, recommendation) without assembling complex pipelines of ASR, captioners, and separate embedding models. The focus on on-device deployment (191MB footprint for text-only) is particularly impactful for privacy-preserving applications. EmbeddingGemma 2 introduces a unified, efficient multimodal embedding model that achieves state-of-the-art performance across text, vision, and audio benchmarks while significantly reducing computational costs through architectural simplifications and modular design. The paper demonstrates that a 744M-parameter model can outperform much larger specialists by leveraging a multi-stage curriculum learning strategy that transitions from distillation to contrastive learning, effectively transferring knowledge from a larger teacher model while optimizing for the student's specific architecture and deployment constraints.
Real-world audio applications require precise, queryable representations grounded in physical time. Yet modern Large Audio-Language Models (LALMs) increasingly favor conversational generation, producing free-form descriptions that conflate acoustic perception with higher-level reasoning and lack precise temporal grounding. We introduce TAC2, a foundation model that casts timestamped audio captioning as composable functions to construct granular, speaker-attributed acoustic timelines directly from in-the-wild audio. To avoid prohibitively expensive manual annotation, we programmatically fuse complementary predictions from specialized weak teachers through time-registered IoU matching. We model the resulting multi-granularity schema of timestamped events, speaker turns, and fine-grained speech attributes using an Instruction-Input-Output (IIO) completion format, in which each task is a function of the audio, an instruction, and an optional input. A single call decodes the full description autonomously, while composing calls lets earlier outputs or external priors, such as a reference transcript, condition later ones. We further extend TAC2 with expressive speech attributes to obtain TAC2-S, using parameter-efficient specialists to elicit latent paralinguistic capabilities from the pre-trained backbone and distill them into a unified model. We validate the resulting representation in two complementary settings. Paired with frozen text-only LLMs, it yields strong spoken audio understanding across diverse reasoning backends, demonstrating the utility of decoupling fine-grained acoustic perception from downstream reasoning. Used as paired supervision for text-to-audio generation, it supports semantically and temporally grounded reconstruction, together with explicit zero-shot control over event timing, speaker attributes, and word-level emphasis.
Primary: Adobe Research
All Institutions: University of Surrey, Adobe Research
TAC2 introduces a scalable framework for timestamped audio captioning by fusing weak teacher predictions and eliciting latent paralinguistic capabilities, resulting in a composable, temporally grounded representation that enhances both audio understanding and controllable generation.
The paper proposes a two-stage framework, TAC2, for timestamped audio captioning. Stage 1 addresses the data scarcity problem for fine-grained temporal grounding by programmatically fusing predictions from weak teachers (sound event detectors, diarizers, captioners) using a time-registered IoU matching algorithm. This creates a structured, speaker-attributed timeline without expensive manual annotation. The model is trained using an Instruction-Input-Output (IIO) completion format, which allows for composable functions where outputs of one task can condition subsequent tasks. Stage 2 extends this by eliciting latent paralinguistic capabilities (stress, emotion, etc.) from a frozen backbone using lightweight LoRA specialists trained on small datasets, then distilling these into a unified student model. The methodology is sound, leveraging the strengths of weak supervision and parameter-efficient fine-tuning to build a complex, structured representation.
The authors validate the model in two settings: audio understanding (cascading with frozen LLMs) and audio generation (paired supervision). In understanding, TAC2 achieves strong results on MMSU and MSU-Bench, outperforming baselines like TAC and Qwen3Omni-Captioner, particularly in perception tasks. The generation experiments demonstrate zero-shot control over event timing and speaker attributes. The evaluation is comprehensive, covering both the utility of the structured representation for reasoning and its application in generative tasks.
The paper provides detailed descriptions of the fusion algorithm, the IIO format, and the specialist training procedure. However, specific hyperparameters, dataset sizes for the weak supervision corpus, and the exact implementation details of the IoU matching thresholds are not fully specified in the provided text, which may hinder exact reproduction. The use of standard tools like LoRA and common benchmarks aids reproducibility.
The reliance on weak teachers means the quality of the fused timeline is bounded by the accuracy of the individual detectors and diarizers. Errors in the weak supervision stage can propagate to the final model. Additionally, the "latent capability elicitation" approach, while efficient, may not capture all nuances of paralinguistic features if they are not strongly encoded in the backbone. The paper is truncated, so the full extent of the generation results and ablations is not visible.
This work has significant potential for applications requiring precise audio understanding, such as automated video captioning, audio search, and accessible media. By decoupling perception from reasoning, it enables more robust and controllable audio pipelines. The composable nature of the IIO format makes it adaptable to various downstream tasks. TAC2 introduces a scalable framework for timestamped audio captioning by fusing weak teacher predictions and eliciting latent paralinguistic capabilities, resulting in a composable, temporally grounded representation that enhances both audio understanding and controllable generation.
Reinforcement learning has made Audio Large Language Models (Audio LLMs) better reasoners, but the reward that drives it still comes from outside the model: a final-answer check, a hand-crafted process score, or an external LLM judge that stays fixed while the policy improves. We show that an Audio LLM can supply this reward itself, and can do so recursively: each improved model becomes the critic that trains its successor, so the training signal improves along with the model. We introduce AudioRSI (Recursively Self-Improving Audio Reasoners), a reinforcement learning framework in which a single model plays two roles: as an actor it generates reasoning paths, and as a critic it compares them pairwise, turning its own judgments into a process-level reward. The critic is periodically replaced by the current policy, which closes the recursive loop. Because the critic hears the same recording as the actor, it can reject reasoning that cites acoustic evidence the audio does not contain, rather than reward reasoning that merely sounds plausible. The recursion is what sustains progress: under a frozen external judge, out-of-distribution accuracy plateaus during training, while a critic refreshed from the improving policy keeps it rising. Across four out-of-distribution benchmarks, all disjoint from the AVQA training distribution, AudioRSI outperforms strong baselines including Audio Flamingo 3 and Gemini 2.5 Pro on MMAU test-mini, ranks first on MMSU, achieves a clear lead on the challenging Multi-Audio split of MMAU-Pro, and leads the 7B open-source tier on MMAR among methods without external reward models. With no reward specifying how to reason, AudioRSI also develops structured reasoning behaviors (logical reasoning, perception-aware analysis, and reduced hallucination), corroborated by an external LLM-as-Judge evaluation and a keyword-based behavior analysis.
Primary: Amazon
All Institutions: University of Illinois Urbana-Champaign (UIUC), Amazon
AudioRSI introduces a recursively self-improving reinforcement learning framework for audio LLMs where the model serves as its own evolving critic. By periodically synchronizing the critic with the actor, the method ensures that the reward signal for reasoning quality improves alongside the policy, leading to superior out-of-distribution performance and emergent structured reasoning behaviors compared to static external judges or outcome-only rewards.
The paper proposes AudioRSI, a framework where an Audio LLM acts as both actor and critic within a GRPO (Group Relative Policy Optimization) loop. The core innovation is the "recursive" aspect: the critic is periodically synchronized with the actor's weights, allowing the reward signal to improve as the model improves. The methodology is sound, leveraging the unique advantage of audio modality where the critic can verify acoustic claims against the raw waveform, unlike text-only self-rewarding models. The use of pairwise comparison for process-level rewards is a solid adaptation of DPO-style thinking to RLVR (Reinforcement Learning with Verifiable Rewards).
The experiments are conducted on Qwen2.5-Omni-7B using AVQA for training and four OOD benchmarks (MMAU, MMSU, MMAU-Pro, MMAR) for evaluation. The results show consistent improvements over baseline RL methods (Ke-Omni-R) and strong performance against proprietary models like Gemini 2.5 Pro on specific splits. The ablation on synchronization period $T$ is crucial and well-executed, demonstrating the trade-off between stability and reward freshness. The claim that the model develops structured reasoning behaviors is supported by LLM-as-Judge and keyword analysis, though these are somewhat proxy metrics.
The paper provides detailed algorithmic descriptions and hyperparameters in the appendix. The reliance on specific base models (Qwen2.5-Omni) and datasets (AVQA) makes reproduction feasible for groups with similar compute resources. The code is not explicitly linked in the provided text, which is a minor gap for immediate reproducibility.
The approach is computationally expensive due to the dual role of the model (generating and critiquing). The "guarded handoff" mechanism, while practical, adds complexity. The evaluation relies heavily on LLM-as-Judge for reasoning quality, which may introduce bias if the judge model shares biases with the actor. The gains are modest in some domains (e.g., Music), suggesting the method may be more effective for domains with clearer acoustic evidence.
This work pushes the boundary of self-improving AI systems in multimodal domains. It demonstrates that models can provide meaningful process-level supervision for their own reasoning, reducing reliance on external, static reward models. This has implications for scaling audio reasoning capabilities without massive human annotation of reasoning steps. AudioRSI introduces a recursively self-improving reinforcement learning framework for audio LLMs where the model serves as its own evolving critic. By periodically synchronizing the critic with the actor, the method ensures that the reward signal for reasoning quality improves alongside the policy, leading to superior out-of-distribution performance and emergent structured reasoning behaviors compared to static external judges or outcome-only rewards.
Pretrained image-generation models have been successfully repurposed for various downstream visual tasks. Whether their learned representations and generative priors can transfer beyond vision to other modalities remains largely unresolved, particularly for speech synthesis, which requires linguistic fidelity and preservation of speaker identity from a short reference. We present ImageTTS, which repurposes FLUX.2 for speech synthesis by encoding pseudo-RGB log-mel spectrograms with its frozen image VAE and adapting only its pretrained Diffusion Transformer (DiT) using conditional latent flow matching. This formulation disentangles two forms of vision-to-speech transfer: representation transfer through the image VAE and generative-prior transfer through DiT initialization. The frozen image VAE largely preserves intelligibility and speaker identity with 0.99 STOI and 0.95 speaker similarity, while image-pretrained DiT initialization accelerates convergence and improves linguistic accuracy, reducing WER from 4.2\% to 2.7\% at 100k matched updates relative to random initialization. The same framework supports both zero-shot and instruction-guided TTS and, when scaled to 100K hours of speech data, achieves 1.12\% WER on Seed-TTS test-en and a Style-ACC of 0.66 on CapSpeech, outperforming speech-native baselines trained on comparable data across both tasks. These results suggest that pretrained image models can transfer beyond architectural reuse, supplying useful latent representations and generative priors for intelligible and controllable speech synthesis.
Primary: Microsoft Corporation
All Institutions: Wuhan University, Microsoft Corporation, The Chinese University of Hong Kong
The experimental evaluation is comprehensive and addresses the three core research questions effectively. 1. **Speech reconstruction with frozen image VAE:** This experiment rigorously evaluates the representation transfer. By varying the frequency resolution ($F$) and comparing with a spectrogram-oriented VAE (Riffusion), the authors demonstrate that FLUX.2's VAE can largely preserve speech intelligibility (STOI 0.99) and speaker identity (SIM 0.95) at $F=512$, approaching vocoder-only baselines. This provides strong evidence for the utility of image VAEs for speech representation. 2. **Impact of image-pretrained DiT initialization:** This experiment directly assesses generative-prior transfer. Comparing FLUX.2 initialization against random initialization under matched training conditions shows significant improvements in convergence speed and linguistic accuracy (WER reduced from 4.2% to 2.7%). This is a clear demonstration that image-learned priors are beneficial for speech generation. 3. **Scaling to zero-shot and instruction-guided TTS:** This section evaluates the full system's competitiveness at scale (100K hours of speech data). ImageTTS-base achieves competitive zero-shot TTS performance (1.1
The methodology proposes ImageTTS, a novel framework that repurposes a pretrained image diffusion transformer (FLUX.2) for speech synthesis. A key strength is the systematic disentanglement of two forms of vision-to-speech transfer: representation transfer via a frozen image VAE and generative-prior transfer via DiT initialization. The approach involves converting log-mel spectrograms into pseudo-RGB images, encoding them with the frozen FLUX.2 image VAE, and then fine-tuning only the pretrained DiT backbone using conditional latent flow matching. This pseudo-RGB conversion and subsequent VAE encoding is a creative way to bridge the modalities, though it relies on the image VAE's ability to meaningfully compress and reconstruct spectrograms. The formulation of zero-shot TTS as reference-conditioned latent infilling and instruction-guided TTS as full latent generation conditioned on text and description is well-defined. The use of classifier-free guidance for both text and speech conditions (and style for instruction-guided) is standard practice in generative models and appropriately applied here. The architecture leverages FLUX.2's double-stream and single-stream Transformer blocks for multimodal interaction, which is a sensible adaptation for handling both text and speech latents. The overall methodology is sound, innovative, and systematically designed to answer the research questions posed.
The experimental evaluation is comprehensive and addresses the three core research questions effectively. 1. **Speech reconstruction with frozen image VAE:** This experiment rigorously evaluates the representation transfer. By varying the frequency resolution ($F$) and comparing with a spectrogram-oriented VAE (Riffusion), the authors demonstrate that FLUX.2's VAE can largely preserve speech intelligibility (STOI 0.99) and speaker identity (SIM 0.95) at $F=512$, approaching vocoder-only baselines. This provides strong evidence for the utility of image VAEs for speech representation. 2. **Impact of image-pretrained DiT initialization:** This experiment directly assesses generative-prior transfer. Comparing FLUX.2 initialization against random initialization under matched training conditions shows significant improvements in convergence speed and linguistic accuracy (WER reduced from 4.2% to 2.7%). This is a clear demonstration that image-learned priors are beneficial for speech generation. 3. **Scaling to zero-shot and instruction-guided TTS:** This section evaluates the full system's competitiveness at scale (100K hours of speech data). ImageTTS-base achieves competitive zero-shot TTS performance (1.1
Spatial audio authoring coordinates source decisions with soundfield synthesis. For editing, plans specify what to change and retain, while reference audio supplies further acoustic detail. We present AMBIT, a unified framework for native First-Order Ambisonics (FOA) generation and editing with three connected contributions. First, a shared Transfusion backbone couples autoregressive planning with rectified-flow rendering through executable ScenePlans that specify source content, activity, and 3D trajectories. For editing, the plan describes the full target scene, including retained sources; contrastive features ground the requested change, while full reference latents supply residual acoustic detail for preservation. Second, an FOA-aware continuous VAE combines shared-decoder omnidirectional recovery, asymmetric grouped KL, and spatial covariance matching to preserve content and inter-channel structure. Third, joint Transfusion on-policy self-distillation (OPSD) refines planning on student-generated prefixes and rendering at sampling states reached under student plans, with target-conditioned teachers supervising both modes of the shared student. Evaluations across speech, music, and sound show improved generation content and spatial quality, acoustic fidelity during instruction-guided editing, and unedited-source preservation over the compared baselines. Project page: https://wjm-george.github.io/AMBIT/
Primary: HKUST
All Institutions: HKUST, Noiz AI, University of Washington
The paper explicitly states a primary limitation:
The paper introduces AMBIT, a unified framework for First-Order Ambisonics (FOA) generation and editing. The core methodology is built upon three interconnected contributions. First, a shared Transfusion backbone effectively couples autoregressive planning with rectified-flow rendering. This is a technically sound approach to integrate discrete planning (ScenePlans) with continuous generation, allowing for explicit control over source content, activity, and 3D trajectories. The ScenePlans are executable and deterministically compiled into semantic and frame-aligned spatial controls, which is a strong design choice for grounding language instructions in 3D audio. For editing, the use of contrastive features for grounding changes and full reference latents for preserving acoustic detail is well-conceived, addressing a key challenge in instruction-guided editing. Second, the proposed FOA-aware continuous Variational Autoencoder (VAE) is a significant improvement over standard audio VAEs for spatial audio. The combination of shared-decoder omnidirectional (W-channel) recovery, asymmetric grouped KL regularization, and especially Spatial Covariance Matching (SCM) directly addresses the unique requirements of FOA data. SCM, by supervising relative energy, coherence, and phase between channels, is a novel and effective way to preserve inter-channel structure crucial for accurate spatial perception. The detailed formulation of SCM, including stabilization and weighting, demonstrates a deep understanding of FOA signal processing. Third, the joint Transfusion On-Policy Self-Distillation (OPSD) mechanism is a sophisticated method for refining both the discrete planner and the continuous renderer. Adapting OPSD to jointly refine both modes, where student plans drive DiT trajectories and target-conditioned teachers supervise on student-generated prefixes and visited sampling states, is a non-trivial technical contribution. This approach addresses the common challenge of distribution shift between training with ground-truth plans and inference with student-generated plans, ensuring that the student learns to perform well in its own operational context. The detailed loss formulations for planning and rendering distillation, including the use of detached weights and target plan memory/audio reference tokens, are well-justified. The overall architecture is complex but appears coherently integrated, leveraging existing strong components (DiT, CLAP) while introducing novel adaptations for spatial audio.
The experimental evaluation is comprehensive and rigorous, covering generation, editing, and VAE performance across speech, music, and sound domains. The use of a large dataset (1.6M synthetic and recorded FOA examples, 1M editing pairs, 1M VAE clips) is commendable, although the reliance on synthetic data for a significant portion is a common limitation in spatial audio research. The evaluation metrics are extensive and appropriate for spatial audio, including content-based metrics (paired CLAP, FAD, WER/CER, UTMOS), spatial metrics (directional and binaural errors, GCC, CRW, wCS), and preservation metrics for editing (unchanged-source direction/waveform/level errors). The inclusion of listener ratings in the appendix further strengthens the subjective evaluation. The VAE evaluation clearly demonstrates the benefits of the FOA-aware design, with SCM and grouped KL significantly improving reconstruction quality and spatial accuracy over baselines like DAC, EnCodec, and WavTokenizer. The generation and editing quality results show AMBIT outperforming strong baselines like AudioX-Turbo and SwanWeave across multiple metrics, indicating superior content fidelity, acoustic fidelity, and spatial accuracy. The fixed-renderer comparisons effectively isolate the planner's contribution, while the unedited-source preservation metrics highlight a critical aspect of editing. The post-training OPSD experiments are particularly strong, showing that joint OPSD consistently improves all nine reported metrics over continued supervised training, ReFL, GRPO, and individual branch supervision variants. This provides compelling evidence for the effectiveness of the proposed joint refinement strategy. The detailed ablation studies for the VAE and OPSD components add significant credibility to the individual contributions. The use of Qwen3.5-0.8B as a frozen text backbone and CLAP features for grounding is a practical choice that leverages powerful pre-trained models. The experimental setup, including the use of 8 NVIDIA L40S GPUs, indicates a substantial computational effort.
The paper provides a good level of detail for reproducibility. The architecture details (15 shared Transfusion blocks, width, attention heads, VAE base width, channel multipliers, strides, latent channels, hop size) are specified. Training details, including learning rates, optimizers (AdamW), batch sizes, loss weights for OPSD, and specific coefficients for various loss terms (STFT, SI-SDR, SCM, phase supervision, etc.), are provided in the main text and appendices. The flow time sampling distributions for training and inference settings (Euler steps, guidance scales) are also given. The datasets are described in detail, including sources (LibriTTS, AudioCaps, AudioSet, FSD50K, MusicCaps, Sphere360, STARSS23, Spatial LibriSpeech) and how synthetic scenes are constructed (Pyroomacoustics, room distributions, source placement, trajectories). The process for generating pseudo-annotations using Qwen3-Omni is also outlined. The project page URL is provided, which typically hosts code, models, and/or audio samples, further aiding reproducibility. While the full code is not directly embedded, the level of detail suggests a strong commitment to enabling replication.
The paper explicitly states a primary limitation:
Learning audio representations that effectively bridge holistic global soundscapes with fine-grained acoustic moments is a long-standing goal in self-supervised learning. Recording-level contrastive objectives encourage agreement between pooled audio representations, but do not explicitly supervise local time-frequency representations. We introduce AudioFM, a self-supervised framework that combines recording-level contrast with patch-wise masked latent discrimination. Using visible acoustic context, a student predicts representations of masked patches and identifies their corresponding full-view teacher targets among candidates from other recordings. We further incorporate weaker supervision from eligible temporal neighbors to encourage local temporal consistency. Extensive evaluations show that AudioFM achieves state-of-the-art performance across diverse audio domains in linear probing, fine-tuning, and zero-shot transfer, demonstrating superior event detection, robust scene-level understanding, and strong domain generalizability.
Primary: Google Research
All Institutions: Google Research, Google DeepMind, Vanderbilt University, University of Washington
The paper introduces AudioFM, a self-supervised audio foundation model that effectively bridges global and local representation learning through a novel masked latent discrimination objective. By combining recording-level contrast with patch-wise discrimination against cross-recording negatives and incorporating temporal neighborhood consistency, AudioFM achieves state-of-the-art performance across diverse domains, including general audio, bioacoustics, and medical acoustics, demonstrating superior transferability and robustness in both fine-tuning and zero-shot settings.
The paper proposes AudioFM, a self-supervised framework that integrates recording-level contrastive learning with a novel local objective called Masked Latent Discrimination (MLD). The core innovation lies in the local branch, where a student model predicts representations for masked time-frequency patches and discriminates the correct full-view teacher target from negative candidates drawn from *other* recordings at the same spatial coordinate. This design avoids the common pitfall of forcing the model to distinguish acoustically similar regions within the same recording solely based on position. Additionally, the method incorporates a temporal neighborhood extension, providing weaker supervision from adjacent masked patches to enforce local temporal consistency. The use of a 2D Transformer predictor with axis-separated positional encodings is a solid architectural choice for handling the grid structure of spectrograms. The joint optimization of global and local losses is well-motivated, addressing the gap between holistic scene understanding and fine-grained event detection.
The experimental evaluation is extensive and rigorous. The authors compare AudioFM against strong baselines (AudioMAE, BEATs, EAT, SSLAM, BAT, AudioMosaic) across multiple settings: supervised fine-tuning, linear probing (frozen encoder), and zero-shot transfer. A standout feature is the systematic cross-domain evaluation spanning general audio, bioacoustics, medical acoustics, and speech emotion recognition. The inclusion of medical acoustics (PSG-Audio, CirCor) and bioacoustics (BEANS, BirdSet) is particularly valuable, as these domains are often underrepresented in standard audio SSL benchmarks. The results show consistent state-of-the-art performance, especially in linear probing where the quality of the frozen representation is isolated. The scaling analysis with model size and data volume further strengthens the contribution by demonstrating empirical scaling laws for audio pretraining.
The paper provides detailed hyperparameters, architecture specifications, and training protocols in the appendix. The use of standard datasets (AudioSet, ESC-50, FSD50K, etc.) and clear evaluation metrics (mAP, Macro-F1, PSDS) enhances reproducibility. However, the specific implementation details of the "mask-aware target selection" and the exact handling of the EMA teacher updates could benefit from code release to ensure full reproducibility. The paper mentions that code is not yet available or provides no link, which is a minor drawback for immediate replication.
The pretraining corpus is limited to AudioSet (approx. 1.68M clips), which is smaller than some recent large-scale efforts. The model is trained on single-channel audio, ignoring spatial cues that could be beneficial for certain tasks. The evaluation, while broad, does not include generative tasks or audio-text alignment, limiting the scope of "foundation model" claims. The medical acoustics evaluation, while impressive, uses relatively small datasets, and the generalizability to larger clinical datasets remains to be seen.
This work has significant potential for applications in environmental monitoring, healthcare (respiratory and cardiac sound analysis), and assistive technologies. By demonstrating strong transferability to specialized domains like bioacoustics and medical acoustics, the paper highlights the utility of general-purpose audio representations in niche but critical fields. The systematic evaluation framework provided could serve as a benchmark for future audio SSL research. The paper introduces AudioFM, a self-supervised audio foundation model that effectively bridges global and local representation learning through a novel masked latent discrimination objective. By combining recording-level contrast with patch-wise discrimination against cross-recording negatives and incorporating temporal neighborhood consistency, AudioFM achieves state-of-the-art performance across diverse domains, including general audio, bioacoustics, and medical acoustics, demonstrating superior transferability and robustness in both fine-tuning and zero-shot settings.
Reasoning is increasingly important for large audio-language models (LALMs), yet remains relatively underexplored compared with text-only and vision-language models. We introduce Audio-R1, a reinforcement learning (RL) post-training framework that revisits audio self-supervised learning as a source of verifiable rewards for improving audio understanding and reasoning task performance. Audio-R1 uses answers automatically determined during the construction of self-supervised tasks to verify model responses and compute rewards. Experiments across multiple LALMs and diverse audio benchmarks show consistent performance gains, while ablations demonstrate the benefits of each self-supervised task and additional gains from joint training. While both supervised fine-tuning and Audio-R1 improve self-supervised task accuracy, Audio-R1 further improves performance on downstream tasks distinct from the training tasks. These results highlight the potential of RL post-training with self-supervised audio tasks to improve downstream performance and reasoning-trace quality.
Primary: Mila-Quebec AI Institute
All Institutions: Mila-Quebec AI Institute, Universite Laval, Concordia University, Adobe Research
Audio-R1 introduces a scalable RL post-training framework using self-supervised audio tasks to improve reasoning in LALMs. The paper demonstrates consistent gains across multiple models and benchmarks, with strong ablation studies showing the complementary benefits of multi-task training and the superiority of RL over SFT for downstream generalization and reasoning quality.
The paper proposes Audio-R1, a reinforcement learning post-training framework that leverages self-supervised audio tasks to generate verifiable rewards. The core innovation is the design of three specific tasks—Identity Detection, Region Inpainting, and Temporal Ordering—that require the model to rely on acoustic evidence rather than linguistic priors. By using Group Relative Policy Optimization (GRPO) with rule-based verifiers, the method avoids the need for human-annotated reasoning traces or learned reward models. The approach effectively bridges the gap between self-supervised representation learning and reasoning-oriented post-training, offering a scalable solution to the data scarcity problem in audio reasoning.
The experiments are comprehensive, evaluating four different LALM backbones across three distinct benchmarks (MMAR, MMAU-mini, MMAU-Pro). The ablation studies are particularly strong, demonstrating that multi-task training yields complementary benefits over single-task training. The comparison between SFT and RL is a key strength, showing that while SFT improves task accuracy, it degrades reasoning trace quality, whereas Audio-R1 improves both. The use of an LLM-as-judge to evaluate reasoning quality provides valuable qualitative insights, although it remains a proxy metric.
The paper provides detailed descriptions of the task construction, reward functions, and training hyperparameters. The inclusion of prompt templates and judge rubrics in the appendix enhances reproducibility. However, the specific implementation details of the audio processing pipeline and the exact composition of the 24K source recordings could be more transparent. The use of standard libraries and frameworks (implied by the use of GRPO and LoRA) suggests reasonable reproducibility for researchers with similar resources.
The primary limitation is the reliance on self-supervised tasks that may not fully capture the complexity of open-ended audio understanding. The paper acknowledges that the tasks are designed for specific capabilities (correspondence, completion, ordering) and may not generalize to all forms of audio reasoning. Additionally, the use of an LLM-as-judge for reasoning quality evaluation is a proxy and does not establish causal links between reasoning traces and accuracy gains. The method is also conditioned on the pretrained backbone's capabilities, meaning it cannot develop reasoning skills absent from pretraining.
This work has significant implications for the development of large audio-language models, providing a scalable and efficient method for improving audio reasoning without the need for extensive human annotation. The approach could be extended to other modalities and tasks, potentially leading to more general-purpose multimodal models. The emphasis on verifiable rewards and self-supervised tasks aligns with broader trends in AI research towards more autonomous and scalable learning methods. Audio-R1 introduces a scalable RL post-training framework using self-supervised audio tasks to improve reasoning in LALMs. The paper demonstrates consistent gains across multiple models and benchmarks, with strong ablation studies showing the complementary benefits of multi-task training and the superiority of RL over SFT for downstream generalization and reasoning quality.
Localized AI manipulations in long-form videos often affect brief events within otherwise authentic content, making video-level classification insufficient for forensic analysis. Existing audio-visual deepfake benchmarks largely focus on short talking-head clips, while long-form temporal forensics remains predominantly visual. We introduce AV-FABLE, a dual-timeline benchmark for sparse manipulations in open-scene long-form audio-visual videos. AV-FABLE contains 16,650 pristine--manipulated twin pairs (998 hours) created with 27 editing configurations, with separate audio and visual timelines that enable modality-specific temporal localization and expose errors hidden by joint-timeline evaluation. On AV-FABLE, zero-shot prompting of audio-visual large language models (AV-LLMs) yields near-chance detection and poor temporal localization. In contrast, supervised readouts of their frozen perception features recover strong forensic signals, suggesting manipulation evidence is encoded in these representations but not effectively exposed through language outputs. Motivated by this observation, we propose NFSR(Native Forensics via Separated Readouts), which reads frozen AV-LLM features with separate audio and visual temporal decoders. NFSR achieves the best audio-visual localization performance in-domain and across all three out-of-distribution settings, including transfer to held-out videos from an external long-video forensic benchmark. AV-FABLE, code, and model weights will be publicly released.
Primary: Tsinghua University
All Institutions: Tsinghua University, Peking University
The paper presents a high-quality, large-scale benchmark (AV-FABLE) and a novel method (NFSR) for audio-visual manipulation forensics in long-form videos, demonstrating that frozen AV-LLM features are more effective for forensic localization than zero-shot language outputs.
The paper introduces AV-FABLE, a substantial benchmark for sparse audio-visual manipulations in long-form videos, featuring a dual-timeline evaluation protocol that distinguishes audio and visual tracks. This is a significant methodological advancement over joint-timeline evaluations which can mask modality-specific errors. The proposed model, NFSR, utilizes frozen features from AV-LLMs (Qwen3-Omni) with separate temporal decoders for each modality. The key methodological insight is the observation that while zero-shot prompting of AV-LLMs fails at forensic detection, their frozen perceptual features contain strong forensic signals accessible via supervised readouts. The use of host-conditioned feature modulation (FiLM) is a clever architectural choice to contextualize local anomalies within the broader video content.
The experimental setup is rigorous, including in-domain and three out-of-distribution (OOD) settings (source transfer, held-out generators, held-out routes). The dataset scale (16,650 pairs, 998 hours) is impressive for a forensic benchmark. Results show NFSR outperforming strong baselines like UMMAFormer and ActionFormer in AV-mAP across all settings. The ablation studies effectively demonstrate the contribution of intermediate-layer features, host conditioning, and modality separation. The comparison with zero-shot LLMs highlights a critical gap in current multimodal LLM capabilities for fine-grained forensic tasks.
The authors commit to releasing the dataset, code, and model weights. The paper provides detailed descriptions of the data construction pipeline (SEE-Forge), including the use of Gemini 3.7 Flash for planning and human review protocols for quality control. The evaluation metrics (AV-mAP, Event-IoU) and protocols are clearly defined. The reliance on specific, potentially proprietary, LLMs for data generation (Gemini) might pose minor reproducibility challenges for the dataset creation itself, but the benchmark and model are designed to be open.
The benchmark relies on synthetic manipulations generated by specific AI models, which may not fully capture the diversity of real-world human-edited forgeries or future, unseen generation techniques. The "explanation" task is noted as challenging and not fully solved by current baselines. The paper acknowledges that visual coverage under distribution shift remains an open problem, with NFSR showing lower V-IoU compared to some visual-only baselines in certain OOD settings.
This work addresses a critical gap in AI safety and media integrity by providing tools to localize and attribute manipulations in long-form content, which is increasingly relevant for misinformation detection. The finding that AV-LLMs encode forensic evidence in their features but fail to expose it via language outputs is a valuable insight for the broader multimodal AI community, suggesting that current alignment or decoding strategies in LLMs may be suboptimal for fine-grained perception tasks. The paper presents a high-quality, large-scale benchmark (AV-FABLE) and a novel method (NFSR) for audio-visual manipulation forensics in long-form videos, demonstrating that frozen AV-LLM features are more effective for forensic localization than zero-shot language outputs.
Music-driven group dance generation must decide not only how each dancer moves, but also what formation the group should form, when it should change, and who should execute each transition. Existing methods can generate orderly and rhythmically synchronized group motion, but remain limited in formation evolution and inter-dancer interaction. In human choreography, formation transitions often align with changes in high-level musical structure, such as the progression from an introduction to a chorus or from a chorus to a climax. We present ChoreoStudio, which makes choreography planning explicit. A music disentangler first decomposes the audio into frame-level rhythmic features and segment-level structural features. Given the segment-level features and sparse labels, a frozen omni-modal model acts as a scriptwriter, producing a global semantic description that guides the Director's choreography planning. Based on this context, the Director produces three coupled interaction decisions: a centroid-relative point set specifies what formation to produce, a transition gate determines when it should change, and a mobility allocation indicates who should execute the transition. The transition gate and mobility allocation are further conditioned on the planned geometric change, coupling formation targets with their temporal and dancer-specific execution. At a finer temporal scale, frame-aligned music features condition the diffusion generator to control detailed body motion and preserve beat-level alignment. The segment-level interaction plan is injected in parallel through frame-aligned instruction tokens and cross-dancer attention biases, coordinating how the generator realizes each transition. Experiments on AIOZ-GDance demonstrate that ChoreoStudio follows the planned interactions while achieving state-of-the-art performance in motion quality, group organization, and formation dynamics.
Primary: Monash University
All Institutions: Monash University
ChoreoStudio introduces a novel explicit choreography planning framework that decouples high-level formation, timing, and role decisions from low-level motion synthesis, achieving state-of-the-art performance in group dance generation by leveraging multi-scale music conditioning and frozen LLM semantic intent to guide a structured Director module, thereby addressing the critical gap in macro-level formation evolution and inter-dancer interaction control in existing methods.
The paper proposes ChoreoStudio, a hierarchical framework that explicitly decouples choreography planning (what, when, who) from motion synthesis (how). The core novelty lies in the "Director" module, which generates a structured plan consisting of relative formation geometry, transition timing gates, and dancer mobility allocations. This plan is derived from multi-scale music features (frame-level rhythm and segment-level structure) and semantic intent from a frozen omni-modal LLM (Qwen3-Omni). The method addresses a specific gap in existing group dance generation: the lack of explicit control over macro-level formation evolution and asymmetric role assignment. The use of a frozen LLM for high-level semantic planning is a strong architectural choice, leveraging recent advances in multimodal understanding without requiring fine-tuning of the large model. The explicit plan-to-motion interface, using frame-aligned instruction tokens and cross-dancer attention biases, is technically sound and ensures that the generated motion adheres to the planned choreography. The verification-guided refinement loop adds robustness by allowing the system to iteratively correct physical or plan-consistency errors.
Experiments are conducted on the AIOZ-GDance dataset, which is a standard benchmark for group dance generation. The paper introduces novel frequency-based metrics (IDDV, Jitter, FMR) to evaluate formation dynamics, which is a significant contribution as standard metrics often miss macro-level structural issues. The quantitative results show state-of-the-the-art performance in Group Motion Realism (GMR) and Group Motion Correlation (GMC) across various group sizes (2-7 dancers). The ablation studies effectively demonstrate the contribution of each component, particularly the Director and the semantic description. The controlled intervention experiments are a strong point, providing causal evidence that the model responds to changes in the planned formation, timing, and mobility. The comparison with baselines like TCDiff++ and ST-GDance is fair and uses a shared protocol.
The paper provides detailed descriptions of the architecture, training objectives, and evaluation metrics. The use of a frozen LLM (Qwen3-Omni) and standard diffusion models enhances reproducibility compared to custom large models. However, the specific implementation details of the "Director" and the exact prompts used for the LLM are not fully specified in the main text (referenced to appendices). The code availability is not explicitly stated in the provided text, but the detailed methodology suggests it is reproducible for a skilled team. The use of standard datasets and metrics further aids reproducibility.
The method relies on a frozen LLM for semantic planning, which may limit the adaptability to novel or complex musical styles not well-represented in the LLM's training data. The planning is done at a segment level, which may not capture very rapid or micro-level choreographic changes. The evaluation is limited to the AIOZ-GDance dataset, and generalization to other datasets or real-world scenarios is not tested. The computational cost of the iterative refinement loop and the LLM inference may be significant for real-time applications.
This work has significant implications for the field of motion generation, particularly for applications in virtual reality, gaming, and automated video production. By making choreography planning explicit and editable, the system enables more controllable and interpretable group dance generation. The approach could be extended to other multi-agent motion tasks, such as crowd simulation or robotic swarm coordination. The integration of LLMs for high-level planning in motion generation is a promising direction that could lead to more sophisticated and semantically aware generative models. ChoreoStudio introduces a novel explicit choreography planning framework that decouples high-level formation, timing, and role decisions from low-level motion synthesis, achieving state-of-the-art performance in group dance generation by leveraging multi-scale music conditioning and frozen LLM semantic intent to guide a structured Director module, thereby addressing the critical gap in macro-level formation evolution and inter-dancer interaction control in existing methods.
Auditory attention decoding (AAD) seeks to identify the attended talker from brief noninvasive brain recordings by comparing neural responses with competing speech streams. Recent electroencephalography (EEG) decoders report nearly perfect accuracy from one second of data, yet we show that this performance fails to generalize to unseen speech because models exploit trial leakage, eye gaze, and acoustic cues instead of speech evoked neural activity. We introduce CoupleFlow, which decodes attention through temporal coupling between neural and acoustic embeddings. Its predictions remain at chance when the brain signal contains no informative coupling, preventing shortcuts based on speech content or talker identity. A conditional flow matching head implements a likelihood ratio decision rule based on pointwise mutual information between brain and speech. This rule is Bayes optimal when the recording depends only on the attended talker and removes preferences for speech that is likely a priori. We evaluate CoupleFlow with content disjoint splits recovered from the audio and null tests that distinguish genuine temporal coupling from trial recognition. Across nine EEG and magnetoencephalography (MEG) datasets processed through a common pipeline, spanning four languages, high density EEG recordings, listeners with hearing impairment, and a room with multiple talkers, CoupleFlow outperforms linear and deep baselines while all null tests remain at chance. Finally, we derive an evidence accumulation law that predicts accuracy across window lengths from a single parameter per dataset and predicts that parameter from the strength of neural speech tracking. This law identifies recording signal to noise ratio as a limit on decoding speed and reliability and estimates the window length required to reach a target accuracy.
Primary: The Ohio State University
All Institutions: The Ohio State University
CoupleFlow introduces a null-calibrated auditory attention decoder that uses flow-matching likelihood ratios to ensure predictions are driven by genuine brain-speech coupling rather than acoustic shortcuts. The paper makes a substantial contribution to the field by exposing the fragility of previous AAD models through rigorous null testing and providing a theoretically grounded, robust alternative that generalizes across diverse datasets and languages.
The paper introduces CoupleFlow, a decoder for Auditory Attention Decoding (AAD) that addresses the critical issue of shortcut learning in previous EEG/MEG studies. The core methodological innovation is the "null-calibrated" design: the model relies on temporal coupling between neural and acoustic embeddings, ensuring that if the brain signal contains no information about the speech (e.g., shuffled or unrelated trials), the decoder's performance drops to chance. This is a significant architectural constraint that prevents the model from exploiting static acoustic cues (like loudness or speaker identity) or trial leakage. The use of a conditional flow-matching head to implement a likelihood ratio decision rule based on pointwise mutual information (PMI) is a sophisticated application of generative modeling to a discriminative task. The theoretical grounding, particularly the derivation of the "evidence accumulation law" which predicts accuracy based on window length and signal-to-noise ratio, provides a strong analytical framework for understanding the limits of non-invasive decoding.
The experimental evaluation is exceptionally rigorous. The authors test on nine diverse EEG and MEG datasets, spanning four languages, different electrode densities, and populations with hearing impairments. Crucially, they employ content-disjoint splits and null tests (shuffled EEG, time-shifted EEG) to verify that performance is driven by genuine neural coupling rather than data leakage. The finding that previous state-of-the-art models fail these null tests while CoupleFlow maintains performance is a strong contribution to the field's integrity. The comparison with linear and deep baselines under a unified preprocessing pipeline further strengthens the validity of the results.
The paper provides detailed descriptions of the preprocessing pipeline, model architecture, and loss functions. The use of public datasets and a common pipeline enhances reproducibility. However, specific hyperparameters and code availability are not explicitly confirmed in the provided text (no GitHub link found), which is a minor deduction. The detailed theoretical derivations and ablation studies (implied by the discussion of loss terms) support reproducibility.
The primary limitation is the reliance on non-invasive signals (EEG/MEG), which inherently have lower spatial resolution and signal-to-noise ratios compared to invasive recordings. The "evidence accumulation law" assumes certain statistical properties (e.g., independent samples, Gaussian noise) that may not perfectly hold in all real-world noisy environments. Additionally, the model's performance on very short windows (sub-second) may still be limited by the physical constraints of neural response latency and SNR, as predicted by their own law.
This work has significant implications for the development of neuro-steered hearing devices and brain-computer interfaces. By providing a robust, leakage-free benchmark and a decoder that genuinely tracks attended speech, it paves the way for more reliable real-world applications. The theoretical framework for predicting decoding accuracy based on recording quality is valuable for designing future experiments and hardware. It also serves as a cautionary tale for the ML community regarding the importance of null tests and proper data splitting in neural decoding tasks. CoupleFlow introduces a null-calibrated auditory attention decoder that uses flow-matching likelihood ratios to ensure predictions are driven by genuine brain-speech coupling rather than acoustic shortcuts. The paper makes a substantial contribution to the field by exposing the fragility of previous AAD models through rigorous null testing and providing a theoretically grounded, robust alternative that generalizes across diverse datasets and languages.
Streaming audio-visual target speaker extraction (AVTSE) is essential for latency-sensitive applications. However, current causal systems face two coupled gaps. First, standard benchmarks fix the number of speakers at two and ignore the near-/far-field interference, reverberation, and sudden noise of real scenes. Second, high-performance causal architectures are too heavy for edge deployment. Lightweight alternatives try to reclaim capacity by repeatedly invoking a small separator, which offsets the savings. To address the first gap, we release RealSSA, a realistic AVTSE benchmark consisting of RealSSA-Sim and RealSSA-Real. RealSSA-Sim provides controllable simulated scenes with 1--5 dynamic near-field interfering speakers; scenes with far-field interference additionally include 10--15 far-field interfering speakers. RealSSA-Real provides a held-out real-recorded evaluation set. To address the second gap, we propose Falcon, a causal AVTSE method. Its Separator performs multi-scale time-frequency separation in a single encoder--decoder pass. We train Falcon with an encoder-space speaker contrastive loss. This loss suppresses near-field leakage at zero inference cost and transfers across backbones. Falcon achieves state-of-the-art extraction quality on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix, and VoxCeleb2-2Mix. Compared with the causal AV-TFGridNet baseline, it cuts parameters by 91.9%, computation by 20.2 times, and GPU inference latency by 11.1 times. Code and demo are available at https://cslikai.cn/Falcon/.
Primary: Tsinghua University
All Institutions: Qinghai University, Tsinghua University, Intelligent Computing and Application Laboratory of Qinghai Province, Chinese Institute for Brain Research (CIBR)
The paper introduces Falcon, a lightweight causal AVTSE model, and RealSSA, a realistic benchmark, achieving significant efficiency gains over existing causal baselines while maintaining state-of-the-art extraction quality. The technical contribution is strong, particularly in addressing the efficiency-quality trade-off in causal systems and providing a more realistic evaluation environment for the field.
The paper proposes Falcon, a causal audio-visual target speaker extraction (AVTSE) model, and RealSSA, a new benchmark. The methodology addresses two specific gaps: the lack of realistic acoustic conditions in existing benchmarks and the high computational cost of causal models. Falcon utilizes a single-pass asymmetric U-Net separator with multi-scale time-frequency modeling, avoiding the iterative recurrence found in prior lightweight models like SwiftNet. A key technical contribution is the "encoder-space speaker contrastive loss," which explicitly suppresses leakage from near-field interferers during training without adding inference cost. The architecture uses a frozen FRCNN visual backbone and a causal LSTM for video processing, fused with audio features via addition. The design is logically sound, aiming to balance latency constraints with extraction quality by replacing iterative small separators with a single, more efficient pass.
The experimental setup is robust, evaluating Falcon on the new RealSSA-Sim, RealSSA-Real, and standard benchmarks (LRS2-2Mix, LRS3-2Mix, VoxCeleb2-2Mix). The paper claims state-of-the-art performance on these benchmarks. Specifically, it reports a 91.9% reduction in parameters and an 11.1x reduction in GPU inference latency compared to the causal AV-TFGridNet baseline. The inclusion of a real-world recorded set (RealSSA-Real) is a significant strength, as it tests generalization beyond simulated mixtures. The use of metrics like SI-SNR, WER, DNSMOS, and NISQA provides a comprehensive evaluation of both separation quality and perceptual/recognizability aspects.
The paper provides a code and demo URL (https://cslikai.cn/Falcon/), which enhances reproducibility. The dataset generation pipeline for RealSSA-Sim is detailed, including specific parameters for room impulse responses (gpuRIR), source placement, and mixing levels (EBU R128). The architectural details of the Separator, including convolution types, padding strategies, and the definition of the causal layer normalization, are described in sufficient detail for re-implementation.
The visual backbone (FRCNN) is frozen, which may limit the model's ability to adapt to diverse visual conditions or lighting changes not present in the training data. The contrastive loss relies on having clean references for interferers during training, which is feasible in simulation but might be challenging in unsupervised or semi-supervised real-world settings. The paper is an arXiv preprint with 0 citations, so long-term impact is yet to be determined. The "RealSSA-Real" set is described as having pixelated faces, which is good for privacy but may introduce artifacts that affect visual feature extraction.
This work has significant implications for edge deployment of speech enhancement systems, such as in hearing aids, video conferencing, and smart speakers. By providing a lightweight causal model and a realistic benchmark, it facilitates the development of more practical, low-latency audio-visual systems. The benchmark addresses a critical need for evaluating models under realistic, non-anechoic conditions with varying numbers of speakers. The paper introduces Falcon, a lightweight causal AVTSE model, and RealSSA, a realistic benchmark, achieving significant efficiency gains over existing causal baselines while maintaining state-of-the-art extraction quality. The technical contribution is strong, particularly in addressing the efficiency-quality trade-off in causal systems and providing a more realistic evaluation environment for the field.
This paper introduces Spark-Audio 1.0, a speech foundation model trained on domestically produced computing infrastructure. The model accepts text and speech inputs and generates text outputs. It supports multilingual (99 languages) and multi-dialect (202 dialects) speech recognition, speech translation, audio perception and understanding, and both text- and speech-based dialogue, marking a shift from merely ``hearing clearly'' to genuinely ``understanding'' speech. Architecturally, Spark-Audio 1.0 couples a 0.65B-parameter dense audio encoder with a 30B-A3B Mixture-of-Experts (MoE) large language model (LLM). The audio encoder is pretrained with representation-enhanced BEST-RQ and then refined through multi-task supervised training. The audio--LLM is trained through staged pretraining, followed by post-training that combines supervised fine-tuning with on-policy distillation. In total, the model is trained on 13 million hours of audio data together with large-scale text data. We evaluate Spark-Audio 1.0 on mainstream audio and text benchmarks, as well as on in-house test sets drawn from real-world application scenarios. Despite using substantially less audio training data than Qwen3.5-Omni-Flash, Spark-Audio 1.0 achieves comparable performance on most audio and text tasks, outperforms it in multilingual speech recognition, spoken dialogue, and real-world application scenarios, and approaches the performance of larger closed-source models. Nevertheless, there remains room for improvement in instruction following and complex audio reasoning. Finally, the successful training of Spark-Audio 1.0 on domestic hardware, including Huawei Ascend 910B and Hygon BW1000 chips, demonstrates the feasibility of a domestic technical pathway for building speech foundation models.
Primary: iFlytek
All Institutions: iFlytek
The main contribution is the demonstration of a competitive, data-efficient speech foundation model trained entirely on domestic Chinese hardware. The paper provides a detailed technical report on the Spark-Audio 1.0 architecture, highlighting a novel representation-enhanced pretraining strategy and a unified streaming/non-streaming encoder design that achieves strong multilingual performance with significantly less data than comparable competitors.
The paper proposes a unified speech foundation model architecture combining a 0.65B dense audio encoder (based on Whisper-Large v3) with a 30B-A3B MoE LLM. The core methodological contribution is the "representation-enhanced BEST-RQ" pretraining strategy, which uses frozen semantic representations from a multilingual ASR teacher to generate discrete targets for the student encoder, aiming to inject semantic priors without additional trainable tokenizers. The training pipeline is staged: self-supervised pretraining, supervised ASR fine-tuning, and multi-task perception training (emotion, gender, etc.). The audio-LLM integration uses a staged recipe including adapter alignment, joint optimization, and long-context extension, followed by SFT and on-policy distillation (OPD) to preserve text capabilities. The unified streaming/non-streaming encoder design via dynamic chunk training is a practical engineering contribution.
The model is evaluated on 99 languages and 202 dialects, covering ASR, translation, audio understanding, and dialogue. The paper claims comparable performance to Qwen3.5-Omni-Flash with significantly less data (13M hours vs. implied larger scale for Qwen) and outperforms it in multilingual recognition and real-world scenarios. It also approaches larger closed-source models. However, the evaluation relies heavily on "in-house test sets" which are not publicly available, limiting independent verification. The comparison with Qwen3.5-Omni-Flash is the primary baseline, but details on the specific data scale of the competitor are somewhat vague ("substantially less").
Reproducibility is low for the general community. The model is trained on specific domestic hardware (Huawei Ascend 910B, Hygon BW1000), and the code/weights are not released in the paper. The "in-house" benchmarks are proprietary. While the architectural details (Whisper-based encoder, MoE LLM) are standard, the specific data curation (13M hours) and the "representation-enhanced" teacher setup are not fully detailed enough for exact replication without access to iFlytek's internal resources.
The paper explicitly acknowledges room for improvement in instruction following and complex audio reasoning. The reliance on proprietary in-house benchmarks makes it difficult for the community to verify the claimed superiority over Qwen3.5-Omni-Flash. The "domestic hardware" aspect, while strategically important, limits the immediate technical applicability for researchers using standard CUDA stacks. The text-only output modality restricts it to understanding rather than generation (speech-to-speech).
The paper demonstrates the feasibility of training large-scale speech foundation models on non-NVIDIA hardware, which is significant for the Chinese AI ecosystem and potentially other regions with hardware restrictions. It contributes to the trend of data-efficient training in audio-LLMs. The support for 202 dialects is a substantial resource for multilingual NLP research, assuming the model or data becomes available. The main contribution is the demonstration of a competitive, data-efficient speech foundation model trained entirely on domestic Chinese hardware. The paper provides a detailed technical report on the Spark-Audio 1.0 architecture, highlighting a novel representation-enhanced pretraining strategy and a unified streaming/non-streaming encoder design that achieves strong multilingual performance with significantly less data than comparable competitors.
We introduce MixAgent, an agentic pipeline for music post-production that follows the workflow of audio engineers to build interpretable and editable mixing plans guided by a reference song. A grouping agent first routes related input tracks, such as drum recordings, into subgroups. A planning agent then uses retrieved examples to choose and order audio effects, such as volume control and reverb, for tracks, subgroups, and the full mix. To refine the mix, a detector identifies what needs correction, and a critic proposes numerical parameter adjustments from tracks to the full mix. In our experiments, prompting recent multimodal models to listen and revise often worsens the mix, making reliable correction a central challenge. We therefore build MixCritic, a dataset of 45,657 repair critiques derived from parameters fitted to professional mixes, and train the critic with supervised fine-tuning followed by preference optimization on rendered audio outcomes. On MedleyDB, learned refinement reduces error by 19% relative to the initial mix, with preference optimization reducing it by 8.4% over supervised training alone. Retrieved examples further improve effects selection, and a human listening test shows that MixAgent gets higher quality ratings than mixes produced by Qwen3-Omni, Gemini 3.8 Flash and GPT-4o-audio.
Primary: Sony Group Corporation
All Institutions: Rice University, Sony Corporate Technology Center America, Inc., Sony Group Corporation
MixAgent introduces a novel agentic pipeline for music post-production that combines retrieval-augmented planning with learned numerical refinement, achieving state-of-the-art results in reference-guided mixing and providing a robust framework for interpretable audio editing.
The paper proposes MixAgent, a multi-agent framework for reference-guided music mixing. The methodology is structured into three distinct phases: grouping (routing tracks to subgroups), planning (selecting and ordering audio effects via retrieval-augmented generation), and refinement (detecting and correcting parameter errors). A key methodological contribution is the construction of the MixCritic dataset, which pairs corrupted mixing states with repair critiques derived from professional mixes. The authors employ a two-stage training process for the critic: Supervised Fine-Tuning (SFT) followed by Direct Preference Optimization (DPO) on rendered audio outcomes. The system explicitly models the hierarchical nature of mixing (tracks -> subgroups -> full mix) and uses a bottom-up refinement strategy, which is shown to be superior to top-down or flat approaches. The use of a frozen planner with retrieved examples to guide effect selection is a practical and effective design choice that mitigates the tendency of LLMs to select suboptimal or overly short effect chains.
The experiments are rigorous and comprehensive. The authors evaluate on MedleyDB and AlbumDB, using non-overlapping excerpts to prevent data leakage. They compare against strong baselines, including recent multimodal LLMs (Qwen3-Omni, Gemini 3.8 Flash, GPT-4o-audio) under various prompting strategies (Single-pass, Self-Refine, Self-Critic). The results demonstrate that MixAgent outperforms these baselines in reconstruction error (MSS) and predicted quality metrics. A particularly insightful analysis is the comparison of feedback forms (perceptual, tool-specific, numerical), showing that numerical corrections trained via DPO yield the best performance. The human listening test, while small (11 participants), provides qualitative validation that the system produces mixes rated higher than LLM baselines, though still below professional human mixes. The ablation studies on retrieval, refinement order, and detector selection are thorough and provide clear insights into the system's components.
The paper provides high reproducibility. The authors release the code and demo on GitHub. They detail the architecture of the detector and critic, the training procedures (SFT and DPO), and the construction of the MixCritic dataset. The prompts used for the agents are included in the appendix. The use of standard datasets (MedleyDB, AlbumDB) and public models (Qwen, Gemini, GPT) further enhances reproducibility. The specific hyperparameters and training settings are listed in tables, allowing for faithful replication.
The primary limitation is the small size of the evaluation sets (20 MedleyDB clips, 14 AlbumDB clips), which limits statistical power and generalizability. The listening test with only 11 participants is also a weakness. The system relies on GRAFX-fitted parameters for supervision, which may not perfectly capture the intent of professional engineers. The audio effects library is limited to a subset of professional tools, and the plan is fixed after the planning stage, meaning the critic cannot add or remove effects. The computational cost of multiple LLM calls for mixing is also noted as a practical limitation.
This work has significant implications for the field of AI-assisted creative tools. By providing an interpretable and editable mixing plan, it addresses a key barrier to adoption of AI in professional audio workflows. The findings on the effectiveness of numerical feedback and preference optimization for audio parameter adjustment are broadly applicable to other domains where LLMs control numerical tools. The construction of the MixCritic dataset is a valuable resource for the community. The paper also highlights the challenges of evaluating audio quality with current metrics, pointing to the need for better perceptual evaluation methods. MixAgent introduces a novel agentic pipeline for music post-production that combines retrieval-augmented planning with learned numerical refinement, achieving state-of-the-art results in reference-guided mixing and providing a robust framework for interpretable audio editing.
Most voice-cloning systems are built for audio alone: they copy the reference recording's delivery and acoustics, so a voice cloned on a phone sounds like a phone recording whether the speaker is pictured in a studio, on a street, or in a cathedral. Speech for a talking video should sound like the pictured person speaking in the pictured place. We present HeyGen Voice, a scene-aware voice-cloning model that separates who is speaking from how and where: it keeps the speaker's voice from an everyday recording while the target image sets the delivery and acoustics. Prior image-conditioned TTS does half of this job: it either adds scene acoustics without keeping a specific speaker or infers a voice from a face. HeyGen Voice is a multimodal autoregressive transformer conditioned on the image, speaker and scene captions, and the voice recording, trained in five stages that move from multilingual TTS pretraining through simulated and real cross-scene adaptation to reinforcement and preference optimization. Two clone policies trade scene adaptation against fidelity to the original recording, and an audio anchor holds long scripts in one voice and one acoustic environment. Listeners prefer HeyGen Voice over leading commercial systems across a third-party arena, production votes, and talking-video studies: it ranks first (Elo 1201) on the public Artificial Analysis controlled-voice TTS leaderboard (October 2026); in a blind test, users cloning their own voice chose it over ElevenLabs v4 and Cartesia Sonic 3.6 in 62% and 54% of votes; and volunteers rated talking videos of their own digital twin about 10 to 30 points higher (out of 100) when voiced by HeyGen Voice than by leading commercial alternatives. Try HeyGen Voice and hear samples at https://heygen.com/heygen-voice.
Primary: HeyGen Research
All Institutions: HeyGen Research
HeyGen Voice introduces a scene-aware voice cloning framework that decouples speaker identity from acoustic environment using multimodal conditioning and a cross-scene training curriculum. The paper demonstrates state-of-the-art performance in both objective cross-scene benchmarks and subjective human evaluations, establishing a new standard for audio-visual coherence in talking video generation.
The paper proposes "HeyGen Voice," a multimodal autoregressive transformer for scene-aware voice cloning. The core methodological contribution is the decoupling of speaker identity (timbre, accent) from scene acoustics (reverberation, background noise) and delivery style. This is achieved through a conditioning prefix that combines pooled reference-voice tokens, image tokens (from a frozen VLM), and explicit text captions for speaker and background. A key technical innovation is the "cross-scene training curriculum," which includes a simulated adaptation stage using reference perturbations to remove the shortcut of relying on the reference recording's acoustics, followed by fine-tuning on real cross-scene pairs. The use of GRPO for intelligibility/speaker similarity and DPO with a low-rank adapter for expressiveness is a robust application of modern alignment techniques to TTS. The "clone policies" (voice-clone vs. full-clone) provide a practical mechanism for users to control the trade-off between fidelity to the source recording and adaptation to the target scene.
The evaluation is extensive and multi-faceted, covering objective benchmarks (cross-scene similarity, multilingual CER), subjective human evaluations (paired listening tests in 5 languages), production user votes, third-party arena rankings (Artificial Analysis), and video-based studies. The cross-scene benchmark (331 cases) is a valuable contribution to the field, as it specifically tests the ability to adapt voice to new environments, a task often ignored in standard TTS benchmarks. The human studies are well-designed, including blind tests with users cloning their own voices and volunteer studies rating talking videos. The results show consistent superiority over leading commercial systems (ElevenLabs, Cartesia, Inworld) in scene alignment and user preference, particularly in the "voice-clone" mode. The inclusion of a "corrected" multilingual benchmark addresses known issues with transcription errors in standard datasets.
Reproducibility is limited by the proprietary nature of the system. While the architecture is described in detail, the specific data pipelines (especially the "cross-scene pair mining" and the 5,000+ GPU training infrastructure) are not open-sourced. The paper references internal infrastructure (HELIOS) and data reports (Avatar V) that are not publicly available. The benchmarks (cross-scene, multilingual) are described but not released as public datasets with code. The use of specific commercial APIs for baselines limits the ability to reproduce the exact comparisons without access to those services.
The primary limitation is the lack of open-source code and data, which hinders independent verification and community adoption. The reliance on a frozen VLM for image understanding may limit the model's ability to handle complex or abstract scene descriptions not well-captured by the VLM. The "captioning" step at inference (where an LLM writes captions from the image) introduces an additional point of failure and latency. The paper does not extensively discuss the computational cost of the multimodal conditioning compared to standard TTS, although it mentions an RTF of 0.1. The evaluation, while strong, is largely conducted by the authors or their platform users, which may introduce bias, although the third-party arena ranking helps mitigate this.
This work has significant implications for the creation of digital humans and talking avatars, where audio-visual coherence is crucial. By enabling voice cloning that adapts to the visual scene, it enhances the realism and immersion of AI-generated video content. The techniques developed for cross-scene adaptation and consistency in long-form synthesis are applicable to other multimodal generation tasks. The focus on user-controlled trade-offs (clone policies) sets a precedent for more flexible and user-centric AI audio tools. HeyGen Voice introduces a scene-aware voice cloning framework that decouples speaker identity from acoustic environment using multimodal conditioning and a cross-scene training curriculum. The paper demonstrates state-of-the-art performance in both objective cross-scene benchmarks and subjective human evaluations, establishing a new standard for audio-visual coherence in talking video generation.
Driven by recent advancements in hearing assistive devices, target speaker extraction (TSE) in multi-talker environments has drawn significant attention. To address this, hearing technologies identify attention using neural signals recorded via electroencephalography (EEG) to guide TSE, enabling neuro-steered auditory attention decoding. However, most existing techniques rely on time-frequency masking, which suffers from low fidelity in spectro-temporally overlapping regions and is often restricted to narrowband audio. In contrast, we introduce NeuroCodec, which, to the best of our knowledge, is the first generative EEG-guided TSE framework operating in the latent space of a neural audio codec. Specifically, NeuroCodec models spatial EEG dependencies via graph convolutions and fuses cross-modal EEG and speech representations through attention mechanisms and Mamba-based temporal modeling. We further introduce a six-phase training curriculum that progressively incorporates latent reconstruction, multi-scale spectral, and adversarial objectives to ensure stable, high-fidelity speech generation. Experiments on the public Cocktail Party and KUL datasets show NeuroCodec outperforms baselines, across several perceptual metrics. A listening experiment with 20 participants validated that NeuroCodec offers greater perceptual accuracy, achieving a mean score of 82.24 compared with 49.63 for M3ANet. Additionally, the proposed method is seen to enable 44.1 kHz reconstruction with spectral content extending up to 20 kHz, further supporting the perceptual results. Finally, we extend NeuroCodec with a causal streaming formulation suited for real-time processing, providing a basis for hearing assistive devices.
Primary: University of Moratuwa
All Institutions: University of Moratuwa
The paper presents a novel generative approach to EEG-guided target speaker extraction by operating in the latent space of a neural audio codec, effectively addressing the fidelity limitations of traditional time-frequency masking. The combination of Mamba-based temporal modeling, graph-based EEG encoding, and a stable multi-phase training curriculum yields strong objective and subjective performance improvements, particularly in wideband reconstruction and real-time streaming efficiency.
The paper proposes NeuroCodec, a generative framework for EEG-guided Target Speaker Extraction (TSE) that operates in the latent space of a frozen neural audio codec (DAC) rather than predicting time-frequency masks. The methodology is technically sound and innovative in its combination of components: it uses Chebyshev graph convolutions for spatial EEG modeling, Mamba (selective state-space models) for efficient temporal modeling, and cross-attention for modality fusion. The introduction of a six-phase training curriculum to stabilize the adversarial and spectral losses is a practical contribution to the training stability of generative speech models. The causal streaming extension (OnlineNeuroCodec) introduces an "auditory attractor" mechanism using an Exponential Moving Average (EMA) of previous target latents to maintain speaker identity across hops, which is a clever, parameter-free alternative to recurrent speaker encoders.
The experiments are conducted on two standard datasets (Cocktail Party and KUL) under a Subject-Independent (SI) setting, which is rigorous. The paper reports improvements in STOI, ESTOI, and PESQ over strong baselines like M3ANet and NeuroSpex+. A significant strength is the inclusion of a subjective MUSHRA listening test with 20 participants, which validates the perceptual quality claims (82.24 vs 49.63 for M3ANet). The ablation studies effectively isolate the contributions of the Mamba backbone, EEG encoder components, and loss functions. The streaming evaluation demonstrates a significant speed advantage (RTF) over the NeuroHeed baseline while maintaining higher accuracy.
The paper states that source code will be released upon acceptance, but no code or demo links are currently provided in the text. The implementation details are reasonably specific (e.g., DAC downsampling factors, window sizes, EMA rates), which aids reproducibility. However, the reliance on specific pretrained DAC models and the complex multi-phase training schedule may make exact reproduction difficult without the authors' code.
The primary limitation is the computational cost of the generative approach, although the RTF analysis suggests it is viable for streaming. The subjective study size (n=20) is modest. The paper acknowledges that the system is not yet ready for embedded hearing-aid hardware. The comparison with some baselines (NeuroHeed+, MSFNet) is limited to reported numbers from other papers due to lack of public code, which weakens the comparative analysis slightly.
This work has significant potential for assistive hearing technologies by enabling high-fidelity, wideband speech extraction guided by neural attention. The shift from masking to generative latent synthesis could reduce artifacts in overlapping speech regions. The use of Mamba for efficient temporal modeling in audio-EEG fusion is a trend that may influence future multimodal audio architectures. The paper presents a novel generative approach to EEG-guided target speaker extraction by operating in the latent space of a neural audio codec, effectively addressing the fidelity limitations of traditional time-frequency masking. The combination of Mamba-based temporal modeling, graph-based EEG encoding, and a stable multi-phase training curriculum yields strong objective and subjective performance improvements, particularly in wideband reconstruction and real-time streaming efficiency.
This paper presents XingChen-UniFE, a unified large audio language model (LALM) for general speech restoration (GSR) and target speaker extraction (TSE). The proposed model employs complementary semantic and acoustic representations through a dual-stream encoder, where a self-supervised learning model provides hierarchical semantic features and a learnable acoustic branch captures fine-grained structures from complex spectra. A bidirectional cross-attention module further facilitates information exchange between the two representations. The fused representation conditions the LALM for discrete speech token generation with task and speaker information. Experiments demonstrate strong performance on both GSR and TSE and robust generalization to unseen degradation conditions.
Primary: China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd.
All Institutions: China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd.
The paper presents a unified LALM-based framework for speech restoration and speaker extraction that effectively combines semantic and acoustic features via bidirectional cross-attention. The technical contribution is solid, with strong empirical results demonstrating improved performance over existing generative baselines and good generalization capabilities, though the lack of subjective evaluation and multilingual testing limits the comprehensiveness of the validation.
The paper proposes XingChen-UniFE, a unified Large Audio Language Model (LALM) for General Speech Restoration (GSR) and Target Speaker Extraction (TSE). The core methodological contribution is a dual-stream encoder that fuses semantic representations from a frozen self-supervised learning (SSL) model (WavLM) with acoustic representations from a learnable branch processing complex STFT spectrograms. A bidirectional cross-attention module is introduced to facilitate explicit information exchange between these two streams before conditioning the LALM backbone. The LALM generates interleaved acoustic and semantic tokens autoregressively, which are then decoded by a neural audio codec (H-Codec). The approach addresses the limitation of SSL features lacking fine-grained acoustic details (like phase) by explicitly modeling complex spectra, which is a reasonable and technically sound design choice for restoration tasks.
The experiments are conducted on the Libri2Mix dataset for TSE and various simulated degradation conditions for GSR. The model is compared against strong baselines like LLaSE-G1, UniSE, and UniTok. The proposed model achieves state-of-the-art or competitive results across PESQ, STOI, SpeechBERTScore (SBS), and WER metrics. Notably, the paper demonstrates robust generalization to unseen degradation types (codec distortion, wind noise) that were excluded from training, which is a significant strength. Ablation studies confirm the complementary roles of the semantic and acoustic branches and the importance of the bidirectional fusion mechanism.
The paper provides sufficient implementation details, including model architecture dimensions (28 layers, 1024 hidden dim), training hyperparameters (400k steps, batch size 64, AdamW), and data composition (approx. 10k hours target speech, 405k hours noise). However, the specific codebase and pre-trained checkpoints are not linked in the provided text, and the reliance on specific internal or less common datasets (e.g., Emilia, CN-Celeb) may pose minor barriers for full reproduction without access to those resources.
The model is computationally expensive due to the large LALM backbone (502M trainable parameters) and the dual-stream encoder, which may limit real-time deployment on edge devices. The evaluation is primarily limited to English speech (Libri2Mix, LibriSpeech); performance on multilingual or non-English speech is not assessed. Additionally, the paper lacks subjective listening tests (MOS), relying solely on objective metrics, which may not fully capture perceptual quality nuances.
This work contributes to the trend of unifying speech enhancement tasks under generative frameworks. By leveraging LALMs, it opens avenues for more complex speech processing tasks that require semantic understanding alongside acoustic restoration. The demonstrated generalization to unseen degradations suggests potential for robust deployment in diverse real-world environments. The paper presents a unified LALM-based framework for speech restoration and speaker extraction that effectively combines semantic and acoustic features via bidirectional cross-attention. The technical contribution is solid, with strong empirical results demonstrating improved performance over existing generative baselines and good generalization capabilities, though the lack of subjective evaluation and multilingual testing limits the comprehensiveness of the validation.