Unsupervised multimodal intent discovery aims to uncover latent intents from unlabeled multimodal dialogues, but remains challenging due to the lack of explicit semantic supervision. Existing methods often provide limited interpretability, as their refinement mainly relies on geometric similarity rather than high-level semantic guidance. To address these limitations, we propose MCSP, a fully unsupervised method that introduces semantic refinement based on concepts into multimodal intent discovery. To obtain reliable semantic evidence for intent discovery, we identify high-quality representative samples for each cluster and use them to support MLLM-guided contrastive reasoning against neighboring clusters, which produces interpretable high-level semantic concepts. Building on these concepts, we perform semantic propagation over a semantically weighted graph to align conceptual information with local structural consistency and generate reliable pseudo-labels for representation refinement. Extensive experiments on three challenging multimodal intent datasets show that MCSP consistently outperforms state-of-the-art methods while producing interpretable clusters grounded in semantic concepts.
Primary: The Chinese University of Hong Kong, Shenzhen
All Institutions: The Chinese University of Hong Kong, Shenzhen, Tsinghua University
The paper presents MCSP, a novel unsupervised multimodal intent discovery method that leverages MLLM-guided concept generation and semantic propagation to achieve state-of-the-art performance and enhanced interpretability. By integrating high-level semantic priors from MLLMs into the clustering process, it overcomes the limitations of geometric-only approaches, offering a robust and transparent framework for uncovering latent structures in multimodal dialogues.
The paper proposes MCSP, a novel framework for unsupervised multimodal intent discovery that integrates Large Multimodal Language Models (MLLMs) into the clustering pipeline. The core innovation lies in a two-stage semantic refinement process: (1) MLLM-guided concept generation, where high-quality representative samples from initial clusters are used to prompt MLLMs (Gemini/Qwen) to generate discriminative semantic concepts via contrastive reasoning against neighboring clusters; and (2) semantic propagation, where these concepts are diffused across the data manifold using a concept-aware graph with semantically weighted edges. This approach addresses the "semantic opacity" of purely geometric clustering methods. The methodology is technically sound, leveraging recent advances in MLLMs to inject high-level semantic priors into unsupervised learning, which is a significant shift from traditional embedding-space optimization. The integration of random walk-based propagation with concept-supervised contrastive learning creates a robust feedback loop for representation refinement.
The experimental evaluation is comprehensive, covering three challenging multimodal intent discovery benchmarks: MIntRec, MIntRec2.0, and MELD-DA. The authors compare MCSP against strong baselines including SCCL, CC, MCN, USNID, SPILL, and UMC. Results demonstrate consistent improvements in ACC, ARI, NMI, and FMI metrics, with MCSP outperforming the current state-of-the-art UMC. The ablation studies effectively isolate the contribution of each component (pre-training, concept refinement, semantic propagation, confidence filtering), showing significant performance drops when key modules are removed. The sensitivity analysis for hyperparameters adds depth to the evaluation. The use of two different MLLMs (Qwen3-VL and Gemini-3.0-Pro) further validates the robustness of the approach.
The paper provides detailed descriptions of the architecture, loss functions, and hyperparameters. The datasets are standard and publicly available. However, the reliance on proprietary MLLMs (Gemini-Pro) and the specific prompt engineering details (though mentioned in the appendix) may introduce some variability in reproducibility across different API versions. The code is not explicitly linked in the provided text, which is a minor drawback for immediate reproducibility, but the methodological description is sufficient for implementation.
The primary limitation is the computational cost and latency associated with querying MLLMs for concept generation, although the authors note that this is done on a sparse set of representatives to mitigate cost. The method's performance is inherently tied to the reasoning capabilities of the underlying MLLM; if the MLLM fails to generate discriminative concepts (e.g., for highly ambiguous or pragmatic intents like sarcasm, as noted in the case study), the clustering performance may suffer. Additionally, the approach assumes that the initial clustering provides reasonable representatives, which might be challenging in very high-dimensional or noisy multimodal spaces without careful initialization.
This work significantly advances the field of multimodal understanding by making unsupervised intent discovery more interpretable and semantically grounded. The ability to generate human-readable concepts for latent clusters has broad implications for human-AI interaction, dialogue systems, and content recommendation, where understanding the "why" behind cluster assignments is crucial. It bridges the gap between low-level feature learning and high-level semantic reasoning, setting a new standard for interpretable multimodal clustering. The paper presents MCSP, a novel unsupervised multimodal intent discovery method that leverages MLLM-guided concept generation and semantic propagation to achieve state-of-the-art performance and enhanced interpretability. By integrating high-level semantic priors from MLLMs into the clustering process, it overcomes the limitations of geometric-only approaches, offering a robust and transparent framework for uncovering latent structures in multimodal dialogues.
Large Audio Language models (LALMs) have made rapid progress on acoustic understanding, yet they still struggle with fine-grained audio reasoning (e.g., recognizing event order, repetitions and duration). Existing post-training methods heavily rely on expensive external labels or provide only coarse semantic signals. To bridge this gap, we introduce Audio-Zero, the first label-free self-evolution framework in the field of LALMs that improves fine-grained auditory perception and reasoning. Audio-Zero constructs an auditory self-play game from unlabeled audio contrast pairs: most players hear a reference audio, while one odd listener hears a subtle variant. The model first generates clues describing what it hears and then identifies the odd listener by reasoning over inconsistencies among clues. Since the odd listener is known by construction, the game provides verifiable rewards without any annotated answers. Experiments with Qwen2-Audio-7B-Instruct and Qwen2.5-Omni-7B on TREA, MMAU Test-mini and MMAR show that Audio-Zero improves fine-grained audio reasoning while preserving broad audio understanding. Evolutionary and diagnostic analyses further reveal that increasingly fine-grained auditory descriptions emerge naturally from game pressure.
Primary: Institute of Computing Technology, Chinese Academy of Sciences
All Institutions: Institute of Acoustics, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing Academy of Artificial Intelligence, Institute of Computing Technology, Chinese Academy of Sciences, University of California, Merced, The University of Queensland
[One sentence main contribution]. Audio-Zero introduces a label-free self-evolution framework for LALMs that uses an auditory self-play game to improve fine-grained reasoning without external annotations. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a novel and technically sophisticated approach to enhancing the fine-grained reasoning capabilities of Large Audio Language Models. By formulating the learning process as a self-play game with verifiable intrinsic rewards, Audio-Zero effectively circumvents the bottleneck of scarce fine-grained annotations. The methodology is well-grounded in recent advances in self-evolution and reinforcement learning, and the experimental results demonstrate its efficacy across multiple benchmarks and model architectures. The diagnostic analysis provides valuable insights into how fine-grained auditory descriptions emerge from game pressure. This work is a significant contribution to the field of audio AI, offering a scalable and efficient path to improving model performance.
The paper proposes Audio-Zero, a label-free self-evolution framework for Large Audio Language Models (LALMs). The core innovation is an "auditory self-play game" that converts unlabeled audio contrast pairs into verifiable training signals. The method involves two coupled stages: a listening stage where the model generates descriptive clues for multiple "players" (some hearing a reference, one hearing a variant), and an attribution stage where the model acts as a judge to identify the "odd listener" based on inconsistencies in the clues. The reward signal is intrinsic to the game mechanics (correct identification) and a rule-based content utility score for the clues. The optimization uses Group Relative Policy Optimization (GRPO). The approach is theoretically sound and addresses a genuine gap in the field: the lack of fine-grained, label-free supervision for audio reasoning. The mechanism of using the "odd-one-out" task to force fine-grained perceptual description is clever and aligns well with recent trends in self-play and self-evolution for LLMs and VLMs.
The authors evaluate Audio-Zero on three benchmarks: TREA (temporal reasoning), MMAU Test-mini (general audio understanding), and MMAR (deep audio reasoning). Experiments are conducted on two strong backbones: Qwen2-Audio-7B-Instruct and Qwen2.5-Omni-7B. The results show consistent improvements over the base policies and several strong baselines (both label-dependent and label-free). Notably, the gains are most pronounced on TREA, the fine-grained temporal reasoning benchmark, which supports the paper's claim of improving fine-grained perception. The ablation studies effectively demonstrate the necessity of each component (listening stage, attribution stage, vote-aware feedback). The self-evolution dynamics analysis (tracking win rates, clue length, and benchmark scores over iterations) provides compelling evidence that the model is indeed learning and evolving rather than just overfitting to a static reward. The use of only 2k unlabeled pairs is a significant efficiency advantage.
The paper provides detailed implementation details, including hyperparameters (learning rate, KL coefficient, number of players/rounds), dataset construction (filtering 2k pairs from Audio-Alpaca), and prompt templates. The GRPO implementation details are standard but well-specified. The diagnostic metrics for analyzing clue quality are clearly defined. While the code is not explicitly linked in the provided text, the description is sufficient for reproduction by researchers familiar with RLHF/GRPO pipelines. The reliance on rule-based scorers for the listening reward is a potential point of variability, but the rules are described.
The method relies on the availability of audio contrast pairs. While the paper claims to use "unlabeled" pairs, it sources them from Audio-Alpaca, which is a preference dataset. The filtering process to create contrast pairs (ensuring they share semantics but differ in fine-grained details) is not fully detailed, raising questions about the scalability and generalizability of the pair generation process. The rule-based listening reward, while effective, may not capture all nuances of good auditory description compared to a learned reward model. Furthermore, the performance gains, while significant, are still modest in absolute terms on some benchmarks (e.g., MMAU Avg. 60.50 vs 57.50 for Qwen2-Audio), suggesting that self-evolution alone may not solve all reasoning deficits without some form of structured supervision or larger-scale data.
This work contributes to the development of more autonomous and data-efficient AI systems. By reducing reliance on expensive human annotations, it lowers the barrier to training high-capability audio models. The improved fine-grained audio reasoning capabilities could benefit applications in accessibility (e.g., better descriptions for the visually impaired), audio forensics, and interactive AI assistants. The self-play paradigm could be extended to other modalities or tasks where verifiable rewards are hard to construct. [One sentence main contribution]. Audio-Zero introduces a label-free self-evolution framework for LALMs that uses an auditory self-play game to improve fine-grained reasoning without external annotations. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a novel and technically sophisticated approach to enhancing the fine-grained reasoning capabilities of Large Audio Language Models. By formulating the learning process as a self-play game with verifiable intrinsic rewards, Audio-Zero effectively circumvents the bottleneck of scarce fine-grained annotations. The methodology is well-grounded in recent advances in self-evolution and reinforcement learning, and the experimental results demonstrate its efficacy across multiple benchmarks and model architectures. The diagnostic analysis provides valuable insights into how fine-grained auditory descriptions emerge from game pressure. This work is a significant contribution to the field of audio AI, offering a scalable and efficient path to improving model performance.
Large-scale visual generators are increasingly capable but costly to train, fine-tune, and deploy. We introduce Mage-Flow, a compact 4B-scale generative stack for efficient text-to-image generation and instruction-based image editing. The stack is built from two co-designed components: Mage-VAE, a lightweight high-fidelity latent tokenizer, and a Native-Resolution Multimodal Diffusion Transformer trained with rectified flow matching. Mage-VAE uses one-step diffusion-style encoding and decoding with anchor-latent regularization, preserving the reconstruction quality of strong public VAEs while reducing tokenization cost by more than an order of magnitude. Together with native-resolution packing and stack-level CUDA kernel fusion, the stack supports flexible-resolution training and improves end-to-end training throughput by about $2.5\times$. Built on this foundation, we develop a complete model family with Base, RL-aligned, and Turbo variants for both generation and editing. Diffusion-NFT improves prompt following, text rendering, aesthetic quality, and editing fidelity, while few-step distillation with adversarial perceptual guidance produces 4-step Turbo models for low-latency inference. Despite its compact scale, Mage-Flow and Mage-Flow-Edit achieves competitive performance across standard generation and editing benchmarks. More importantly, the Turbo variants make high-resolution generation and editing practical for interactive use: at $1024^2$ resolution on a single NVIDIA A100 GPU, Mage-Flow-Turbo generates an image in 0.59s, and Mage-Flow-Edit-Turbo edits an image in 1.02s, while maintaining a small memory footprint. These results show that careful tokenizer--backbone--system co-design can deliver strong high-resolution generation and editing within an efficient 4B model family.
Primary: Microsoft
All Institutions: Microsoft
Mage-Flow presents a compelling case for system-level co-design in generative AI, demonstrating that a compact 4B model family can achieve state-of-the-art efficiency and competitive quality in text-to-image generation and editing through innovative tokenizer design, native-resolution training, and kernel fusion.
The paper proposes a co-designed generative stack, Mage-Flow, centered on system-level efficiency rather than just model scaling. The core methodological contributions are threefold. First, the introduction of Mage-VAE, a lightweight latent tokenizer that replaces standard convolutional VAEs with one-step diffusion-style encoding and decoding. This is a significant architectural shift, treating the tokenizer as a learned image codec distilled from a strong teacher (FLUX.2-VAE) using anchor-latent regularization to preserve the generation-ready latent geometry. Second, the Native-Resolution MMDiT (NR-MMDiT) backbone abandons fixed-resolution bucketing in favor of native-resolution packing. By utilizing variable-length sequence packing with FlashAttention and per-sample 2D RoPE, the model handles arbitrary aspect ratios and resolutions natively, improving training diversity and inference efficiency for classifier-free guidance. Third, the paper details a comprehensive system-level optimization strategy, including stack-level CUDA kernel fusion to mitigate memory-bound overheads in the VAE, text encoder, and diffusion transformer. The approach is technically sound, leveraging recent advances in diffusion codecs (CoD-Lite) and efficient attention mechanisms to solve the latency bottleneck inherent in high-resolution generation.
The authors provide extensive evaluations covering reconstruction quality, inference speed, memory footprint, and downstream generation/editing performance. Mage-VAE demonstrates reconstruction fidelity comparable to FLUX.2-VAE while reducing computational cost by over an order of magnitude. The generation and editing benchmarks (GenEval, GEdit-Bench, TIIF, etc.) show that the 4B Mage-Flow models achieve competitive or superior performance against much larger open-source models (6B-80B parameters). The Turbo variants (4-step distillation) achieve impressive latency metrics (0.59s for generation, 1.02s for editing on a single A100), making interactive use practical. The ablation studies on tokenizer compatibility and native-resolution packing effectively validate the design choices. The results are robust and clearly demonstrate the quality-efficiency trade-off.
The paper provides significant details for reproduction, including the architecture of the VAE, the training objectives (flow matching, DMD, adversarial guidance), and the data curation pipeline. The release of code, models, and a project page enhances reproducibility. However, the specific hyperparameters for the anchor-latent KL regularization and the exact composition of the 1.3B curated dataset are not fully detailed in the text, though the curation pipeline is described qualitatively. The reliance on proprietary text encoders (Qwen3-VL) and the specific CUDA kernel implementations may require additional effort to replicate exactly, but the overall framework is well-documented.
The primary limitation is the reliance on a proprietary text encoder (Qwen3-VL) for conditioning, which prevents full open-source reproducibility of the conditioning pipeline. Additionally, while the VAE is efficient, the one-step diffusion decoding still requires more compute than standard convolutional decoders, though significantly less than multi-step approaches. The native-resolution packing, while efficient, may introduce complexity in distributed training synchronization compared to fixed-bucket approaches. The model's performance, while competitive, is still behind closed-source giants in absolute aesthetic quality, particularly in complex text rendering and fine-grained compositional control, which is expected given the 4B parameter scale.
This work significantly lowers the barrier to entry for high-quality, high-resolution image generation and editing by providing a compact, efficient, and research-friendly foundation model. By demonstrating that strong performance is achievable at 4B parameters through careful co-design, it challenges the trend of endless scaling and promotes sustainable AI development. The efficient inference capabilities enable deployment on consumer-grade hardware, democratizing access to advanced generative tools for developers and researchers. The focus on editing also supports creative workflows and iterative design processes. Mage-Flow presents a compelling case for system-level co-design in generative AI, demonstrating that a compact 4B model family can achieve state-of-the-art efficiency and competitive quality in text-to-image generation and editing through innovative tokenizer design, native-resolution training, and kernel fusion.
Test-time search lets small video diffusion models rival larger ones, but costs 2-10x more. All candidates are fully denoised, although most are discarded. Training-free caching makes each rollout 2-3x faster at near-lossless quality. Composition is safe only if lossy caching preserves verifier rankings. We present the first study of whether caching corrupts candidate ranking in video test-time search. On Wan2.1-T2V-1.3B with an adaptive caching wrapper (~2x per-candidate speedup), ImageReward scores seed-matched cached and full rollouts. Median per-prompt Spearman rank correlation is 0.905, with 72% top-1 agreement on the VBench suite. VBench-2.0 replicates this result on a harder suite. Recomputing the cached winner at full compute retains 90-94% of the full-search gain. Errors cluster among near-tied candidates, making corruption self-limiting. This finding leads to CachedSearch. It explores every candidate with aggressive caching, then re-generates only the winner at full compute. At N=8, it captures 94.7% of best-of-N's gain at 63% of the cost. Capture rises with width. At matched budget, it searches twice as wide for 38% more gain. The result holds from 1.3B-14B across six models and four families: Wan, LTX, CogVideoX, and Hunyuan. Wan2.1-14B matches the 1.3B model's fidelity. Mid-trajectory pruning multiplies the exploration saving to 3.11x at 88.6% capture. Ports to other model families require recalibrating a single parameter, showing that fidelity tracks architecture rather than parameter count. CachedSearch is training-free, verifier-agnostic, and orthogonal to the search algorithm, making it a plug-in multiplier for test-time scaling.
Primary: The University of Texas at Austin
All Institutions: The University of Texas at Austin, Google, Inc., University of Colorado Boulder
CachedSearch is a practical and theoretically grounded method that effectively decouples exploration from delivery in test-time search for video diffusion, achieving significant cost savings with minimal quality loss by leveraging the robustness of candidate rankings to caching-induced noise.
The paper proposes "CachedSearch," a training-free method that integrates inference-time caching (specifically an adaptive transformation-vector caching wrapper based on EasyCache) into test-time search (Best-of-N) for video diffusion models. The core insight is that while caching introduces approximation errors, these errors largely preserve the relative ranking of candidates under a verifier (e.g., ImageReward), provided the candidates are not near-tied. The method explores all N candidates using cheap cached rollouts, selects the winner based on cached scores, and then regenerates only the winning seed at full compute. This "explore-cheap, commit-full" strategy is orthogonal to the search algorithm and verifier. The methodology is sound, leveraging existing caching techniques in a novel compositional manner for search, rather than for single-sample generation.
The evaluation is comprehensive and rigorous. The authors conduct a detailed analysis of ranking preservation (Spearman correlation) across different caching thresholds and model families. They demonstrate that CachedSearch retains ~95% of the quality gain of full Best-of-N at ~63% of the cost for N=8. They validate this across six models (1.3B to 14B) and four architecture families (Wan, LTX, CogVideoX, Hunyuan). The paper includes ablation studies comparing caching against step truncation (showing caching is superior for ranking preservation) and composes with mid-trajectory pruning for multiplicative speedups. The use of VBench and VBench-2.0 suites adds robustness. The statistical analysis, including regret calculations and spread-correlation analysis, is thorough.
The paper provides a clear algorithm, detailed descriptions of the caching wrapper, and specific hyperparameters. The authors explicitly state they will release the caching wrapper, search harness, evaluation scripts, prompt lists, seeds, and per-candidate scores. The code repository is linked. The deterministic nature of the diffusion sampling (given seed and threshold) further aids reproducibility.
The method relies on the assumption that the verifier's ranking is robust to the specific type of noise introduced by caching. While the paper shows this holds for ImageReward and VideoScore, it notes that learned verifiers can be hacked or miss temporal artifacts (motion dampening in cached samples). The "commit" step mitigates this for the final output but adds latency. The method requires recalibration of the caching threshold ($\tau$) for different model families, though this is a minor overhead. The speedup is limited by the overhead of the wrapper and the fact that caching is not applicable to all layers/steps equally.
This work significantly lowers the inference cost of high-quality video generation, making test-time scaling more accessible. It contributes to the broader understanding of how approximation errors affect selection tasks in generative models. By enabling cheaper search, it could accelerate research in verifier-guided generation and make high-fidelity video synthesis more sustainable. CachedSearch is a practical and theoretically grounded method that effectively decouples exploration from delivery in test-time search for video diffusion, achieving significant cost savings with minimal quality loss by leveraging the robustness of candidate rankings to caching-induced noise.
Unsupervised multimodal intent discovery aims to uncover latent intents from unlabeled multimodal dialogues, but remains challenging due to the lack of explicit semantic supervision. Existing methods often provide limited interpretability, as their refinement mainly relies on geometric similarity rather than high-level semantic guidance. To address these limitations, we propose MCSP, a fully unsupervised method that introduces semantic refinement based on concepts into multimodal intent discovery. To obtain reliable semantic evidence for intent discovery, we identify high-quality representative samples for each cluster and use them to support MLLM-guided contrastive reasoning against neighboring clusters, which produces interpretable high-level semantic concepts. Building on these concepts, we perform semantic propagation over a semantically weighted graph to align conceptual information with local structural consistency and generate reliable pseudo-labels for representation refinement. Extensive experiments on three challenging multimodal intent datasets show that MCSP consistently outperforms state-of-the-art methods while producing interpretable clusters grounded in semantic concepts.
Primary: The Chinese University of Hong Kong, Shenzhen
All Institutions: The Chinese University of Hong Kong, Shenzhen, Tsinghua University
The paper presents MCSP, a novel unsupervised multimodal intent discovery method that leverages MLLM-guided concept generation and semantic propagation to achieve state-of-the-art performance and enhanced interpretability. By integrating high-level semantic priors from MLLMs into the clustering process, it overcomes the limitations of geometric-only approaches, offering a robust and transparent framework for uncovering latent structures in multimodal dialogues.
The paper proposes MCSP, a novel framework for unsupervised multimodal intent discovery that integrates Large Multimodal Language Models (MLLMs) into the clustering pipeline. The core innovation lies in a two-stage semantic refinement process: (1) MLLM-guided concept generation, where high-quality representative samples from initial clusters are used to prompt MLLMs (Gemini/Qwen) to generate discriminative semantic concepts via contrastive reasoning against neighboring clusters; and (2) semantic propagation, where these concepts are diffused across the data manifold using a concept-aware graph with semantically weighted edges. This approach addresses the "semantic opacity" of purely geometric clustering methods. The methodology is technically sound, leveraging recent advances in MLLMs to inject high-level semantic priors into unsupervised learning, which is a significant shift from traditional embedding-space optimization. The integration of random walk-based propagation with concept-supervised contrastive learning creates a robust feedback loop for representation refinement.
The experimental evaluation is comprehensive, covering three challenging multimodal intent discovery benchmarks: MIntRec, MIntRec2.0, and MELD-DA. The authors compare MCSP against strong baselines including SCCL, CC, MCN, USNID, SPILL, and UMC. Results demonstrate consistent improvements in ACC, ARI, NMI, and FMI metrics, with MCSP outperforming the current state-of-the-art UMC. The ablation studies effectively isolate the contribution of each component (pre-training, concept refinement, semantic propagation, confidence filtering), showing significant performance drops when key modules are removed. The sensitivity analysis for hyperparameters adds depth to the evaluation. The use of two different MLLMs (Qwen3-VL and Gemini-3.0-Pro) further validates the robustness of the approach.
The paper provides detailed descriptions of the architecture, loss functions, and hyperparameters. The datasets are standard and publicly available. However, the reliance on proprietary MLLMs (Gemini-Pro) and the specific prompt engineering details (though mentioned in the appendix) may introduce some variability in reproducibility across different API versions. The code is not explicitly linked in the provided text, which is a minor drawback for immediate reproducibility, but the methodological description is sufficient for implementation.
The primary limitation is the computational cost and latency associated with querying MLLMs for concept generation, although the authors note that this is done on a sparse set of representatives to mitigate cost. The method's performance is inherently tied to the reasoning capabilities of the underlying MLLM; if the MLLM fails to generate discriminative concepts (e.g., for highly ambiguous or pragmatic intents like sarcasm, as noted in the case study), the clustering performance may suffer. Additionally, the approach assumes that the initial clustering provides reasonable representatives, which might be challenging in very high-dimensional or noisy multimodal spaces without careful initialization.
This work significantly advances the field of multimodal understanding by making unsupervised intent discovery more interpretable and semantically grounded. The ability to generate human-readable concepts for latent clusters has broad implications for human-AI interaction, dialogue systems, and content recommendation, where understanding the "why" behind cluster assignments is crucial. It bridges the gap between low-level feature learning and high-level semantic reasoning, setting a new standard for interpretable multimodal clustering. The paper presents MCSP, a novel unsupervised multimodal intent discovery method that leverages MLLM-guided concept generation and semantic propagation to achieve state-of-the-art performance and enhanced interpretability. By integrating high-level semantic priors from MLLMs into the clustering process, it overcomes the limitations of geometric-only approaches, offering a robust and transparent framework for uncovering latent structures in multimodal dialogues.
Joint Embedding Predictive Architectures (JEPA) have recently emerged as a paradigm for learning world models by predicting latent representations, offering a promising direction for self-supervised learning. While initial attempts have applied JEPA to the music domain, it remains unclear how such frameworks can naturally support the formation of a world model for music. In this work, we propose to learn a world model of piano sound using JEPA by framing music as an action-conditioned system: the audio is treated as the state, and the pianoroll as the instrument action. Given a current audio state and an action, the model predicts the resulting future audio state, mirroring how humans learn musical sound through interaction. The model is trained in a fully offline setting using paired audio-pianoroll data, without environment interaction. Experiments show that the learned model captures the relationships between musical actions and their resulting sound. The resulting representations support downstream tasks, including beat tracking, composer identification, and key estimation, and enable piano transcription via planning, by searching for actions that best explain a target sound.
Primary: McGill University
All Institutions: McGill University, Advanced Machine Intelligence (AMI Labs), Centre for Interdisciplinary Research in Music Media and Technology (CIRMMT), Mohamed bin Zayid University of Artificial Intelligence (MBZUAI), Schulich School of Music
[One sentence main contribution]. The paper introduces Music-JEPA, an action-conditioned world model that learns latent dynamics of piano music by predicting future audio states from current states and pianoroll actions. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. This work makes a significant contribution to self-supervised learning in audio by moving beyond passive representation learning to active, action-conditioned modeling. The technical execution is solid, with clear demonstrations of learned dynamics and competitive performance on downstream tasks. While the novelty is incremental (applying known JEPA techniques to a new domain with a specific conditioning mechanism), the significance lies in the rigorous evaluation of action-conditioned latent spaces for music, providing a strong baseline for future research in controllable music generation and understanding. The limitation of requiring paired data is a notable constraint, but the results validate the potential of this paradigm.
The paper proposes Music-JEPA, an action-conditioned Joint Embedding Predictive Architecture for piano music. The core innovation lies in framing music as a dynamical system where audio spectrograms are states and pianoroll/pedal signals are actions. The model predicts future latent states given current states and actions, leveraging the EMA-based training scheme to prevent collapse. The methodology is technically sound, adapting recent vision JEPA techniques to the audio domain with a specific focus on action-conditioning. The use of ViT backbones for both spectrogram and pianoroll inputs is standard but effective. The introduction of an inverse predictor for transcription via planning is a notable conceptual extension, although the implementation relies on amortized optimization rather than direct latent-space planning, which is a pragmatic choice given the high dimensionality of the action space.
The experimental evaluation is comprehensive, covering temporal dynamics analysis, downstream MIR tasks (beat tracking, composer identification, key recognition), and transcription via planning. The results show that Music-JEPA outperforms a passive audio-only JEPA baseline and remains competitive with MERT, a much larger model, on several tasks. The ablation studies on perturbation sensitivity provide strong evidence that the model learns meaningful action-conditioned dynamics rather than just temporal proximity. The transcription results are mixed; while pedal estimation is strong, note-level transcription lags behind supervised methods, which is expected for a generative/planning approach but highlights current limitations. The comparison with MERT is fair and demonstrates the efficiency of the proposed approach.
The paper provides detailed hyperparameters, dataset splits (MAESTRO v3.0.0), and architectural details. The training algorithm is clearly outlined. However, the code and pretrained models are stated to be "released soon," which currently hinders immediate reproducibility. The specific patch sizes and embedding dimensions are provided, allowing for architectural replication.
The primary limitation is the reliance on paired audio-MIDI data, which restricts the model to domains with precise action annotations (like piano) and prevents direct application to unannotated audio. The transcription performance, particularly for notes, is inferior to supervised methods, indicating that the latent space may not be perfectly invertible or that the planning objective is not yet optimized for high-fidelity reconstruction. The model's ability to generalize to other instruments or genres is untested. Additionally, the "planning" aspect is implemented via an inverse model (amortized planning) rather than search-based planning, which limits the interpretability of the "world model" aspect in terms of explicit trajectory optimization.
This work contributes to the growing field of world models for audio and music information retrieval. By demonstrating that action-conditioned latent dynamics can capture musical structure, it opens avenues for more interactive and controllable music AI systems. The approach could inspire similar frameworks for other domains where actions influence observations, such as robotics or speech synthesis. The focus on self-supervised learning with minimal labeled data aligns with broader trends in efficient AI. [One sentence main contribution]. The paper introduces Music-JEPA, an action-conditioned world model that learns latent dynamics of piano music by predicting future audio states from current states and pianoroll actions. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. This work makes a significant contribution to self-supervised learning in audio by moving beyond passive representation learning to active, action-conditioned modeling. The technical execution is solid, with clear demonstrations of learned dynamics and competitive performance on downstream tasks. While the novelty is incremental (applying known JEPA techniques to a new domain with a specific conditioning mechanism), the significance lies in the rigorous evaluation of action-conditioned latent spaces for music, providing a strong baseline for future research in controllable music generation and understanding. The limitation of requiring paired data is a notable constraint, but the results validate the potential of this paradigm.
Real-time score following from sheet images remains chal- lenging because the model must process streaming au- dio while resolving highly repetitive visual patterns un- der strict latency constraints. Recent image-based meth- ods have attempted to use multi-resolution prediction by simultaneously predicting the positions of the active sys- tem, bar, and note. However, their predictions across these different levels of notation are independent, which makes the predictions unstable and introduces unnecessary ex- tra search space for bar- and note-level predictions. Most existing methods also lack mechanisms to recover from score discontinuities, such as repeats, da capo (D.C.), or coda jumps. This paper proposes CODA, to the best of our knowledge, the first real-time score following system that addresses both gaps. CODA explicitly exploits the cascaded structure of music scores: it first selects the ac- tive system, then the active bar within it, and finally the active note within the selected bar. This enforces pre- diction consistency across resolutions. A silence-driven break mode enables recovery from arbitrary score discon- tinuities without requiring knowledge of the repeat struc- ture. Evaluated on the Multimodal Sheet Music Dataset (MSMD) piano benchmarks, CODA achieves state-of-the- art tracking accuracy and discontinuity-recovery perfor- mance under real-time throughput. Code is available at https://github.com/ValleyC/CODA.
Primary: University of Alberta
All Institutions: University of Alberta, Western University
This paper presents a significant methodological advancement in real-time image-based score following by introducing a cascaded selection architecture and a novel silence-driven recovery mechanism, effectively addressing the critical challenges of geometric consistency and score discontinuity handling.
The paper proposes CODA, a cascaded online alignment framework for real-time image-based score following. The core methodological innovation is the reformulation of the tracking problem from independent multi-resolution object detection (as seen in prior work like CYOLO-SB) to a cascaded selection-and-regression task. By explicitly predicting the system, then the bar within that system, and finally the note within that bar, the model enforces geometric consistency and significantly reduces the search space at each stage. The architecture utilizes a causal Mamba encoder for audio processing, which efficiently handles streaming data with linear complexity, paired with a CNN visual backbone. A key technical contribution is the "silence-driven break mode," which leverages the observation that score discontinuities (repeats, da capo, coda jumps) are often preceded by silence. This allows the system to reset its temporal priors and re-localize without requiring explicit symbolic knowledge of the score's repeat structure. The use of beam search with learned temporal priors further stabilizes the predictions.
The evaluation is conducted on the Multimodal Sheet Music Dataset (MSMD), a standard benchmark for this task. The authors report state-of-the-art results in tracking accuracy on both synthetic and real-audio settings compared to strong baselines like CYOLO-SB. Crucially, they introduce a new "repeat-aware jump test benchmark" with manually annotated repeat structures for the 94 test pieces, addressing a significant gap in the literature where discontinuity handling was previously unstandardized. The results show substantial improvements in system and bar accuracy, and notably, CODA demonstrates robust recovery from jumps, whereas baselines fail. The ablation studies effectively isolate the contributions of the cascade structure, beam search, and cross-attention mechanisms.
The paper provides detailed implementation details, including audio preprocessing (log-filterbank, hop size), model architecture dimensions, training schedules (two-phase curriculum with scheduled sampling), and hyperparameters. The code is made publicly available on GitHub. The dataset (MSMD) is publicly accessible. The inclusion of a specific benchmark for jump recovery enhances reproducibility for future work in this niche area.
The method is evaluated exclusively on solo piano music from the MSMD dataset. The performance on more complex musical textures (chamber music, orchestral) or different instruments is not reported. The reliance on synthesized audio for the primary evaluation setting (Setting I) may not fully capture the challenges of real-world acoustic recordings, although Setting II provides some real-audio validation. The "silence" assumption for jump recovery, while effective for many classical pieces, might fail in continuous musical passages where performers execute jumps without significant pauses.
This work advances the field of Human-Computer Interaction (HCI) in music, specifically interactive accompaniment and automated page-turning systems. By enabling robust real-time score following from images without requiring symbolic scores, it lowers the barrier to entry for using digital sheet music in performance settings. The standardized benchmark for discontinuity handling will facilitate fairer comparisons in future research. This paper presents a significant methodological advancement in real-time image-based score following by introducing a cascaded selection architecture and a novel silence-driven recovery mechanism, effectively addressing the critical challenges of geometric consistency and score discontinuity handling.
Omni models transcribe clean, single-speaker speech well, but their accuracy drops sharply when speakers overlap and the scene is noisy, exactly where knowing who said what matters most. A natural fix is a short scene description. We show why this is risky: answer-bearing text lets the model copy instead of listen, so the score rises although nothing has been heard; a silent test exposes this shortcut at once. We call this failure mode perception bypass and address it with Audio-Grounded Scaffold Context (AGSC). AGSC links three steps: first, we build clues from audio to guide listening without giving the answer; second, answer-overlap and silence tests probe them for leakage and audio dependence; finally, those clues scaffold training but vanish at test time, yielding no-clue capability. Across three heterogeneous Omni models, training on AGSC lowers no-clue capped mean permutation word error rate (mpWER) on overlapping, noisy speech from 25%-71% to 9%-15%. For streaming control, we formulate a joint GDPO task in which the model learns when to use a clue and how to produce a speaker-attributed transcript from separately normalized format, gate, and transcript rewards. After internalization, AGSC adds almost no inference overhead.
Primary: unknown
All Institutions: unknown
This paper introduces AGSC, a training framework that uses audio-grounded scaffolds and a novel "silent test" evaluation to prevent perception bypass in Omni models, significantly improving robustness in overlapping and noisy speech scenarios through internalized learning.
The paper proposes Audio-Grounded Scaffold Context (AGSC), a framework designed to mitigate "perception bypass" in Omni models—where models rely on textual context rather than audio signals. The methodology involves synthesizing "clues" (partial, shuffled words, time windows) from audio using off-the-shelf front-ends (diarization, ASR, VAD). These clues are used during Supervised Fine-Tuning (SFT) and Group Reward-Decoupled Policy Optimization (GDPO). The core innovation lies in the training protocol: using a "silent test" (silencing the audio input) to detect if the model is copying text, and ensuring that performance gains persist even when the scaffold clues are removed at inference time (internalization). The GDPO component introduces a novel reward structure with separate normalization for format, gate, and transcript rewards, and a conditional reward mechanism to prevent gate collapse. While the concept of using scaffolds is not entirely new, the specific application to preventing perception bypass in Omni models and the rigorous "silent test" evaluation metric constitute a meaningful methodological contribution to robust multimodal training.
The authors evaluate their approach on three heterogeneous Omni models (Qwen3-Omni, MiniCPM-o, Ming-flash-omni) using a custom benchmark, Context-Speech Bench (CSB), which includes overlapping and noisy speech scenarios. The results show significant improvements in mean permutation Word Error Rate (mpWER) for the smaller models (Ming, MiniCPM) after SFT, reducing errors from ~25-68% to ~8-15%. The paper provides a thorough ablation study, including the "silent test" to prove that gains are not due to text copying. The GDPO experiments demonstrate the ability to learn gating strategies, although the full-chain joint training shows some instability (gate collapse) which is addressed with a secondary supervised step. The evaluation is comprehensive, covering both offline transcription and streaming control, and the use of multiple models strengthens the generalizability claims.
The paper states that code, benchmark manifests, and result evidence will be open-sourced upon acceptance. The methodology describes the pipeline components (Pyannote, Silero, Sepformer) and the training hyperparameters (LoRA ranks, learning rates). However, the reliance on specific versions of large base models (Qwen3, MiniCPM, Ming) and the complex synthesis pipeline for clues may pose reproducibility challenges. The description of the GDPO reward normalization and the specific "silent test" protocol is detailed enough to allow replication by researchers familiar with RLHF techniques.
The paper acknowledges that the external detector for gating has lower performance than the learned gate but notes the learned gate can suffer from collapse. The improvements are less pronounced for the strongest baseline model (Qwen3), suggesting diminishing returns for already high-performing systems. The "clue" generation relies on imperfect off-the-shelf tools, which might introduce noise or errors that the model must learn to ignore, potentially limiting the ceiling of performance. Furthermore, the "silent test" is an aggregate control and may not catch all forms of shortcut learning. The paper also notes that unconditional clue usage at test time can hurt performance, highlighting the necessity of the learned gate.
This work addresses a critical failure mode in multimodal AI systems, enhancing their reliability in real-world scenarios like meetings and call centers where audio is often noisy or overlapping. By preventing "perception bypass," the method ensures that models are genuinely listening, which is crucial for trust and safety in automated transcription and assistance systems. The Context-Speech Bench provides a valuable resource for evaluating robustness in challenging acoustic conditions. This paper introduces AGSC, a training framework that uses audio-grounded scaffolds and a novel "silent test" evaluation to prevent perception bypass in Omni models, significantly improving robustness in overlapping and noisy speech scenarios through internalized learning.
Audio deepfake detectors are trained to distinguish genuine speech from synthetic speech and often perform well on standard benchmarks. Yet the same detector that achieves less than 1% error on one dataset can see its error rate increase twentyfold when evaluated on a different dataset. We argue that one contributing factor is speaker-identity reliance: standard training corpora correlate speaker identity with the genuine/synthetic label, allowing detectors to partially rely on speaker-related cues rather than synthesis artifacts alone. We propose the Identity Sensitivity Score (ISS), a per-utterance diagnostic that quantifies how much a detector's output changes across different speaker identity contexts. ISS requires no ground-truth labels at inference time and can be computed from the detector score and a pool of reference speaker examples. Across two detectors and two datasets, incorrectly classified utterances have ISS scores 29 to 52 times higher than correctly classified utterances, and ISS alone predicts misclassification with area-under-curve (AUC) up to 0.954. To test whether ISS actually captures identity-sensitive behavior rather than serving only as a proxy for prediction confidence, we apply voice conversion to 500 utterances and measure the resulting detector-score shift. Utterances flagged as identity-sensitive by ISS respond 19 to 30 times more strongly to this manipulation than utterances flagged as stable. These results position ISS as a practical inference-time diagnostic for speaker-dependent failure analysis in audio deepfake detection.
Primary: Michigan State University
All Institutions: Michigan State University
The paper introduces the Identity Sensitivity Score (ISS), a label-free diagnostic metric that quantifies speaker-identity reliance in audio deepfake detectors, demonstrating that identity-sensitive predictions are strongly correlated with misclassification and distributional shift failures.
The paper proposes the Identity Sensitivity Score (ISS), a novel diagnostic metric designed to quantify the extent to which an audio deepfake detector relies on speaker identity cues rather than synthesis-specific artifacts. The methodology is theoretically grounded in the sensitivity of sigmoid decision boundaries; it injects speaker identity context via logit-space perturbation using external ECAPA-TDNN embeddings. This approach is clever because it is label-free at inference time, requiring only the detector's output and a pool of reference speaker embeddings. The method avoids the computational cost of generating new audio samples for probing, instead manipulating the internal representation space. The theoretical motivation linking ISS to decision-boundary proximity provides a strong mathematical justification for why high-ISS scores correlate with misclassification, distinguishing it from standard uncertainty measures like entropy or margin which are monotonic functions of the prediction score itself.
The experimental evaluation is rigorous and well-structured. The authors evaluate ISS across two architecturally distinct detectors (AASIST and RawNet2) and two datasets (ASVspoof 2019 LA and 2021 LA), demonstrating the metric's robustness across different model families. The core finding—that incorrectly classified utterances have ISS scores 29 to 52 times higher than correctly classified ones—is compelling. The use of Voice Conversion (FreeVC) as a controlled intervention to validate that ISS captures genuine identity sensitivity rather than just boundary proximity is a strong empirical contribution. The statistical significance is well-established with Mann-Whitney U tests and Cohen's d. The analysis of attack types (TTS vs. VC vs. Hybrid) adds depth, revealing that different failure modes (identity-driven vs. artifact-driven) can be distinguished by ISS, a nuance lost in aggregate metrics.
The paper provides sufficient detail for reproduction. The datasets (ASVspoof 2019/2021 LA) are standard benchmarks. The detectors (AASIST, RawNet2) are widely available. The external speaker model (ECAPA-TDNN) is also standard. The specific configuration parameters ($K=10$, $\lambda=5.0$) are provided. However, the paper notes that $\lambda$ tuning requires a small labeled development set, which is a minor practical hurdle but does not prevent reproduction. The code is not explicitly linked in the provided text, but the methodology is clear enough that implementation is straightforward.
The authors correctly identify several limitations. ISS relies on an external speaker embedding model, meaning it measures sensitivity to identity as represented in ECAPA-TDNN's space, which may not perfectly align with the deepfake detector's internal identity representations. The method requires a small labeled development set for hyperparameter tuning ($\lambda$), although this is a one-time offline step. The evaluation is limited to two detectors and two datasets; extending to SSL-based detectors (e.g., Wav2Vec2) is noted as future work. Furthermore, while ISS diagnoses the *cause* of potential failure (identity reliance), it does not inherently correct it; training interventions to mitigate this reliance are outside the scope of this work.
This work has significant implications for the reliability and trustworthiness of audio deepfake detection systems, which are critical for security, forensics, and media integrity. By providing a tool to diagnose *why* a detector fails (identity bias vs. artifact ambiguity), ISS enables more targeted improvements in model training and deployment. It highlights a systemic issue in current benchmarks where speaker identity correlates with labels, potentially leading to over-optimistic performance estimates. This diagnostic capability can help researchers build more robust, generalizable detectors that rely on genuine synthesis artifacts rather than spurious correlations. The paper introduces the Identity Sensitivity Score (ISS), a label-free diagnostic metric that quantifies speaker-identity reliance in audio deepfake detectors, demonstrating that identity-sensitive predictions are strongly correlated with misclassification and distributional shift failures.
We present VibeVoice-ASR-BitNet, a compressed variant of VibeVoice-ASR optimized for real-time inference on edge CPUs. We apply heterogeneous quantization tailored to the computational characteristics of each stage: the VAE acoustic tokenizer uses full-pipeline INT8 quantization (I8_S) with kernel fusion and SIMD optimization, while the autoregressive language model adopts BitNet-style ternary weights (I2_S). To preserve accuracy under aggressive compression, we employ a progressive quantization-aware training strategy. For inference, we implement custom SIMD kernels and fused operators within the ggml framework targeting both ARM and x86 platforms, achieving real-time recognition with RTF < 1 using as few as 3 CPU threads. VibeVoice-ASR-BitNet is 1.6-2.3x faster than Whisper.cpp at comparable model sizes (~1.6 GB), with only modest accuracy degradation compared to the FP16 baseline.
Primary: Microsoft Research
All Institutions: Microsoft Research, Shanghai Jiao Tong University, Fudan University, University of Chinese Academy of Sciences
This paper presents a highly effective engineering solution for deploying large LLM-based ASR models on edge CPUs through heterogeneous quantization (INT8 for VAE, ternary for LM) and custom SIMD optimization, achieving real-time performance with minimal accuracy loss and significant compression.
The paper proposes a heterogeneous quantization strategy for the VibeVoice-ASR architecture, specifically targeting edge CPU deployment. The core technical contribution lies in the distinct treatment of the two main components: the VAE tokenizer is quantized to full-pipeline INT8 (I8_S) to address its activation-bound nature, while the autoregressive language model (LM) decoder utilizes BitNet-style ternary weights (I2_S) to address its weight-bound nature during decoding. The authors introduce a progressive quantization-aware training (QAT) schedule using a blending parameter to stabilize training, which is a practical engineering contribution. The inference engine is built on ggml with custom SIMD kernels (AVX2/NEON) for efficient INT8 and ternary (unpacking to INT8 for MAD operations) computation. The methodology is sound and well-reasoned, leveraging established techniques (BitNet, ggml, INT8 quantization) in a novel combination specific to the VAE-LM ASR paradigm.
The evaluation is conducted on a server-grade AMD EPYC CPU, which serves as a proxy for high-end edge hardware, though real-world edge devices (e.g., mobile SoCs) may have different memory bandwidth and cache characteristics. The paper reports a 2.9x compression ratio (4.62 GB to 1.58 GB) and achieves Real-Time Factor (RTF) < 1 with only 3 threads, demonstrating significant efficiency gains over FP16 baselines and competitive performance against Whisper.cpp. Accuracy is evaluated on multiple multilingual benchmarks (MLC, AISHELL, etc.), showing modest degradation (1-4%) compared to the larger FP16 baseline. The comparison with Whisper.cpp is relevant and highlights the benefits of the specific heterogeneous quantization approach for LLM-based ASR.
The paper provides a GitHub link to the codebase (VibeASR.cpp) and Hugging Face links to the models. The methodology describes the quantization schemes, training schedules, and kernel optimizations in sufficient detail for replication by experts in the field. The use of standard frameworks (ggml) and well-defined quantization protocols enhances reproducibility.
The primary limitation is the lack of streaming support, as noted by the authors. The current implementation is batch-oriented, which limits its applicability in real-time interactive scenarios where low latency is critical. Additionally, the evaluation is performed on a specific server CPU; performance on actual resource-constrained edge devices (ARM-based mobile chips, etc.) is not explicitly detailed, although ARM support is mentioned. The generalizability of the heterogeneous strategy to other VAE-LM architectures (e.g., Whisper) is also unexplored.
This work significantly advances the field of on-device AI by demonstrating that large, accurate LLM-based ASR systems can be deployed on commodity CPUs without GPUs. This has profound implications for privacy-preserving speech recognition, reduced cloud infrastructure costs, and enabling advanced voice assistants on low-power devices. It bridges the gap between high-accuracy LLM-based ASR and practical, efficient edge deployment. This paper presents a highly effective engineering solution for deploying large LLM-based ASR models on edge CPUs through heterogeneous quantization (INT8 for VAE, ternary for LM) and custom SIMD optimization, achieving real-time performance with minimal accuracy loss and significant compression.
Transformers with global attention capture long-range dependencies but can miss the fine-grained local continuity crucial for speech separation. We propose TF-MossFormer, a time-frequency transformer that combines local and global attention to jointly model short- and long-range contexts for monaural speech separation. At its core is a content-aware sliding-window attention mechanism that dynamically adapts receptive fields for stronger local interactions, avoiding the rigidity of static convolutions. Unlike time-domain chunk-based methods, TF-MossFormer leverages the 2D spectrogram to model structure along both time and frequency axes. Convolutional gating between attention layers further improves feature selection and information flow. TF-MossFormer achieves SI-SDRi of 22.6, 24.0, and 24.4 dB on WSJ0-2Mix with 5.9M, 16.9M, and 25.4M parameters, respectively, outperforming prior approaches.
Primary: Alibaba Group
All Institutions: Alibaba Group, Token Foundry
TF-MossFormer achieves state-of-the-art performance on WSJ0-2Mix by effectively combining sliding-window local attention with global self-attention and convolutional gating, offering a robust and efficient solution for monaural speech separation.
The paper proposes TF-MossFormer, a time-frequency domain architecture for monaural speech separation. The core innovation lies in integrating a content-aware sliding-window local attention mechanism with global self-attention, augmented by convolutional gating. This hybrid approach aims to balance the computational efficiency and local feature preservation of sliding windows with the long-range dependency modeling of global attention. The architecture follows a dual-path (time-frequency) design, processing spectro-temporal features through stacked blocks. While the combination of local and global attention is not entirely new in the broader transformer literature (e.g., Longformer, BigBird), its specific adaptation to the 2D spectrogram structure with convolutional gating for speech separation is a targeted and reasonable engineering contribution. The use of Conv-SwiGLU and RMSGroupNorm indicates a solid understanding of modern transformer stabilization techniques.
The experimental evaluation is conducted on the standard WSJ0-2Mix benchmark. The authors report results for three model scales (Small, Medium, Large) with parameter counts ranging from 5.9M to 25.4M. The results demonstrate state-of-the-art performance (SI-SDRi up to 24.4 dB) compared to prior works like TF-LocoFormer, TF-GridNet, and SepTDA2. The ablation studies provide valuable insights into the hyperparameters (window sizes) and the architectural choices (order of local/global attention, presence of gating). The comparison with SPMamba highlights efficiency gains. However, the evaluation is limited to a single dataset (WSJ0-2Mix), which restricts the generalizability claims. The computational complexity (MACs) is reported, showing competitive efficiency, which is a strong point.
The paper provides sufficient implementation details, including STFT parameters (16ms window, 8ms hop), optimizer settings (AdamW, learning rate schedule), and loss function (SI-SDR). The architecture diagrams and block descriptions are clear. The use of the ESPnet pipeline is mentioned, which aids reproducibility for those familiar with the framework. However, the specific code repository is not linked in the text provided, and some hyperparameters for the ablation studies (e.g., exact dimension $D$ for the medium/large models) are referenced but not explicitly detailed in the text snippets provided (though implied by parameter counts).
The primary limitation is the lack of evaluation on other benchmarks (e.g., LibriMix, WHAM!, or real-world noisy mixtures), which limits the assessment of robustness. The "content-aware" nature of the sliding window is mentioned but the mechanism for dynamic adaptation is not deeply elaborated in the provided text, raising questions about whether it is truly dynamic or just a fixed window with a specific size. The focus on WSJ0-2Mix, while standard, does not demonstrate performance on more challenging, diverse datasets.
This work contributes to the field of audio signal processing, specifically speech separation, which has applications in hearing aids, voice assistants, and audio post-production. By improving the efficiency and quality of TF-domain separation, it supports the development of more accessible and effective speech processing tools. The hybrid attention mechanism could inspire similar architectures in other sequential data domains. TF-MossFormer achieves state-of-the-art performance on WSJ0-2Mix by effectively combining sliding-window local attention with global self-attention and convolutional gating, offering a robust and efficient solution for monaural speech separation.
We present an agentic soundscape construction framework for controllable compositional audio generation that makes explicit the scene planning, source selection, temporal layout, and rendering steps typically handled implicitly by single-shot text-to-audio models. An LLM-based agent converts user intent into an executable scene plan, acquires assets through retrieval and on-demand generation, renders controllable multi-event mixtures, and exports aligned scene metadata. The framework also supports human-in-the-loop interaction through user-guided tool selection and editable scene plans. Together, these components provide an inspectable and reusable approach to controllable soundscape synthesis and scalable audio-language data construction. Listener studies and objective metrics demonstrate competitive generation performance against text-to-audio baselines, while models trained with agent-generated data consistently outperform real-only baselines in downstream audio reasoning. Code, demos, and listening-test results are available at https://haozhang6720.github.io/SoundscapeAgentDemoPage/.
Primary: Tencent Hunyuan
All Institutions: Wuhan University, Tencent Hunyuan
SoundscapeAgent presents a structured, agentic approach to soundscape synthesis that enables controllable generation and provides valuable structured supervision for audio-language reasoning tasks. The paper demonstrates that explicit scene planning and hybrid asset acquisition can produce high-quality, multi-event soundscapes that outperform single-shot baselines in complex scenarios and effectively augment training data for downstream audio reasoning models.
The paper proposes SoundscapeAgent, an agentic framework that decomposes soundscape synthesis into explicit planning, asset acquisition, and rendering stages. The core innovation lies in using an LLM to generate a structured scene plan (events, roles, timing, gain) which is then executed by deterministic rendering primitives. This contrasts with end-to-end text-to-audio models by offering inspectability and controllability. The methodology combines retrieval from a curated asset library with on-demand generation for missing assets, followed by a timeline-based mixer. While the concept of "agentic" audio is emerging, the specific implementation here is a composite of existing techniques (LLM planning + T2A generation + audio mixing) rather than a novel algorithmic breakthrough. The novelty is moderate, primarily in the system-level integration and the specific focus on structured supervision for downstream reasoning.
The evaluation is split into two tracks. Track A compares generation quality against baselines (TangoFlux, EzAudio, AudioLDM2) using human listening tests and CLAP scores. The results show competitive performance, particularly in complex, multi-event, or affective scenes, validating the controllability aspect. Track B evaluates the utility of the generated data for training audio-language models on the MMAU benchmark. The results demonstrate that adding agent-generated data improves reasoning accuracy, especially in sound and music domains, over real-only baselines. This is a strong empirical result supporting the value of structured synthetic data. However, the downstream model uses a frozen encoder and only trains a small projector, which limits the assessment of the data's full potential compared to fine-tuning larger models. The evaluation is solid but relies on standard benchmarks and metrics.
The paper provides a demo page with code links (implied by the URL, though the specific GitHub repo isn't explicitly named in the text, the URL points to a project page). The methodology is described in detail, including the asset filtering criteria (CLAP score, production complexity) and the rendering pipeline. The use of open-source models (Qwen, Whisper, CLAP) aids reproducibility. However, the "private in-house data" mentioned in the asset library is not accessible, which may limit the exact replication of the asset acquisition phase. The prompt engineering for the LLM agent is not fully detailed, which could affect reproducibility of the scene plans.
The authors acknowledge limitations including dependence on asset inventory quality, the domain gap between synthetic and real audio, and the bias towards non-speech events in the asset library. The evaluation of speech capabilities is limited. The reliance on an LLM for planning introduces potential latency and error propagation if the plan is flawed. The "offline prior mode" improves throughput but may reduce the adaptability of the generation to specific user intents compared to online planning.
This work contributes to the field of audio-language modeling by providing a scalable method for generating structured, high-quality synthetic data for training. It addresses the need for fine-grained temporal and compositional supervision in audio understanding. The agentic approach could inspire similar frameworks in other multimodal domains. However, the generation of synthetic audio data raises concerns about data contamination and the potential for propagating biases present in the underlying models or asset libraries. SoundscapeAgent presents a structured, agentic approach to soundscape synthesis that enables controllable generation and provides valuable structured supervision for audio-language reasoning tasks. The paper demonstrates that explicit scene planning and hybrid asset acquisition can produce high-quality, multi-event soundscapes that outperform single-shot baselines in complex scenarios and effectively augment training data for downstream audio reasoning models.
In this work, we propose an interpretable framework for speech deepfake detection based on artifact-specific expert models. Rather than relying on black-box decisions, the framework provides human-understandable evidence, which is critical in high-stakes settings. Each expert is trained to detect a specific speech synthesis artifact, and its output is calibrated into a log-likelihood ratio that serves as an interpretable evidence score. We evaluate five artifact-specific experts and show that, with proper calibration, they can capture their target artifacts and produce meaningful evidence. Importantly, each expert estimates only the presence of its assigned artifact rather than directly performing the final decision. Their outputs are aggregated into an ensemble to produce the actual real-versus-fake classification, while maintaining interpretability by indicating how strongly each expert supports or contradicts a fake classification. Results show that artifact-specific experts capture interpretable signals of synthetic speech across multiple generation pipelines.
Primary: Politecnico di Milano
All Institutions: Politecnico di Milano, National Institute of Informatics
This paper presents a novel, interpretable framework for speech deepfake detection by decomposing the problem into artifact-specific experts trained on pseudo-fake data and aggregating their calibrated log-likelihood ratios, offering a transparent alternative to black-box models in forensic contexts.
The paper proposes a modular, interpretable framework for speech deepfake detection that decomposes the detection task into five specific synthesis artifacts: F0 inconsistencies, oversmoothing, degraded phase, energy inconsistencies, and tonal artifacts. The core methodological innovation lies in the use of "pseudo-fake" data generation—applying controlled, algorithmic manipulations to real speech to train artifact-specific expert models. This avoids the confounding factors present in real deepfake data. The outputs of these experts are calibrated into Log-Likelihood Ratios (LLRs) using either Kernel Density Estimation (KDE) or Constrained Maximum-Likelihood Gaussian (CMLG) methods, allowing for a principled, forensic-style aggregation of evidence. The approach is grounded in signal processing principles rather than purely end-to-end deep learning, aiming for intrinsic interpretability.
The authors evaluate their framework on ASVspoof 2019, ASVspoof 5, and SpoofCeleb datasets. They demonstrate that individual experts can detect their target artifacts with above-chance performance, though not state-of-the-art accuracy. The ensemble of calibrated experts achieves competitive results, with the sum-aggregation strategy outperforming max and learned gating mechanisms. The interpretability is validated by showing that experts activate strongly for attacks known to exhibit those specific artifacts (e.g., the phase expert for Neural Source-Filter vocoders). The paper also includes an ablation study on calibration data composition, finding that a pooled calibration set across experts yields the most stable LLRs.
The paper provides detailed descriptions of the pseudo-fake generation pipelines, including specific parameters for manipulations (e.g., Gaussian filter standard deviation, F0 scaling factors). The feature extraction methods (pYIN, LSF, STFT) and model architectures (MLPs, LCNN) are specified. However, the exact code for the pseudo-fake generation and the specific random seeds for the experiments are not provided in the text, which may hinder exact reproduction. The reliance on external libraries (e.g., pYIN) is noted but implementation details are standard.
The primary limitation is the reliance on predefined artifacts. The framework cannot detect novel or unknown artifacts that do not fit the five modeled categories. The performance is lower than state-of-the-art black-box detectors, which is acknowledged but remains a practical constraint. The assumption of independence among artifact features for LLR summation is a simplification that may not hold perfectly in complex synthesis pipelines. Furthermore, the "pseudo-fake" approach may not fully capture the complexity and distribution of artifacts found in real-world, multi-stage deepfake generation systems.
This work contributes to the critical field of audio forensics and security by providing a transparent method for deepfake detection. The interpretable nature of the system makes it suitable for high-stakes applications where explaining the decision to a human judge or jury is required. It shifts the paradigm from opaque black-box detectors to evidence-based reasoning, potentially increasing trust in automated detection systems. This paper presents a novel, interpretable framework for speech deepfake detection by decomposing the problem into artifact-specific experts trained on pseudo-fake data and aggregating their calibrated log-likelihood ratios, offering a transparent alternative to black-box models in forensic contexts.
Large Audio Language models (LALMs) have made rapid progress on acoustic understanding, yet they still struggle with fine-grained audio reasoning (e.g., recognizing event order, repetitions and duration). Existing post-training methods heavily rely on expensive external labels or provide only coarse semantic signals. To bridge this gap, we introduce Audio-Zero, the first label-free self-evolution framework in the field of LALMs that improves fine-grained auditory perception and reasoning. Audio-Zero constructs an auditory self-play game from unlabeled audio contrast pairs: most players hear a reference audio, while one odd listener hears a subtle variant. The model first generates clues describing what it hears and then identifies the odd listener by reasoning over inconsistencies among clues. Since the odd listener is known by construction, the game provides verifiable rewards without any annotated answers. Experiments with Qwen2-Audio-7B-Instruct and Qwen2.5-Omni-7B on TREA, MMAU Test-mini and MMAR show that Audio-Zero improves fine-grained audio reasoning while preserving broad audio understanding. Evolutionary and diagnostic analyses further reveal that increasingly fine-grained auditory descriptions emerge naturally from game pressure.
Primary: Institute of Computing Technology, Chinese Academy of Sciences
All Institutions: Institute of Acoustics, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing Academy of Artificial Intelligence, Institute of Computing Technology, Chinese Academy of Sciences, University of California, Merced, The University of Queensland
[One sentence main contribution]. Audio-Zero introduces a label-free self-evolution framework for LALMs that uses an auditory self-play game to improve fine-grained reasoning without external annotations. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a novel and technically sophisticated approach to enhancing the fine-grained reasoning capabilities of Large Audio Language Models. By formulating the learning process as a self-play game with verifiable intrinsic rewards, Audio-Zero effectively circumvents the bottleneck of scarce fine-grained annotations. The methodology is well-grounded in recent advances in self-evolution and reinforcement learning, and the experimental results demonstrate its efficacy across multiple benchmarks and model architectures. The diagnostic analysis provides valuable insights into how fine-grained auditory descriptions emerge from game pressure. This work is a significant contribution to the field of audio AI, offering a scalable and efficient path to improving model performance.
The paper proposes Audio-Zero, a label-free self-evolution framework for Large Audio Language Models (LALMs). The core innovation is an "auditory self-play game" that converts unlabeled audio contrast pairs into verifiable training signals. The method involves two coupled stages: a listening stage where the model generates descriptive clues for multiple "players" (some hearing a reference, one hearing a variant), and an attribution stage where the model acts as a judge to identify the "odd listener" based on inconsistencies in the clues. The reward signal is intrinsic to the game mechanics (correct identification) and a rule-based content utility score for the clues. The optimization uses Group Relative Policy Optimization (GRPO). The approach is theoretically sound and addresses a genuine gap in the field: the lack of fine-grained, label-free supervision for audio reasoning. The mechanism of using the "odd-one-out" task to force fine-grained perceptual description is clever and aligns well with recent trends in self-play and self-evolution for LLMs and VLMs.
The authors evaluate Audio-Zero on three benchmarks: TREA (temporal reasoning), MMAU Test-mini (general audio understanding), and MMAR (deep audio reasoning). Experiments are conducted on two strong backbones: Qwen2-Audio-7B-Instruct and Qwen2.5-Omni-7B. The results show consistent improvements over the base policies and several strong baselines (both label-dependent and label-free). Notably, the gains are most pronounced on TREA, the fine-grained temporal reasoning benchmark, which supports the paper's claim of improving fine-grained perception. The ablation studies effectively demonstrate the necessity of each component (listening stage, attribution stage, vote-aware feedback). The self-evolution dynamics analysis (tracking win rates, clue length, and benchmark scores over iterations) provides compelling evidence that the model is indeed learning and evolving rather than just overfitting to a static reward. The use of only 2k unlabeled pairs is a significant efficiency advantage.
The paper provides detailed implementation details, including hyperparameters (learning rate, KL coefficient, number of players/rounds), dataset construction (filtering 2k pairs from Audio-Alpaca), and prompt templates. The GRPO implementation details are standard but well-specified. The diagnostic metrics for analyzing clue quality are clearly defined. While the code is not explicitly linked in the provided text, the description is sufficient for reproduction by researchers familiar with RLHF/GRPO pipelines. The reliance on rule-based scorers for the listening reward is a potential point of variability, but the rules are described.
The method relies on the availability of audio contrast pairs. While the paper claims to use "unlabeled" pairs, it sources them from Audio-Alpaca, which is a preference dataset. The filtering process to create contrast pairs (ensuring they share semantics but differ in fine-grained details) is not fully detailed, raising questions about the scalability and generalizability of the pair generation process. The rule-based listening reward, while effective, may not capture all nuances of good auditory description compared to a learned reward model. Furthermore, the performance gains, while significant, are still modest in absolute terms on some benchmarks (e.g., MMAU Avg. 60.50 vs 57.50 for Qwen2-Audio), suggesting that self-evolution alone may not solve all reasoning deficits without some form of structured supervision or larger-scale data.
This work contributes to the development of more autonomous and data-efficient AI systems. By reducing reliance on expensive human annotations, it lowers the barrier to training high-capability audio models. The improved fine-grained audio reasoning capabilities could benefit applications in accessibility (e.g., better descriptions for the visually impaired), audio forensics, and interactive AI assistants. The self-play paradigm could be extended to other modalities or tasks where verifiable rewards are hard to construct. [One sentence main contribution]. Audio-Zero introduces a label-free self-evolution framework for LALMs that uses an auditory self-play game to improve fine-grained reasoning without external annotations. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a novel and technically sophisticated approach to enhancing the fine-grained reasoning capabilities of Large Audio Language Models. By formulating the learning process as a self-play game with verifiable intrinsic rewards, Audio-Zero effectively circumvents the bottleneck of scarce fine-grained annotations. The methodology is well-grounded in recent advances in self-evolution and reinforcement learning, and the experimental results demonstrate its efficacy across multiple benchmarks and model architectures. The diagnostic analysis provides valuable insights into how fine-grained auditory descriptions emerge from game pressure. This work is a significant contribution to the field of audio AI, offering a scalable and efficient path to improving model performance.
Long-form streaming speech-to-speech translation (S2ST) requires incremental, unbounded translation under strict latency constraints. Existing methods typically suffer from sentence-bounded supervision or demand massive paired-S2ST supervision. We introduce a training recipe enabling a speech language model for sentence-level and long-form streaming S2ST using only $\sim$2k hours of paired cross-lingual S2ST data, layered atop auxiliary supervision. Anchored by auxiliary multitask training, our approach remains robust even when the paired-S2ST budget itself is reduced by 90\%. Our core contribution, joint text-code trajectory supervision, schedules target text and acoustic semantic codes as a unified commitment path, eliminating the need for separate, unstable speech-side emission controllers. Furthermore, our two-stream Thinker--Talker factorization significantly outperforms unified-decoder baselines by decoupling linguistic reasoning from dense acoustic prediction to mitigate modality interference. Finally, our system achieves highly competitive quality-latency trade-offs on RealSI and ACL60/60-dev, matching state-of-the-art, closed-source S2ST systems such as LiveInterpret~2.0 on ASR-BLEU.
Primary: The Chinese University of Hong Kong, Shenzhen
All Institutions: The Chinese University of Hong Kong, Shenzhen
SimulS2ST-Omni presents a significant technical contribution to streaming speech-to-speech translation by introducing a joint text-code trajectory supervision method and a decoupled two-stream architecture that achieves state-of-the-art performance with substantially reduced data requirements, effectively addressing the critical bottleneck of paired S2ST data scarcity.
The paper proposes SimulS2ST-Omni, a data-efficient framework for long-form streaming Speech-to-Speech Translation (S2ST). The core methodological contributions are threefold: (1) A joint text-code trajectory supervision mechanism that schedules target text and acoustic semantic codes simultaneously, eliminating the need for unstable, separate speech-side emission controllers. (2) A two-stream "Thinker-Talker" architecture that decouples linguistic reasoning (Thinker) from dense acoustic prediction (Talker), mitigating modality interference common in unified decoders. (3) A training recipe leveraging auxiliary multitask supervision (ASR, S2TT, MT, TTS) to enable robust performance with only ~2k hours of paired S2ST data, a significant reduction from prior works requiring tens of thousands of hours. The approach extends trajectory-based simultaneous translation to speech output by using cross-lingual word alignments and monotonicity filtering (NIR) to construct stable read/wait/write trajectories.
The authors conduct comprehensive experiments on offline S2ST (CVSS-T), sentence-level streaming (RealSI), and long-form streaming (RealSI, ACL60/60-dev). They compare against state-of-the-art public baselines (UniSS, Seamless-M4T) and closed-source systems (LiveInterpret 2.0). Results show that the Thinker-Talker architecture significantly outperforms the matched unified-decoder baseline in translation quality (ASR-BLEU, Text-BLEU) while maintaining competitive or superior speech quality metrics (A.PCP, SIM-O). The system matches or exceeds LiveInterpret 2.0 in several latency tiers. Ablation studies confirm the importance of NIR-based trajectory filtering and the architectural decoupling. The evaluation is rigorous, covering multiple metrics and latency constraints.
The paper provides detailed descriptions of the data construction process, including the use of public ASR/S2TT corpora to synthesize paired S2ST data, the alignment tools (SimAlign), and the filtering criteria (NIR, ASR error rates). Training hyperparameters, stages, and model initialization details are provided in the appendix. The use of open-source components (Qwen2.5-Omni, DualCodec, Whisper, Paraformer) enhances reproducibility. However, the exact codebase for the trajectory construction and the specific synthesized data pool are not explicitly linked to a public repository, which may hinder exact replication.
The experiments are limited to Chinese-English directions due to data scarcity. The paired S2ST data is synthesized rather than naturally recorded simultaneous interpretation, which may not fully capture real-world interpreting phenomena (e.g., disfluencies, interruptions). The speech generation backend (chunk-wise Flow Matching) is not fully streaming-native, potentially limiting inter-chunk coherence and inference efficiency. The work does not address duplex interaction or broader multilingual settings.
This work advances the field of real-time cross-lingual communication by demonstrating that high-quality streaming S2ST is achievable with limited paired data. It addresses ethical concerns regarding voice cloning and impersonation by acknowledging the risks and suggesting safeguards. The data-efficient approach lowers the barrier to entry for developing robust S2ST systems. SimulS2ST-Omni presents a significant technical contribution to streaming speech-to-speech translation by introducing a joint text-code trajectory supervision method and a decoupled two-stream architecture that achieves state-of-the-art performance with substantially reduced data requirements, effectively addressing the critical bottleneck of paired S2ST data scarcity.
Existing symbolic music generation models typically use bars as the basic structural unit. However, human perception of musical phrases often does not align with notated bar lines, leading to long-term structural fragmentation. This paper proposes RPPNet-a two-stage deep learning architecture with variable structural boundaries. It first generates variable-length Rhythm-Pitch Primitive (RPP) sequences, where each RPP encodes note count, rhythm, and contour; then decodes the RPP sequences into concrete notes. The grouping of RPPs is automatically derived from acoustic cues, auditory inertia, and similarity perception based on music psychology. Experiments show that melodies generated by RPPNet are superior in both long-term structure and musicality, with significant improvements across all subjective evaluation dimensions. Ablation studies confirm that the performance gain stems from the structural correctness of the psychological representation, rather than from model capacity. This work offers an interdisciplinary perspective for music generation, integrating music theory, computational modeling, and music psychology.
Primary: Zhejiang University
All Institutions: Zhejiang University, ZJU-ZJCM MusicAI Lab, Zhejiang Conservatory of Music
[One sentence main contribution]. [RPPNet introduces a perceptually-grounded, variable-length structural representation for symbolic music generation, demonstrating that aligning model boundaries with human auditory grouping principles significantly improves long-term structural coherence compared to fixed bar-level approaches.]
The paper proposes RPPNet, a hierarchical two-stage architecture for symbolic melody generation. The core innovation lies in replacing fixed bar-level tokenization with perceptually-grouped Rhythm-Pitch Primitives (RPPs). The authors employ a heuristic algorithm based on music psychology (acoustic cues, auditory inertia, similarity perception) to segment melodies into variable-length RPPs. The generation process is decoupled: an RPP-level Transformer generates the structural skeleton, which is then instantiated into note-level events via a time-scale expansion mapping. The methodology is theoretically grounded in cognitive musicology, offering a novel perspective on structural segmentation that challenges the dominant fixed-grid assumptions in models like Museformer and MELONS. The decoupled serial prediction mechanism at the RPP level is a technically sound approach to managing the complexity of long-sequence generation.
The experimental setup utilizes the MelodyNet dataset. The evaluation relies heavily on subjective listening tests (MOS-style) with 15 participants, which is a standard but resource-intensive method for music generation. RPPNet significantly outperforms baselines (Museformer, MELONS) in subjective metrics, particularly in "Structure" and "Overall Impression." The inclusion of ablation studies (Random-Grouped vs. Heuristic-Grouped) is crucial and effectively isolates the contribution of the psychological grouping algorithm from mere variable-length tokenization. However, the objective metrics are limited; the paper notes that PPL and Structural Error are difficult to compare across different tokenization schemes, leading to a reliance on subjective scores. The gap between generated music and human composition (6.69 vs 7.47) remains significant, though the paper attributes this to rhythmic distribution mismatches.
The paper provides a GitHub repository link, which is a positive step for reproducibility. The heuristic algorithm for RPP grouping is described, but the full formalization of the weight function $W()$ is stated to be in the repository, which may contain implementation-specific details not fully explicit in the text. The dataset (MelodyNet) is public. The subjective evaluation protocol is described, but the small sample size (15 participants) limits the statistical power of the findings, although significance tests are provided.
The primary limitation is the reliance on a heuristic, rule-based algorithm for RPP segmentation, which may not generalize well to all musical styles or complex polyphonic textures. The current model is limited to monophonic melody generation. The subjective evaluation, while showing statistical significance, suffers from a small participant pool and potential bias. The objective metrics do not fully capture the perceptual improvements claimed, as standard metrics like PPL are not directly comparable across different tokenization granularities.
This work bridges the gap between computational music generation and music psychology/cognition. By demonstrating that perceptually-grounded structural units lead to more coherent music, it offers a new paradigm for designing generative models that align with human auditory perception. This could influence future research in hierarchical modeling and structured generation beyond music, such as in language or video, where fixed-grid assumptions may also be limiting. [One sentence main contribution]. [RPPNet introduces a perceptually-grounded, variable-length structural representation for symbolic music generation, demonstrating that aligning model boundaries with human auditory grouping principles significantly improves long-term structural coherence compared to fixed bar-level approaches.]
Keyword spotting (KWS) models on embedded devices often need to add new keywords after deployment, but updates are difficult when original training data are unavailable and regressions on existing triggers are unacceptable. At a fixed operating point, our method reduces average new-keyword false reject rate (FRR) from 6.46 to 4.37 versus a parameter-matched separate-model baseline and outperforms parameter-efficient tuning baselines (adapters, LoRA), while using fewer multiply-accumulate operations (MACs) under the same added-parameter budget ($\leq$10k): 16.34M vs 18.45M/20.52M. We achieve this via parameter-capped modular expansion: the base network, including batch-normalization statistics and the core classifier, is frozen, and only a lightweight expansion branch with a separate new-keyword head is trained, preserving core logits, shipped outputs, and thresholds for existing keywords.
Primary: Yandex
All Institutions: Yandex
The paper presents a practical and effective method for modular expansion of embedded KWS models, achieving strict non-regulation and improved new-keyword detection through a frozen-base, parallel-branch architecture.
The paper proposes a "modular network expansion" technique for Keyword Spotting (KWS) on embedded devices. The core innovation is architectural: it freezes the entire base SVDF network (including Batch Normalization statistics) and attaches a lightweight, trainable expansion branch with a separate classifier head for new keywords. This design ensures a strict non-regression guarantee for existing keywords because the core logits remain identical to the pre-deployment model. The method is compared against parameter-efficient fine-tuning (LoRA, Adapters), ensemble methods, and continual learning (EWC). The approach is pragmatic and well-suited for the specific constraints of embedded deployment (no original data, strict latency/parameter budgets). While the concept of "parallel heads" or "expert routing" is not new in the broader ML landscape (e.g., Mixture of Experts, Progressive Neural Networks), its specific application to strict non-regression KWS updates with frozen BN layers is a solid, incremental engineering contribution. It lacks deep theoretical novelty but offers a robust practical solution.
The experiments are conducted on Google Speech Commands v2 (GSC) using five held-out keyword pairs. The evaluation metric is macro False Reject Rate (FRR) at a fixed False Accept Rate (FAR). The results show that the proposed method significantly outperforms the Ensemble baseline (6.46% vs 4.37% FRR) and other parameter-efficient methods (LoRA, Adapters) under a 10k parameter budget. The ablation study on expansion depth is useful, identifying that tapping 4 blocks is optimal. The comparison with Full Fine-tuning highlights the catastrophic forgetting problem, and EWC is shown to be less effective without replay data. The use of Mozilla Common Voice for negative samples is a good practice for realistic FAR calibration. The results are statistically significant (95% confidence intervals provided). However, the evaluation is limited to GSC, which is a relatively clean, controlled dataset. Real-world embedded KWS faces more challenges with noise, channel variation, and long-tail distributions.
The paper provides sufficient detail regarding the architecture (SVDF, BN, Hard-Swish), training setup (Adam, cosine LR, SpecAugment), and evaluation protocol (FAR calibration, 8 runs). The parameter budget (10k) and MAC counts are specified. The code is not explicitly linked, but the description is detailed enough for reproduction. The use of standard datasets (GSC, Common Voice) aids reproducibility.
The primary limitation is the reliance on GSC, which may not reflect the complexity of real-world embedded audio environments. The method assumes that the base model's features are sufficiently generalizable to the new keywords via the expansion branch; if the new keywords are phonetically very distinct from the core set, performance might degrade. The "core-first" decision rule introduces a slight latency overhead (checking core first, then new), although the paper argues this is negligible or manageable. The method is specific to KWS and may not generalize easily to other incremental learning tasks without modification.
This work has significant practical impact for the deployment of always-on voice assistants on resource-constrained devices. By enabling safe, data-efficient updates to keyword inventories, it addresses a major pain point in product lifecycle management for IoT devices. It reduces the need for retraining on large datasets and mitigates the risk of degrading user experience for existing features. The paper presents a practical and effective method for modular expansion of embedded KWS models, achieving strict non-regulation and improved new-keyword detection through a frozen-base, parallel-branch architecture.
Large-scale visual generators are increasingly capable but costly to train, fine-tune, and deploy. We introduce Mage-Flow, a compact 4B-scale generative stack for efficient text-to-image generation and instruction-based image editing. The stack is built from two co-designed components: Mage-VAE, a lightweight high-fidelity latent tokenizer, and a Native-Resolution Multimodal Diffusion Transformer trained with rectified flow matching. Mage-VAE uses one-step diffusion-style encoding and decoding with anchor-latent regularization, preserving the reconstruction quality of strong public VAEs while reducing tokenization cost by more than an order of magnitude. Together with native-resolution packing and stack-level CUDA kernel fusion, the stack supports flexible-resolution training and improves end-to-end training throughput by about $2.5\times$. Built on this foundation, we develop a complete model family with Base, RL-aligned, and Turbo variants for both generation and editing. Diffusion-NFT improves prompt following, text rendering, aesthetic quality, and editing fidelity, while few-step distillation with adversarial perceptual guidance produces 4-step Turbo models for low-latency inference. Despite its compact scale, Mage-Flow and Mage-Flow-Edit achieves competitive performance across standard generation and editing benchmarks. More importantly, the Turbo variants make high-resolution generation and editing practical for interactive use: at $1024^2$ resolution on a single NVIDIA A100 GPU, Mage-Flow-Turbo generates an image in 0.59s, and Mage-Flow-Edit-Turbo edits an image in 1.02s, while maintaining a small memory footprint. These results show that careful tokenizer--backbone--system co-design can deliver strong high-resolution generation and editing within an efficient 4B model family.
Primary: Microsoft
All Institutions: Microsoft
Mage-Flow presents a compelling case for system-level co-design in generative AI, demonstrating that a compact 4B model family can achieve state-of-the-art efficiency and competitive quality in text-to-image generation and editing through innovative tokenizer design, native-resolution training, and kernel fusion.
The paper proposes a co-designed generative stack, Mage-Flow, centered on system-level efficiency rather than just model scaling. The core methodological contributions are threefold. First, the introduction of Mage-VAE, a lightweight latent tokenizer that replaces standard convolutional VAEs with one-step diffusion-style encoding and decoding. This is a significant architectural shift, treating the tokenizer as a learned image codec distilled from a strong teacher (FLUX.2-VAE) using anchor-latent regularization to preserve the generation-ready latent geometry. Second, the Native-Resolution MMDiT (NR-MMDiT) backbone abandons fixed-resolution bucketing in favor of native-resolution packing. By utilizing variable-length sequence packing with FlashAttention and per-sample 2D RoPE, the model handles arbitrary aspect ratios and resolutions natively, improving training diversity and inference efficiency for classifier-free guidance. Third, the paper details a comprehensive system-level optimization strategy, including stack-level CUDA kernel fusion to mitigate memory-bound overheads in the VAE, text encoder, and diffusion transformer. The approach is technically sound, leveraging recent advances in diffusion codecs (CoD-Lite) and efficient attention mechanisms to solve the latency bottleneck inherent in high-resolution generation.
The authors provide extensive evaluations covering reconstruction quality, inference speed, memory footprint, and downstream generation/editing performance. Mage-VAE demonstrates reconstruction fidelity comparable to FLUX.2-VAE while reducing computational cost by over an order of magnitude. The generation and editing benchmarks (GenEval, GEdit-Bench, TIIF, etc.) show that the 4B Mage-Flow models achieve competitive or superior performance against much larger open-source models (6B-80B parameters). The Turbo variants (4-step distillation) achieve impressive latency metrics (0.59s for generation, 1.02s for editing on a single A100), making interactive use practical. The ablation studies on tokenizer compatibility and native-resolution packing effectively validate the design choices. The results are robust and clearly demonstrate the quality-efficiency trade-off.
The paper provides significant details for reproduction, including the architecture of the VAE, the training objectives (flow matching, DMD, adversarial guidance), and the data curation pipeline. The release of code, models, and a project page enhances reproducibility. However, the specific hyperparameters for the anchor-latent KL regularization and the exact composition of the 1.3B curated dataset are not fully detailed in the text, though the curation pipeline is described qualitatively. The reliance on proprietary text encoders (Qwen3-VL) and the specific CUDA kernel implementations may require additional effort to replicate exactly, but the overall framework is well-documented.
The primary limitation is the reliance on a proprietary text encoder (Qwen3-VL) for conditioning, which prevents full open-source reproducibility of the conditioning pipeline. Additionally, while the VAE is efficient, the one-step diffusion decoding still requires more compute than standard convolutional decoders, though significantly less than multi-step approaches. The native-resolution packing, while efficient, may introduce complexity in distributed training synchronization compared to fixed-bucket approaches. The model's performance, while competitive, is still behind closed-source giants in absolute aesthetic quality, particularly in complex text rendering and fine-grained compositional control, which is expected given the 4B parameter scale.
This work significantly lowers the barrier to entry for high-quality, high-resolution image generation and editing by providing a compact, efficient, and research-friendly foundation model. By demonstrating that strong performance is achievable at 4B parameters through careful co-design, it challenges the trend of endless scaling and promotes sustainable AI development. The efficient inference capabilities enable deployment on consumer-grade hardware, democratizing access to advanced generative tools for developers and researchers. The focus on editing also supports creative workflows and iterative design processes. Mage-Flow presents a compelling case for system-level co-design in generative AI, demonstrating that a compact 4B model family can achieve state-of-the-art efficiency and competitive quality in text-to-image generation and editing through innovative tokenizer design, native-resolution training, and kernel fusion.
Modern ASR models trained on heterogeneously annotated data treat transcription style (verbatim vs. intended) as an uncontrolled latent variable, causing measurable decoding instability, evaluation confounding (up to 60% of reported WER attributable to style mismatch), and unreliable word-level timing. We show that models already encode both styles; the challenge is controlled activation. Using coverage-aware decoder task tokens trained on parallel verbatim/intended transcript pairs, we raise German disfluency F1 from 10% to 79% zero-shot, despite English-only training. Full English-only fine-tuning surpasses all baselines in verbatim accuracy, disfluency detection, and intended-mode quality across both languages. We further introduce supervised cross-attention fine-tuning that improves word-level timestamps on disfluent speech beyond forced-alignment baselines. Finally, we propose verbatimize, a new task enabling scalable creation and enrichment of speech corpora with high-quality canonical verbatim transcriptions.
Primary: nyra labs
All Institutions: nyra labs
The paper presents a significant advancement in controllable ASR by explicitly parameterizing transcription policy, demonstrating that pretrained models possess latent capabilities for verbatim transcription that can be activated through targeted fine-tuning and explicit conditioning.
The paper proposes a novel framework for controlling transcription style (verbatim vs. intended) in Automatic Speech Recognition (ASR) models by treating the transcription policy as an explicit latent variable. The core technical contribution is the introduction of "mode tags" (discrete decoder-prefix tokens) that condition the model on the desired output style. This is complemented by a supervised cross-attention mechanism for word-level timing, which addresses the instability of emergent alignments in disfluent speech. Additionally, the authors introduce "verbatimize," a task for retroactively annotating verbatim transcripts from intended ones. The methodology is sound, leveraging the latent capabilities of pretrained Whisper models through efficient fine-tuning (Stage 1: embeddings only; Stage 2: decoder unfrozen). The approach to decoupling style control from acoustic modeling is innovative and addresses a significant gap in current ASR systems.
The experimental evaluation is comprehensive, covering disfluency detection, intended transcript quality, and word-level timing across English and German. The results demonstrate significant improvements: German disfluency F1 jumps from 10% to 79% in a zero-shot setting with only mode tag embeddings trained. The timing experiments show that the supervised cross-attention method outperforms strong baselines like MFA and WhisperX on disfluent speech (FluencyBank), achieving lower Mean Absolute Error (MAE). The ablation studies on head selection and sharpening techniques provide valuable insights into the mechanics of attention-based alignment. The use of a newly created German DisfluencySpeech dataset adds value, although the small size (202 utterances) limits the statistical power of the German results.
The paper provides detailed descriptions of the training stages, hyperparameters (learning rates, batch size), and data preprocessing steps. The code repository is linked, which enhances reproducibility. The use of standard benchmarks (TIMIT, FluencyBank, AMI) allows for comparison with existing work. However, the reliance on GPT-4o for generating intended transcripts and the creation of the German dataset by non-trained speakers introduces potential biases that are not fully quantified. The specific implementation details of the "virtual blank" pause detection and Viterbi decoding are described but might require careful implementation to replicate exactly.
The primary limitation is the reliance on paired verbatim/intended data for training, which is scarce. The authors mitigate this by using GPT-4o to generate intended transcripts from verbatim ones, which may not perfectly capture human annotation nuances. The German evaluation set is small and recorded by non-native speakers, potentially affecting the naturalness and generalizability of the results. The method is evaluated primarily on English and German; cross-lingual transfer to more distant languages is not thoroughly tested. Furthermore, the "verbatimize" task, while innovative, depends on the accuracy of the underlying intended transcript and the model's ability to correctly insert disfluencies, which may not always align with human perception of "verbatim" speech.
This work has significant implications for ASR systems used in clinical, legal, and linguistic contexts where verbatim transcription is crucial. By enabling controllable verbatim ASR, it opens up new possibilities for analyzing spontaneous speech, detecting speech disorders, and improving the quality of transcripts for accessibility. The "verbatimize" tool could facilitate the creation of large-scale annotated datasets, addressing a major bottleneck in speech research. However, the potential for misuse in generating realistic but fake verbatim transcripts (deepfakes) should be considered. The improvement in timing accuracy also benefits downstream tasks like speaker diarization and video captioning. The paper presents a significant advancement in controllable ASR by explicitly parameterizing transcription policy, demonstrating that pretrained models possess latent capabilities for verbatim transcription that can be activated through targeted fine-tuning and explicit conditioning.
Auditory attention decoding (AAD) identifies the speaker a listener attends to from neural responses like electroencephalography (EEG), making it a key algorithm in neuro-steered hearing aids. However, most neural AAD models are trained as independent short-window classifiers, despite auditory attention being a temporally persistent cognitive state and short-window EEG--audio evidence often being noisy and ambiguous. We propose an end-to-end Markov AAD framework based on conditional random field (CRF) that trains window-level neural emissions under a two-state attention prior. The framework treats the logits of any AAD backbone as Markov emissions, learns the transition rate from a standard HMM initialization, and jointly optimizes cross-entropy and CRF objectives, allowing temporal continuity to guide representation learning rather than merely smoothing predictions after training. We also introduce ESCNet, an EEG--speech correlation backbone that preserves time-aligned features and converts the difference between two mean Pearson correlations into state logits. We evaluate the framework with four emission backbones spanning correlation-based, convolutional, recurrent, and attention-based designs. On the dynamic AVGC dataset, CRF training generally outperforms post-hoc HMM smoothing; with ESCNet, it achieves $86.5\%$ causal and $92.4\%$ non-causal accuracy using $1$s windows. On the static KUL and USTC datasets, it improves causal decoding over fixed-rate post-hoc HMM baselines by $5.6\%$ and $2.0\%$, respectively, showing the superiority of learning AAD as attention state sequence over isolated-window classification.
Primary: University of Science and Technology of China
All Institutions: University of Science and Technology of China, Guangzhou Maritime University, iFLYTEK Company, Ltd.
The paper presents a robust and theoretically grounded advancement in Auditory Attention Decoding by integrating end-to-end CRF training with neural backbones, demonstrating consistent improvements in accuracy and responsiveness across multiple datasets and architectures.
The paper proposes a significant methodological shift in Auditory Attention Decoding (AAD) by moving from independent window classification to end-to-end Markov State Sequence Learning using Conditional Random Fields (CRF). The core innovation lies in treating the logits of any AAD backbone as emissions for a two-state CRF, allowing the temporal structure of attention (modeled via a transition prior) to guide the learning of neural representations directly, rather than just smoothing predictions post-hoc. The authors also introduce ESCNet, a specialized correlation-based backbone that preserves time alignment and outputs interpretable logits. This approach effectively bridges the gap between local feature extraction and global sequence coherence, addressing the inherent noise and ambiguity in short-window EEG data. The formulation is mathematically sound, leveraging standard forward-backward algorithms for differentiable sequence scoring.
The experimental evaluation is comprehensive and well-structured. The authors test their framework on three distinct datasets (AVGC, KUL, USTC) covering both dynamic attention switching and static persistence scenarios. They evaluate four heterogeneous backbones (ESCNet, AADNet, LSTM, Attn-GRU), demonstrating that the CRF training paradigm is architecture-agnostic and provides consistent improvements over fixed-rate post-hoc HMM baselines. The results show statistically significant gains, particularly in causal decoding on the dynamic AVGC dataset (86.5% accuracy with 1s windows). The inclusion of ablation studies isolating the effects of transition rate learning versus sequence-aware emission learning adds rigor to the claims. The switch delay analysis further validates the responsiveness of the model.
The paper provides detailed implementation details, including preprocessing steps, network architectures, hyperparameters (learning rates, batch sizes, optimizer settings), and training schedules. Crucially, the source code is made publicly available on GitHub, which significantly enhances reproducibility. The datasets used are public (AVGC, KUL, USTC), and the evaluation protocols (leave-one-trial-out) are clearly defined. The description of the CRF training phase, including the warm-up strategy and joint optimization, is sufficiently detailed for replication.
The primary limitation is the reliance on controlled laboratory datasets with normal-hearing participants. The performance on real-world, noisy, or hearing-impaired populations remains unverified. Additionally, the causal switch delay, while improved, is still around 23 seconds for the 1s window setting, which may be too slow for some real-time hearing aid applications requiring immediate adaptation. The study also assumes access to two candidate speech streams, which is a constraint for general multi-talker scenarios with more than two speakers. The static datasets (KUL, USTC) do not capture the complexity of naturalistic attention switching, potentially overestimating the robustness of the model in highly dynamic environments.
This work has significant implications for the development of neuro-steered hearing aids and brain-computer interfaces. By improving the accuracy and responsiveness of auditory attention decoding, it can enhance speech understanding in noisy environments for individuals with hearing loss. The end-to-end framework offers a generalizable template for incorporating temporal priors into other neural decoding tasks where state persistence is a key factor. The open-source release of the code and the detailed benchmarking contribute to the broader ML and neuroscience communities by providing a robust baseline for future research in EEG-based attention decoding. The paper presents a robust and theoretically grounded advancement in Auditory Attention Decoding by integrating end-to-end CRF training with neural backbones, demonstrating consistent improvements in accuracy and responsiveness across multiple datasets and architectures.
Almost every piece of recorded music you have ever heard was modified before it reached you; commercial releases rarely spring fully-formed from the mind of a musician. Despite the promise of music generation models for one-shot output, such fine-grained iterative refinement workflows are a complementary problem, and largely out of their reach. There is also a gap for musicians: while they can express what they want to hear, not all have the facility with studio production tools to implement the complex set of actions needed to realize these intuitions. We formalize this task as agentic post-production, wherein individual aspects of a song are targeted, refined, and combined into a final track. We argue the bottleneck is data: existing corpora do not reflect how realistic post-production chains map onto the vocabulary musicians and engineers actually use. We argue there is a language for modifying recorded music that is dense, consistent, and learnable. We introduce the Rule-based Instructions for Music Editing (RIME) framework, which generates realistic paired edit-instruction data from any baseline music dataset grounded in canonical methods, design patterns, and constraints derived from real production workflows. RIME leverages POEMS, a new toolkit that combines stem separation, mixing, and common studio effects for use by multimodal agents. We use RIME and POEMS to generate 3,000 pairs of edit instructions and ground truth audio, and use this data to evaluate existing multimodal LLMs as agents on this task, showing persistent challenges in current models' post-production capabilities. We also demonstrate RIME's ability to improve post-production agent performance via supervised fine-tuning. We see RIME as an early step toward iterative musical agents, collaborative systems that could transform music production much as interactive coding agents have reshaped software engineering.
Primary: Dartmouth College
All Institutions: Dartmouth College
This paper presents RIME and POEMS, a framework and toolkit for generating synthetic data and benchmarking agentic post-production agents, addressing the gap between one-shot music generation and iterative studio workflows.
The paper introduces RIME, a framework for generating synthetic edit-instruction data for music post-production, and POEMS, a toolkit exposing audio processing tools via the Model Context Protocol (MCP). The methodology is sound and addresses a specific gap: the lack of structured, iterative post-production data compared to one-shot generation data. The use of rule-based recipes, motifs, and parameter priors to generate grounded (input, output, instruction) triples is a clever approach to synthesizing training data. However, the novelty is somewhat limited by the reliance on existing, well-understood signal processing libraries (Pedalboard, Demucs) and standard LLM tool-calling architectures. The core contribution is the *system design* and *benchmark creation* rather than a new algorithmic breakthrough in audio synthesis or model architecture. The abstraction layer generation for prompts is a nice touch for simulating real-world ambiguity.
The experiments evaluate zero-shot and fine-tuned multimodal LLMs on the generated RIME benchmark. The results clearly demonstrate that current models struggle with agentic post-production, particularly in parameter inference and chain assembly. The supervised fine-tuning results show improvement, validating the utility of the synthetic data. The evaluation metrics (FAD, KAD, Edit Similarity, Graph F1) are appropriate for the task. The inclusion of an artifact-removal ("poisoning") split adds depth. However, the dataset size (3,000 pairs) is relatively small for modern LLM benchmarks, though reasonable for this specific, high-fidelity audio task. The reliance on MERT embeddings for audio similarity is standard but has known limitations in capturing perceptual quality compared to human listening tests, which are absent here.
The paper provides detailed descriptions of the POEMS toolkit, recipe schemas, and agent architecture. The use of open-weight models for fine-tuning and standard libraries for audio processing aids reproducibility. However, the specific implementation details of the "prompt templates" and the exact "parameter priors" are referenced but not fully enumerated in the main text (though likely in the appendix, which is partially provided). The lack of a public code repository URL in the provided text is a minor hindrance, but the description is sufficient for a competent researcher to reconstruct the system.
The primary limitation is the dependence on the quality of the underlying stem separation model (Demucs), which can introduce artifacts. The rule-based nature of RIME means it can only generate edits present in the recipe catalog, limiting the diversity of edits compared to fully generative approaches. The evaluation relies heavily on embedding-based metrics, which may not correlate perfectly with human preference for audio quality. The dataset size is small, potentially limiting the generalizability of the fine-tuned models. The focus on specific instrument stems (vocals, drums, bass, etc.) via Demucs limits the scope to tracks where these stems can be reliably separated.
This work has significant potential impact on the field of AI-assisted music production. By formalizing agentic post-production and providing a benchmark, it opens up a new avenue for research beyond one-shot generation. It highlights the challenges of iterative, tool-using agents in the audio domain. The framework could empower musicians with less technical expertise to achieve professional-quality mixes through natural language interaction. It also serves as a testbed for improving multimodal LLMs' reasoning and tool-use capabilities in a complex, structured domain. This paper presents RIME and POEMS, a framework and toolkit for generating synthetic data and benchmarking agentic post-production agents, addressing the gap between one-shot music generation and iterative studio workflows.
DCASE~2026 Task~5 introduces Audio-Dependent Question Answering (ADQA), which tests whether large audio-language models answer from the audio rather than from textual priors. An Audio-Dependency Filtering (ADF) pipeline combines silent-audio probing, per-option perplexity, a large language model (LLM) commonsense check, and human review to remove items solvable from text alone. The 3000 items that pass form the ADQA-Bench evaluation set, spanning music, speech, and environmental audio. The inaugural edition draws 14 teams and 36 submissions across two tracks defined by total parameter count (up to 100B and under 10B). A Chung-Ang University ensemble of MOSS-Audio-8B-Thinking and Qwen3-Omni-30B reaches the top overall accuracy at \pct{58.33}, and a MOSS-only configuration from the same team leads the sub-10B track at \pct{57.30}. Across the 30 submissions with a comparable development score, evaluation accuracy falls by 11.91 percentage points (pp) on average (median 10.91\,pp) on the hidden evaluation split, which is designed to be harder than the development split. The most common building blocks are: the MOSS-Audio-8B-Thinking backbone (13 of 36 submissions), Low-Rank Adaptation (LoRA) fine-tuning on AudioMCQ-StrongAC, and preference or reinforcement-learning objectives -- Group Relative Policy Optimization (GRPO) in five teams, Group reward-Decoupled Normalization Policy Optimization (GDPO) in two. At test time, prompt engineering is near-universal, and majority or choice-permutation voting is common. Every system misses the same set of 233 evaluation items.
Primary: The Chinese University of Hong Kong
All Institutions: The Chinese University of Hong Kong, Ant Group, University of Rochester, Shanghai Jiao Tong University, King's College London
This paper presents the DCASE 2026 Task 5 challenge, introducing the ADQA-Bench, a rigorously filtered benchmark for Audio-Dependent Question Answering that effectively exposes and mitigates textual shortcutting in Large Audio-Language Models, thereby setting a new standard for evaluating genuine audio understanding capabilities.
The paper introduces Audio-Dependent Question Answering (ADQA), a novel benchmark designed to mitigate textual priors in Large Audio-Language Models (LALMs). The core methodological contribution is the Audio-Dependency Filtering (ADF) pipeline, a multi-stage process combining silent-audio probing, perplexity filtering, LLM commonsense checks, and human verification to curate a dataset where answers cannot be derived from text alone. The paper also provides a comprehensive analysis of 36 submissions, identifying common techniques such as LoRA fine-tuning, GRPO/GDPO reinforcement learning, and choice-permutation ensembling. While the benchmark creation is methodologically sound, the paper itself is a challenge summary rather than a proposal of a new algorithmic architecture, focusing on empirical observation of existing methods.
The evaluation is rigorous in scope, covering 3000 items across music, speech, and environmental audio. The paper highlights a significant performance gap between development and hidden evaluation sets (average drop of ~12 pp), effectively demonstrating the difficulty of the benchmark and the prevalence of overfitting to development artifacts. The analysis of universal failures (233 items missed by all systems) provides valuable insight into current model limitations. The use of multiple tracks (parameter size) allows for a nuanced comparison of efficiency versus performance. However, the lack of ablation studies isolating specific components (e.g., the acoustic tagger vs. ensemble) limits the depth of causal claims regarding what drives success.
The paper provides detailed descriptions of the ADF pipeline and the training data (AudioMCQ-StrongAC). It lists the base models and techniques used by top teams, offering high transparency. However, as a challenge summary, it does not provide a single unified codebase. Reproducibility depends on the individual team submissions, most of which are open-source. The specific hyperparameters and exact filtering thresholds for the ADF pipeline are described conceptually but may require checking individual team papers for full implementation details.
The paper acknowledges that the evaluation set, while filtered, may still contain subtle biases. The performance drop from dev to eval suggests that the development set, while harder than previous benchmarks, is not perfectly representative of the generalization capability required for the hidden test set. Furthermore, the reliance on multiple-choice questions inherently limits the assessment of open-ended audio reasoning capabilities. The "universal failure" set indicates that current LALM backbones have fundamental gaps in audio grounding that no amount of prompt engineering or RL can fully bridge.
This work significantly advances the field of multimodal AI by establishing a more rigorous standard for evaluating audio-language models. By explicitly targeting and removing textual shortcuts, ADQA-Bench pushes the community to develop models that truly understand audio signals rather than exploiting dataset artifacts. This has implications for improving the reliability of audio assistants, accessibility tools, and automated content analysis systems. The findings also highlight the need for better audio encoders and pre-training strategies that integrate acoustic features more deeply with language models. This paper presents the DCASE 2026 Task 5 challenge, introducing the ADQA-Bench, a rigorously filtered benchmark for Audio-Dependent Question Answering that effectively exposes and mitigates textual shortcutting in Large Audio-Language Models, thereby setting a new standard for evaluating genuine audio understanding capabilities.
Multi-channel speech enhancement (SE) systems exhibit superior performance over single-channel methods but are constrained to fixed microphone array configurations. This restricts their real-world deployment across devices with diverse array geometries. While recent array-agnostic SE methods address variable microphone numbers and permutations, they largely fail to exploit explicit array geometry priors when available, missing a crucial cue for optimal spatial filtering. A Geometry-Aware Dynamic Convolution (Geo-DConv) framework is proposed, which explicitly leverages microphone coordinates to transform standard fixed-array SE models into robust array-invariant systems. Experiments are conducted on the recent real-recorded RealMAN multi-channel speech dataset. Results demonstrate that the proposed architecture enables two widely used fixed-array models to adapt to array-invariant settings, with consistent performance improvements across diverse array topologies.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University, VUI Labs
The paper presents a practical and effective method for adapting fixed-array speech enhancement models to arbitrary microphone geometries using explicit geometric priors via dynamic convolution, demonstrating improved generalization on real-world data.
The paper proposes a Geometry-Aware Dynamic Convolution (Geo-DConv) module to enable fixed-array speech enhancement models to handle arbitrary microphone array geometries. The core innovation lies in using a Topology-Aware Coordinate Transformer (TACT) to generate dynamic convolution weights based on relative microphone coordinates. This approach attempts to bridge the gap between fixed-array models (which exploit geometry implicitly through training) and array-agnostic models (which often ignore explicit geometry). The method is theoretically sound, leveraging Fourier positional encoding and self-attention to ensure permutation equivariance. However, the novelty is moderate as dynamic convolution and coordinate-based conditioning are established techniques in other domains (e.g., medical imaging, computer vision), and their application here is an incremental adaptation rather than a fundamental algorithmic breakthrough.
The experiments are conducted on the RealMAN dataset, which is a significant strength as it uses real-recorded data, addressing the common sim-to-real gap in multi-channel SE. The authors compare their method against strong baselines (FaSNet-TAC, USES2-comp, SpatialNet, TF-GridNet). The results show that Geo-DConv improves upon existing array-agnostic methods and allows fixed-array models to generalize to unseen array configurations. The evaluation includes metrics like SDR, SI-SDR, PESQ, STOI, and DNSMOS. The generalization to the CHiME-4 dataset (6 mics) is a positive finding. However, the performance gains, while consistent, are not massive, and the computational overhead of the TACT module is not extensively analyzed in terms of inference latency compared to the baseline fixed-array models.
The paper provides detailed architectural descriptions, including hyperparameters for the TACT module (hidden dimension, number of heads, layers) and the Geo-DConv layer (basis dimension, output channels). The dataset (RealMAN) is publicly available, and the implementation details (STFT parameters, sampling rate) are clear. The code is not explicitly linked, but the mathematical formulations are sufficient for reproduction by a competent researcher. The use of standard libraries for STFT and common neural network components further aids reproducibility.
The primary limitation is the assumption that microphone coordinates are known or can be accurately estimated. In real-world scenarios with unknown or dynamically changing arrays, this prior might not be available. Additionally, the method relies on a fixed basis dimension for dynamic convolution, which might limit the capacity to model highly complex or large-scale array geometries compared to methods that process raw channel interactions directly. The performance on very small arrays (1-2 mics) is still constrained by the fundamental limits of single-channel or stereo enhancement. The paper does not extensively discuss the sensitivity of the TACT module to coordinate normalization or scaling.
This work contributes to the broader goal of robust, device-agnostic audio processing. By enabling fixed-array models to adapt to various geometries, it reduces the need for device-specific training and deployment, facilitating the integration of advanced SE into diverse consumer devices (smartphones, laptops, IoT devices) with varying hardware configurations. It highlights the importance of explicit geometric priors in multi-channel audio tasks. The paper presents a practical and effective method for adapting fixed-array speech enhancement models to arbitrary microphone geometries using explicit geometric priors via dynamic convolution, demonstrating improved generalization on real-world data.
Synthesizing physically plausible impact sounds from visual observations remains a great challenge in multi-modal AI. Existing 3D-aware audio generation methods primarily model the surface geometry of hollow rigid bodies. However, they fundamentally overlook internal filling states, a critical physical factor that drastically modulates acoustic resonance and damping. To address this issue, we have defined a new task called Fine-Grained Filling-Aware Impact Sound Generation. As a foundational step, we first introduce the fine-grained fill-aware dataset (FillImpact), a pioneering multi-modal collection comprising over 5,000 rigorous acoustic recordings from 88 diverse real-world objects. It captures impact interactions with varying internal contents (i.e., water, rice), a continuous range of fill levels, and distinct striker materials. Furthermore, comprehensive acoustic analysis confirms that the collected data closely aligns with established physical laws governing acoustic resonance and damping, indicating its suitability for physically grounded modeling. Building on this dataset, we propose a novel generative framework (FillGauss) that integrates 3D Gaussian Splatting (3DGS) with internal state conditioning for sound generation. By fusing 3DGS geometric features, precise 3D spatial strike coordinates, and fine-grained textual physical conditions within a latent diffusion architecture, FillGauss enables position-aware, striker-aware, and filling-aware audio generation. Extensive experiments demonstrate that our approach could generate high-fidelity impact sounds that adhere to underlying physical principles, establishing a new state-of-the-art for physically grounded cross-modal audio generation.
Primary: Unknown (Affiliations not provided in text)
All Institutions: Unknown
This paper introduces FillGauss, a novel framework for fine-grained, filling-aware impact sound generation that leverages 3D Gaussian Splatting and latent diffusion to synthesize physically plausible audio conditioned on internal object states, supported by the pioneering FillImpact dataset.
The paper proposes FillGauss, a framework for fine-grained, filling-aware impact sound generation using 3D Gaussian Splatting (3DGS). The methodology integrates three modalities: 3DGS geometric features, precise 3D strike coordinates, and textual descriptions of internal states (fill material and level). The core technical contribution lies in the cross-modal attention fusion mechanism, which entangles spatial strike context with internal physical semantics before conditioning a Latent Diffusion Transformer (DiT). The use of 3DGS as a geometric prior for audio generation is a novel application, moving beyond traditional mesh or point-cloud representations. However, the underlying diffusion architecture (TangoFlux) is a pre-existing model, and the primary innovation is in the conditioning pipeline and dataset construction rather than a new generative backbone. The approach of using text prompts to encode physical states (fill level, material) is effective but relies heavily on the semantic capabilities of the pre-trained language model, which may limit fine-grained control compared to continuous numerical conditioning.
The experimental setup is robust, centered on the introduction of the FillImpact dataset, which contains over 5,000 recordings from 88 objects with varying internal states. This dataset is a significant contribution, addressing a gap in existing literature where internal filling states are largely ignored. The evaluation includes objective metrics (FAD, KL, IS) and subjective human perceptual studies (MOS, Win Rate, Attribute Matching). The results demonstrate that FillGauss outperforms baselines (SonicGauss, TangoFlux) in generating physically plausible sounds, particularly in capturing the acoustic differences between liquid and granular fills. The ablation studies effectively validate the importance of the cross-attention modules and the text-prompting strategy. The sensitivity analysis regarding 3DGS sparsity provides valuable insights into the model's reliance on geometric detail.
The paper provides detailed descriptions of the data collection pipeline, including the use of specific hardware (DJI Mic, Revo Scan) and software (Hunyuan 3D, Stable Audio Open VAE). The normalization procedures for 3DGS and strike coordinates are clearly defined. The training details, including the use of Huber loss and the specific architecture of the DiT, are described. However, as the code and dataset are not explicitly linked in the provided text (URLs are "none"), reproducibility depends on the authors' future release of these assets. The description of the dataset construction is sufficiently detailed for another group to replicate the data collection process.
The paper acknowledges that extreme sparsification of 3DGS degrades performance, indicating a dependency on high-fidelity geometric reconstruction. The model's ability to handle complex internal dynamics like sloshing fluids or soft-body deformations is limited to static fill levels and materials. The reliance on text prompts for internal state conditioning might introduce semantic ambiguities or lack the precision of direct numerical inputs for continuous variables like fill level. Furthermore, the subjective evaluation, while rigorous, is limited by the number of participants (20) and the specific test set.
This work significantly advances the field of physically grounded multimodal AI, particularly for applications in virtual reality, gaming, and robotic simulation where accurate auditory feedback is crucial for immersion and task performance. By explicitly modeling internal states, it enables more realistic interactions with a wider variety of objects. The FillImpact dataset serves as a valuable resource for future research in acoustic physics and multimodal learning. The approach could be extended to other physical properties, such as temperature or material degradation, further enriching the realism of synthetic environments. This paper introduces FillGauss, a novel framework for fine-grained, filling-aware impact sound generation that leverages 3D Gaussian Splatting and latent diffusion to synthesize physically plausible audio conditioned on internal object states, supported by the pioneering FillImpact dataset.
Real-time speech-to-speech translation (S2ST) systems must balance translation quality, latency, speech naturalness, and speaker consistency. Publicly documented S2ST systems have advanced direct, multilingual, streaming, and expressive modeling, while proprietary products and APIs increasingly expose real-time translation capabilities to users. However, practical deployment remains challenging for open and reproducible systems, especially in long-form and multi-speaker conversations where partial ASR hypotheses are unstable, turn boundaries are ambiguous, and target speech must be generated with an appropriate speaker prompt. We present X-Translator, a low-cost modular cascaded S2ST system that combines streaming ASR, machine translation, and prompt-conditioned TTS through a session-level runtime controller. The system uses incremental segment commitment to convert unstable ASR streams into translation-ready units, and an online speaker prompt manager to bind source speech spans to speaker-specific voice prompts for synthesis. We evaluate translation, speech quality, and latency with OpenSTBench, compare against proprietary speech translation APIs as behavioral baselines, measure long-form voice stability, evaluate speaker preservation in multi-speaker conversations, and assess multilingual translation quality. X-Translator provides an open platform for understanding the practical trade-offs of deployment-oriented S2ST. Code and demo are available at https://github.com/zhaoyx239/X-Translator.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University, Shanghai Innovation Institute, Microsoft, AISpeech Co., Ltd.
X-Translator presents a practical, modular, and open-source framework for real-time, multi-speaker speech-to-speech translation, offering valuable insights into the system-level engineering challenges of balancing latency, quality, and speaker consistency in deployment-oriented scenarios.
The paper proposes X-Translator, a modular, cascaded Speech-to-Speech Translation (S2ST) system designed for real-time, long-form, and multi-speaker scenarios. The core technical contribution lies not in novel model architectures (e.g., new ASR or TTS backbones) but in the system-level runtime controller. Specifically, it introduces an "incremental segment commitment" layer to handle unstable streaming ASR hypotheses and an "online speaker prompt manager" to maintain speaker consistency in multi-speaker conversations. The methodology is pragmatic, leveraging existing state-of-the-art open-source components (Qwen3-ASR, LMT-60-8B, X-Voice) and orchestrating them via explicit runtime policies. While the engineering is sound and addresses real deployment pain points (latency, speaker drift, partial hypothesis instability), the novelty is incremental relative to the broader S2ST literature, which has seen significant advances in direct, end-to-end models (e.g., Translatotron, SeamlessM4T). The paper positions itself as a complementary "deployment-oriented" perspective rather than a fundamental algorithmic breakthrough.
The evaluation is comprehensive and well-structured, covering short-form benchmarks (OpenSTBench), long-form single-speaker voice stability, multi-speaker preservation, and multilingual translation. The use of proprietary APIs (Doubao, Qwen LiveTranslate, GPT Realtime Translate) as behavioral baselines is a strong point, providing a realistic comparison against current industry standards. The metrics are appropriate, including translation quality (COMET, BLEU), speech quality (UTMOS), and specific S2ST metrics like speaker similarity (Spk. SIM) and purity measures for multi-speaker settings. The results show that X-Translator achieves competitive speaker preservation and translation quality, though it lags behind proprietary systems in latency and sometimes translation accuracy. The analysis of trade-offs between commitment aggressiveness and quality/latency is insightful. However, the reliance on open-source backbones means the system's performance is bounded by those components, and the "low-cost" aspect comes at the expense of the peak performance seen in proprietary, heavily optimized black-box systems.
The paper provides a high level of reproducibility. The code and demo are available on GitHub. The modular design allows other researchers to swap out components (ASR, MT, TTS) and test different runtime policies. The detailed description of the segment commitment triggers and speaker prompt management logic is clear. The evaluation protocol is well-documented, including the specific datasets (TEDLIUM-3, WenetSpeech, VoxConverse, FLEURS) and the metrics used. The release of runtime logs for error analysis further enhances reproducibility and interpretability.
The paper acknowledges several limitations. First, as a cascaded system, it suffers from error propagation (ASR errors affect MT and TTS) and accumulated latency. Second, the speaker-aware mechanism depends on the quality of online speaker tracking and the availability of clean prompt audio; overlapping speech or rapid turn-taking can degrade performance. Third, the comparison with proprietary systems is behavioral; the internal optimizations of commercial APIs (e.g., joint optimization, larger models, caching) are not visible, making a direct architectural comparison difficult. Finally, the system's performance is limited by the underlying open-source models, which may not match the quality of proprietary counterparts.
X-Translator provides a valuable open-source platform for studying real-time S2ST deployment challenges. By exposing the intermediate decisions (segment commitment, prompt routing), it facilitates research into latency-quality trade-offs and speaker consistency. It democratizes access to real-time S2ST capabilities, allowing researchers to build upon and improve upon existing systems. The focus on multi-speaker and long-form scenarios addresses a significant gap in current S2ST research, which often focuses on short, single-speaker utterances. This work encourages the development of more robust, interpretable, and reproducible speech translation systems. X-Translator presents a practical, modular, and open-source framework for real-time, multi-speaker speech-to-speech translation, offering valuable insights into the system-level engineering challenges of balancing latency, quality, and speaker consistency in deployment-oriented scenarios.
Recent audio large language models (ALLMs) are typically built upon audio encoders trained with large amounts of supervised data. Since self-supervised learning (SSL) audio encoder models are known to learn general-purpose and transferable representations, we investigate whether general-purpose SSL audio representations can serve as an effective foundation for ALLMs. We present SALMONN-2, an ALLM built upon a unified SSL encoder. To better exploit the hierarchical representations learned by SSL encoders, we propose a multi-layer feature fusion (MLF) adapter that aggregates information from all encoder layers before projecting them into the language model. Beyond conventional audio understanding tasks, we further explore multimodal in-context learning (MICL) in ALLMs and study how this capability can be acquired through contextual biasing training. Experimental results show that a general-purpose SSL encoder achieves performance comparable to, or better than, specialised supervised audio encoders while providing a more balanced capability across speech, audio, music and paralinguistic tasks. SALMONN-2 further achieves state-of-the-art performance among comparable-scale open-weight models on ALLM understanding benchmarks, obtaining the best results on MMAU-Pro, MMAR and MMSU. We also show that MICL does not emerge naturally in ALLMs, but can be effectively acquired through targeted contextual biasing training.
Primary: Shanghai Artificial Intelligence Laboratory
All Institutions: University of Cambridge, Shanghai Artificial Intelligence Laboratory, Tsinghua University
SALMONN-2 presents a compelling argument for the use of unified self-supervised audio representations in Audio Large Language Models, achieving state-of-the-art performance with greater efficiency and architectural simplicity than previous multi-encoder approaches. The introduction of the Multi-Layer Feature Fusion adapter and the systematic study of Multimodal In-Context Learning provide valuable technical contributions and insights for the broader field of multimodal AI.
The paper proposes SALMONN-2, an Audio Large Language Model (ALLM) built upon a unified Self-Supervised Learning (SSL) encoder (SPEAR). The core methodological contributions are: 1) The adoption of a single, general-purpose SSL encoder (SPEAR) as a unified front-end for diverse audio tasks (speech, music, environmental sound), challenging the prevalent dual-encoder or supervised-encoder paradigms. 2) A Multi-Layer Feature Fusion (MLF) adapter that aggregates hidden states from all layers of the SSL encoder before projection, aiming to exploit the hierarchical nature of SSL representations (low-level acoustic to high-level semantic). 3) A timestamp injection mechanism using textual tokens from the LLM vocabulary to provide explicit temporal grounding without extra parameters. 4) Explicit training for Multimodal In-Context Learning (MICL) in the context of Contextual ASR, demonstrating that this capability does not emerge naturally from standard instruction tuning. The approach is technically sound, leveraging recent advances in unified SSL models (SPEAR) and applying them effectively to the ALLM paradigm. The MLF adapter is a relatively simple but effective architectural choice compared to more complex cross-attention or Q-Former styles.
The experimental evaluation is comprehensive and rigorous. The authors evaluate SALMONN-2 on a wide range of canonical tasks (ASR, AAC, Music Captioning, Emotion Recognition, Speech Translation, Overlapped Speech Recognition, Speaker Verification) and general audio understanding benchmarks (MMAU-Pro, MMAR, MMSU). They also include extended audio analysis tasks (Sound Event Detection, Spoofing Analysis, Speech Quality Assessment) and a novel MICL evaluation for Contextual ASR. The results show that SALMONN-2 achieves state-of-the-art performance among comparable-scale open-weight models, often outperforming models with larger LLM backbones or more complex encoder setups. The ablation studies effectively isolate the contributions of the SSL encoder, the MLF adapter, and timestamp injection. The MICL experiments provide valuable insights into the limitations of current ALLMs and how targeted training can address them. The use of less than 20K hours of instruction-tuning data compared to competitors is a significant efficiency claim supported by the results.
The paper provides detailed implementation details, including the architecture of the SPEAR encoder, the MLF adapter, the LLM backbones (Qwen series), and the training stages. The code and model are open-sourced on GitHub, which significantly aids reproducibility. The dataset construction for MICL is described, though the specific processing of some external datasets (e.g., HoliAntiSpoof, AudioSet-strong) might require careful attention to licensing and availability. The training hyperparameters and data mix are summarized in tables, providing a clear recipe for replication.
The paper acknowledges that scaling the LLM backbone from 9B to 30B-A3B did not yield substantial improvements on canonical tasks, suggesting these tasks may be perception-heavy rather than reasoning-heavy. However, it does not deeply explore the failure modes of the SSL encoder on out-of-distribution audio or the potential biases introduced by the SSL pre-training data. The MICL capability is demonstrated primarily on Contextual ASR; its generalizability to other multimodal in-context learning scenarios (e.g., audio QA with few-shot examples) is less explored. The reliance on SPEAR, while effective, ties the model to a specific SSL architecture; the generalizability of the MLF adapter to other SSL models (e.g., WavLM, HuBERT) is not extensively tested, although the principles are likely transferable.
SALMONN-2 contributes to the development of more efficient and general-purpose audio intelligence systems. By demonstrating that a single SSL encoder can replace specialized supervised encoders, it reduces the computational and data overhead required to build ALLMs. The explicit handling of multimodal context has implications for robust speech recognition in noisy or domain-specific environments. The open-sourcing of the model and code facilitates further research in audio-language models. Potential negative impacts include the misuse of powerful audio understanding models for surveillance or deepfake detection evasion, though the model's focus on understanding rather than generation mitigates some generation-specific risks. SALMONN-2 presents a compelling argument for the use of unified self-supervised audio representations in Audio Large Language Models, achieving state-of-the-art performance with greater efficiency and architectural simplicity than previous multi-encoder approaches. The introduction of the Multi-Layer Feature Fusion adapter and the systematic study of Multimodal In-Context Learning provide valuable technical contributions and insights for the broader field of multimodal AI.