Neural audio codecs with residual vector quantization (RVQ) normally treat all frequencies uniformly, so their codebooks become spectrally entangled. Truncating stages then removes an unpredictable mix of frequencies. Parallel band decomposition addresses this by splitting audio into independent bands, but fragments the latent space and loses cross-frequency coherence. We introduce HARP (Harmonic-Aware Residual Partitioning), a training strategy that partitions RVQ stages into frequency-ordered groups where each group refines its target band while the decoder retains access to all lower frequencies. Overtones are reconstructed in the context of their fundamentals, preserving coherence that parallel methods lose. HARP requires no architectural changes; it only modifies the training loss, leaving inference identical to standard RVQ. On speech, music, and general audio, HARP outperforms both standard RVQ and parallel decomposition. MUSHRA listening tests also show perceptual improvements.
Primary: Georgia Institute of Technology
All Institutions: Georgia Institute of Technology, The Chinese University of Hong Kong, Tencent Music Entertainment
HARP introduces a training-only strategy for neural audio codecs that partitions RVQ stages into frequency-ordered groups with cumulative decoding and soft subband supervision, effectively addressing spectral entanglement and improving both objective metrics and perceptual quality, particularly at low bitrates, without architectural changes.
The paper proposes HARP, a training-time strategy for Residual Vector Quantization (RVQ) in neural audio codecs. The core innovation lies in partitioning RVQ stages into frequency-ordered groups and enforcing spectral specialization through a novel loss function. Key components include: 1) Cumulative Decoding: Ensuring higher-frequency stages are reconstructed in the context of lower-frequency content to preserve harmonic coherence. 2) Subband Contribution Supervision: Supervising the isolated contribution of each group to its target band using stop-gradient operators to prevent gradient interference. 3) Soft Band Weighting: Using learnable Gaussian weights over mel bins to guide specialization without hard spectral cutoffs. The approach is architecturally agnostic, requiring only modifications to the training objective. This is a clever and relatively simple modification that addresses the "spectral entanglement" problem inherent in standard RVQ, where early and late stages capture similar frequency content.
The authors evaluate HARP against standard DAC (same architecture, no band supervision) and BSCodec (parallel band decomposition). They use a diverse dataset including MUSDB18-HQ, LibriTTS, and AudioSet. Metrics include SI-SDR, KAD, PESQ, STOI, and MUSHRA listening tests. Results show consistent improvements in SI-SDR and KAD over both baselines, with particularly notable gains at low bitrates (2.6 kbps). The ablation studies effectively isolate the contributions of cumulative decoding, soft boundaries, and stop-gradient mechanisms. The harmonic coherence analysis using synthetic tones provides strong evidence for the preservation of phase/amplitude relationships. The MUSHRA tests confirm perceptual improvements, especially in bass fullness and timbral naturalness.
The paper provides detailed descriptions of the architecture, training objectives, and hyperparameters. The code is made publicly available on GitHub. The use of standard components (DAC backbone, EMA updates) and clear algorithmic descriptions enhances reproducibility. The training cost increase (2-3x) is noted, which is a practical consideration but does not hinder reproducibility.
The primary limitation is that the performance gains, while consistent, are modest in absolute terms (e.g., +0.25 to +0.96 dB SI-SDR). The method relies on the assumption that natural audio has a specific energy distribution (low-to-high), which might not hold for all types of audio (e.g., highly inharmonic or noise-heavy content, though the paper argues it still applies). The method is training-only, so it does not offer architectural efficiency gains. The "soft" boundaries mean that some spectral leakage or imperfect separation might still occur, potentially limiting the maximum potential of the specialization compared to hard architectural splits (though the paper argues this is offset by coherence).
This work contributes to the field of neural audio compression by providing a principled way to impose spectral structure on RVQ without architectural complexity. This can lead to more robust and interpretable audio codecs, which are foundational for speech synthesis, music generation, and audio understanding tasks. The improved low-bitrate performance is particularly relevant for streaming applications. The open-source release facilitates further research in this area. HARP introduces a training-only strategy for neural audio codecs that partitions RVQ stages into frequency-ordered groups with cumulative decoding and soft subband supervision, effectively addressing spectral entanglement and improving both objective metrics and perceptual quality, particularly at low bitrates, without architectural changes.
Large Audio Language models (LALMs) have made rapid progress on acoustic understanding, yet they still struggle with fine-grained audio reasoning (e.g., recognizing event order, repetitions and duration). Existing post-training methods heavily rely on expensive external labels or provide only coarse semantic signals. To bridge this gap, we introduce Audio-Zero, the first label-free self-evolution framework in the field of LALMs that improves fine-grained auditory perception and reasoning. Audio-Zero constructs an auditory self-play game from unlabeled audio contrast pairs: most players hear a reference audio, while one odd listener hears a subtle variant. The model first generates clues describing what it hears and then identifies the odd listener by reasoning over inconsistencies among clues. Since the odd listener is known by construction, the game provides verifiable rewards without any annotated answers. Experiments with Qwen2-Audio-7B-Instruct and Qwen2.5-Omni-7B on TREA, MMAU Test-mini and MMAR show that Audio-Zero improves fine-grained audio reasoning while preserving broad audio understanding. Evolutionary and diagnostic analyses further reveal that increasingly fine-grained auditory descriptions emerge naturally from game pressure.
Primary: Institute of Computing Technology, Chinese Academy of Sciences
All Institutions: Institute of Acoustics, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing Academy of Artificial Intelligence, Institute of Computing Technology, Chinese Academy of Sciences, University of California, Merced, The University of Queensland
[One sentence main contribution]. Audio-Zero introduces a label-free self-evolution framework for LALMs that uses an auditory self-play game to improve fine-grained reasoning without external annotations. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a novel and technically sophisticated approach to enhancing the fine-grained reasoning capabilities of Large Audio Language Models. By formulating the learning process as a self-play game with verifiable intrinsic rewards, Audio-Zero effectively circumvents the bottleneck of scarce fine-grained annotations. The methodology is well-grounded in recent advances in self-evolution and reinforcement learning, and the experimental results demonstrate its efficacy across multiple benchmarks and model architectures. The diagnostic analysis provides valuable insights into how fine-grained auditory descriptions emerge from game pressure. This work is a significant contribution to the field of audio AI, offering a scalable and efficient path to improving model performance.
The paper proposes Audio-Zero, a label-free self-evolution framework for Large Audio Language Models (LALMs). The core innovation is an "auditory self-play game" that converts unlabeled audio contrast pairs into verifiable training signals. The method involves two coupled stages: a listening stage where the model generates descriptive clues for multiple "players" (some hearing a reference, one hearing a variant), and an attribution stage where the model acts as a judge to identify the "odd listener" based on inconsistencies in the clues. The reward signal is intrinsic to the game mechanics (correct identification) and a rule-based content utility score for the clues. The optimization uses Group Relative Policy Optimization (GRPO). The approach is theoretically sound and addresses a genuine gap in the field: the lack of fine-grained, label-free supervision for audio reasoning. The mechanism of using the "odd-one-out" task to force fine-grained perceptual description is clever and aligns well with recent trends in self-play and self-evolution for LLMs and VLMs.
The authors evaluate Audio-Zero on three benchmarks: TREA (temporal reasoning), MMAU Test-mini (general audio understanding), and MMAR (deep audio reasoning). Experiments are conducted on two strong backbones: Qwen2-Audio-7B-Instruct and Qwen2.5-Omni-7B. The results show consistent improvements over the base policies and several strong baselines (both label-dependent and label-free). Notably, the gains are most pronounced on TREA, the fine-grained temporal reasoning benchmark, which supports the paper's claim of improving fine-grained perception. The ablation studies effectively demonstrate the necessity of each component (listening stage, attribution stage, vote-aware feedback). The self-evolution dynamics analysis (tracking win rates, clue length, and benchmark scores over iterations) provides compelling evidence that the model is indeed learning and evolving rather than just overfitting to a static reward. The use of only 2k unlabeled pairs is a significant efficiency advantage.
The paper provides detailed implementation details, including hyperparameters (learning rate, KL coefficient, number of players/rounds), dataset construction (filtering 2k pairs from Audio-Alpaca), and prompt templates. The GRPO implementation details are standard but well-specified. The diagnostic metrics for analyzing clue quality are clearly defined. While the code is not explicitly linked in the provided text, the description is sufficient for reproduction by researchers familiar with RLHF/GRPO pipelines. The reliance on rule-based scorers for the listening reward is a potential point of variability, but the rules are described.
The method relies on the availability of audio contrast pairs. While the paper claims to use "unlabeled" pairs, it sources them from Audio-Alpaca, which is a preference dataset. The filtering process to create contrast pairs (ensuring they share semantics but differ in fine-grained details) is not fully detailed, raising questions about the scalability and generalizability of the pair generation process. The rule-based listening reward, while effective, may not capture all nuances of good auditory description compared to a learned reward model. Furthermore, the performance gains, while significant, are still modest in absolute terms on some benchmarks (e.g., MMAU Avg. 60.50 vs 57.50 for Qwen2-Audio), suggesting that self-evolution alone may not solve all reasoning deficits without some form of structured supervision or larger-scale data.
This work contributes to the development of more autonomous and data-efficient AI systems. By reducing reliance on expensive human annotations, it lowers the barrier to training high-capability audio models. The improved fine-grained audio reasoning capabilities could benefit applications in accessibility (e.g., better descriptions for the visually impaired), audio forensics, and interactive AI assistants. The self-play paradigm could be extended to other modalities or tasks where verifiable rewards are hard to construct. [One sentence main contribution]. Audio-Zero introduces a label-free self-evolution framework for LALMs that uses an auditory self-play game to improve fine-grained reasoning without external annotations. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a novel and technically sophisticated approach to enhancing the fine-grained reasoning capabilities of Large Audio Language Models. By formulating the learning process as a self-play game with verifiable intrinsic rewards, Audio-Zero effectively circumvents the bottleneck of scarce fine-grained annotations. The methodology is well-grounded in recent advances in self-evolution and reinforcement learning, and the experimental results demonstrate its efficacy across multiple benchmarks and model architectures. The diagnostic analysis provides valuable insights into how fine-grained auditory descriptions emerge from game pressure. This work is a significant contribution to the field of audio AI, offering a scalable and efficient path to improving model performance.
Large-scale visual generators are increasingly capable but costly to train, fine-tune, and deploy. We introduce Mage-Flow, a compact 4B-scale generative stack for efficient text-to-image generation and instruction-based image editing. The stack is built from two co-designed components: Mage-VAE, a lightweight high-fidelity latent tokenizer, and a Native-Resolution Multimodal Diffusion Transformer trained with rectified flow matching. Mage-VAE uses one-step diffusion-style encoding and decoding with anchor-latent regularization, preserving the reconstruction quality of strong public VAEs while reducing tokenization cost by more than an order of magnitude. Together with native-resolution packing and stack-level CUDA kernel fusion, the stack supports flexible-resolution training and improves end-to-end training throughput by about $2.5\times$. Built on this foundation, we develop a complete model family with Base, RL-aligned, and Turbo variants for both generation and editing. Diffusion-NFT improves prompt following, text rendering, aesthetic quality, and editing fidelity, while few-step distillation with adversarial perceptual guidance produces 4-step Turbo models for low-latency inference. Despite its compact scale, Mage-Flow and Mage-Flow-Edit achieves competitive performance across standard generation and editing benchmarks. More importantly, the Turbo variants make high-resolution generation and editing practical for interactive use: at $1024^2$ resolution on a single NVIDIA A100 GPU, Mage-Flow-Turbo generates an image in 0.59s, and Mage-Flow-Edit-Turbo edits an image in 1.02s, while maintaining a small memory footprint. These results show that careful tokenizer--backbone--system co-design can deliver strong high-resolution generation and editing within an efficient 4B model family.
Primary: Microsoft
All Institutions: Microsoft
Mage-Flow presents a compelling case for system-level co-design in generative AI, demonstrating that a compact 4B model family can achieve state-of-the-art efficiency and competitive quality in text-to-image generation and editing through innovative tokenizer design, native-resolution training, and kernel fusion.
The paper proposes a co-designed generative stack, Mage-Flow, centered on system-level efficiency rather than just model scaling. The core methodological contributions are threefold. First, the introduction of Mage-VAE, a lightweight latent tokenizer that replaces standard convolutional VAEs with one-step diffusion-style encoding and decoding. This is a significant architectural shift, treating the tokenizer as a learned image codec distilled from a strong teacher (FLUX.2-VAE) using anchor-latent regularization to preserve the generation-ready latent geometry. Second, the Native-Resolution MMDiT (NR-MMDiT) backbone abandons fixed-resolution bucketing in favor of native-resolution packing. By utilizing variable-length sequence packing with FlashAttention and per-sample 2D RoPE, the model handles arbitrary aspect ratios and resolutions natively, improving training diversity and inference efficiency for classifier-free guidance. Third, the paper details a comprehensive system-level optimization strategy, including stack-level CUDA kernel fusion to mitigate memory-bound overheads in the VAE, text encoder, and diffusion transformer. The approach is technically sound, leveraging recent advances in diffusion codecs (CoD-Lite) and efficient attention mechanisms to solve the latency bottleneck inherent in high-resolution generation.
The authors provide extensive evaluations covering reconstruction quality, inference speed, memory footprint, and downstream generation/editing performance. Mage-VAE demonstrates reconstruction fidelity comparable to FLUX.2-VAE while reducing computational cost by over an order of magnitude. The generation and editing benchmarks (GenEval, GEdit-Bench, TIIF, etc.) show that the 4B Mage-Flow models achieve competitive or superior performance against much larger open-source models (6B-80B parameters). The Turbo variants (4-step distillation) achieve impressive latency metrics (0.59s for generation, 1.02s for editing on a single A100), making interactive use practical. The ablation studies on tokenizer compatibility and native-resolution packing effectively validate the design choices. The results are robust and clearly demonstrate the quality-efficiency trade-off.
The paper provides significant details for reproduction, including the architecture of the VAE, the training objectives (flow matching, DMD, adversarial guidance), and the data curation pipeline. The release of code, models, and a project page enhances reproducibility. However, the specific hyperparameters for the anchor-latent KL regularization and the exact composition of the 1.3B curated dataset are not fully detailed in the text, though the curation pipeline is described qualitatively. The reliance on proprietary text encoders (Qwen3-VL) and the specific CUDA kernel implementations may require additional effort to replicate exactly, but the overall framework is well-documented.
The primary limitation is the reliance on a proprietary text encoder (Qwen3-VL) for conditioning, which prevents full open-source reproducibility of the conditioning pipeline. Additionally, while the VAE is efficient, the one-step diffusion decoding still requires more compute than standard convolutional decoders, though significantly less than multi-step approaches. The native-resolution packing, while efficient, may introduce complexity in distributed training synchronization compared to fixed-bucket approaches. The model's performance, while competitive, is still behind closed-source giants in absolute aesthetic quality, particularly in complex text rendering and fine-grained compositional control, which is expected given the 4B parameter scale.
This work significantly lowers the barrier to entry for high-quality, high-resolution image generation and editing by providing a compact, efficient, and research-friendly foundation model. By demonstrating that strong performance is achievable at 4B parameters through careful co-design, it challenges the trend of endless scaling and promotes sustainable AI development. The efficient inference capabilities enable deployment on consumer-grade hardware, democratizing access to advanced generative tools for developers and researchers. The focus on editing also supports creative workflows and iterative design processes. Mage-Flow presents a compelling case for system-level co-design in generative AI, demonstrating that a compact 4B model family can achieve state-of-the-art efficiency and competitive quality in text-to-image generation and editing through innovative tokenizer design, native-resolution training, and kernel fusion.
We present VibeVoice-ASR-BitNet, a compressed variant of VibeVoice-ASR optimized for real-time inference on edge CPUs. We apply heterogeneous quantization tailored to the computational characteristics of each stage: the VAE acoustic tokenizer uses full-pipeline INT8 quantization (I8_S) with kernel fusion and SIMD optimization, while the autoregressive language model adopts BitNet-style ternary weights (I2_S). To preserve accuracy under aggressive compression, we employ a progressive quantization-aware training strategy. For inference, we implement custom SIMD kernels and fused operators within the ggml framework targeting both ARM and x86 platforms, achieving real-time recognition with RTF < 1 using as few as 3 CPU threads. VibeVoice-ASR-BitNet is 1.6-2.3x faster than Whisper.cpp at comparable model sizes (~1.6 GB), with only modest accuracy degradation compared to the FP16 baseline.
Primary: Microsoft Research
All Institutions: Microsoft Research, Shanghai Jiao Tong University, Fudan University, University of Chinese Academy of Sciences
This paper presents a highly effective engineering solution for deploying large LLM-based ASR models on edge CPUs through heterogeneous quantization (INT8 for VAE, ternary for LM) and custom SIMD optimization, achieving real-time performance with minimal accuracy loss and significant compression.
The paper proposes a heterogeneous quantization strategy for the VibeVoice-ASR architecture, specifically targeting edge CPU deployment. The core technical contribution lies in the distinct treatment of the two main components: the VAE tokenizer is quantized to full-pipeline INT8 (I8_S) to address its activation-bound nature, while the autoregressive language model (LM) decoder utilizes BitNet-style ternary weights (I2_S) to address its weight-bound nature during decoding. The authors introduce a progressive quantization-aware training (QAT) schedule using a blending parameter to stabilize training, which is a practical engineering contribution. The inference engine is built on ggml with custom SIMD kernels (AVX2/NEON) for efficient INT8 and ternary (unpacking to INT8 for MAD operations) computation. The methodology is sound and well-reasoned, leveraging established techniques (BitNet, ggml, INT8 quantization) in a novel combination specific to the VAE-LM ASR paradigm.
The evaluation is conducted on a server-grade AMD EPYC CPU, which serves as a proxy for high-end edge hardware, though real-world edge devices (e.g., mobile SoCs) may have different memory bandwidth and cache characteristics. The paper reports a 2.9x compression ratio (4.62 GB to 1.58 GB) and achieves Real-Time Factor (RTF) < 1 with only 3 threads, demonstrating significant efficiency gains over FP16 baselines and competitive performance against Whisper.cpp. Accuracy is evaluated on multiple multilingual benchmarks (MLC, AISHELL, etc.), showing modest degradation (1-4%) compared to the larger FP16 baseline. The comparison with Whisper.cpp is relevant and highlights the benefits of the specific heterogeneous quantization approach for LLM-based ASR.
The paper provides a GitHub link to the codebase (VibeASR.cpp) and Hugging Face links to the models. The methodology describes the quantization schemes, training schedules, and kernel optimizations in sufficient detail for replication by experts in the field. The use of standard frameworks (ggml) and well-defined quantization protocols enhances reproducibility.
The primary limitation is the lack of streaming support, as noted by the authors. The current implementation is batch-oriented, which limits its applicability in real-time interactive scenarios where low latency is critical. Additionally, the evaluation is performed on a specific server CPU; performance on actual resource-constrained edge devices (ARM-based mobile chips, etc.) is not explicitly detailed, although ARM support is mentioned. The generalizability of the heterogeneous strategy to other VAE-LM architectures (e.g., Whisper) is also unexplored.
This work significantly advances the field of on-device AI by demonstrating that large, accurate LLM-based ASR systems can be deployed on commodity CPUs without GPUs. This has profound implications for privacy-preserving speech recognition, reduced cloud infrastructure costs, and enabling advanced voice assistants on low-power devices. It bridges the gap between high-accuracy LLM-based ASR and practical, efficient edge deployment. This paper presents a highly effective engineering solution for deploying large LLM-based ASR models on edge CPUs through heterogeneous quantization (INT8 for VAE, ternary for LM) and custom SIMD optimization, achieving real-time performance with minimal accuracy loss and significant compression.
Transformers with global attention capture long-range dependencies but can miss the fine-grained local continuity crucial for speech separation. We propose TF-MossFormer, a time-frequency transformer that combines local and global attention to jointly model short- and long-range contexts for monaural speech separation. At its core is a content-aware sliding-window attention mechanism that dynamically adapts receptive fields for stronger local interactions, avoiding the rigidity of static convolutions. Unlike time-domain chunk-based methods, TF-MossFormer leverages the 2D spectrogram to model structure along both time and frequency axes. Convolutional gating between attention layers further improves feature selection and information flow. TF-MossFormer achieves SI-SDRi of 22.6, 24.0, and 24.4 dB on WSJ0-2Mix with 5.9M, 16.9M, and 25.4M parameters, respectively, outperforming prior approaches.
Primary: Alibaba Group
All Institutions: Alibaba Group, Token Foundry
TF-MossFormer achieves state-of-the-art performance on WSJ0-2Mix by effectively combining sliding-window local attention with global self-attention and convolutional gating, offering a robust and efficient solution for monaural speech separation.
The paper proposes TF-MossFormer, a time-frequency domain architecture for monaural speech separation. The core innovation lies in integrating a content-aware sliding-window local attention mechanism with global self-attention, augmented by convolutional gating. This hybrid approach aims to balance the computational efficiency and local feature preservation of sliding windows with the long-range dependency modeling of global attention. The architecture follows a dual-path (time-frequency) design, processing spectro-temporal features through stacked blocks. While the combination of local and global attention is not entirely new in the broader transformer literature (e.g., Longformer, BigBird), its specific adaptation to the 2D spectrogram structure with convolutional gating for speech separation is a targeted and reasonable engineering contribution. The use of Conv-SwiGLU and RMSGroupNorm indicates a solid understanding of modern transformer stabilization techniques.
The experimental evaluation is conducted on the standard WSJ0-2Mix benchmark. The authors report results for three model scales (Small, Medium, Large) with parameter counts ranging from 5.9M to 25.4M. The results demonstrate state-of-the-art performance (SI-SDRi up to 24.4 dB) compared to prior works like TF-LocoFormer, TF-GridNet, and SepTDA2. The ablation studies provide valuable insights into the hyperparameters (window sizes) and the architectural choices (order of local/global attention, presence of gating). The comparison with SPMamba highlights efficiency gains. However, the evaluation is limited to a single dataset (WSJ0-2Mix), which restricts the generalizability claims. The computational complexity (MACs) is reported, showing competitive efficiency, which is a strong point.
The paper provides sufficient implementation details, including STFT parameters (16ms window, 8ms hop), optimizer settings (AdamW, learning rate schedule), and loss function (SI-SDR). The architecture diagrams and block descriptions are clear. The use of the ESPnet pipeline is mentioned, which aids reproducibility for those familiar with the framework. However, the specific code repository is not linked in the text provided, and some hyperparameters for the ablation studies (e.g., exact dimension $D$ for the medium/large models) are referenced but not explicitly detailed in the text snippets provided (though implied by parameter counts).
The primary limitation is the lack of evaluation on other benchmarks (e.g., LibriMix, WHAM!, or real-world noisy mixtures), which limits the assessment of robustness. The "content-aware" nature of the sliding window is mentioned but the mechanism for dynamic adaptation is not deeply elaborated in the provided text, raising questions about whether it is truly dynamic or just a fixed window with a specific size. The focus on WSJ0-2Mix, while standard, does not demonstrate performance on more challenging, diverse datasets.
This work contributes to the field of audio signal processing, specifically speech separation, which has applications in hearing aids, voice assistants, and audio post-production. By improving the efficiency and quality of TF-domain separation, it supports the development of more accessible and effective speech processing tools. The hybrid attention mechanism could inspire similar architectures in other sequential data domains. TF-MossFormer achieves state-of-the-art performance on WSJ0-2Mix by effectively combining sliding-window local attention with global self-attention and convolutional gating, offering a robust and efficient solution for monaural speech separation.
In this work, we propose an interpretable framework for speech deepfake detection based on artifact-specific expert models. Rather than relying on black-box decisions, the framework provides human-understandable evidence, which is critical in high-stakes settings. Each expert is trained to detect a specific speech synthesis artifact, and its output is calibrated into a log-likelihood ratio that serves as an interpretable evidence score. We evaluate five artifact-specific experts and show that, with proper calibration, they can capture their target artifacts and produce meaningful evidence. Importantly, each expert estimates only the presence of its assigned artifact rather than directly performing the final decision. Their outputs are aggregated into an ensemble to produce the actual real-versus-fake classification, while maintaining interpretability by indicating how strongly each expert supports or contradicts a fake classification. Results show that artifact-specific experts capture interpretable signals of synthetic speech across multiple generation pipelines.
Primary: Politecnico di Milano
All Institutions: Politecnico di Milano, National Institute of Informatics
This paper presents a novel, interpretable framework for speech deepfake detection by decomposing the problem into artifact-specific experts trained on pseudo-fake data and aggregating their calibrated log-likelihood ratios, offering a transparent alternative to black-box models in forensic contexts.
The paper proposes a modular, interpretable framework for speech deepfake detection that decomposes the detection task into five specific synthesis artifacts: F0 inconsistencies, oversmoothing, degraded phase, energy inconsistencies, and tonal artifacts. The core methodological innovation lies in the use of "pseudo-fake" data generation—applying controlled, algorithmic manipulations to real speech to train artifact-specific expert models. This avoids the confounding factors present in real deepfake data. The outputs of these experts are calibrated into Log-Likelihood Ratios (LLRs) using either Kernel Density Estimation (KDE) or Constrained Maximum-Likelihood Gaussian (CMLG) methods, allowing for a principled, forensic-style aggregation of evidence. The approach is grounded in signal processing principles rather than purely end-to-end deep learning, aiming for intrinsic interpretability.
The authors evaluate their framework on ASVspoof 2019, ASVspoof 5, and SpoofCeleb datasets. They demonstrate that individual experts can detect their target artifacts with above-chance performance, though not state-of-the-art accuracy. The ensemble of calibrated experts achieves competitive results, with the sum-aggregation strategy outperforming max and learned gating mechanisms. The interpretability is validated by showing that experts activate strongly for attacks known to exhibit those specific artifacts (e.g., the phase expert for Neural Source-Filter vocoders). The paper also includes an ablation study on calibration data composition, finding that a pooled calibration set across experts yields the most stable LLRs.
The paper provides detailed descriptions of the pseudo-fake generation pipelines, including specific parameters for manipulations (e.g., Gaussian filter standard deviation, F0 scaling factors). The feature extraction methods (pYIN, LSF, STFT) and model architectures (MLPs, LCNN) are specified. However, the exact code for the pseudo-fake generation and the specific random seeds for the experiments are not provided in the text, which may hinder exact reproduction. The reliance on external libraries (e.g., pYIN) is noted but implementation details are standard.
The primary limitation is the reliance on predefined artifacts. The framework cannot detect novel or unknown artifacts that do not fit the five modeled categories. The performance is lower than state-of-the-art black-box detectors, which is acknowledged but remains a practical constraint. The assumption of independence among artifact features for LLR summation is a simplification that may not hold perfectly in complex synthesis pipelines. Furthermore, the "pseudo-fake" approach may not fully capture the complexity and distribution of artifacts found in real-world, multi-stage deepfake generation systems.
This work contributes to the critical field of audio forensics and security by providing a transparent method for deepfake detection. The interpretable nature of the system makes it suitable for high-stakes applications where explaining the decision to a human judge or jury is required. It shifts the paradigm from opaque black-box detectors to evidence-based reasoning, potentially increasing trust in automated detection systems. This paper presents a novel, interpretable framework for speech deepfake detection by decomposing the problem into artifact-specific experts trained on pseudo-fake data and aggregating their calibrated log-likelihood ratios, offering a transparent alternative to black-box models in forensic contexts.
Large Audio Language models (LALMs) have made rapid progress on acoustic understanding, yet they still struggle with fine-grained audio reasoning (e.g., recognizing event order, repetitions and duration). Existing post-training methods heavily rely on expensive external labels or provide only coarse semantic signals. To bridge this gap, we introduce Audio-Zero, the first label-free self-evolution framework in the field of LALMs that improves fine-grained auditory perception and reasoning. Audio-Zero constructs an auditory self-play game from unlabeled audio contrast pairs: most players hear a reference audio, while one odd listener hears a subtle variant. The model first generates clues describing what it hears and then identifies the odd listener by reasoning over inconsistencies among clues. Since the odd listener is known by construction, the game provides verifiable rewards without any annotated answers. Experiments with Qwen2-Audio-7B-Instruct and Qwen2.5-Omni-7B on TREA, MMAU Test-mini and MMAR show that Audio-Zero improves fine-grained audio reasoning while preserving broad audio understanding. Evolutionary and diagnostic analyses further reveal that increasingly fine-grained auditory descriptions emerge naturally from game pressure.
Primary: Institute of Computing Technology, Chinese Academy of Sciences
All Institutions: Institute of Acoustics, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing Academy of Artificial Intelligence, Institute of Computing Technology, Chinese Academy of Sciences, University of California, Merced, The University of Queensland
[One sentence main contribution]. Audio-Zero introduces a label-free self-evolution framework for LALMs that uses an auditory self-play game to improve fine-grained reasoning without external annotations. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a novel and technically sophisticated approach to enhancing the fine-grained reasoning capabilities of Large Audio Language Models. By formulating the learning process as a self-play game with verifiable intrinsic rewards, Audio-Zero effectively circumvents the bottleneck of scarce fine-grained annotations. The methodology is well-grounded in recent advances in self-evolution and reinforcement learning, and the experimental results demonstrate its efficacy across multiple benchmarks and model architectures. The diagnostic analysis provides valuable insights into how fine-grained auditory descriptions emerge from game pressure. This work is a significant contribution to the field of audio AI, offering a scalable and efficient path to improving model performance.
The paper proposes Audio-Zero, a label-free self-evolution framework for Large Audio Language Models (LALMs). The core innovation is an "auditory self-play game" that converts unlabeled audio contrast pairs into verifiable training signals. The method involves two coupled stages: a listening stage where the model generates descriptive clues for multiple "players" (some hearing a reference, one hearing a variant), and an attribution stage where the model acts as a judge to identify the "odd listener" based on inconsistencies in the clues. The reward signal is intrinsic to the game mechanics (correct identification) and a rule-based content utility score for the clues. The optimization uses Group Relative Policy Optimization (GRPO). The approach is theoretically sound and addresses a genuine gap in the field: the lack of fine-grained, label-free supervision for audio reasoning. The mechanism of using the "odd-one-out" task to force fine-grained perceptual description is clever and aligns well with recent trends in self-play and self-evolution for LLMs and VLMs.
The authors evaluate Audio-Zero on three benchmarks: TREA (temporal reasoning), MMAU Test-mini (general audio understanding), and MMAR (deep audio reasoning). Experiments are conducted on two strong backbones: Qwen2-Audio-7B-Instruct and Qwen2.5-Omni-7B. The results show consistent improvements over the base policies and several strong baselines (both label-dependent and label-free). Notably, the gains are most pronounced on TREA, the fine-grained temporal reasoning benchmark, which supports the paper's claim of improving fine-grained perception. The ablation studies effectively demonstrate the necessity of each component (listening stage, attribution stage, vote-aware feedback). The self-evolution dynamics analysis (tracking win rates, clue length, and benchmark scores over iterations) provides compelling evidence that the model is indeed learning and evolving rather than just overfitting to a static reward. The use of only 2k unlabeled pairs is a significant efficiency advantage.
The paper provides detailed implementation details, including hyperparameters (learning rate, KL coefficient, number of players/rounds), dataset construction (filtering 2k pairs from Audio-Alpaca), and prompt templates. The GRPO implementation details are standard but well-specified. The diagnostic metrics for analyzing clue quality are clearly defined. While the code is not explicitly linked in the provided text, the description is sufficient for reproduction by researchers familiar with RLHF/GRPO pipelines. The reliance on rule-based scorers for the listening reward is a potential point of variability, but the rules are described.
The method relies on the availability of audio contrast pairs. While the paper claims to use "unlabeled" pairs, it sources them from Audio-Alpaca, which is a preference dataset. The filtering process to create contrast pairs (ensuring they share semantics but differ in fine-grained details) is not fully detailed, raising questions about the scalability and generalizability of the pair generation process. The rule-based listening reward, while effective, may not capture all nuances of good auditory description compared to a learned reward model. Furthermore, the performance gains, while significant, are still modest in absolute terms on some benchmarks (e.g., MMAU Avg. 60.50 vs 57.50 for Qwen2-Audio), suggesting that self-evolution alone may not solve all reasoning deficits without some form of structured supervision or larger-scale data.
This work contributes to the development of more autonomous and data-efficient AI systems. By reducing reliance on expensive human annotations, it lowers the barrier to training high-capability audio models. The improved fine-grained audio reasoning capabilities could benefit applications in accessibility (e.g., better descriptions for the visually impaired), audio forensics, and interactive AI assistants. The self-play paradigm could be extended to other modalities or tasks where verifiable rewards are hard to construct. [One sentence main contribution]. Audio-Zero introduces a label-free self-evolution framework for LALMs that uses an auditory self-play game to improve fine-grained reasoning without external annotations. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a novel and technically sophisticated approach to enhancing the fine-grained reasoning capabilities of Large Audio Language Models. By formulating the learning process as a self-play game with verifiable intrinsic rewards, Audio-Zero effectively circumvents the bottleneck of scarce fine-grained annotations. The methodology is well-grounded in recent advances in self-evolution and reinforcement learning, and the experimental results demonstrate its efficacy across multiple benchmarks and model architectures. The diagnostic analysis provides valuable insights into how fine-grained auditory descriptions emerge from game pressure. This work is a significant contribution to the field of audio AI, offering a scalable and efficient path to improving model performance.
Long-form streaming speech-to-speech translation (S2ST) requires incremental, unbounded translation under strict latency constraints. Existing methods typically suffer from sentence-bounded supervision or demand massive paired-S2ST supervision. We introduce a training recipe enabling a speech language model for sentence-level and long-form streaming S2ST using only $\sim$2k hours of paired cross-lingual S2ST data, layered atop auxiliary supervision. Anchored by auxiliary multitask training, our approach remains robust even when the paired-S2ST budget itself is reduced by 90\%. Our core contribution, joint text-code trajectory supervision, schedules target text and acoustic semantic codes as a unified commitment path, eliminating the need for separate, unstable speech-side emission controllers. Furthermore, our two-stream Thinker--Talker factorization significantly outperforms unified-decoder baselines by decoupling linguistic reasoning from dense acoustic prediction to mitigate modality interference. Finally, our system achieves highly competitive quality-latency trade-offs on RealSI and ACL60/60-dev, matching state-of-the-art, closed-source S2ST systems such as LiveInterpret~2.0 on ASR-BLEU.
Primary: The Chinese University of Hong Kong, Shenzhen
All Institutions: The Chinese University of Hong Kong, Shenzhen
SimulS2ST-Omni presents a significant technical contribution to streaming speech-to-speech translation by introducing a joint text-code trajectory supervision method and a decoupled two-stream architecture that achieves state-of-the-art performance with substantially reduced data requirements, effectively addressing the critical bottleneck of paired S2ST data scarcity.
The paper proposes SimulS2ST-Omni, a data-efficient framework for long-form streaming Speech-to-Speech Translation (S2ST). The core methodological contributions are threefold: (1) A joint text-code trajectory supervision mechanism that schedules target text and acoustic semantic codes simultaneously, eliminating the need for unstable, separate speech-side emission controllers. (2) A two-stream "Thinker-Talker" architecture that decouples linguistic reasoning (Thinker) from dense acoustic prediction (Talker), mitigating modality interference common in unified decoders. (3) A training recipe leveraging auxiliary multitask supervision (ASR, S2TT, MT, TTS) to enable robust performance with only ~2k hours of paired S2ST data, a significant reduction from prior works requiring tens of thousands of hours. The approach extends trajectory-based simultaneous translation to speech output by using cross-lingual word alignments and monotonicity filtering (NIR) to construct stable read/wait/write trajectories.
The authors conduct comprehensive experiments on offline S2ST (CVSS-T), sentence-level streaming (RealSI), and long-form streaming (RealSI, ACL60/60-dev). They compare against state-of-the-art public baselines (UniSS, Seamless-M4T) and closed-source systems (LiveInterpret 2.0). Results show that the Thinker-Talker architecture significantly outperforms the matched unified-decoder baseline in translation quality (ASR-BLEU, Text-BLEU) while maintaining competitive or superior speech quality metrics (A.PCP, SIM-O). The system matches or exceeds LiveInterpret 2.0 in several latency tiers. Ablation studies confirm the importance of NIR-based trajectory filtering and the architectural decoupling. The evaluation is rigorous, covering multiple metrics and latency constraints.
The paper provides detailed descriptions of the data construction process, including the use of public ASR/S2TT corpora to synthesize paired S2ST data, the alignment tools (SimAlign), and the filtering criteria (NIR, ASR error rates). Training hyperparameters, stages, and model initialization details are provided in the appendix. The use of open-source components (Qwen2.5-Omni, DualCodec, Whisper, Paraformer) enhances reproducibility. However, the exact codebase for the trajectory construction and the specific synthesized data pool are not explicitly linked to a public repository, which may hinder exact replication.
The experiments are limited to Chinese-English directions due to data scarcity. The paired S2ST data is synthesized rather than naturally recorded simultaneous interpretation, which may not fully capture real-world interpreting phenomena (e.g., disfluencies, interruptions). The speech generation backend (chunk-wise Flow Matching) is not fully streaming-native, potentially limiting inter-chunk coherence and inference efficiency. The work does not address duplex interaction or broader multilingual settings.
This work advances the field of real-time cross-lingual communication by demonstrating that high-quality streaming S2ST is achievable with limited paired data. It addresses ethical concerns regarding voice cloning and impersonation by acknowledging the risks and suggesting safeguards. The data-efficient approach lowers the barrier to entry for developing robust S2ST systems. SimulS2ST-Omni presents a significant technical contribution to streaming speech-to-speech translation by introducing a joint text-code trajectory supervision method and a decoupled two-stream architecture that achieves state-of-the-art performance with substantially reduced data requirements, effectively addressing the critical bottleneck of paired S2ST data scarcity.
Existing symbolic music generation models typically use bars as the basic structural unit. However, human perception of musical phrases often does not align with notated bar lines, leading to long-term structural fragmentation. This paper proposes RPPNet-a two-stage deep learning architecture with variable structural boundaries. It first generates variable-length Rhythm-Pitch Primitive (RPP) sequences, where each RPP encodes note count, rhythm, and contour; then decodes the RPP sequences into concrete notes. The grouping of RPPs is automatically derived from acoustic cues, auditory inertia, and similarity perception based on music psychology. Experiments show that melodies generated by RPPNet are superior in both long-term structure and musicality, with significant improvements across all subjective evaluation dimensions. Ablation studies confirm that the performance gain stems from the structural correctness of the psychological representation, rather than from model capacity. This work offers an interdisciplinary perspective for music generation, integrating music theory, computational modeling, and music psychology.
Primary: Zhejiang University
All Institutions: Zhejiang University, ZJU-ZJCM MusicAI Lab, Zhejiang Conservatory of Music
[One sentence main contribution]. [RPPNet introduces a perceptually-grounded, variable-length structural representation for symbolic music generation, demonstrating that aligning model boundaries with human auditory grouping principles significantly improves long-term structural coherence compared to fixed bar-level approaches.]
The paper proposes RPPNet, a hierarchical two-stage architecture for symbolic melody generation. The core innovation lies in replacing fixed bar-level tokenization with perceptually-grouped Rhythm-Pitch Primitives (RPPs). The authors employ a heuristic algorithm based on music psychology (acoustic cues, auditory inertia, similarity perception) to segment melodies into variable-length RPPs. The generation process is decoupled: an RPP-level Transformer generates the structural skeleton, which is then instantiated into note-level events via a time-scale expansion mapping. The methodology is theoretically grounded in cognitive musicology, offering a novel perspective on structural segmentation that challenges the dominant fixed-grid assumptions in models like Museformer and MELONS. The decoupled serial prediction mechanism at the RPP level is a technically sound approach to managing the complexity of long-sequence generation.
The experimental setup utilizes the MelodyNet dataset. The evaluation relies heavily on subjective listening tests (MOS-style) with 15 participants, which is a standard but resource-intensive method for music generation. RPPNet significantly outperforms baselines (Museformer, MELONS) in subjective metrics, particularly in "Structure" and "Overall Impression." The inclusion of ablation studies (Random-Grouped vs. Heuristic-Grouped) is crucial and effectively isolates the contribution of the psychological grouping algorithm from mere variable-length tokenization. However, the objective metrics are limited; the paper notes that PPL and Structural Error are difficult to compare across different tokenization schemes, leading to a reliance on subjective scores. The gap between generated music and human composition (6.69 vs 7.47) remains significant, though the paper attributes this to rhythmic distribution mismatches.
The paper provides a GitHub repository link, which is a positive step for reproducibility. The heuristic algorithm for RPP grouping is described, but the full formalization of the weight function $W()$ is stated to be in the repository, which may contain implementation-specific details not fully explicit in the text. The dataset (MelodyNet) is public. The subjective evaluation protocol is described, but the small sample size (15 participants) limits the statistical power of the findings, although significance tests are provided.
The primary limitation is the reliance on a heuristic, rule-based algorithm for RPP segmentation, which may not generalize well to all musical styles or complex polyphonic textures. The current model is limited to monophonic melody generation. The subjective evaluation, while showing statistical significance, suffers from a small participant pool and potential bias. The objective metrics do not fully capture the perceptual improvements claimed, as standard metrics like PPL are not directly comparable across different tokenization granularities.
This work bridges the gap between computational music generation and music psychology/cognition. By demonstrating that perceptually-grounded structural units lead to more coherent music, it offers a new paradigm for designing generative models that align with human auditory perception. This could influence future research in hierarchical modeling and structured generation beyond music, such as in language or video, where fixed-grid assumptions may also be limiting. [One sentence main contribution]. [RPPNet introduces a perceptually-grounded, variable-length structural representation for symbolic music generation, demonstrating that aligning model boundaries with human auditory grouping principles significantly improves long-term structural coherence compared to fixed bar-level approaches.]
Keyword spotting (KWS) models on embedded devices often need to add new keywords after deployment, but updates are difficult when original training data are unavailable and regressions on existing triggers are unacceptable. At a fixed operating point, our method reduces average new-keyword false reject rate (FRR) from 6.46 to 4.37 versus a parameter-matched separate-model baseline and outperforms parameter-efficient tuning baselines (adapters, LoRA), while using fewer multiply-accumulate operations (MACs) under the same added-parameter budget ($\leq$10k): 16.34M vs 18.45M/20.52M. We achieve this via parameter-capped modular expansion: the base network, including batch-normalization statistics and the core classifier, is frozen, and only a lightweight expansion branch with a separate new-keyword head is trained, preserving core logits, shipped outputs, and thresholds for existing keywords.
Primary: Yandex
All Institutions: Yandex
The paper presents a practical and effective method for modular expansion of embedded KWS models, achieving strict non-regulation and improved new-keyword detection through a frozen-base, parallel-branch architecture.
The paper proposes a "modular network expansion" technique for Keyword Spotting (KWS) on embedded devices. The core innovation is architectural: it freezes the entire base SVDF network (including Batch Normalization statistics) and attaches a lightweight, trainable expansion branch with a separate classifier head for new keywords. This design ensures a strict non-regression guarantee for existing keywords because the core logits remain identical to the pre-deployment model. The method is compared against parameter-efficient fine-tuning (LoRA, Adapters), ensemble methods, and continual learning (EWC). The approach is pragmatic and well-suited for the specific constraints of embedded deployment (no original data, strict latency/parameter budgets). While the concept of "parallel heads" or "expert routing" is not new in the broader ML landscape (e.g., Mixture of Experts, Progressive Neural Networks), its specific application to strict non-regression KWS updates with frozen BN layers is a solid, incremental engineering contribution. It lacks deep theoretical novelty but offers a robust practical solution.
The experiments are conducted on Google Speech Commands v2 (GSC) using five held-out keyword pairs. The evaluation metric is macro False Reject Rate (FRR) at a fixed False Accept Rate (FAR). The results show that the proposed method significantly outperforms the Ensemble baseline (6.46% vs 4.37% FRR) and other parameter-efficient methods (LoRA, Adapters) under a 10k parameter budget. The ablation study on expansion depth is useful, identifying that tapping 4 blocks is optimal. The comparison with Full Fine-tuning highlights the catastrophic forgetting problem, and EWC is shown to be less effective without replay data. The use of Mozilla Common Voice for negative samples is a good practice for realistic FAR calibration. The results are statistically significant (95% confidence intervals provided). However, the evaluation is limited to GSC, which is a relatively clean, controlled dataset. Real-world embedded KWS faces more challenges with noise, channel variation, and long-tail distributions.
The paper provides sufficient detail regarding the architecture (SVDF, BN, Hard-Swish), training setup (Adam, cosine LR, SpecAugment), and evaluation protocol (FAR calibration, 8 runs). The parameter budget (10k) and MAC counts are specified. The code is not explicitly linked, but the description is detailed enough for reproduction. The use of standard datasets (GSC, Common Voice) aids reproducibility.
The primary limitation is the reliance on GSC, which may not reflect the complexity of real-world embedded audio environments. The method assumes that the base model's features are sufficiently generalizable to the new keywords via the expansion branch; if the new keywords are phonetically very distinct from the core set, performance might degrade. The "core-first" decision rule introduces a slight latency overhead (checking core first, then new), although the paper argues this is negligible or manageable. The method is specific to KWS and may not generalize easily to other incremental learning tasks without modification.
This work has significant practical impact for the deployment of always-on voice assistants on resource-constrained devices. By enabling safe, data-efficient updates to keyword inventories, it addresses a major pain point in product lifecycle management for IoT devices. It reduces the need for retraining on large datasets and mitigates the risk of degrading user experience for existing features. The paper presents a practical and effective method for modular expansion of embedded KWS models, achieving strict non-regulation and improved new-keyword detection through a frozen-base, parallel-branch architecture.
Large-scale visual generators are increasingly capable but costly to train, fine-tune, and deploy. We introduce Mage-Flow, a compact 4B-scale generative stack for efficient text-to-image generation and instruction-based image editing. The stack is built from two co-designed components: Mage-VAE, a lightweight high-fidelity latent tokenizer, and a Native-Resolution Multimodal Diffusion Transformer trained with rectified flow matching. Mage-VAE uses one-step diffusion-style encoding and decoding with anchor-latent regularization, preserving the reconstruction quality of strong public VAEs while reducing tokenization cost by more than an order of magnitude. Together with native-resolution packing and stack-level CUDA kernel fusion, the stack supports flexible-resolution training and improves end-to-end training throughput by about $2.5\times$. Built on this foundation, we develop a complete model family with Base, RL-aligned, and Turbo variants for both generation and editing. Diffusion-NFT improves prompt following, text rendering, aesthetic quality, and editing fidelity, while few-step distillation with adversarial perceptual guidance produces 4-step Turbo models for low-latency inference. Despite its compact scale, Mage-Flow and Mage-Flow-Edit achieves competitive performance across standard generation and editing benchmarks. More importantly, the Turbo variants make high-resolution generation and editing practical for interactive use: at $1024^2$ resolution on a single NVIDIA A100 GPU, Mage-Flow-Turbo generates an image in 0.59s, and Mage-Flow-Edit-Turbo edits an image in 1.02s, while maintaining a small memory footprint. These results show that careful tokenizer--backbone--system co-design can deliver strong high-resolution generation and editing within an efficient 4B model family.
Primary: Microsoft
All Institutions: Microsoft
Mage-Flow presents a compelling case for system-level co-design in generative AI, demonstrating that a compact 4B model family can achieve state-of-the-art efficiency and competitive quality in text-to-image generation and editing through innovative tokenizer design, native-resolution training, and kernel fusion.
The paper proposes a co-designed generative stack, Mage-Flow, centered on system-level efficiency rather than just model scaling. The core methodological contributions are threefold. First, the introduction of Mage-VAE, a lightweight latent tokenizer that replaces standard convolutional VAEs with one-step diffusion-style encoding and decoding. This is a significant architectural shift, treating the tokenizer as a learned image codec distilled from a strong teacher (FLUX.2-VAE) using anchor-latent regularization to preserve the generation-ready latent geometry. Second, the Native-Resolution MMDiT (NR-MMDiT) backbone abandons fixed-resolution bucketing in favor of native-resolution packing. By utilizing variable-length sequence packing with FlashAttention and per-sample 2D RoPE, the model handles arbitrary aspect ratios and resolutions natively, improving training diversity and inference efficiency for classifier-free guidance. Third, the paper details a comprehensive system-level optimization strategy, including stack-level CUDA kernel fusion to mitigate memory-bound overheads in the VAE, text encoder, and diffusion transformer. The approach is technically sound, leveraging recent advances in diffusion codecs (CoD-Lite) and efficient attention mechanisms to solve the latency bottleneck inherent in high-resolution generation.
The authors provide extensive evaluations covering reconstruction quality, inference speed, memory footprint, and downstream generation/editing performance. Mage-VAE demonstrates reconstruction fidelity comparable to FLUX.2-VAE while reducing computational cost by over an order of magnitude. The generation and editing benchmarks (GenEval, GEdit-Bench, TIIF, etc.) show that the 4B Mage-Flow models achieve competitive or superior performance against much larger open-source models (6B-80B parameters). The Turbo variants (4-step distillation) achieve impressive latency metrics (0.59s for generation, 1.02s for editing on a single A100), making interactive use practical. The ablation studies on tokenizer compatibility and native-resolution packing effectively validate the design choices. The results are robust and clearly demonstrate the quality-efficiency trade-off.
The paper provides significant details for reproduction, including the architecture of the VAE, the training objectives (flow matching, DMD, adversarial guidance), and the data curation pipeline. The release of code, models, and a project page enhances reproducibility. However, the specific hyperparameters for the anchor-latent KL regularization and the exact composition of the 1.3B curated dataset are not fully detailed in the text, though the curation pipeline is described qualitatively. The reliance on proprietary text encoders (Qwen3-VL) and the specific CUDA kernel implementations may require additional effort to replicate exactly, but the overall framework is well-documented.
The primary limitation is the reliance on a proprietary text encoder (Qwen3-VL) for conditioning, which prevents full open-source reproducibility of the conditioning pipeline. Additionally, while the VAE is efficient, the one-step diffusion decoding still requires more compute than standard convolutional decoders, though significantly less than multi-step approaches. The native-resolution packing, while efficient, may introduce complexity in distributed training synchronization compared to fixed-bucket approaches. The model's performance, while competitive, is still behind closed-source giants in absolute aesthetic quality, particularly in complex text rendering and fine-grained compositional control, which is expected given the 4B parameter scale.
This work significantly lowers the barrier to entry for high-quality, high-resolution image generation and editing by providing a compact, efficient, and research-friendly foundation model. By demonstrating that strong performance is achievable at 4B parameters through careful co-design, it challenges the trend of endless scaling and promotes sustainable AI development. The efficient inference capabilities enable deployment on consumer-grade hardware, democratizing access to advanced generative tools for developers and researchers. The focus on editing also supports creative workflows and iterative design processes. Mage-Flow presents a compelling case for system-level co-design in generative AI, demonstrating that a compact 4B model family can achieve state-of-the-art efficiency and competitive quality in text-to-image generation and editing through innovative tokenizer design, native-resolution training, and kernel fusion.
Modern ASR models trained on heterogeneously annotated data treat transcription style (verbatim vs. intended) as an uncontrolled latent variable, causing measurable decoding instability, evaluation confounding (up to 60% of reported WER attributable to style mismatch), and unreliable word-level timing. We show that models already encode both styles; the challenge is controlled activation. Using coverage-aware decoder task tokens trained on parallel verbatim/intended transcript pairs, we raise German disfluency F1 from 10% to 79% zero-shot, despite English-only training. Full English-only fine-tuning surpasses all baselines in verbatim accuracy, disfluency detection, and intended-mode quality across both languages. We further introduce supervised cross-attention fine-tuning that improves word-level timestamps on disfluent speech beyond forced-alignment baselines. Finally, we propose verbatimize, a new task enabling scalable creation and enrichment of speech corpora with high-quality canonical verbatim transcriptions.
Primary: nyra labs
All Institutions: nyra labs
The paper presents a significant advancement in controllable ASR by explicitly parameterizing transcription policy, demonstrating that pretrained models possess latent capabilities for verbatim transcription that can be activated through targeted fine-tuning and explicit conditioning.
The paper proposes a novel framework for controlling transcription style (verbatim vs. intended) in Automatic Speech Recognition (ASR) models by treating the transcription policy as an explicit latent variable. The core technical contribution is the introduction of "mode tags" (discrete decoder-prefix tokens) that condition the model on the desired output style. This is complemented by a supervised cross-attention mechanism for word-level timing, which addresses the instability of emergent alignments in disfluent speech. Additionally, the authors introduce "verbatimize," a task for retroactively annotating verbatim transcripts from intended ones. The methodology is sound, leveraging the latent capabilities of pretrained Whisper models through efficient fine-tuning (Stage 1: embeddings only; Stage 2: decoder unfrozen). The approach to decoupling style control from acoustic modeling is innovative and addresses a significant gap in current ASR systems.
The experimental evaluation is comprehensive, covering disfluency detection, intended transcript quality, and word-level timing across English and German. The results demonstrate significant improvements: German disfluency F1 jumps from 10% to 79% in a zero-shot setting with only mode tag embeddings trained. The timing experiments show that the supervised cross-attention method outperforms strong baselines like MFA and WhisperX on disfluent speech (FluencyBank), achieving lower Mean Absolute Error (MAE). The ablation studies on head selection and sharpening techniques provide valuable insights into the mechanics of attention-based alignment. The use of a newly created German DisfluencySpeech dataset adds value, although the small size (202 utterances) limits the statistical power of the German results.
The paper provides detailed descriptions of the training stages, hyperparameters (learning rates, batch size), and data preprocessing steps. The code repository is linked, which enhances reproducibility. The use of standard benchmarks (TIMIT, FluencyBank, AMI) allows for comparison with existing work. However, the reliance on GPT-4o for generating intended transcripts and the creation of the German dataset by non-trained speakers introduces potential biases that are not fully quantified. The specific implementation details of the "virtual blank" pause detection and Viterbi decoding are described but might require careful implementation to replicate exactly.
The primary limitation is the reliance on paired verbatim/intended data for training, which is scarce. The authors mitigate this by using GPT-4o to generate intended transcripts from verbatim ones, which may not perfectly capture human annotation nuances. The German evaluation set is small and recorded by non-native speakers, potentially affecting the naturalness and generalizability of the results. The method is evaluated primarily on English and German; cross-lingual transfer to more distant languages is not thoroughly tested. Furthermore, the "verbatimize" task, while innovative, depends on the accuracy of the underlying intended transcript and the model's ability to correctly insert disfluencies, which may not always align with human perception of "verbatim" speech.
This work has significant implications for ASR systems used in clinical, legal, and linguistic contexts where verbatim transcription is crucial. By enabling controllable verbatim ASR, it opens up new possibilities for analyzing spontaneous speech, detecting speech disorders, and improving the quality of transcripts for accessibility. The "verbatimize" tool could facilitate the creation of large-scale annotated datasets, addressing a major bottleneck in speech research. However, the potential for misuse in generating realistic but fake verbatim transcripts (deepfakes) should be considered. The improvement in timing accuracy also benefits downstream tasks like speaker diarization and video captioning. The paper presents a significant advancement in controllable ASR by explicitly parameterizing transcription policy, demonstrating that pretrained models possess latent capabilities for verbatim transcription that can be activated through targeted fine-tuning and explicit conditioning.
Auditory attention decoding (AAD) identifies the speaker a listener attends to from neural responses like electroencephalography (EEG), making it a key algorithm in neuro-steered hearing aids. However, most neural AAD models are trained as independent short-window classifiers, despite auditory attention being a temporally persistent cognitive state and short-window EEG--audio evidence often being noisy and ambiguous. We propose an end-to-end Markov AAD framework based on conditional random field (CRF) that trains window-level neural emissions under a two-state attention prior. The framework treats the logits of any AAD backbone as Markov emissions, learns the transition rate from a standard HMM initialization, and jointly optimizes cross-entropy and CRF objectives, allowing temporal continuity to guide representation learning rather than merely smoothing predictions after training. We also introduce ESCNet, an EEG--speech correlation backbone that preserves time-aligned features and converts the difference between two mean Pearson correlations into state logits. We evaluate the framework with four emission backbones spanning correlation-based, convolutional, recurrent, and attention-based designs. On the dynamic AVGC dataset, CRF training generally outperforms post-hoc HMM smoothing; with ESCNet, it achieves $86.5\%$ causal and $92.4\%$ non-causal accuracy using $1$s windows. On the static KUL and USTC datasets, it improves causal decoding over fixed-rate post-hoc HMM baselines by $5.6\%$ and $2.0\%$, respectively, showing the superiority of learning AAD as attention state sequence over isolated-window classification.
Primary: University of Science and Technology of China
All Institutions: University of Science and Technology of China, Guangzhou Maritime University, iFLYTEK Company, Ltd.
The paper presents a robust and theoretically grounded advancement in Auditory Attention Decoding by integrating end-to-end CRF training with neural backbones, demonstrating consistent improvements in accuracy and responsiveness across multiple datasets and architectures.
The paper proposes a significant methodological shift in Auditory Attention Decoding (AAD) by moving from independent window classification to end-to-end Markov State Sequence Learning using Conditional Random Fields (CRF). The core innovation lies in treating the logits of any AAD backbone as emissions for a two-state CRF, allowing the temporal structure of attention (modeled via a transition prior) to guide the learning of neural representations directly, rather than just smoothing predictions post-hoc. The authors also introduce ESCNet, a specialized correlation-based backbone that preserves time alignment and outputs interpretable logits. This approach effectively bridges the gap between local feature extraction and global sequence coherence, addressing the inherent noise and ambiguity in short-window EEG data. The formulation is mathematically sound, leveraging standard forward-backward algorithms for differentiable sequence scoring.
The experimental evaluation is comprehensive and well-structured. The authors test their framework on three distinct datasets (AVGC, KUL, USTC) covering both dynamic attention switching and static persistence scenarios. They evaluate four heterogeneous backbones (ESCNet, AADNet, LSTM, Attn-GRU), demonstrating that the CRF training paradigm is architecture-agnostic and provides consistent improvements over fixed-rate post-hoc HMM baselines. The results show statistically significant gains, particularly in causal decoding on the dynamic AVGC dataset (86.5% accuracy with 1s windows). The inclusion of ablation studies isolating the effects of transition rate learning versus sequence-aware emission learning adds rigor to the claims. The switch delay analysis further validates the responsiveness of the model.
The paper provides detailed implementation details, including preprocessing steps, network architectures, hyperparameters (learning rates, batch sizes, optimizer settings), and training schedules. Crucially, the source code is made publicly available on GitHub, which significantly enhances reproducibility. The datasets used are public (AVGC, KUL, USTC), and the evaluation protocols (leave-one-trial-out) are clearly defined. The description of the CRF training phase, including the warm-up strategy and joint optimization, is sufficiently detailed for replication.
The primary limitation is the reliance on controlled laboratory datasets with normal-hearing participants. The performance on real-world, noisy, or hearing-impaired populations remains unverified. Additionally, the causal switch delay, while improved, is still around 23 seconds for the 1s window setting, which may be too slow for some real-time hearing aid applications requiring immediate adaptation. The study also assumes access to two candidate speech streams, which is a constraint for general multi-talker scenarios with more than two speakers. The static datasets (KUL, USTC) do not capture the complexity of naturalistic attention switching, potentially overestimating the robustness of the model in highly dynamic environments.
This work has significant implications for the development of neuro-steered hearing aids and brain-computer interfaces. By improving the accuracy and responsiveness of auditory attention decoding, it can enhance speech understanding in noisy environments for individuals with hearing loss. The end-to-end framework offers a generalizable template for incorporating temporal priors into other neural decoding tasks where state persistence is a key factor. The open-source release of the code and the detailed benchmarking contribute to the broader ML and neuroscience communities by providing a robust baseline for future research in EEG-based attention decoding. The paper presents a robust and theoretically grounded advancement in Auditory Attention Decoding by integrating end-to-end CRF training with neural backbones, demonstrating consistent improvements in accuracy and responsiveness across multiple datasets and architectures.
Almost every piece of recorded music you have ever heard was modified before it reached you; commercial releases rarely spring fully-formed from the mind of a musician. Despite the promise of music generation models for one-shot output, such fine-grained iterative refinement workflows are a complementary problem, and largely out of their reach. There is also a gap for musicians: while they can express what they want to hear, not all have the facility with studio production tools to implement the complex set of actions needed to realize these intuitions. We formalize this task as agentic post-production, wherein individual aspects of a song are targeted, refined, and combined into a final track. We argue the bottleneck is data: existing corpora do not reflect how realistic post-production chains map onto the vocabulary musicians and engineers actually use. We argue there is a language for modifying recorded music that is dense, consistent, and learnable. We introduce the Rule-based Instructions for Music Editing (RIME) framework, which generates realistic paired edit-instruction data from any baseline music dataset grounded in canonical methods, design patterns, and constraints derived from real production workflows. RIME leverages POEMS, a new toolkit that combines stem separation, mixing, and common studio effects for use by multimodal agents. We use RIME and POEMS to generate 3,000 pairs of edit instructions and ground truth audio, and use this data to evaluate existing multimodal LLMs as agents on this task, showing persistent challenges in current models' post-production capabilities. We also demonstrate RIME's ability to improve post-production agent performance via supervised fine-tuning. We see RIME as an early step toward iterative musical agents, collaborative systems that could transform music production much as interactive coding agents have reshaped software engineering.
Primary: Dartmouth College
All Institutions: Dartmouth College
This paper presents RIME and POEMS, a framework and toolkit for generating synthetic data and benchmarking agentic post-production agents, addressing the gap between one-shot music generation and iterative studio workflows.
The paper introduces RIME, a framework for generating synthetic edit-instruction data for music post-production, and POEMS, a toolkit exposing audio processing tools via the Model Context Protocol (MCP). The methodology is sound and addresses a specific gap: the lack of structured, iterative post-production data compared to one-shot generation data. The use of rule-based recipes, motifs, and parameter priors to generate grounded (input, output, instruction) triples is a clever approach to synthesizing training data. However, the novelty is somewhat limited by the reliance on existing, well-understood signal processing libraries (Pedalboard, Demucs) and standard LLM tool-calling architectures. The core contribution is the *system design* and *benchmark creation* rather than a new algorithmic breakthrough in audio synthesis or model architecture. The abstraction layer generation for prompts is a nice touch for simulating real-world ambiguity.
The experiments evaluate zero-shot and fine-tuned multimodal LLMs on the generated RIME benchmark. The results clearly demonstrate that current models struggle with agentic post-production, particularly in parameter inference and chain assembly. The supervised fine-tuning results show improvement, validating the utility of the synthetic data. The evaluation metrics (FAD, KAD, Edit Similarity, Graph F1) are appropriate for the task. The inclusion of an artifact-removal ("poisoning") split adds depth. However, the dataset size (3,000 pairs) is relatively small for modern LLM benchmarks, though reasonable for this specific, high-fidelity audio task. The reliance on MERT embeddings for audio similarity is standard but has known limitations in capturing perceptual quality compared to human listening tests, which are absent here.
The paper provides detailed descriptions of the POEMS toolkit, recipe schemas, and agent architecture. The use of open-weight models for fine-tuning and standard libraries for audio processing aids reproducibility. However, the specific implementation details of the "prompt templates" and the exact "parameter priors" are referenced but not fully enumerated in the main text (though likely in the appendix, which is partially provided). The lack of a public code repository URL in the provided text is a minor hindrance, but the description is sufficient for a competent researcher to reconstruct the system.
The primary limitation is the dependence on the quality of the underlying stem separation model (Demucs), which can introduce artifacts. The rule-based nature of RIME means it can only generate edits present in the recipe catalog, limiting the diversity of edits compared to fully generative approaches. The evaluation relies heavily on embedding-based metrics, which may not correlate perfectly with human preference for audio quality. The dataset size is small, potentially limiting the generalizability of the fine-tuned models. The focus on specific instrument stems (vocals, drums, bass, etc.) via Demucs limits the scope to tracks where these stems can be reliably separated.
This work has significant potential impact on the field of AI-assisted music production. By formalizing agentic post-production and providing a benchmark, it opens up a new avenue for research beyond one-shot generation. It highlights the challenges of iterative, tool-using agents in the audio domain. The framework could empower musicians with less technical expertise to achieve professional-quality mixes through natural language interaction. It also serves as a testbed for improving multimodal LLMs' reasoning and tool-use capabilities in a complex, structured domain. This paper presents RIME and POEMS, a framework and toolkit for generating synthetic data and benchmarking agentic post-production agents, addressing the gap between one-shot music generation and iterative studio workflows.
DCASE~2026 Task~5 introduces Audio-Dependent Question Answering (ADQA), which tests whether large audio-language models answer from the audio rather than from textual priors. An Audio-Dependency Filtering (ADF) pipeline combines silent-audio probing, per-option perplexity, a large language model (LLM) commonsense check, and human review to remove items solvable from text alone. The 3000 items that pass form the ADQA-Bench evaluation set, spanning music, speech, and environmental audio. The inaugural edition draws 14 teams and 36 submissions across two tracks defined by total parameter count (up to 100B and under 10B). A Chung-Ang University ensemble of MOSS-Audio-8B-Thinking and Qwen3-Omni-30B reaches the top overall accuracy at \pct{58.33}, and a MOSS-only configuration from the same team leads the sub-10B track at \pct{57.30}. Across the 30 submissions with a comparable development score, evaluation accuracy falls by 11.91 percentage points (pp) on average (median 10.91\,pp) on the hidden evaluation split, which is designed to be harder than the development split. The most common building blocks are: the MOSS-Audio-8B-Thinking backbone (13 of 36 submissions), Low-Rank Adaptation (LoRA) fine-tuning on AudioMCQ-StrongAC, and preference or reinforcement-learning objectives -- Group Relative Policy Optimization (GRPO) in five teams, Group reward-Decoupled Normalization Policy Optimization (GDPO) in two. At test time, prompt engineering is near-universal, and majority or choice-permutation voting is common. Every system misses the same set of 233 evaluation items.
Primary: The Chinese University of Hong Kong
All Institutions: The Chinese University of Hong Kong, Ant Group, University of Rochester, Shanghai Jiao Tong University, King's College London
This paper presents the DCASE 2026 Task 5 challenge, introducing the ADQA-Bench, a rigorously filtered benchmark for Audio-Dependent Question Answering that effectively exposes and mitigates textual shortcutting in Large Audio-Language Models, thereby setting a new standard for evaluating genuine audio understanding capabilities.
The paper introduces Audio-Dependent Question Answering (ADQA), a novel benchmark designed to mitigate textual priors in Large Audio-Language Models (LALMs). The core methodological contribution is the Audio-Dependency Filtering (ADF) pipeline, a multi-stage process combining silent-audio probing, perplexity filtering, LLM commonsense checks, and human verification to curate a dataset where answers cannot be derived from text alone. The paper also provides a comprehensive analysis of 36 submissions, identifying common techniques such as LoRA fine-tuning, GRPO/GDPO reinforcement learning, and choice-permutation ensembling. While the benchmark creation is methodologically sound, the paper itself is a challenge summary rather than a proposal of a new algorithmic architecture, focusing on empirical observation of existing methods.
The evaluation is rigorous in scope, covering 3000 items across music, speech, and environmental audio. The paper highlights a significant performance gap between development and hidden evaluation sets (average drop of ~12 pp), effectively demonstrating the difficulty of the benchmark and the prevalence of overfitting to development artifacts. The analysis of universal failures (233 items missed by all systems) provides valuable insight into current model limitations. The use of multiple tracks (parameter size) allows for a nuanced comparison of efficiency versus performance. However, the lack of ablation studies isolating specific components (e.g., the acoustic tagger vs. ensemble) limits the depth of causal claims regarding what drives success.
The paper provides detailed descriptions of the ADF pipeline and the training data (AudioMCQ-StrongAC). It lists the base models and techniques used by top teams, offering high transparency. However, as a challenge summary, it does not provide a single unified codebase. Reproducibility depends on the individual team submissions, most of which are open-source. The specific hyperparameters and exact filtering thresholds for the ADF pipeline are described conceptually but may require checking individual team papers for full implementation details.
The paper acknowledges that the evaluation set, while filtered, may still contain subtle biases. The performance drop from dev to eval suggests that the development set, while harder than previous benchmarks, is not perfectly representative of the generalization capability required for the hidden test set. Furthermore, the reliance on multiple-choice questions inherently limits the assessment of open-ended audio reasoning capabilities. The "universal failure" set indicates that current LALM backbones have fundamental gaps in audio grounding that no amount of prompt engineering or RL can fully bridge.
This work significantly advances the field of multimodal AI by establishing a more rigorous standard for evaluating audio-language models. By explicitly targeting and removing textual shortcuts, ADQA-Bench pushes the community to develop models that truly understand audio signals rather than exploiting dataset artifacts. This has implications for improving the reliability of audio assistants, accessibility tools, and automated content analysis systems. The findings also highlight the need for better audio encoders and pre-training strategies that integrate acoustic features more deeply with language models. This paper presents the DCASE 2026 Task 5 challenge, introducing the ADQA-Bench, a rigorously filtered benchmark for Audio-Dependent Question Answering that effectively exposes and mitigates textual shortcutting in Large Audio-Language Models, thereby setting a new standard for evaluating genuine audio understanding capabilities.
Multi-channel speech enhancement (SE) systems exhibit superior performance over single-channel methods but are constrained to fixed microphone array configurations. This restricts their real-world deployment across devices with diverse array geometries. While recent array-agnostic SE methods address variable microphone numbers and permutations, they largely fail to exploit explicit array geometry priors when available, missing a crucial cue for optimal spatial filtering. A Geometry-Aware Dynamic Convolution (Geo-DConv) framework is proposed, which explicitly leverages microphone coordinates to transform standard fixed-array SE models into robust array-invariant systems. Experiments are conducted on the recent real-recorded RealMAN multi-channel speech dataset. Results demonstrate that the proposed architecture enables two widely used fixed-array models to adapt to array-invariant settings, with consistent performance improvements across diverse array topologies.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University, VUI Labs
The paper presents a practical and effective method for adapting fixed-array speech enhancement models to arbitrary microphone geometries using explicit geometric priors via dynamic convolution, demonstrating improved generalization on real-world data.
The paper proposes a Geometry-Aware Dynamic Convolution (Geo-DConv) module to enable fixed-array speech enhancement models to handle arbitrary microphone array geometries. The core innovation lies in using a Topology-Aware Coordinate Transformer (TACT) to generate dynamic convolution weights based on relative microphone coordinates. This approach attempts to bridge the gap between fixed-array models (which exploit geometry implicitly through training) and array-agnostic models (which often ignore explicit geometry). The method is theoretically sound, leveraging Fourier positional encoding and self-attention to ensure permutation equivariance. However, the novelty is moderate as dynamic convolution and coordinate-based conditioning are established techniques in other domains (e.g., medical imaging, computer vision), and their application here is an incremental adaptation rather than a fundamental algorithmic breakthrough.
The experiments are conducted on the RealMAN dataset, which is a significant strength as it uses real-recorded data, addressing the common sim-to-real gap in multi-channel SE. The authors compare their method against strong baselines (FaSNet-TAC, USES2-comp, SpatialNet, TF-GridNet). The results show that Geo-DConv improves upon existing array-agnostic methods and allows fixed-array models to generalize to unseen array configurations. The evaluation includes metrics like SDR, SI-SDR, PESQ, STOI, and DNSMOS. The generalization to the CHiME-4 dataset (6 mics) is a positive finding. However, the performance gains, while consistent, are not massive, and the computational overhead of the TACT module is not extensively analyzed in terms of inference latency compared to the baseline fixed-array models.
The paper provides detailed architectural descriptions, including hyperparameters for the TACT module (hidden dimension, number of heads, layers) and the Geo-DConv layer (basis dimension, output channels). The dataset (RealMAN) is publicly available, and the implementation details (STFT parameters, sampling rate) are clear. The code is not explicitly linked, but the mathematical formulations are sufficient for reproduction by a competent researcher. The use of standard libraries for STFT and common neural network components further aids reproducibility.
The primary limitation is the assumption that microphone coordinates are known or can be accurately estimated. In real-world scenarios with unknown or dynamically changing arrays, this prior might not be available. Additionally, the method relies on a fixed basis dimension for dynamic convolution, which might limit the capacity to model highly complex or large-scale array geometries compared to methods that process raw channel interactions directly. The performance on very small arrays (1-2 mics) is still constrained by the fundamental limits of single-channel or stereo enhancement. The paper does not extensively discuss the sensitivity of the TACT module to coordinate normalization or scaling.
This work contributes to the broader goal of robust, device-agnostic audio processing. By enabling fixed-array models to adapt to various geometries, it reduces the need for device-specific training and deployment, facilitating the integration of advanced SE into diverse consumer devices (smartphones, laptops, IoT devices) with varying hardware configurations. It highlights the importance of explicit geometric priors in multi-channel audio tasks. The paper presents a practical and effective method for adapting fixed-array speech enhancement models to arbitrary microphone geometries using explicit geometric priors via dynamic convolution, demonstrating improved generalization on real-world data.
Synthesizing physically plausible impact sounds from visual observations remains a great challenge in multi-modal AI. Existing 3D-aware audio generation methods primarily model the surface geometry of hollow rigid bodies. However, they fundamentally overlook internal filling states, a critical physical factor that drastically modulates acoustic resonance and damping. To address this issue, we have defined a new task called Fine-Grained Filling-Aware Impact Sound Generation. As a foundational step, we first introduce the fine-grained fill-aware dataset (FillImpact), a pioneering multi-modal collection comprising over 5,000 rigorous acoustic recordings from 88 diverse real-world objects. It captures impact interactions with varying internal contents (i.e., water, rice), a continuous range of fill levels, and distinct striker materials. Furthermore, comprehensive acoustic analysis confirms that the collected data closely aligns with established physical laws governing acoustic resonance and damping, indicating its suitability for physically grounded modeling. Building on this dataset, we propose a novel generative framework (FillGauss) that integrates 3D Gaussian Splatting (3DGS) with internal state conditioning for sound generation. By fusing 3DGS geometric features, precise 3D spatial strike coordinates, and fine-grained textual physical conditions within a latent diffusion architecture, FillGauss enables position-aware, striker-aware, and filling-aware audio generation. Extensive experiments demonstrate that our approach could generate high-fidelity impact sounds that adhere to underlying physical principles, establishing a new state-of-the-art for physically grounded cross-modal audio generation.
Primary: Unknown (Affiliations not provided in text)
All Institutions: Unknown
This paper introduces FillGauss, a novel framework for fine-grained, filling-aware impact sound generation that leverages 3D Gaussian Splatting and latent diffusion to synthesize physically plausible audio conditioned on internal object states, supported by the pioneering FillImpact dataset.
The paper proposes FillGauss, a framework for fine-grained, filling-aware impact sound generation using 3D Gaussian Splatting (3DGS). The methodology integrates three modalities: 3DGS geometric features, precise 3D strike coordinates, and textual descriptions of internal states (fill material and level). The core technical contribution lies in the cross-modal attention fusion mechanism, which entangles spatial strike context with internal physical semantics before conditioning a Latent Diffusion Transformer (DiT). The use of 3DGS as a geometric prior for audio generation is a novel application, moving beyond traditional mesh or point-cloud representations. However, the underlying diffusion architecture (TangoFlux) is a pre-existing model, and the primary innovation is in the conditioning pipeline and dataset construction rather than a new generative backbone. The approach of using text prompts to encode physical states (fill level, material) is effective but relies heavily on the semantic capabilities of the pre-trained language model, which may limit fine-grained control compared to continuous numerical conditioning.
The experimental setup is robust, centered on the introduction of the FillImpact dataset, which contains over 5,000 recordings from 88 objects with varying internal states. This dataset is a significant contribution, addressing a gap in existing literature where internal filling states are largely ignored. The evaluation includes objective metrics (FAD, KL, IS) and subjective human perceptual studies (MOS, Win Rate, Attribute Matching). The results demonstrate that FillGauss outperforms baselines (SonicGauss, TangoFlux) in generating physically plausible sounds, particularly in capturing the acoustic differences between liquid and granular fills. The ablation studies effectively validate the importance of the cross-attention modules and the text-prompting strategy. The sensitivity analysis regarding 3DGS sparsity provides valuable insights into the model's reliance on geometric detail.
The paper provides detailed descriptions of the data collection pipeline, including the use of specific hardware (DJI Mic, Revo Scan) and software (Hunyuan 3D, Stable Audio Open VAE). The normalization procedures for 3DGS and strike coordinates are clearly defined. The training details, including the use of Huber loss and the specific architecture of the DiT, are described. However, as the code and dataset are not explicitly linked in the provided text (URLs are "none"), reproducibility depends on the authors' future release of these assets. The description of the dataset construction is sufficiently detailed for another group to replicate the data collection process.
The paper acknowledges that extreme sparsification of 3DGS degrades performance, indicating a dependency on high-fidelity geometric reconstruction. The model's ability to handle complex internal dynamics like sloshing fluids or soft-body deformations is limited to static fill levels and materials. The reliance on text prompts for internal state conditioning might introduce semantic ambiguities or lack the precision of direct numerical inputs for continuous variables like fill level. Furthermore, the subjective evaluation, while rigorous, is limited by the number of participants (20) and the specific test set.
This work significantly advances the field of physically grounded multimodal AI, particularly for applications in virtual reality, gaming, and robotic simulation where accurate auditory feedback is crucial for immersion and task performance. By explicitly modeling internal states, it enables more realistic interactions with a wider variety of objects. The FillImpact dataset serves as a valuable resource for future research in acoustic physics and multimodal learning. The approach could be extended to other physical properties, such as temperature or material degradation, further enriching the realism of synthetic environments. This paper introduces FillGauss, a novel framework for fine-grained, filling-aware impact sound generation that leverages 3D Gaussian Splatting and latent diffusion to synthesize physically plausible audio conditioned on internal object states, supported by the pioneering FillImpact dataset.
Real-time speech-to-speech translation (S2ST) systems must balance translation quality, latency, speech naturalness, and speaker consistency. Publicly documented S2ST systems have advanced direct, multilingual, streaming, and expressive modeling, while proprietary products and APIs increasingly expose real-time translation capabilities to users. However, practical deployment remains challenging for open and reproducible systems, especially in long-form and multi-speaker conversations where partial ASR hypotheses are unstable, turn boundaries are ambiguous, and target speech must be generated with an appropriate speaker prompt. We present X-Translator, a low-cost modular cascaded S2ST system that combines streaming ASR, machine translation, and prompt-conditioned TTS through a session-level runtime controller. The system uses incremental segment commitment to convert unstable ASR streams into translation-ready units, and an online speaker prompt manager to bind source speech spans to speaker-specific voice prompts for synthesis. We evaluate translation, speech quality, and latency with OpenSTBench, compare against proprietary speech translation APIs as behavioral baselines, measure long-form voice stability, evaluate speaker preservation in multi-speaker conversations, and assess multilingual translation quality. X-Translator provides an open platform for understanding the practical trade-offs of deployment-oriented S2ST. Code and demo are available at https://github.com/zhaoyx239/X-Translator.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University, Shanghai Innovation Institute, Microsoft, AISpeech Co., Ltd.
X-Translator presents a practical, modular, and open-source framework for real-time, multi-speaker speech-to-speech translation, offering valuable insights into the system-level engineering challenges of balancing latency, quality, and speaker consistency in deployment-oriented scenarios.
The paper proposes X-Translator, a modular, cascaded Speech-to-Speech Translation (S2ST) system designed for real-time, long-form, and multi-speaker scenarios. The core technical contribution lies not in novel model architectures (e.g., new ASR or TTS backbones) but in the system-level runtime controller. Specifically, it introduces an "incremental segment commitment" layer to handle unstable streaming ASR hypotheses and an "online speaker prompt manager" to maintain speaker consistency in multi-speaker conversations. The methodology is pragmatic, leveraging existing state-of-the-art open-source components (Qwen3-ASR, LMT-60-8B, X-Voice) and orchestrating them via explicit runtime policies. While the engineering is sound and addresses real deployment pain points (latency, speaker drift, partial hypothesis instability), the novelty is incremental relative to the broader S2ST literature, which has seen significant advances in direct, end-to-end models (e.g., Translatotron, SeamlessM4T). The paper positions itself as a complementary "deployment-oriented" perspective rather than a fundamental algorithmic breakthrough.
The evaluation is comprehensive and well-structured, covering short-form benchmarks (OpenSTBench), long-form single-speaker voice stability, multi-speaker preservation, and multilingual translation. The use of proprietary APIs (Doubao, Qwen LiveTranslate, GPT Realtime Translate) as behavioral baselines is a strong point, providing a realistic comparison against current industry standards. The metrics are appropriate, including translation quality (COMET, BLEU), speech quality (UTMOS), and specific S2ST metrics like speaker similarity (Spk. SIM) and purity measures for multi-speaker settings. The results show that X-Translator achieves competitive speaker preservation and translation quality, though it lags behind proprietary systems in latency and sometimes translation accuracy. The analysis of trade-offs between commitment aggressiveness and quality/latency is insightful. However, the reliance on open-source backbones means the system's performance is bounded by those components, and the "low-cost" aspect comes at the expense of the peak performance seen in proprietary, heavily optimized black-box systems.
The paper provides a high level of reproducibility. The code and demo are available on GitHub. The modular design allows other researchers to swap out components (ASR, MT, TTS) and test different runtime policies. The detailed description of the segment commitment triggers and speaker prompt management logic is clear. The evaluation protocol is well-documented, including the specific datasets (TEDLIUM-3, WenetSpeech, VoxConverse, FLEURS) and the metrics used. The release of runtime logs for error analysis further enhances reproducibility and interpretability.
The paper acknowledges several limitations. First, as a cascaded system, it suffers from error propagation (ASR errors affect MT and TTS) and accumulated latency. Second, the speaker-aware mechanism depends on the quality of online speaker tracking and the availability of clean prompt audio; overlapping speech or rapid turn-taking can degrade performance. Third, the comparison with proprietary systems is behavioral; the internal optimizations of commercial APIs (e.g., joint optimization, larger models, caching) are not visible, making a direct architectural comparison difficult. Finally, the system's performance is limited by the underlying open-source models, which may not match the quality of proprietary counterparts.
X-Translator provides a valuable open-source platform for studying real-time S2ST deployment challenges. By exposing the intermediate decisions (segment commitment, prompt routing), it facilitates research into latency-quality trade-offs and speaker consistency. It democratizes access to real-time S2ST capabilities, allowing researchers to build upon and improve upon existing systems. The focus on multi-speaker and long-form scenarios addresses a significant gap in current S2ST research, which often focuses on short, single-speaker utterances. This work encourages the development of more robust, interpretable, and reproducible speech translation systems. X-Translator presents a practical, modular, and open-source framework for real-time, multi-speaker speech-to-speech translation, offering valuable insights into the system-level engineering challenges of balancing latency, quality, and speaker consistency in deployment-oriented scenarios.
Recent audio large language models (ALLMs) are typically built upon audio encoders trained with large amounts of supervised data. Since self-supervised learning (SSL) audio encoder models are known to learn general-purpose and transferable representations, we investigate whether general-purpose SSL audio representations can serve as an effective foundation for ALLMs. We present SALMONN-2, an ALLM built upon a unified SSL encoder. To better exploit the hierarchical representations learned by SSL encoders, we propose a multi-layer feature fusion (MLF) adapter that aggregates information from all encoder layers before projecting them into the language model. Beyond conventional audio understanding tasks, we further explore multimodal in-context learning (MICL) in ALLMs and study how this capability can be acquired through contextual biasing training. Experimental results show that a general-purpose SSL encoder achieves performance comparable to, or better than, specialised supervised audio encoders while providing a more balanced capability across speech, audio, music and paralinguistic tasks. SALMONN-2 further achieves state-of-the-art performance among comparable-scale open-weight models on ALLM understanding benchmarks, obtaining the best results on MMAU-Pro, MMAR and MMSU. We also show that MICL does not emerge naturally in ALLMs, but can be effectively acquired through targeted contextual biasing training.
Primary: Shanghai Artificial Intelligence Laboratory
All Institutions: University of Cambridge, Shanghai Artificial Intelligence Laboratory, Tsinghua University
SALMONN-2 presents a compelling argument for the use of unified self-supervised audio representations in Audio Large Language Models, achieving state-of-the-art performance with greater efficiency and architectural simplicity than previous multi-encoder approaches. The introduction of the Multi-Layer Feature Fusion adapter and the systematic study of Multimodal In-Context Learning provide valuable technical contributions and insights for the broader field of multimodal AI.
The paper proposes SALMONN-2, an Audio Large Language Model (ALLM) built upon a unified Self-Supervised Learning (SSL) encoder (SPEAR). The core methodological contributions are: 1) The adoption of a single, general-purpose SSL encoder (SPEAR) as a unified front-end for diverse audio tasks (speech, music, environmental sound), challenging the prevalent dual-encoder or supervised-encoder paradigms. 2) A Multi-Layer Feature Fusion (MLF) adapter that aggregates hidden states from all layers of the SSL encoder before projection, aiming to exploit the hierarchical nature of SSL representations (low-level acoustic to high-level semantic). 3) A timestamp injection mechanism using textual tokens from the LLM vocabulary to provide explicit temporal grounding without extra parameters. 4) Explicit training for Multimodal In-Context Learning (MICL) in the context of Contextual ASR, demonstrating that this capability does not emerge naturally from standard instruction tuning. The approach is technically sound, leveraging recent advances in unified SSL models (SPEAR) and applying them effectively to the ALLM paradigm. The MLF adapter is a relatively simple but effective architectural choice compared to more complex cross-attention or Q-Former styles.
The experimental evaluation is comprehensive and rigorous. The authors evaluate SALMONN-2 on a wide range of canonical tasks (ASR, AAC, Music Captioning, Emotion Recognition, Speech Translation, Overlapped Speech Recognition, Speaker Verification) and general audio understanding benchmarks (MMAU-Pro, MMAR, MMSU). They also include extended audio analysis tasks (Sound Event Detection, Spoofing Analysis, Speech Quality Assessment) and a novel MICL evaluation for Contextual ASR. The results show that SALMONN-2 achieves state-of-the-art performance among comparable-scale open-weight models, often outperforming models with larger LLM backbones or more complex encoder setups. The ablation studies effectively isolate the contributions of the SSL encoder, the MLF adapter, and timestamp injection. The MICL experiments provide valuable insights into the limitations of current ALLMs and how targeted training can address them. The use of less than 20K hours of instruction-tuning data compared to competitors is a significant efficiency claim supported by the results.
The paper provides detailed implementation details, including the architecture of the SPEAR encoder, the MLF adapter, the LLM backbones (Qwen series), and the training stages. The code and model are open-sourced on GitHub, which significantly aids reproducibility. The dataset construction for MICL is described, though the specific processing of some external datasets (e.g., HoliAntiSpoof, AudioSet-strong) might require careful attention to licensing and availability. The training hyperparameters and data mix are summarized in tables, providing a clear recipe for replication.
The paper acknowledges that scaling the LLM backbone from 9B to 30B-A3B did not yield substantial improvements on canonical tasks, suggesting these tasks may be perception-heavy rather than reasoning-heavy. However, it does not deeply explore the failure modes of the SSL encoder on out-of-distribution audio or the potential biases introduced by the SSL pre-training data. The MICL capability is demonstrated primarily on Contextual ASR; its generalizability to other multimodal in-context learning scenarios (e.g., audio QA with few-shot examples) is less explored. The reliance on SPEAR, while effective, ties the model to a specific SSL architecture; the generalizability of the MLF adapter to other SSL models (e.g., WavLM, HuBERT) is not extensively tested, although the principles are likely transferable.
SALMONN-2 contributes to the development of more efficient and general-purpose audio intelligence systems. By demonstrating that a single SSL encoder can replace specialized supervised encoders, it reduces the computational and data overhead required to build ALLMs. The explicit handling of multimodal context has implications for robust speech recognition in noisy or domain-specific environments. The open-sourcing of the model and code facilitates further research in audio-language models. Potential negative impacts include the misuse of powerful audio understanding models for surveillance or deepfake detection evasion, though the model's focus on understanding rather than generation mitigates some generation-specific risks. SALMONN-2 presents a compelling argument for the use of unified self-supervised audio representations in Audio Large Language Models, achieving state-of-the-art performance with greater efficiency and architectural simplicity than previous multi-encoder approaches. The introduction of the Multi-Layer Feature Fusion adapter and the systematic study of Multimodal In-Context Learning provide valuable technical contributions and insights for the broader field of multimodal AI.
Neural audio codecs with residual vector quantization (RVQ) normally treat all frequencies uniformly, so their codebooks become spectrally entangled. Truncating stages then removes an unpredictable mix of frequencies. Parallel band decomposition addresses this by splitting audio into independent bands, but fragments the latent space and loses cross-frequency coherence. We introduce HARP (Harmonic-Aware Residual Partitioning), a training strategy that partitions RVQ stages into frequency-ordered groups where each group refines its target band while the decoder retains access to all lower frequencies. Overtones are reconstructed in the context of their fundamentals, preserving coherence that parallel methods lose. HARP requires no architectural changes; it only modifies the training loss, leaving inference identical to standard RVQ. On speech, music, and general audio, HARP outperforms both standard RVQ and parallel decomposition. MUSHRA listening tests also show perceptual improvements.
Primary: Georgia Institute of Technology
All Institutions: Georgia Institute of Technology, The Chinese University of Hong Kong, Tencent Music Entertainment
HARP introduces a training-only strategy for neural audio codecs that partitions RVQ stages into frequency-ordered groups with cumulative decoding and soft subband supervision, effectively addressing spectral entanglement and improving both objective metrics and perceptual quality, particularly at low bitrates, without architectural changes.
The paper proposes HARP, a training-time strategy for Residual Vector Quantization (RVQ) in neural audio codecs. The core innovation lies in partitioning RVQ stages into frequency-ordered groups and enforcing spectral specialization through a novel loss function. Key components include: 1) Cumulative Decoding: Ensuring higher-frequency stages are reconstructed in the context of lower-frequency content to preserve harmonic coherence. 2) Subband Contribution Supervision: Supervising the isolated contribution of each group to its target band using stop-gradient operators to prevent gradient interference. 3) Soft Band Weighting: Using learnable Gaussian weights over mel bins to guide specialization without hard spectral cutoffs. The approach is architecturally agnostic, requiring only modifications to the training objective. This is a clever and relatively simple modification that addresses the "spectral entanglement" problem inherent in standard RVQ, where early and late stages capture similar frequency content.
The authors evaluate HARP against standard DAC (same architecture, no band supervision) and BSCodec (parallel band decomposition). They use a diverse dataset including MUSDB18-HQ, LibriTTS, and AudioSet. Metrics include SI-SDR, KAD, PESQ, STOI, and MUSHRA listening tests. Results show consistent improvements in SI-SDR and KAD over both baselines, with particularly notable gains at low bitrates (2.6 kbps). The ablation studies effectively isolate the contributions of cumulative decoding, soft boundaries, and stop-gradient mechanisms. The harmonic coherence analysis using synthetic tones provides strong evidence for the preservation of phase/amplitude relationships. The MUSHRA tests confirm perceptual improvements, especially in bass fullness and timbral naturalness.
The paper provides detailed descriptions of the architecture, training objectives, and hyperparameters. The code is made publicly available on GitHub. The use of standard components (DAC backbone, EMA updates) and clear algorithmic descriptions enhances reproducibility. The training cost increase (2-3x) is noted, which is a practical consideration but does not hinder reproducibility.
The primary limitation is that the performance gains, while consistent, are modest in absolute terms (e.g., +0.25 to +0.96 dB SI-SDR). The method relies on the assumption that natural audio has a specific energy distribution (low-to-high), which might not hold for all types of audio (e.g., highly inharmonic or noise-heavy content, though the paper argues it still applies). The method is training-only, so it does not offer architectural efficiency gains. The "soft" boundaries mean that some spectral leakage or imperfect separation might still occur, potentially limiting the maximum potential of the specialization compared to hard architectural splits (though the paper argues this is offset by coherence).
This work contributes to the field of neural audio compression by providing a principled way to impose spectral structure on RVQ without architectural complexity. This can lead to more robust and interpretable audio codecs, which are foundational for speech synthesis, music generation, and audio understanding tasks. The improved low-bitrate performance is particularly relevant for streaming applications. The open-source release facilitates further research in this area. HARP introduces a training-only strategy for neural audio codecs that partitions RVQ stages into frequency-ordered groups with cumulative decoding and soft subband supervision, effectively addressing spectral entanglement and improving both objective metrics and perceptual quality, particularly at low bitrates, without architectural changes.
Singing Quality Assessment (SQA) has become increasingly important for practical multimedia applications and Music AI systems, yet existing studies predominantly focus on short singing clips and remain insufficient for full-length songs. Unlike clip-level assessment, full-length song SQA requires modeling how singing quality varies across different audio segments and how these local variations influence the overall evaluation of vocal performance. Moreover, the scarcity of segment-level annotations makes effective supervision challenging, as directly assigning a single overall score label to every segment tends to treat different segment qualities as equivalent. To address these challenges, we propose SongSQA, a two-stage framework for full-length song SQA. In the first stage, a Segment Score Predictor is trained with pseudo labels generated by a pre-trained teacher model, enabling segment-level singing quality prediction without requiring manual segment annotations. In the second stage, a Song Quality Aggregator integrates segment features and predicted segment scores into unified segment embeddings, and employs a learnable song embedding together with self-attention to capture the connection between segment-level vocal performance and overall song quality. In this way, SongSQA dynamically aggregates critical quality cues across the song to produce a holistic quality prediction, while also generating a temporal segment-level quality curve. Experimental results demonstrate the effectiveness of SongSQA for full-length song SQA, achieving up to a 13.95% relative improvement in KTAU over the strongest baseline, while consistently improving other evaluation metrics across all datasets.
Primary: The Chinese University of Hong Kong, Shenzhen
All Institutions: Xi'an Jiaotong University, The Chinese University of Hong Kong, Shenzhen
[One sentence main contribution]. SongSQA introduces a two-stage framework for full-length song singing quality assessment that utilizes pseudo-labels from a teacher model for segment-level supervision and a Transformer-based aggregator to capture non-linear segment interactions. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a solid technical contribution to the field of audio quality assessment by shifting the paradigm from clip-level to full-song evaluation. The proposed methodology is innovative in its handling of the annotation scarcity problem through a teacher-student pseudo-labeling strategy, which is a clever adaptation of semi-supervised learning principles to a specific perceptual task. The integration of segment-level predictions into a global context via self-attention allows the model to mimic human perceptual weighting of vocal highlights and flaws. The experimental results are strong and well-supported by ablation studies and subjective evaluations. While the novelty is incremental in terms of architectural components (using Transformers and SSL features is common), the specific application to full-song SQA with temporal curves and the pseudo-labeling strategy for segment supervision is a valuable addition to the literature. The work is technically sound and addresses a clear gap in current SQA systems.
The paper proposes SongSQA, a two-stage framework for full-length song singing quality assessment (SQA). The core methodological innovation lies in addressing the lack of segment-level annotations by using a pre-trained teacher model to generate pseudo-labels for a Segment Score Predictor. This predictor extracts segment-level features and scores, which are then fused and aggregated by a Song Quality Aggregator using a Transformer with a learnable [CLS]-like token. The approach effectively bridges the gap between clip-level SQA and full-song assessment. The use of pseudo-labels from a specialized teacher (SongEval) rather than simple label sharing is a pragmatic and technically sound solution to the data scarcity problem. The aggregation mechanism using self-attention to model the non-linear influence of segments on overall quality is well-motivated by human perception studies.
The experimental evaluation is robust, utilizing two datasets: the public Lyra-SA dataset and a proprietary Internal Dataset annotated by experts. The paper compares SongSQA against strong baselines including clip-level SOTA models (UTMOSv2, PS-SQA, RAMP+) and SSL-based full-song adaptations. SongSQA achieves significant relative improvements (up to 13.95% in KTAU) over the strongest baseline. The ablation studies are thorough, validating the contributions of the teacher model choice, feature fusion strategy, and aggregation design. The case study and subjective evaluation further support the perceptual validity of the temporal score curves generated by the model. The results on the Internal Dataset show even higher performance, suggesting the model benefits from high-quality expert annotations.
The paper provides detailed implementation details, including dataset splits, segment duration (10s with 5s overlap), model architecture parameters (number of layers, loss weights), and optimizer settings. The use of publicly available datasets (Lyra-SA) and pre-trained models (MuQ, SongEval) enhances reproducibility. However, the Internal Dataset is proprietary, which limits the ability to fully replicate the results on that specific benchmark. The code is not explicitly linked in the text provided, though a demo page is available.
The primary limitation is the reliance on a pre-trained teacher model for pseudo-label generation. The quality of the final model is thus bounded by the teacher's ability to discern segment-level singing quality. If the teacher model has biases or errors, these will propagate to the student model. Additionally, the performance on the proprietary Internal Dataset, while impressive, cannot be directly compared to other works due to the lack of public access. The model's performance on non-Mandarin/Cantonese singing (only a small proportion in the internal dataset) is not thoroughly evaluated, limiting generalizability claims.
This work has significant implications for Music AI, particularly in the evaluation of Singing Voice Synthesis (SVS) and Singing Voice Conversion (SVC) systems. By providing a tool for fine-grained, full-song assessment, it enables more precise feedback for model improvement and content moderation on multimedia platforms. The generated temporal score curves can also be used for music education and analysis. The methodology of using pseudo-labels for segment-level supervision in audio quality assessment could be adapted to other domains lacking granular annotations. [One sentence main contribution]. SongSQA introduces a two-stage framework for full-length song singing quality assessment that utilizes pseudo-labels from a teacher model for segment-level supervision and a Transformer-based aggregator to capture non-linear segment interactions. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a solid technical contribution to the field of audio quality assessment by shifting the paradigm from clip-level to full-song evaluation. The proposed methodology is innovative in its handling of the annotation scarcity problem through a teacher-student pseudo-labeling strategy, which is a clever adaptation of semi-supervised learning principles to a specific perceptual task. The integration of segment-level predictions into a global context via self-attention allows the model to mimic human perceptual weighting of vocal highlights and flaws. The experimental results are strong and well-supported by ablation studies and subjective evaluations. While the novelty is incremental in terms of architectural components (using Transformers and SSL features is common), the specific application to full-song SQA with temporal curves and the pseudo-labeling strategy for segment supervision is a valuable addition to the literature. The work is technically sound and addresses a clear gap in current SQA systems.
End-to-end speech language models increasingly represent user speech with speech tokens rather than relying exclusively on cascaded ASR--LLM--TTS pipelines. Although these tokens support expressive and low-latency spoken interaction, they may also preserve sensitive speaker characteristics. We investigate whether exposed speech tokens leak voiceprints and formulate this risk as a speaker inversion attack. We introduce Audio BERT (AuB), a trainable model that constructs token embeddings from discrete codebooks and aggregates them into speaker-sensitive representations, and propose SpInv, a two-stage inversion method built on AuB to recover embeddings in the space of an attacker-specified speaker encoder. We evaluate Moshi, Higgs3, Kimi-Audio, and Qwen3-Omni using speaker-disjoint protocols on the VoxCeleb dataset. Extensive experiments show that, with only three seconds of frontend output, SpInv achieves cosine similarities above 0.70 in the attacker-specified speaker-encoder space.
Primary: Harbin Institute of Technology
All Institutions: Harbin Institute of Technology
The paper presents a significant and timely contribution to the intersection of speech processing and machine learning security. By formulating and executing a speaker inversion attack against modern end-to-end speech language models, it exposes a critical privacy vulnerability in emerging AI architectures. The proposed SpInv method is technically robust, leveraging advanced distillation and contrastive learning techniques to recover speaker embeddings from discrete tokens with high fidelity. The comprehensive evaluation across multiple state-of-the-art models and rigorous experimental protocols strengthens the validity of the findings. This work is essential reading for researchers and practitioners in speech AI, highlighting the urgent need for privacy-preserving design patterns in next-generation speech interfaces.
The paper proposes a novel threat model called "Speaker Inversion Attack" (SIA) against end-to-end speech language models (SLMs). The core methodological contribution is SpInv, a two-stage attack framework. Stage A uses a distillation pretraining approach with an auxiliary model called Audio BERT (AuB) to map discrete speech tokens (or continuous features) to a target speaker embedding space defined by a frozen encoder (e.g., ECAPA-TDNN). This involves alignment loss, relational loss, and self-distillation (DINO-style) to prevent collapse and preserve geometry. Stage B fine-tunes the model with discriminative supervision (ArcFace) to enhance speaker discrimination. The approach is technically sound, leveraging established contrastive and distillation techniques (DINO, VICReg) in a new context. The mapping from discrete tokens to continuous embeddings via a learned lookup table is a clever adaptation to handle the black-box nature of the tokenizers.
The evaluation is comprehensive and rigorous. The authors test against four state-of-the-art SLM frontends (Moshi, Higgs3, Kimi-Audio, Qwen3-Omni), covering different tokenization strategies (residual codebooks, hybrid, continuous). They use speaker-disjoint protocols on VoxCeleb1 and VoxCeleb2, which is crucial for validating generalization. The results are strong: cosine similarities >0.70 are achieved with only 3 seconds of audio. The ablation studies on input duration and target speaker encoder spaces add depth. The fact that they test on "Qwen3-Omni" (which appears to be a very recent or potentially misnamed model in the context of current public releases, possibly referring to Qwen-Audio or a specific internal variant, but treated as a valid target here) suggests the work is timely. The metrics (CosSim, EER, Acc) are appropriate for the task.
The paper provides detailed descriptions of the AuB architecture, loss functions, and training hyperparameters (learning rates, batch sizes, loss weights). The datasets (VoxCeleb1/2) and target encoders (ECAPA-TDNN, etc.) are publicly available. The threat model assumes query access to the frontend, which is a standard and realistic assumption for API-based services. The code is not explicitly linked, but the methodological description is sufficient for reproduction by a competent researcher.
The authors acknowledge two main limitations: focusing on embedding recovery rather than waveform reconstruction, and using only VoxCeleb datasets. The reliance on VoxCeleb, while standard, may not fully capture the diversity of "in-the-wild" speech (accents, noise, channel variations) that might be present in real-world SLM deployments. Additionally, the attack requires a training phase where the adversary collects data from the target frontend; while feasible, this is a resource-intensive prerequisite. The paper does not explore defense mechanisms, only the attack.
This work has significant implications for the privacy of users interacting with AI voice assistants and SLMs. It demonstrates that even when raw audio is not transmitted, the intermediate representations (tokens) can leak sensitive biometric information (voiceprints). This necessitates the development of privacy-preserving tokenization techniques or anonymization methods for speech tokens. It serves as a wake-up call for the SLM community to consider privacy in the design of frontend tokenizers. The paper presents a significant and timely contribution to the intersection of speech processing and machine learning security. By formulating and executing a speaker inversion attack against modern end-to-end speech language models, it exposes a critical privacy vulnerability in emerging AI architectures. The proposed SpInv method is technically robust, leveraging advanced distillation and contrastive learning techniques to recover speaker embeddings from discrete tokens with high fidelity. The comprehensive evaluation across multiple state-of-the-art models and rigorous experimental protocols strengthens the validity of the findings. This work is essential reading for researchers and practitioners in speech AI, highlighting the urgent need for privacy-preserving design patterns in next-generation speech interfaces.
Interpretability is critical in clinical decision support. Concept bottleneck frameworks improve it by representing inputs as human-understandable concepts and restricting predictions solely on them. However, research on their use for voice-based health assessment remains limited. In this study, we propose a voice concept bottleneck framework for interpretable health assessment using an audio language model (ALM). The ALM is fine-tuned on a voice quality assessment dataset to enhance its understanding of voice concepts and serves as an independent concept extractor, producing discrete, interpretable scores for a lightweight downstream classifier. The discrete concept scores provide intuitive interpretation, while the lightweight classifier facilitates post-hoc interpretability analyses. Results on depression and dysarthria assessment tasks demonstrate that the proposed framework can flexibly adapt voice concepts to different health conditions and consistently outperforms openSMILE-based and self-supervised speech model-based baselines.
Primary: Columbia University
All Institutions: Columbia University
This paper presents a compelling application of Audio Language Models as interpretable concept bottlenecks for health assessment, demonstrating that decoupling concept extraction from classification can yield both high performance and clinical interpretability, although validation on larger, more diverse datasets is needed to confirm robustness.
The paper proposes a novel framework that leverages Audio Language Models (ALMs), specifically Audio Flamingo Next, as independent concept extractors for interpretable health assessment. The core innovation lies in decoupling the concept extraction (via fine-tuned ALM producing discrete 1-5 ordinal scores) from the downstream classification (via XGBoost). This approach aims to mitigate shortcut learning and enhance interpretability by restricting predictions to human-understandable concepts (Clinical Voice Quality and Speaking Behavior). The methodology is sound in its intent, utilizing LoRA for efficient fine-tuning and aggregating speaker-level statistics for the classifier. However, the novelty is somewhat incremental, as concept bottleneck models (CBMs) are an established paradigm, and applying them to audio via LLMs is a logical but not radically new extension. The specific adaptation of ALMs to produce discrete ordinal concepts rather than continuous embeddings or text descriptions is a notable technical choice.
The experimental evaluation covers two distinct health conditions: depression (Android Corpus) and dysarthria (TORGO corpus). The results demonstrate that the proposed ALM-based framework outperforms baselines using openSMILE and self-supervised models (Vox-Profile). The paper provides SHAP analysis to validate interpretability, showing that features like pitch variability and speech tempo align with clinical expectations for depression. For dysarthria, the separation of CVQ scores across severity levels is shown. While the performance gains are consistent, the baselines are relatively standard for the domain. The comparison with Vox-Profile is fair, but the lack of comparison with other recent SSL-based CBMs or end-to-end interpretable models limits the comprehensiveness of the benchmark. The results are promising but rely on small datasets, particularly for dysarthria (only 15 speakers total), which raises questions about generalizability.
The paper provides sufficient detail regarding the model architecture (Audio Flamingo Next), fine-tuning parameters (LoRA rank, alpha, dropout), and downstream classifier settings (XGBoost hyperparameters). The datasets are publicly available, and the voice activity detection tool is cited. However, the code is not explicitly linked in the provided text, and specific random seeds or exact data splitting procedures beyond the standard protocols are not detailed. The reliance on a specific Hugging Face model checkpoint aids reproducibility, but the fine-tuning process on the PVQD dataset requires careful handling of the discrete label generation which is described but could be more precise.
The primary limitation is the small sample size of the evaluation datasets, especially the dysarthria corpus, which limits the statistical power of the results. The ALM's pretraining bias is acknowledged and partially corrected via fine-tuning, but the paper notes that the model still struggles with certain concepts like abnormal loudness and pitch due to definition mismatches. The discrete 1-5 scale, while interpretable, may lose nuance compared to continuous ratings. Furthermore, the "independent" nature of the concept extractor assumes that the fine-tuned ALM captures the true underlying concepts without leakage from the downstream task, which is hard to guarantee without rigorous ablation studies on concept fidelity.
This work has significant potential impact in clinical decision support systems by providing interpretable voice biomarkers. By making AI predictions transparent through human-understandable concepts, it can help clinicians trust and validate AI outputs. The framework could be extended to other health conditions affecting voice, such as Parkinson's disease or Alzheimer's. However, ethical considerations regarding privacy, bias in health assessment, and the risk of over-reliance on automated diagnostics must be addressed. The use of ALMs also raises questions about the environmental cost and accessibility of such models in resource-constrained clinical settings. This paper presents a compelling application of Audio Language Models as interpretable concept bottlenecks for health assessment, demonstrating that decoupling concept extraction from classification can yield both high performance and clinical interpretability, although validation on larger, more diverse datasets is needed to confirm robustness.
We propose the concept of noise-aware audio self-supervised learning (SSL), whose goal is to encode audio mixtures while suppressing undesired noise, and present Noise-Aware BEATs (NABEATs) as a BEATs-based realization of this framework. Audio SSL models are designed to handle a wide range of audio signals. Consequently, under noisy conditions, they cannot effectively focus on the target sounds relevant to a downstream task, resulting in degraded performance. To address this issue, NABEATs is trained to estimate clean BEATs representations from a noisy audio signal with an auxiliary reference noise input. This reference noise enables the model to account for specific noise characteristics at inference time, thereby achieving better generalization across operating environments. Our experimental evaluations demonstrate that NABEATs significantly improves performance of various downstream tasks under noisy conditions and also generalizes well to unseen noise types.
Primary: Mitsubishi Electric Research Laboratories (MERL)
All Institutions: Mitsubishi Electric Research Laboratories (MERL), Nagoya University
This paper presents a practical and effective method for enhancing the noise robustness of audio self-supervised learning models by conditioning representation learning on reference noise, demonstrating significant improvements in downstream task performance under both seen and unseen noisy conditions.
The paper proposes NABEATs, a noise-aware adaptation of the BEATs self-supervised learning (SSL) model. The core innovation lies in conditioning the representation learning process on a reference noise signal. The authors introduce two mechanisms to integrate this reference noise: Cross-Attention (CA) and Feature-wise Linear Modulation (FiLM). The model is trained via distillation from a frozen, pre-trained BEATs model, aiming to estimate clean representations from noisy inputs. This approach is theoretically sound and addresses a specific gap in general-purpose audio SSL, which typically lacks robustness to environmental noise. The distinction between using a reference noise (NABEATs) versus unconditional denoising (DBEATs) is clearly defined. However, the method is essentially an architectural modification to an existing SSL framework rather than a fundamentally new learning paradigm. The use of reference noise is analogous to techniques in speech enhancement and speaker verification, making the novelty incremental rather than transformative.
The experimental setup is rigorous and well-structured. The authors evaluate NABEATs on six diverse downstream tasks (environmental sound, speech, music) under various noisy conditions using multiple noise datasets (WHAM!, CHiME-3, MUSDB18). The inclusion of an "unseen noise" generalization test (MUSDB18) is a strong point, demonstrating the practical utility of the reference noise mechanism. The results show consistent improvements over the baseline BEATs and the unconditional denoising baseline (DBEATs), particularly in unseen noise conditions. The use of t-SNE visualizations and SNR analysis of representations provides good qualitative and quantitative support for the claims. The evaluation on the DCASE 2025 Task 2 benchmark adds real-world relevance, although the performance gain is modest. The ablation between CA and FiLM variants provides useful insight into how reference information is utilized.
The paper provides sufficient detail for reproduction. It specifies the base model (BEATs_iter3.pt), the additional layer architectures (Transformer layers with MHSA/MHCA, RMSNorm, SwiGLU), training hyperparameters (AdamW, learning rate, batch size, epochs), and dataset sources (FSD50K, WHAM!, DEMAND, QUT-NOISE). The training procedure for distillation is clearly described. However, the code is not publicly linked in the provided text, and some details regarding the exact mixing ratios and SNR distributions for the downstream evaluation splits could be more precise. The use of standard datasets and a widely available base model enhances reproducibility.
The primary limitation is the dependency on a reference noise signal. While the authors argue this is practical in many scenarios (e.g., pre-recorded noise), it is not universally available. The method does not address the case where no reference noise is present. Furthermore, the performance gains, while statistically significant, are not massive, and the model introduces additional complexity and computational overhead compared to the base BEATs. The reliance on MSE loss for distillation may not perfectly align with downstream classification objectives, as noted in the discussion of the CRM-D task. The evaluation is limited to 16kHz audio, which may not reflect high-fidelity audio applications.
This work contributes to the development of robust audio AI systems that can operate effectively in real-world, noisy environments. By improving the generalization of SSL models, it facilitates the deployment of audio understanding systems in diverse settings, from smart homes to industrial monitoring. The concept of noise-aware SSL could inspire similar approaches in other modalities where background interference is a major challenge. However, the potential for misuse in creating robust surveillance or eavesdropping systems is a minor concern, though mitigated by the requirement for reference noise. This paper presents a practical and effective method for enhancing the noise robustness of audio self-supervised learning models by conditioning representation learning on reference noise, demonstrating significant improvements in downstream task performance under both seen and unseen noisy conditions.
Audio mixing style encompasses the artistic and technical decisions a mix engineer makes, including level balancing, spatialization, and the choice, ordering, and parameterization of audio effects (FX) on each stem. FX chains are a key determinant of this style, yet existing approaches to modeling them remain limited. Some operate on stereo mixtures without explicit per-stem FX chain modeling, others fix the number or type of effects per track, and many require differentiable effect implementations or scarce multitrack datasets. We present StemFX, a framework that learns mixing style representations by autoregressively predicting variable-length FX chains on source-separated stems. A Transformer decoder predicts tokenized FX chains autoregressively, while a band-split multi-band CNN encoder with FiLM conditioning captures per-stem spectral structure. To enable large-scale paired training, we extract pseudo-stems from about 105K songs via source separation and augment them using MultiAFx, a toolkit unifying 85 audio effects from 7 Python libraries. Evaluated on mixing style retrieval, StemFX outperforms all baseline models across all tested chain lengths. On paired mixing style transfer, StemFX achieves the best spectral fidelity and the highest listener preference, over 4000 times faster than iterative optimization.
Primary: National Taiwan University
All Institutions: National Taiwan University
This paper presents a significant advancement in automatic mixing and audio representation learning by introducing a scalable, autoregressive framework for FX chain prediction that overcomes data scarcity through a novel pseudo-stem generation pipeline, achieving state-of-the-art performance in both style retrieval and transfer tasks.
The paper proposes StemFX, a novel framework for learning mixing style representations by treating FX chain prediction as an autoregressive sequence generation problem. The architecture combines a band-split multi-band CNN encoder with FiLM conditioning (incorporating hand-crafted mixing features) and a Transformer decoder. A key methodological innovation is the "StemFX Pipeline," which leverages source separation on large-scale unpaired datasets (FMA) to create pseudo-stems, which are then augmented with random FX chains using a unified toolkit (MultiAFx) to generate paired training data at scale. This approach circumvents the scarcity of real multitrack datasets with ground-truth FX metadata. The tokenization scheme for variable-length FX chains (effect names, parameter names, and quantized values) is well-designed for autoregressive modeling. The use of FiLM conditioning to inject explicit mixing statistics into the CNN layers is a strong technical choice that bridges the gap between low-level spectral features and high-level mixing decisions.
The experimental evaluation is comprehensive and robust. The authors evaluate on two tasks: mixing style retrieval and paired mixing style transfer. For retrieval, StemFX significantly outperforms baselines (AFx-Rep, Fx-Encoder++, CLAP, and a controlled contrastive variant BSFiLM-CL) across various chain lengths, demonstrating that the learned embeddings capture mixing style distinct from musical content. The ablation studies effectively isolate the contributions of the encoder architecture, FiLM conditioning, and the data pipeline. The style transfer experiments are particularly compelling, showing superior spectral fidelity (MRSTFT) and human preference (MUSHRA scores) compared to fixed-chain methods and iterative optimization techniques (ST-ITO). The claim of being 4000x faster than iterative optimization is a significant practical advantage. The use of out-of-domain effects (pedalboard) in evaluation tests generalization well.
The paper provides detailed implementation details, including hyperparameters, dataset sizes, and architecture specifications. The release of both the StemFX codebase and the MultiAFx toolkit on GitHub greatly enhances reproducibility. The description of the data generation pipeline, including source separation models and augmentation strategies, is clear enough for other researchers to replicate the training data generation process. The use of standard metrics (MRSTFT, MUSHRA) also aids in reproducibility of evaluation.
The authors acknowledge several limitations. The model is restricted to the 85 effects included in the MultiAFx toolkit; adding new effects requires retraining. The reliance on source separation introduces potential artifacts from the separation model (e.g., SCNet), which may propagate into the learned representations. The use of pseudo-stems from different songs (cross-song mixing) may introduce musical incoherence, although the authors argue this decorrelates content from style. The model currently operates on four stems (vocals, bass, drums, other), which is a coarser decomposition than professional multitracks. Finally, the training data uses random effect chains, which may not reflect the structured, genre-dependent processing choices of human engineers.
StemFX has significant potential impact on music production technology. By enabling efficient and accurate mixing style transfer and retrieval, it can assist producers in learning mixing techniques, automate parts of the mixing workflow, and enhance music information retrieval systems. The open-source release of the toolkit and models promotes further research in audio effect modeling and automatic mixing. The ability to generate interpretable FX chains also provides insights into the decision-making process of mix engineers, potentially serving as an educational tool. This paper presents a significant advancement in automatic mixing and audio representation learning by introducing a scalable, autoregressive framework for FX chain prediction that overcomes data scarcity through a novel pseudo-stem generation pipeline, achieving state-of-the-art performance in both style retrieval and transfer tasks.
In X-lingual automatic speaker verification (ASV), fixed front-end scores vary in reliability with language match, duration, and score source. We propose AMECxSV, an adaptive metadata-driven embedding-fusion calibration backend for metadata-available settings. AMECxSV fuses trial scores with metadata to produce calibrated target posteriors, with optional posterior-confidence abstention; metadata serve as calibration context, not speaker evidence. On a development-derived speaker-disjoint held-out split, score+metadata heads reduce equal error rate (EER) from 3.15% to 2.42% for the official TidyVoice score source and from 0.64% to 0.43% for LI-MSV; the dual-score head reaches 0.43% full-coverage EER. At 0.79 coverage, abstention yields 0.03% accepted-trial EER, not a full-coverage metric. Matched score-only, metadata-permutation, and metadata-only controls support a calibration-context interpretation and limit claims to metadata-available scoring.
Primary: University of Southern California
All Institutions: University of Southern California, Signal Analysis and Interpretation Laboratory (SAIL), University of Southern California
AMECxSV presents a practical and effective metadata-driven calibration framework that improves the reliability of X-lingual speaker verification systems by adaptively adjusting scores based on language and duration context, demonstrating significant gains in calibration quality and error rates through rigorous ablation and control studies.
The paper proposes AMECxSV, a metadata-driven calibration backend for X-lingual Automatic Speaker Verification (ASV). The core methodology involves fusing trial scores from fixed embedding front-ends with metadata (language match, duration) using a lightweight Multi-Layer Perceptron (MLP). The approach is framed as "embedding-fusion calibration," where metadata serves as context for score reliability rather than direct speaker evidence. The method includes a full-coverage variant (AMEC-FC) and a confidence-based abstention variant (AMEC-ABS). The technical contribution is relatively incremental, building upon established logistic/isotonic calibration and multi-system fusion techniques by adding a structured metadata input to an MLP. While the specific application to X-lingual reliability is well-motivated, the architectural novelty is limited to the feature engineering (score summaries, order statistics, interactions) and the specific conditioning on language/duration.
The evaluation is conducted on the TidyVoiceX-ASV development protocol, using a speaker-disjoint held-out split. The authors compare AMECxSV against single-score baselines, global calibration methods, and a multi-score fusion baseline (MultiScore-FC). Results show consistent improvements in Equal Error Rate (EER) and $C_llr$ when metadata is added. The paper includes rigorous controls, such as metadata permutation and matched score-only architectures, to validate that gains are due to the metadata content and not just model capacity. External evaluations on TidyVoice and LI-MSV score sources further support the method's generalizability. The statistical significance is addressed via bootstrap replicates. However, the results are primarily on development splits, and the magnitude of improvement, while consistent, is modest in absolute terms (e.g., 3.15% to 2.42% EER on one source).
The paper provides detailed implementation details, including the architecture of the MLP, training hyperparameters (AdamW, learning rates, batch sizes), and the specific metadata features used. The codebase is linked via an anonymous repository. The dataset split is clearly defined as development-derived, which aids reproducibility but limits direct comparison to official challenge results. The use of fixed, publicly available embedding systems (SpeechBrain, WeSpeaker, etc.) ensures that the front-end components are reproducible.
The primary limitation is the dependency on metadata. The system requires language match and duration information, which may not be available or reliable in all deployment scenarios (e.g., language-blind protocols). The authors acknowledge that language metadata might need to be predicted by a separate LID module, introducing potential error propagation. Additionally, the improvements are source-dependent, and the method does not address the underlying representation learning issues of the front-ends. The abstention mechanism, while effective for reducing risk, reduces coverage, which may not be desirable for all applications.
This work contributes to the robustness and reliability of ASV systems, particularly in multilingual contexts. By providing a framework for adaptive calibration, it helps mitigate the performance degradation caused by language mismatch and short utterances. This can lead to more fair and accurate speaker verification in real-world applications where trial conditions are heterogeneous. The emphasis on calibration and abstention also promotes safer deployment by allowing the system to defer uncertain decisions. AMECxSV presents a practical and effective metadata-driven calibration framework that improves the reliability of X-lingual speaker verification systems by adaptively adjusting scores based on language and duration context, demonstrating significant gains in calibration quality and error rates through rigorous ablation and control studies.