This week’s shortlist
Video rough-cut is a core stage in media production workflows, organizing raw media materials into an initial video cut. Despite recent advances in large vision-language models (LVLMs) for video-related tasks, automatic rough-cut remains underexplored, without a comprehensive…
Abstract
Video rough-cut is a core stage in media production workflows, organizing raw media materials into an initial video cut. Despite recent advances in large vision-language models (LVLMs) for video-related tasks, automatic rough-cut remains underexplored, without a comprehensive benchmark for evaluation or mature workflows for producing executable cut plans. To address the evaluation gap, we introduce RoughCut-Bench, which covers two practical settings: (1) a visually driven setting over video cases, covering highlight detection and selection, source-video ordering, and adherence to user requirements when provided; and (2) a speech-driven setting over automatic speech recognition (ASR) transcripts, covering filler removal, expression deduplication, and low-value content trimming. Across these settings, the benchmark contains 101 visually driven cases, including 831 raw videos with 1,476 highlight annotations. It also includes 84 speech-driven cases, each consisting of an ASR transcript with annotated cleanup spans. The evaluation compares cut plans with these annotations in both settings and additionally uses a human-aligned agentic evaluator to assess rendered-cut quality in the visual setting. To apply LVLMs to automatic rough-cut, we further propose agentic workflows that guide them to produce structured cut plans from input materials. Experiments show that current LVLMs remain limited in producing expert-aligned cut plans. Our benchmark, code, and prompts are released at https://whalesong-zrs.github.io/RoughCut-Bench-Project-Page.
AI-generated analysis. Check claims against the paper.
Summary
The paper establishes a rigorous benchmark and evaluation framework for automatic video rough-cutting, demonstrating that current LVLMs exhibit uneven capabilities across compositional editing tasks and providing a scalable, human-aligned evaluation protocol for assessing rendered cut quality.
Methodology
The paper introduces RoughCut-Bench, a benchmark for automatic video rough-cutting, which is a practical and underexplored task in media production. The methodology is sound, defining two distinct settings: visually driven (highlight detection, ordering) and speech-driven (ASR transcript cleanup). The proposed agentic workflows (Video-Understanding Agent + Cut-Planning Agent) are logical decompositions of the complex task, addressing context length and reasoning stability issues inherent in long-video processing. The inclusion of a human-aligned agentic evaluator is a strong methodological choice, as it addresses the subjectivity of "quality" in video editing that standard IoU metrics cannot capture. The iterative refinement of the evaluator against professional scores is a rigorous approach to ensuring the automated metric is valid.
Results & evaluation
The experiments are comprehensive, evaluating 8 representative LVLMs (Gemini, GPT, Qwen, GLM, Seed) across both settings. The results provide valuable insights into the "capability dissociation" of current models, showing that strong performance in one aspect (e.g., filler removal) does not guarantee performance in another (e.g., source-video ordering). The ablation studies on frame sampling rates and input representations (time, frame, description) are well-designed and provide actionable insights for workflow optimization. The human-aligned evaluation results, showing high correlation with professional scores after refinement, validate the benchmark's utility.
Reproducibility
The paper commits to releasing code, prompts, and data, which is crucial for a benchmark paper. The detailed description of the annotation protocol (two-stage, professional annotators) and the evaluation metrics (IoU, Kendall's tau, specific precision/recall definitions for speech cleanup) enhances reproducibility. The specific model versions and workflow configurations are clearly stated.
Limitations
The benchmark size is moderate (101 visual cases, 84 speech cases), which may limit generalizability to all video types. The speech-driven setting relies on ASR transcripts, ignoring audio-visual cues like prosody or emotion, which the authors acknowledge as a limitation. The agentic workflows, while robust, may not represent the optimal performance of models that can handle longer contexts natively, as the authors prioritize cross-model comparability.
Broader impact
This work has significant potential impact on the video production industry and the development of multimodal agents. By providing a standardized benchmark and evaluation protocol, it enables fair comparison of LVLMs for creative tasks. The insights into model capabilities (e.g., the need for multi-granularity reasoning in speech cleanup) will guide future model training and architecture design. The release of professional annotations and a human-aligned evaluator will accelerate research in automated video editing. The paper establishes a rigorous benchmark and evaluation framework for automatic video rough-cutting, demonstrating that current LVLMs exhibit uneven capabilities across compositional editing tasks and providing a scalable, human-aligned evaluation protocol for assessing rendered cut quality.
Multi-source audio-to-image generation exhibits consistent compositional failures when audio tokens occupy geometrically displaced, semantically unanchored positions in the CLIP text-token space, where the diffusion U-Net lacks the prior to reliably attend to them or ground them…
Abstract
Multi-source audio-to-image generation exhibits consistent compositional failures when audio tokens occupy geometrically displaced, semantically unanchored positions in the CLIP text-token space, where the diffusion U-Net lacks the prior to reliably attend to them or ground them in distinct spatial regions. We formalize this as the geometric displacement phenomenon and expose it through a two-level clustering analysis comparing BEATs and CLAP embeddings at the raw encoder and projected CLIP token levels. Three geometric properties emerge as important: intra-class compactness, angular alignment with the CLIP text manifold, and inter-token spatial separation. We show that CLAP, whose contrastive language-audio pretraining aligns embeddings with natural language semantics, substantially outperforms BEATs on all three properties, improving the silhouette score by more than 3x and the Audio-Image Content score as well. Building on this analysis, we introduce a multi-token extension that requires no diffusion-backbone retraining and assigns each audio source to a distinct placeholder token in the CLIP tokenizer, complemented by a mixed inference-time objective that combines Attend-and-Excite with an A* spatial separation loss to resolve residual cross-token interference. For evaluation, we construct multi-audio compositional benchmarks from VGGSound (VGG2, VGG3, and VGGN) and extend six standard audio-image metrics to the multi-source setting. Across five benchmarks and four strong baselines, our method achieves strong results on the metrics most directly tied to compositional faithfulness, with AIC improvements of up to 2.25x over the strongest baseline and FID reduced by nearly 2x on two-source generation. We propose geometric displacement as a diagnostic framework for cross-modal generation systems that project learned embeddings into a pretrained contrastive space.
AI-generated analysis. Check claims against the paper.
Summary
The paper identifies a critical geometric failure mode in multi-source audio-to-image generation and provides a principled, effective remedy through encoder substitution and inference-time optimization. By formalizing the "geometric displacement" phenomenon and constructing new compositional benchmarks, the work offers both a diagnostic tool and a state-of-the-art solution, significantly advancing the field of audio-conditioned generation.
Methodology
The paper proposes a geometric diagnosis for the failure of multi-source audio-to-image generation. The core insight is that audio embeddings (specifically from BEATs) are geometrically displaced in the CLIP text-token space, leading to poor cross-attention behavior. The authors formalize this using clustering metrics (silhouette, Davies-Bouldin) and angular alignment. The remedy involves two main components: (1) switching to CLAP embeddings, which are naturally aligned with the text manifold due to contrastive pretraining, and (2) a Multi-Token Extension (MTE) that assigns distinct placeholder tokens to each audio source, combined with an inference-time optimization objective that blends Attend-and-Excite (to ensure activation) and A* spatial separation losses (to prevent overlap). The methodology is sound, leveraging existing components (textual inversion, frozen diffusion backbone) but applying them in a novel geometric context. The redesign of the embedder for CLAP's global representation is a necessary and well-explained technical detail.
Results & evaluation
The experimental setup is robust. The authors construct three new benchmarks (VGG2, VGG3, VGGn) from VGGSound to evaluate compositional generation, which is a significant contribution as prior work lacked such multi-source benchmarks. They compare against four strong baselines (AudioToken, SonicDiffusion, Sound2Scene, MACS). The results show substantial improvements in Audio-Image Content (AIC) and FID, particularly for the CLAP-based variant. The ablation studies clearly isolate the contributions of the encoder choice (CLAP vs BEATs) and the inference-time losses. The observation that BEATs performs better out-of-distribution (IS3) while CLAP dominates in-distribution is a nuanced and valuable finding.
Reproducibility
The paper provides extensive implementation details in the supplementary material, including hyperparameters for the embedder, training schedules, and the specific configuration of the inference-time optimization. The code for the benchmarks and metrics is implied to be available or easily reconstructible given the detailed descriptions. The use of standard open-source components (Stable Diffusion, CLAP, BEATs) further aids reproducibility.
Limitations
The method assumes separated audio streams, which may not always be available in real-world scenarios (though they suggest using Demucs as a preprocessing step). The embedder is trained on a curated 50-class subset, limiting the vocabulary size compared to the full VGGSound dataset. The inference-time optimization increases computational cost by roughly 2.8-3.1x. The out-of-distribution performance of the CLAP variant is weaker than BEATs, indicating a potential vulnerability to unseen classes.
Broader impact
The geometric diagnosis framework is broadly applicable to any cross-modal generation system that projects learned embeddings into a pretrained contrastive space (e.g., video, 3D). The new benchmarks will be useful for the community to evaluate future multi-source audio generation models. The insights into the importance of angular alignment over Euclidean distance in cross-attention are valuable for designing future conditioning mechanisms. The paper identifies a critical geometric failure mode in multi-source audio-to-image generation and provides a principled, effective remedy through encoder substitution and inference-time optimization. By formalizing the "geometric displacement" phenomenon and constructing new compositional benchmarks, the work offers both a diagnostic tool and a state-of-the-art solution, significantly advancing the field of audio-conditioned generation.
Recent generative techniques have facilitated the rapid proliferation of speech enhancement approaches built upon learnable compact latent representations and large generative backbones. Nevertheless, it remains inherently challenging to balance between acoustic fidelity…
Abstract
Recent generative techniques have facilitated the rapid proliferation of speech enhancement approaches built upon learnable compact latent representations and large generative backbones. Nevertheless, it remains inherently challenging to balance between acoustic fidelity, perceptual quality, and processing efficiency. In this paper, we revisit the Mel spectrum as a structured acoustic representation and propose BAME, a flow-matching-based framework that exploits the complementary priors for generative speech enhancement task. First, leveraging the explicit hierarchical frequency distribution of Mel spectrum, we elaborately devise a subband-aware Transformer (SaT) for efficient generative modeling. Using only 2.9% of the parameters, the proposed SaT achieves superior enhancement performance compared with a 331 M DiT backbone. Second, motivated by range-null decomposition theory of Mel filtering in the magnitude-spectral domain, we introduce a phase-aware neural vocoder that incorporates degraded-phase prior, together with a quality-guided dual-phase strategy to adaptively mitigate the effects of unreliable phase information. Third, we devise a simple yet effective one-step fidelity-perception alignment strategy that fine-tunes the flow model with complementary constraints from fidelity and perception domains. Extensive experiments are conducted on DNS-Challenge, EARS, GSR, and URGENT benchmarks, and both objective and subjective results demonstrate promising performance on various intrusive and non-intrusive evaluation metrics. These results further advocate the potential of exploiting structured acoustic representations for reliable and efficient generative speech enhancement. Audio samples and codes are available at https://github.com/AndongLi-speech/BAME.
AI-generated analysis. Check claims against the paper.
Summary
The paper proposes BAME, a flow-matching-based generative speech enhancement framework that leverages the structural priors of the Mel spectrum through a subband-aware Transformer and a phase-aware vocoder, achieving efficient one-step generation with balanced fidelity and perceptual quality. The technical contribution is significant due to the efficient backbone design, the rigorous exploitation of Mel's mathematical properties (RND), and the effective one-step fine-tuning strategy that addresses the fidelity-perception trade-off.
Methodology
The paper proposes BAME, a flow-matching-based framework for generative speech enhancement that revisits the Mel spectrum as a structured acoustic representation. The core novelty lies in the Subband-aware Transformer (SaT), which exploits the hierarchical frequency distribution of Mel bands to achieve efficient generative modeling with significantly fewer parameters (2.9% of a DiT baseline) while maintaining superior performance. The methodology also introduces a phase-aware neural vocoder that leverages the Range-Null Decomposition (RND) theory to recover magnitude information from Mel spectra and incorporates degraded phase priors. A key contribution is the Quality-guided Dual-Phase (QDV) strategy, which adaptively selects between zero-phase and degraded-phase vocoding based on non-intrusive quality scores to mitigate the instability of degraded phase in reverberant conditions. Finally, the One-Step Fidelity-Perception Alignment (OS-FPA) strategy fine-tunes the flow model to balance intrusive fidelity and non-intrusive perceptual quality in a single step, addressing the trade-off typically observed in iterative sampling methods.
Results & evaluation
The experiments are extensive, covering four major benchmarks: DNS-Challenge 2020, EARS, GSR, and URGENT. The evaluation includes a comprehensive suite of intrusive metrics (PESQ, eSTOI, LPS, SpkSim, SBS, WER) and non-intrusive metrics (DNSMOS, UTMOS, DiMOS, WVMOS, SCOREQ). Subjective A/B preference tests were conducted with 22 participants. Ablation studies thoroughly investigate the impact of the subband structure, the choice of backbone modules (GRU vs LSTM vs Conv), the trajectory type (FM vs Schrödinger Bridge), and the specific components of the OS-FPA strategy. The results demonstrate that the proposed SaT backbone outperforms larger DiT backbones and that the QDV strategy effectively improves robustness against phase artifacts.
Reproducibility
The paper provides a GitHub link with code and audio samples. Detailed hyperparameters, dataset configurations, and training procedures are described in the appendix and main text. The use of standard components (GRU, MHSA, WavLM) and open-source datasets enhances reproducibility. The specific implementation of the RND-based vocoder and the OS-FPA fine-tuning procedure are clearly defined.
Limitations
The method relies on a pre-trained WavLM model for semantic conditioning, which adds computational overhead during inference if not optimized. The QDV strategy requires running the vocoder twice and evaluating quality scores, which may increase latency compared to single-pass vocoding. The performance gains in some non-intrusive metrics are marginal compared to state-of-the-art generative methods, suggesting diminishing returns in perceptual quality. The reliance on Mel spectra may still limit the recovery of very fine-grained high-frequency details compared to complex spectrum modeling, although the phase-aware vocoder mitigates this.
Broader impact
This work advocates for the use of structured, handcrafted representations (Mel) over learned latents in generative speech enhancement, offering a more efficient and interpretable alternative. The subband-aware modeling approach could be extended to other audio processing tasks. The one-step generation strategy is particularly relevant for real-time applications where latency is critical. The integration of fidelity and perception alignment provides a framework for balancing objective reconstruction error with subjective quality, which is applicable to other generative audio tasks. The paper proposes BAME, a flow-matching-based generative speech enhancement framework that leverages the structural priors of the Mel spectrum through a subband-aware Transformer and a phase-aware vocoder, achieving efficient one-step generation with balanced fidelity and perceptual quality. The technical contribution is significant due to the efficient backbone design, the rigorous exploitation of Mel's mathematical properties (RND), and the effective one-step fine-tuning strategy that addresses the fidelity-perception trade-off.
Recent advances in large audio-language models (LALMs) have broadened audio understanding across speech, sound, and music, with tool-based agents further supporting these tasks by invoking predefined audio tools. However, a persistent gap remains between perceiving acoustic…
Abstract
Recent advances in large audio-language models (LALMs) have broadened audio understanding across speech, sound, and music, with tool-based agents further supporting these tasks by invoking predefined audio tools. However, a persistent gap remains between perceiving acoustic content and using it as evidence for reasoning: recognizing what is audible does not ensure that a model can identify which cues resolve an uncertain decision. To bridge this gap, we propose Think2Listen, an evidence-centric learning framework for agentic audio understanding. Guided by evidence before commitment, Think2Listen couples iterative reasoning with a targeted LALM interaction to actively seek acoustic cues that distinguish competing hypotheses and revise candidate judgments based on the observations returned. It learns what to ask, how to use evidence, and when to stop, using observation-induced changes in candidate-state to inform training credit. Extensive evaluations on two benchmarks spanning seven audio categories demonstrate that Think2Listen achieves (I) enhanced multi-step audio reasoning, with gains in overall accuracy of 5.08%--17.67% over competing audio agents; and (II) improved audio understanding and reasoning across speech, sound, and music, boosting overall accuracy by 2.29%--13.10% over the corresponding baselines. The source code is available at https://github.com/JinWang79/Think2Listen.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces an evidence-centric learning framework for agentic audio understanding that couples iterative reasoning with targeted LALM interactions, using observation-induced candidate-state changes to inform policy training. This approach effectively bridges the gap between acoustic perception and discriminative reasoning, demonstrating significant performance gains over existing audio agents by learning what to ask, how to use evidence, and when to stop.
Methodology
The paper proposes "Think2Listen," a framework that integrates an "Adaptive Audio Evidence Loop" with a novel training signal called "Transition-as-Credit." The methodology addresses the gap between acoustic perception and evidence-based reasoning by allowing an agent to iteratively query a Large Audio-Language Model (LALM) for specific acoustic cues before committing to an answer. The core technical contribution is the "Transition-as-Credit" mechanism, which uses a semantic extractor to reconstruct how observations change the agent's candidate state (eliminating or confirming hypotheses). This process-level feedback is used to calibrate rewards in a Group Relative Policy Optimization (GRPO) style update, distinguishing between useful evidence-seeking and misleading or redundant queries. The approach is theoretically sound, leveraging intermediate state changes to provide denser supervision signals than terminal accuracy alone.
Results & evaluation
The authors evaluate the framework on two benchmarks spanning seven audio categories (speech, sound, music). The results show significant improvements over competing audio agents (5.08%–17.67% accuracy gain) and baselines (2.29%–13.10% gain). The inclusion of ablation studies on the "Transition-as-Credit" component and the analysis of untrained interaction patterns (Figure 2) provides strong empirical support for the necessity of the proposed learning strategy. The experiments effectively demonstrate that learning *what* to ask and *when* to stop is crucial for agentic audio understanding.
Reproducibility
The paper provides a GitHub repository link, which is a positive indicator for reproducibility. The methodology is described with sufficient detail regarding the state formulation, the semantic extractor's role, and the reward calibration formulas. However, specific hyperparameters for the LALM and the actor policy are likely in the appendix (truncated in the provided text), which is standard. The reliance on a specific LALM for the "listener" role may limit immediate reproducibility if that model is not open-source or easily accessible, but the framework itself is clearly defined.
Limitations
The framework relies on the quality of the underlying LALM for generating observations; if the LALM hallucinates acoustic features, the agent's reasoning will be flawed, although the "misleading trajectory" flag attempts to mitigate this. The computational cost of iterative querying and the semantic extraction step for every turn may be high compared to single-shot inference. Additionally, the method is tested on specific benchmarks; generalization to open-ended audio tasks without predefined candidate sets $C$ is not explicitly addressed.
Broader impact
This work contributes to the emerging field of agentic multimodal AI, specifically in the audio domain. By formalizing "evidence-centric" learning, it offers a template for other modalities where perception and reasoning are decoupled. The technique of using state-transition changes for credit assignment is applicable to other tool-using agents. It advances the capability of AI systems to handle complex, multi-step audio reasoning tasks that require active information gathering rather than passive listening. The paper introduces an evidence-centric learning framework for agentic audio understanding that couples iterative reasoning with targeted LALM interactions, using observation-induced candidate-state changes to inform policy training. This approach effectively bridges the gap between acoustic perception and discriminative reasoning, demonstrating significant performance gains over existing audio agents by learning what to ask, how to use evidence, and when to stop.
Zero-shot voice cloning typically emphasizes preserving the timbre and prosodic characteristics of a reference speaker. However, in real-world use, speech should adapt its delivery to the target text rather than remain tied to a fixed speaking style. In this work, we formulate…
Abstract
Zero-shot voice cloning typically emphasizes preserving the timbre and prosodic characteristics of a reference speaker. However, in real-world use, speech should adapt its delivery to the target text rather than remain tied to a fixed speaking style. In this work, we formulate this problem as text-adaptive prosody synthesis for zero-shot voice cloning and study how post-training can endow a cloning model with this capability. We develop a three-stage framework. First, Direct Preference Optimization (DPO) provides a cold start by aligning the model with text-appropriate expressive speech. Second, Group Sequence Policy Optimization (GSPO) jointly optimizes prosodic expressiveness, acoustic quality, intelligibility, and speaker similarity. Finally, context-conditioned On-Policy Self-Distillation (OPSD) enables self-improvement by transferring expressive behavior from strong-reference to weak-reference conditions. On Qwen3-TTS, our framework improves Emotion from 18.5 to 81.2 and Paralinguistics from 38.4 to 73.9. Extensive ablations validate the role of each stage.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces a three-stage post-training framework (DPO, GSPO, OPSD) to achieve text-adaptive prosody in zero-shot TTS, significantly improving expressiveness while maintaining speaker identity. The technical contribution is strong, with a well-designed methodology that leverages modern RLHF techniques and LLM-based data pipelines, supported by extensive experiments and ablations that validate the effectiveness of each stage.
Methodology
The paper proposes a three-stage post-training framework (DPO, GSPO, OPSD) to enable text-adaptive prosody in zero-shot TTS. The methodology is sound, leveraging established RLHF techniques (DPO, GSPO) and a novel application of self-distillation (OPSD) to decouple speaker identity from reference prosody. The use of LLMs (Gemini) for automated preference data construction and reward modeling is a strong, modern approach that addresses the scarcity of high-quality expressive speech data. The formulation of the problem as "text-adaptive prosody synthesis" rather than just "voice cloning" is a meaningful conceptual shift.
Results & evaluation
Experiments are extensive, covering multiple benchmarks (Emergent TTS-Eval, Seed-TTS-Eval) and subjective evaluations (MOS, ABX). The ablation studies are thorough, isolating the contributions of each stage and hyperparameter choices (e.g., speaker diversity, reward weights). The results show significant improvements in emotion and paralinguistics scores, though there is a noted trade-off with speaker similarity, which is honestly reported. The use of human ABX tests to validate reduced reliance on reference expressiveness is a strong experimental design choice.
Reproducibility
Reproducibility is moderate. While the framework is well-described, reliance on proprietary models (Gemini 3.1, Qwen3-TTS) and specific datasets (Emilia, SeedVC) limits open reproduction. The automated data pipeline is detailed, but the exact prompts and filtering thresholds for Gemini-based verification are not fully specified. The code is not explicitly linked, though a demo page is provided.
Limitations
The primary limitation is the trade-off between expressiveness and speaker similarity, which the authors acknowledge but do not fully resolve. The method relies heavily on the quality of the LLM-based reward model and data generation, introducing potential biases. The evaluation is primarily on English and Chinese, with limited cross-lingual generalization analysis. The computational cost of the multi-stage training and LLM-in-the-loop data generation is not discussed.
Broader impact
This work has significant implications for the development of more natural and expressive TTS systems, moving beyond static voice cloning towards dynamic, context-aware speech generation. The framework could be adapted to other speech tasks and languages. The use of LLMs for data augmentation and reward modeling sets a precedent for future work in speech AI. The paper introduces a three-stage post-training framework (DPO, GSPO, OPSD) to achieve text-adaptive prosody in zero-shot TTS, significantly improving expressiveness while maintaining speaker identity. The technical contribution is strong, with a well-designed methodology that leverages modern RLHF techniques and LLM-based data pipelines, supported by extensive experiments and ablations that validate the effectiveness of each stage.
Audio phase relations encode source locations, yet existing large audio-language models (LALM) reduce audio to single-channel content to stabilise general audio understanding, leaving spatial information for direction, distance, and source-relation questions unused. We present…
Abstract
Audio phase relations encode source locations, yet existing large audio-language models (LALM) reduce audio to single-channel content to stabilise general audio understanding, leaving spatial information for direction, distance, and source-relation questions unused. We present SpheraAudio, which accepts recordings with any channel count -- mono, stereo, First-Order Ambisonics (FOA), or higher -- and maps them into a single token stream. A spatial encoder turns complex multichannel spectra into spatial tokens interleaved with audio tokens at corresponding temporal positions. Spatial Multimodal RoPE (SMRoPE) makes each token's modality identity and temporal slot visible to rotary attention without altering the language model's original RoPE. Training runs in three stages -- spatial-encoder pretraining, projector alignment, and joint supervised-and-reinforcement learning -- progressing from geometric supervision to language-based spatial decisions, with answers trained as explicit spatial chain-of-thought: step-by-step geometric reasoning over the heard scene. Current benchmarks tie questions to a fixed channel-count format and score answers without spatial reasoning evidence. We introduce SpheraQA, which spans mono, stereo, and FOA recordings and organises questions hierarchically from single-attribute perception through cross-source relations to multi-step reasoning, scored programmatically against simulated scene geometry and real-recording annotations with step-by-step solution references. Across SpheraQA and public spatial-audio benchmarks, SpheraAudio sets a new performance standard for spatial audio understanding. It supports high-precision localisation, tracking, and reasoning over static and moving sound sources while preserving strong general-audio capabilities. We will release code, SpheraQA, and evaluation metrics to support reproducibility.
AI-generated analysis. Check claims against the paper.
Summary
The paper presents SpheraAudio, a novel LALM framework that integrates spatial audio understanding via a dedicated spatial encoder and SMRoPE, achieving state-of-the-art performance on spatial reasoning tasks while preserving general audio capabilities. The contribution is significant due to its comprehensive handling of multichannel inputs, the introduction of the SpheraQA benchmark, and the rigorous training pipeline combining geometric pretraining, CoT supervision, and reinforcement learning, effectively bridging the gap between signal processing and language-based spatial reasoning.
Methodology
The paper proposes SpheraAudio, a framework integrating spatial audio understanding into Large Audio-Language Models (LALMs). The core methodological contributions are the "Spatial AuT" encoder, which processes complex multichannel spectra (FOA, stereo, mono) into spatial tokens, and "SMRoPE" (Spatial Multimodal RoPE), a novel rotary position embedding scheme that encodes modality identity and temporal alignment without altering the base LLM's RoPE. The training pipeline is rigorous, involving geometric pretraining of the spatial encoder, projector alignment, and joint SFT with explicit spatial Chain-of-Thought (CoT) supervision, followed by GRPO reinforcement learning. The approach effectively bridges the gap between raw multichannel signal processing and high-level language reasoning by explicitly pairing audio and spatial tokens in time.
Results & evaluation
The authors introduce SpheraQA, a substantial benchmark (20,492 questions) covering perception, relations, and reasoning across mono, stereo, and FOA formats. Experimental results show SpheraAudio outperforming strong baselines like Spatial-Omni-7B and Gemini 3.1 Pro on spatial tasks while retaining or improving general audio capabilities. Ablation studies convincingly demonstrate the necessity of token interleaving (vs. addition) and the specific SMRoPE design for temporal correspondence. The inclusion of stereo-specific experiments and general audio retention tests provides a comprehensive evaluation of the model's versatility.
Reproducibility
The paper provides detailed implementation specifications, including architecture dimensions, loss functions, hyperparameters for SFT and RL, and precise evaluation protocols for SpheraQA. The commitment to release code, the benchmark, and evaluation metrics enhances reproducibility. The use of GPT-4o for answer extraction in free-form responses is a practical but potentially biased evaluation choice, though mitigated by programmatic scoring for structured answers.
Limitations
The model relies on a 4B parameter backbone (Qwen3-Omni), which may limit its performance on extremely complex reasoning compared to larger LLMs. The spatial encoder is pretrained on simulated FOA data; while real recordings are included in evaluation, the generalization to uncalibrated, noisy real-world microphone arrays beyond the tested LOCATA/STARSS23 datasets is not fully explored. The "Motion direction" task shows a negative delta compared to the baseline, indicating a weakness in specific dynamic spatial reasoning.
Broader impact
This work significantly advances the field of spatial audio AI by providing a unified framework for handling arbitrary channel counts. It enables embodied agents to understand sound scenes with precision, supporting applications in navigation, robotics, and immersive media. The introduction of SpheraQA provides a much-needed standardized benchmark for evaluating spatial reasoning in LALMs, moving beyond simple localization to complex geometric reasoning. The paper presents SpheraAudio, a novel LALM framework that integrates spatial audio understanding via a dedicated spatial encoder and SMRoPE, achieving state-of-the-art performance on spatial reasoning tasks while preserving general audio capabilities. The contribution is significant due to its comprehensive handling of multichannel inputs, the introduction of the SpheraQA benchmark, and the rigorous training pipeline combining geometric pretraining, CoT supervision, and reinforcement learning, effectively bridging the gap between signal processing and language-based spatial reasoning.
Rapid progress in text-to-audio (TTA) generation demands reliable and explainable automatic evaluation. However, existing metrics fail to simultaneously deliver high correlation with human judgments and meaningful, audio-grounded explanations, forcing reliance on labor-intensive…
Abstract
Rapid progress in text-to-audio (TTA) generation demands reliable and explainable automatic evaluation. However, existing metrics fail to simultaneously deliver high correlation with human judgments and meaningful, audio-grounded explanations, forcing reliance on labor-intensive human evaluation that bottlenecks model development iteration. To address this, we propose CoTTAGE, a reasoning-based evaluator for text relevance (REL) and audio quality (OVL). Following a reason-then-predict paradigm, CoTTAGE first generates an audio-grounded rationale to diagnose errors before predicting scores. Supported by over 10K newly collected human OVL annotations, CoTTAGE is first trained via supervised fine-tuning and then refined with reinforcement learning using a Thurstone-based preference objective that converts sampled score distributions into pairwise preference probabilities and rewards agreement with rankings derived from human judgments. Across six benchmarks spanning REL and OVL, CoTTAGE achieves state-of-the-art correlation with human judgments while producing human-preferred explanations to accelerate TTA development.
AI-generated analysis. Check claims against the paper.
Summary
CoTTAGE introduces a reasoning-based evaluator for text-to-audio generation that achieves state-of-the-art correlation with human judgments by combining supervised fine-tuning with a novel Thurstone-based reinforcement learning objective, while also providing human-preferred, audio-grounded explanations for its scores.
Methodology
The paper proposes CoTTAGE, a reasoning-based evaluator for text-to-audio (TTA) generation that assesses both text relevance (REL) and overall audio quality (OVL). The core methodological contribution is the "reason-then-predict" paradigm, where the model generates an audio-grounded rationale before predicting a score. This is implemented via a two-stage training pipeline: (1) Supervised Fine-Tuning (SFT) on reasoning traces generated by a proprietary LLM (Gemini-3.1 Pro) conditioned on ground-truth MOS scores, and (2) Reinforcement Learning (RL) using a Thurstone-based pairwise preference objective. The RL stage is particularly interesting as it leverages the variance of sampled scores to create uncertainty-aware preference probabilities, rewarding the model for aligning with human rankings rather than just pointwise scores. The use of a Bhattacharyya coefficient for the reward signal is a solid technical choice for handling probabilistic preferences.
Results & evaluation
The experimental setup is robust, utilizing six benchmarks (RELATE, XACLE, PAM, Clotho) to test both in-domain and out-of-domain generalization. The collection of 10K+ new human OVL annotations is a significant contribution, as OVL data was previously scarce compared to REL data. The results show CoTTAGE outperforming strong baselines, including embedding-based metrics (CLAPScore, ELSA) and general-purpose LALMs (Gemini-3.1 Pro, Qwen2.5-Omni). The ablation study clearly demonstrates the benefit of the RL stage over SFT alone. The human preference evaluation of the rationales further validates the explainability claim, showing that CoTTAGE's explanations are preferred over those of the base model and even the proprietary teacher model in some cases.
Reproducibility
The paper provides high reproducibility. The code and data are publicly available on GitHub. Detailed hyperparameters, prompts, and annotation protocols are included in the appendices. The use of standard models (Qwen2.5-Omni-7B) and LoRA fine-tuning makes the method accessible to the community. The detailed description of the MTurk annotation process, including quality control measures, adds to the transparency of the dataset creation.
Limitations
The primary limitation is the reliance on a proprietary model (Gemini-3.1 Pro) for generating the reasoning traces during SFT. This introduces a dependency on a closed-source system and potential biases from that specific model. Additionally, the evaluation is limited to short audio clips (<30s), and the model does not provide temporal grounding (timestamps) for errors, which limits its utility for fine-grained debugging of long-form audio. The domain coverage is also limited to environmental sounds, with no evaluation on music or speech-centric generation.
Broader impact
This work addresses a critical bottleneck in TTA development: the lack of scalable, explainable automatic evaluation. By providing a model that not only scores audio but also explains *why* it gave that score, CoTTAGE enables more efficient iteration cycles for TTA model developers. The release of the 10K OVL annotations is a valuable resource for the community. The approach of using RL with preference objectives for audio evaluation is a promising direction that could be extended to other audio tasks. CoTTAGE introduces a reasoning-based evaluator for text-to-audio generation that achieves state-of-the-art correlation with human judgments by combining supervised fine-tuning with a novel Thurstone-based reinforcement learning objective, while also providing human-preferred, audio-grounded explanations for its scores.
Speech tokenizers convert audio into discrete units, enabling language models to process and generate speech. The goal is to learn compact, text-like representations while preserving enough detail to reconstruct a speaker's voice and delivery. We propose Facto: a factorized…
Abstract
Speech tokenizers convert audio into discrete units, enabling language models to process and generate speech. The goal is to learn compact, text-like representations while preserving enough detail to reconstruct a speaker's voice and delivery. We propose Facto: a factorized tokenizer that disentangles time-varying information (content and prosody) from static components (speaker identity and recording conditions). We build on a recent linear factorization method for self-supervised features, extending it to handle unseen speakers. Then, we cluster the factorized space and train a lightweight decoder to reconstruct audio from the resulting tokens. By removing the static components before clustering, our tokens are more robust to small noise perturbations. We validate this by testing token stability under various noise conditions. Next, we theoretically analyze Facto's convergence and scaling properties, quantifying how the factorization improves with the number of training speakers. Finally, we evaluate Facto on a range of downstream tasks. We show that greater token stability translates into improved text-to-speech performance and competitive results in voice conversion and spoken language modeling. Code and checkpoints at https://github.com/ubisoft/ubisoft-laforge-facto/ and demo page at https://ubisoft-laforge.github.io/speech/facto/.
AI-generated analysis. Check claims against the paper.
Summary
Facto proposes a factorized speech tokenizer that disentangles content from speaker identity using linear factorization of self-supervised features, providing theoretical guarantees on convergence and stability. The paper makes a significant contribution by extending linear factorization to unseen speakers and rigorously analyzing the scaling properties with the number of training speakers, resulting in more robust tokens for downstream tasks like TTS and voice conversion.
Methodology
The paper proposes "Facto," a factorized speech tokenizer that disentangles time-varying content/prosody from static speaker/channel information. The core method builds on a linear factorization of self-supervised features (WavLM), extending it to handle unseen speakers via a linear content estimator. The methodology is theoretically grounded, providing convergence analysis for the factorization and a bias-variance trade-off analysis for the content estimator. The approach uses PCA and k-means clustering on the factorized content space, followed by a lightweight HiFiGAN-based decoder with FiLM conditioning for speaker identity reintroduction. The theoretical contributions (Theorem 1 and 2, Corollary 1) are a strong differentiator, offering rigorous bounds on error scaling with the number of training speakers.
Results & evaluation
The experiments validate the theoretical predictions, showing that the factorization stabilizes as the number of speakers increases (O(1/K) decay). The paper demonstrates improved token stability under noise perturbations compared to baselines. Downstream tasks include text-to-speech, voice conversion, and spoken language modeling. The results show competitive performance, with clear improvements in TTS due to token stability. The use of LibriSpeech and standard metrics provides a solid empirical foundation.
Reproducibility
The paper provides code and checkpoints on GitHub and a demo page. The methodology is described in sufficient detail, including hyperparameters (e.g., rank R=12, WavLM layer 6) and training setups. The theoretical derivations are provided in the main text or appendices, enhancing reproducibility.
Limitations
The method relies on the quality of the self-supervised features (WavLM) and the linear factorization assumption, which may not capture all non-linear interactions. The content estimator is linear, which might limit its ability to handle highly complex speaker variations. The paper focuses on English speech (LibriSpeech), and generalization to other languages or noisy real-world environments beyond small perturbations is not extensively tested.
Broader impact
The work contributes to the field of speech tokenization by providing a principled, theoretically sound approach to disentangling content and speaker identity. This can benefit downstream tasks like TTS, voice conversion, and speech language modeling by providing more robust and compact representations. The emphasis on stability and theoretical guarantees sets a high standard for future work in speech representation learning. Facto proposes a factorized speech tokenizer that disentangles content from speaker identity using linear factorization of self-supervised features, providing theoretical guarantees on convergence and stability. The paper makes a significant contribution by extending linear factorization to unseen speakers and rigorously analyzing the scaling properties with the number of training speakers, resulting in more robust tokens for downstream tasks like TTS and voice conversion.