CoTTAGE: An Explainable Judge for Text-to-Audio Generation
Abstract
Rapid progress in text-to-audio (TTA) generation demands reliable and explainable automatic evaluation. However, existing metrics fail to simultaneously deliver high correlation with human judgments and meaningful, audio-grounded explanations, forcing reliance on labor-intensive human evaluation that bottlenecks model development iteration. To address this, we propose CoTTAGE, a reasoning-based evaluator for text relevance (REL) and audio quality (OVL). Following a reason-then-predict paradigm, CoTTAGE first generates an audio-grounded rationale to diagnose errors before predicting scores. Supported by over 10K newly collected human OVL annotations, CoTTAGE is first trained via supervised fine-tuning and then refined with reinforcement learning using a Thurstone-based preference objective that converts sampled score distributions into pairwise preference probabilities and rewards agreement with rankings derived from human judgments. Across six benchmarks spanning REL and OVL, CoTTAGE achieves state-of-the-art correlation with human judgments while producing human-preferred explanations to accelerate TTA development.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
CoTTAGE introduces a reasoning-based evaluator for text-to-audio generation that achieves state-of-the-art correlation with human judgments by combining supervised fine-tuning with a novel Thurstone-based reinforcement learning objective, while also providing human-preferred, audio-grounded explanations for its scores.
Methodology
The paper proposes CoTTAGE, a reasoning-based evaluator for text-to-audio (TTA) generation that assesses both text relevance (REL) and overall audio quality (OVL). The core methodological contribution is the "reason-then-predict" paradigm, where the model generates an audio-grounded rationale before predicting a score. This is implemented via a two-stage training pipeline: (1) Supervised Fine-Tuning (SFT) on reasoning traces generated by a proprietary LLM (Gemini-3.1 Pro) conditioned on ground-truth MOS scores, and (2) Reinforcement Learning (RL) using a Thurstone-based pairwise preference objective. The RL stage is particularly interesting as it leverages the variance of sampled scores to create uncertainty-aware preference probabilities, rewarding the model for aligning with human rankings rather than just pointwise scores. The use of a Bhattacharyya coefficient for the reward signal is a solid technical choice for handling probabilistic preferences.
Results & evaluation
The experimental setup is robust, utilizing six benchmarks (RELATE, XACLE, PAM, Clotho) to test both in-domain and out-of-domain generalization. The collection of 10K+ new human OVL annotations is a significant contribution, as OVL data was previously scarce compared to REL data. The results show CoTTAGE outperforming strong baselines, including embedding-based metrics (CLAPScore, ELSA) and general-purpose LALMs (Gemini-3.1 Pro, Qwen2.5-Omni). The ablation study clearly demonstrates the benefit of the RL stage over SFT alone. The human preference evaluation of the rationales further validates the explainability claim, showing that CoTTAGE's explanations are preferred over those of the base model and even the proprietary teacher model in some cases.
Reproducibility
The paper provides high reproducibility. The code and data are publicly available on GitHub. Detailed hyperparameters, prompts, and annotation protocols are included in the appendices. The use of standard models (Qwen2.5-Omni-7B) and LoRA fine-tuning makes the method accessible to the community. The detailed description of the MTurk annotation process, including quality control measures, adds to the transparency of the dataset creation.
Limitations
The primary limitation is the reliance on a proprietary model (Gemini-3.1 Pro) for generating the reasoning traces during SFT. This introduces a dependency on a closed-source system and potential biases from that specific model. Additionally, the evaluation is limited to short audio clips (<30s), and the model does not provide temporal grounding (timestamps) for errors, which limits its utility for fine-grained debugging of long-form audio. The domain coverage is also limited to environmental sounds, with no evaluation on music or speech-centric generation.
Broader impact
This work addresses a critical bottleneck in TTA development: the lack of scalable, explainable automatic evaluation. By providing a model that not only scores audio but also explains *why* it gave that score, CoTTAGE enables more efficient iteration cycles for TTA model developers. The release of the 10K OVL annotations is a valuable resource for the community. The approach of using RL with preference objectives for audio evaluation is a promising direction that could be extended to other audio tasks. CoTTAGE introduces a reasoning-based evaluator for text-to-audio generation that achieves state-of-the-art correlation with human judgments by combining supervised fine-tuning with a novel Thurstone-based reinforcement learning objective, while also providing human-preferred, audio-grounded explanations for its scores.
Author of this paper? Send a correction.