Improving Large Audio-Language Model Embedding Baselines for Instruction-Aware Speech Retrieval
Abstract
Speech retrieval finds relevant spoken documents given a spoken query. The spoken query alone may not reveal the retrieval goal, such as finding an answer or matching speaker identity. Instruction-aware speech retrieval expresses this goal through a natural-language text instruction. Large audio-language models (LALMs) effectively process speech and text, and our analysis shows that different retrieval requirements favor hidden states from different audio-encoder layers. We introduce IRIS-Embed, which adapts LALMs into embedding models through contrastive training and instruction-aware audio layer weighting. We construct about 900K training samples from real data and synthetic data, with instructions expressing requirements for one or more speech attributes. On INSPIRE, an instruction-aware speech retrieval benchmark, IRIS-Embed consistently outperforms the corresponding LALM baselines. The best model achieves 62.92% macro-average Recall@50, surpassing the strongest baseline at 45.15%. Controlled comparisons demonstrate instruction awareness and the benefits of audio layer weighting. On MAEB, a massive audio embedding benchmark, macro-average gains of 9.4-19.8 points over the corresponding baselines show generalization beyond speech retrieval.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces IRIS-Embed, a novel method for instruction-aware speech retrieval that adapts Large Audio-Language Models by learning instruction-conditioned weights to fuse hidden states from different audio encoder layers. This approach significantly outperforms existing baselines on the INSPIRE benchmark and demonstrates strong generalization to broader audio embedding tasks, offering a scalable and effective framework for context-aware audio retrieval.
Methodology
The paper proposes IRIS-Embed, a framework for adapting Large Audio-Language Models (LALMs) into embedding models for instruction-aware speech retrieval. The core methodological contribution is "instruction-aware audio layer weighting." The authors observe that different retrieval tasks (e.g., speaker matching vs. content retrieval) benefit from different layers of the frozen audio encoder. To exploit this, they introduce a lightweight module that takes the text instruction as input and predicts softmax weights to combine hidden states from all layers of the audio encoder. This weighted representation is then projected and fed into the language model (adapted via LoRA) to generate the final embedding. The training objective is InfoNCE contrastive loss, with careful handling of false negatives by masking potentially relevant documents from the batch pool. The data construction pipeline is robust, combining real data from various corpora (VoxCeleb, UrbanSound8K, etc.) with synthetic data generated via TTS to control specific attributes like speaker identity and style.
Results & evaluation
The experiments are extensive and well-controlled. The primary evaluation is on the INSPIRE benchmark, which is specifically designed for instruction-aware speech retrieval. The model significantly outperforms strong baselines, including specialized encoders (ECAPA-TDNN, emotion2vec) and other LALM-based embeddings (LCO-Embedding). The best model achieves 62.92% Recall@50, surpassing the strongest baseline (45.15%) by a large margin. Ablation studies confirm the effectiveness of both instruction awareness and the audio layer weighting mechanism. Furthermore, the model's generalization is tested on the MAEB benchmark, showing consistent gains across 15 diverse audio tasks, indicating that the learned embeddings are not overfitted to the specific retrieval task but capture broad audio-linguistic alignment.
Reproducibility
The authors provide a public GitHub repository containing code and data construction scripts. The paper details the training hyperparameters, data sources, and specific model architectures (Qwen2.5-Omni-3B, Gemma-4-E2B-it). The use of standard components (LoRA, InfoNCE, frozen encoders) and open-source models enhances reproducibility. The detailed description of the data construction pipeline, including how synthetic data is generated and how instructions are formulated, allows other researchers to replicate the dataset.
Limitations
One potential limitation is the reliance on a specific set of LALMs (Qwen2.5-Omni and Gemma-4). While these are state-of-the-art, the method's performance on other architectures or smaller models might vary. Additionally, the instruction-aware weighting module adds computational overhead during inference, as it requires processing the instruction to generate layer weights, although this is likely negligible compared to the LLM inference. The paper focuses on English speech; performance on multilingual or low-resource languages is not evaluated.
Broader impact
This work bridges the gap between general audio-language understanding and specific retrieval tasks. By demonstrating that LALMs can be effectively adapted for retrieval via layer weighting, it opens avenues for more flexible and context-aware audio search systems. The approach could be extended to other multimodal retrieval tasks (video, image) where different modalities or tasks might benefit from different feature layers. The public release of the dataset and code will likely accelerate research in instruction-aware retrieval. The paper introduces IRIS-Embed, a novel method for instruction-aware speech retrieval that adapts Large Audio-Language Models by learning instruction-conditioned weights to fuse hidden states from different audio encoder layers. This approach significantly outperforms existing baselines on the INSPIRE benchmark and demonstrates strong generalization to broader audio embedding tasks, offering a scalable and effective framework for context-aware audio retrieval.
Author of this paper? Send a correction.