Audio ML PapersLandmark papers
← All papers
RESEARCH PAPER / 2026-11-13

Any Channel Count, One Token Stream: Incorporating Spatial Audio Understanding and Reasoning into Large Audio-Language Models

Yuhuan You, Wenxiang Guo, Zhifang Guo, Yuxuan Wang, Jin Xu

Alibaba Group +1 · arXiv

Affiliations: Alibaba Group, Zhejiang University

Abstract

Audio phase relations encode source locations, yet existing large audio-language models (LALM) reduce audio to single-channel content to stabilise general audio understanding, leaving spatial information for direction, distance, and source-relation questions unused. We present SpheraAudio, which accepts recordings with any channel count -- mono, stereo, First-Order Ambisonics (FOA), or higher -- and maps them into a single token stream. A spatial encoder turns complex multichannel spectra into spatial tokens interleaved with audio tokens at corresponding temporal positions. Spatial Multimodal RoPE (SMRoPE) makes each token's modality identity and temporal slot visible to rotary attention without altering the language model's original RoPE. Training runs in three stages -- spatial-encoder pretraining, projector alignment, and joint supervised-and-reinforcement learning -- progressing from geometric supervision to language-based spatial decisions, with answers trained as explicit spatial chain-of-thought: step-by-step geometric reasoning over the heard scene. Current benchmarks tie questions to a fixed channel-count format and score answers without spatial reasoning evidence. We introduce SpheraQA, which spans mono, stereo, and FOA recordings and organises questions hierarchically from single-attribute perception through cross-source relations to multi-step reasoning, scored programmatically against simulated scene geometry and real-recording annotations with step-by-step solution references. Across SpheraQA and public spatial-audio benchmarks, SpheraAudio sets a new performance standard for spatial audio understanding. It supports high-precision localisation, tracking, and reasoning over static and moving sound sources while preserving strong general-audio capabilities. We will release code, SpheraQA, and evaluation metrics to support reproducibility.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

The paper presents SpheraAudio, a novel LALM framework that integrates spatial audio understanding via a dedicated spatial encoder and SMRoPE, achieving state-of-the-art performance on spatial reasoning tasks while preserving general audio capabilities. The contribution is significant due to its comprehensive handling of multichannel inputs, the introduction of the SpheraQA benchmark, and the rigorous training pipeline combining geometric pretraining, CoT supervision, and reinforcement learning, effectively bridging the gap between signal processing and language-based spatial reasoning.

Methodology

The paper proposes SpheraAudio, a framework integrating spatial audio understanding into Large Audio-Language Models (LALMs). The core methodological contributions are the "Spatial AuT" encoder, which processes complex multichannel spectra (FOA, stereo, mono) into spatial tokens, and "SMRoPE" (Spatial Multimodal RoPE), a novel rotary position embedding scheme that encodes modality identity and temporal alignment without altering the base LLM's RoPE. The training pipeline is rigorous, involving geometric pretraining of the spatial encoder, projector alignment, and joint SFT with explicit spatial Chain-of-Thought (CoT) supervision, followed by GRPO reinforcement learning. The approach effectively bridges the gap between raw multichannel signal processing and high-level language reasoning by explicitly pairing audio and spatial tokens in time.

Results & evaluation

The authors introduce SpheraQA, a substantial benchmark (20,492 questions) covering perception, relations, and reasoning across mono, stereo, and FOA formats. Experimental results show SpheraAudio outperforming strong baselines like Spatial-Omni-7B and Gemini 3.1 Pro on spatial tasks while retaining or improving general audio capabilities. Ablation studies convincingly demonstrate the necessity of token interleaving (vs. addition) and the specific SMRoPE design for temporal correspondence. The inclusion of stereo-specific experiments and general audio retention tests provides a comprehensive evaluation of the model's versatility.

Reproducibility

The paper provides detailed implementation specifications, including architecture dimensions, loss functions, hyperparameters for SFT and RL, and precise evaluation protocols for SpheraQA. The commitment to release code, the benchmark, and evaluation metrics enhances reproducibility. The use of GPT-4o for answer extraction in free-form responses is a practical but potentially biased evaluation choice, though mitigated by programmatic scoring for structured answers.

Limitations

The model relies on a 4B parameter backbone (Qwen3-Omni), which may limit its performance on extremely complex reasoning compared to larger LLMs. The spatial encoder is pretrained on simulated FOA data; while real recordings are included in evaluation, the generalization to uncalibrated, noisy real-world microphone arrays beyond the tested LOCATA/STARSS23 datasets is not fully explored. The "Motion direction" task shows a negative delta compared to the baseline, indicating a weakness in specific dynamic spatial reasoning.

Broader impact

This work significantly advances the field of spatial audio AI by providing a unified framework for handling arbitrary channel counts. It enables embodied agents to understand sound scenes with precision, supporting applications in navigation, robotics, and immersive media. The introduction of SpheraQA provides a much-needed standardized benchmark for evaluating spatial reasoning in LALMs, moving beyond simple localization to complex geometric reasoning. The paper presents SpheraAudio, a novel LALM framework that integrates spatial audio understanding via a dedicated spatial encoder and SMRoPE, achieving state-of-the-art performance on spatial reasoning tasks while preserving general audio capabilities. The contribution is significant due to its comprehensive handling of multichannel inputs, the introduction of the SpheraQA benchmark, and the rigorous training pipeline combining geometric pretraining, CoT supervision, and reinforcement learning, effectively bridging the gap between signal processing and language-based spatial reasoning.

Author of this paper? Send a correction.