Machine Hearing: Can Audio LLMs Generalize Beyond Human Auditory Perception?
Abstract
Multimodal LLMs have made remarkable progress in perceiving the physical world, while their perceptual scope remains largely human-centric, focusing on what humans can directly see or hear. This progress raises the prospect of perception beyond human limits, such as extending hearing to sounds that are inaudible to humans. More broadly, the physical world is full of waves, vibrations, and oscillations, only a small fraction of which are directly audible to humans. This invites the extension of multimodal LLMs toward a broader form of machine hearing, which remains largely underexplored. To investigate this direction, we introduce the Machine Hearing Benchmark (MHB), comprising 6,128 instances across 13 tasks and seven domains. MHB spans highly heterogeneous input regimes, with sampling rates from approximately $10^{-5}$ to $10^6$ Hz, sequence lengths from $10^1$ to $10^6$ samples, and both single- and multi-channel inputs. Evaluations of current open-source and proprietary Audio LLMs (ALLMs) reveal limited zero-shot transfer to machine hearing. Supervised adaptation on machine-hearing data alone does not close this gap, suggesting that the challenge extends beyond data exposure to a systematic mismatch between existing audio front ends and machine-hearing inputs. To address this mismatch, we introduce MachEar, a lightweight encoder extension that complements the pretrained Audio LLM for machine-hearing inputs. The results show that MachEar equips a pretrained Audio LLM with stronger machine-hearing capabilities while largely preserving its existing audio understanding performance.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces the Machine Hearing Benchmark and the MachEar encoder extension, demonstrating that current Audio LLMs struggle with non-human-audible signals and proposing a lightweight architectural fix to bridge this gap. The technical contribution is strong, combining a well-curated benchmark with a targeted architectural innovation. The methodology is sound, using parallel processing to handle heterogeneous signal regimes without disrupting the pretrained LLM's capabilities. The significance to the field is high, as it shifts the focus from human-centric audio understanding to broader physical world perception, addressing a critical limitation in current multimodal models. The rigorous evaluation against both general and domain-specific models provides a clear baseline and validates the proposed solution's effectiveness.
Methodology
The paper proposes a novel paradigm of "Machine Hearing," extending Audio LLMs (ALLMs) beyond human-audible frequencies to process signals ranging from $10^{-5}$ Hz to $10^6$ Hz. The methodology involves constructing the Machine Hearing Benchmark (MHB) with 6,128 instances across 13 tasks in 7 domains (Astronomy, Earth Science, Healthcare, etc.). The core technical contribution is MachEar, a lightweight encoder extension that processes inputs in parallel with the pretrained audio encoder. MachEar utilizes parallel FFT and 1D CNN pathways to extract spectral and temporal features, which are then processed by Transformer blocks to generate compact tokens. This design aims to address the systematic mismatch between standard audio front-ends (optimized for 16-48kHz speech/audio) and extreme machine-hearing inputs (ultra-low or ultra-high frequency, short sequences). The approach is elegant in its modularity, allowing it to complement rather than replace the existing LLM backbone.
Results & evaluation
The experimental evaluation is rigorous and comprehensive. The authors benchmark seven state-of-the-art ALLMs (including Qwen2.5-Omni, Audio-Flamingo-3, Gemini-3-Flash, GPT-Audio-Mini, and Doubao) in a zero-shot setting. The results clearly demonstrate that current ALLMs fail significantly on machine-hearing tasks, often performing below random guessing. The paper also evaluates domain-specific foundation models (e.g., SeisLM, SpectrumFM) to provide context, showing that while domain pretraining helps, it doesn't solve the generalization issue for ALLMs. The introduction of MachEar shows substantial improvements in Macro-F1 scores for Qwen2.5-Omni, validating the hypothesis that the bottleneck lies in the audio front-end processing rather than just the LLM's reasoning capabilities. The analysis of failure modes (e.g., short-input failures, unrecognized inputs) provides valuable diagnostic insights.
Reproducibility
The paper provides high reproducibility. The benchmark is released on HuggingFace, and the code for MachEar is presumably available (linked in the project URL). The experimental setup, including model versions, prompting protocols, and evaluation metrics (Macro-F1, Success Rate), is clearly defined. The use of public datasets and standard models further enhances reproducibility.
Limitations
The primary limitation is the reliance on a specific base model (Qwen2.5-Omni) for the MachEar demonstration; while the architecture is general, the specific gains might vary across different LLM backbones. Additionally, the benchmark, while diverse, is relatively small (6,128 instances) for such a broad range of domains, which may limit the statistical robustness of some conclusions. The paper focuses on classification tasks; the ability to perform generative or reasoning tasks on these signals is not deeply explored.
Broader impact
This work has significant broader impact by defining a new frontier for multimodal AI: perception beyond human sensory limits. It opens avenues for AI in scientific discovery (astronomy, geophysics), industrial monitoring, and healthcare diagnostics where signals are not human-audible. The proposed MachEar module is a practical solution that can be integrated into existing ALLMs, potentially accelerating the adoption of AI in these specialized fields. The paper introduces the Machine Hearing Benchmark and the MachEar encoder extension, demonstrating that current Audio LLMs struggle with non-human-audible signals and proposing a lightweight architectural fix to bridge this gap. The technical contribution is strong, combining a well-curated benchmark with a targeted architectural innovation. The methodology is sound, using parallel processing to handle heterogeneous signal regimes without disrupting the pretrained LLM's capabilities. The significance to the field is high, as it shifts the focus from human-centric audio understanding to broader physical world perception, addressing a critical limitation in current multimodal models. The rigorous evaluation against both general and domain-specific models provides a clear baseline and validates the proposed solution's effectiveness.
Author of this paper? Send a correction.