Mix-CLAP: Teaching Audio AI to Work in the Noisy Real World

Introduction

That is exactly the deployment gap that “Mix-CLAP: Adaptive Fusion of Knowledge-Distilled Audio Embeddings for Noise-Aware Audio-Language Models” addresses. The paper was presented at the 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2026) and introduces a lightweight audio classification framework that remains reliable in noisy, resource-constrained environments. The authors are Wataru Kohno (NEC Corporation and NEC Laboratories America), Shaobo Han (NEC Laboratories America), Noriyuki Tonami (NEC Corporation), Tingfeng Li (NEC Laboratories America), Jingchen Sun (NEC Laboratories America and The State University of New York at Buffalo), and Ting Wang (NEC Laboratories America).

Why Clean-Audio AI Fails in the Field

Contrastive Language-Audio Pretraining, or CLAP, is a method that trains an AI model to align sounds with natural language descriptions. A CLAP model learns, without task-specific labels, that the audio of a jackhammer corresponds to text like “construction noise” and that birdsong corresponds to “outdoor nature sounds.” This zero-shot capability, meaning the ability to classify sounds it was not explicitly trained on, makes CLAP attractive for open-ended acoustic monitoring.

The problem is computational weight. State-of-the-art CLAP models rely on Transformer-based audio encoders, the large neural network components that convert audio signals into compact numerical representations. These encoders achieve high accuracy but require significant processing power. Running them on battery-powered edge sensors, fiber-optic Distributed Acoustic Sensing (DAS) arrays, or wireless mesh nodes is not practical.

Previous attempts to build smaller, more efficient encoders using a technique called knowledge distillation, in which a compact “student” model learns to mimic a large “teacher” model, have tended to struggle when conditions deviate from those the student was trained on. A model trained on clean microphone audio, for example, may perform poorly when deployed on a fiber-optic sensor picking up low-signal-to-noise recordings.

The Solution: Two Encoders, One Adaptive Decision

Mix-CLAP addresses this with two connected ideas. The first is training two lightweight student encoders rather than one. One encoder trains on clean audio; the other trains on noisy audio. Both learn to produce embeddings, the numerical fingerprints of a sound, that stay close to what the larger teacher model would produce. Using MobileNetV3-Large backbones, a family of small, efficient neural networks designed for mobile and embedded applications, each student encoder requires only about 5% of the computational operations of the full Transformer-based teacher model.

The second idea is adaptive inference at test time. Rather than committing to one encoder or the other, Mix-CLAP blends their outputs using a single mixing parameter, then finds the blend that produces the most confident, lowest-entropy prediction for each incoming audio sample. Think of it as a dial between “trust the clean-audio model” and “trust the noise-robust model,” automatically adjusted for each recording based on how certain the combined output is. This test-time adaptation (TTA) requires no additional labeled data and incurs negligible computational cost, since it amounts to a small one-dimensional search over the mixing parameter.

One subtle but important detail: the method allows the mixing parameter to go outside the standard zero-to-one interpolation range. This mild extrapolation, rather than just blending the two encoders, can push the combined embedding further in a direction that reduces uncertainty. The paper shows this is especially useful under severe noise, where neither encoder alone gets the prediction right.

When you deploy audio AI in the field, the biggest practical question is not just accuracy on a clean benchmark. It is whether the model still works when the recording device is a fiber-optic sensor or a weathered outdoor microphone picking up background noise you never trained on. Our adaptive fusion approach lets the system figure out, sample by sample, how much to trust each encoder depending on what it is hearing. – Wataru Kohno, NEC Corporation and Laboratories America

The Results: Near-Teacher Accuracy at a Fraction of the Cost

The researchers evaluated Mix-CLAP on ESC-50, a standard benchmark comprising 2,000 environmental sound clips across 50 categories, recorded with three device types: an electric microphone, a fiber mandrel sensor, and a fiber coil sensor. The fiber-based devices introduce the kinds of low-signal-to-noise conditions encountered in real infrastructure deployments.

Mix-CLAP, with its best encoder configuration (MobileNetV3-Large with a width multiplier of 2.0 for the clean encoder and 1.0 for the noise encoder), achieved an average zero-shot Sound Event Classification accuracy of 52.58% across all three device conditions, compared to 52.83% for the full Transformer-based teacher model. That gap of 0.25 percentage points comes with a substantial reduction in compute: the student encoders together require only about 10% of the multiply-accumulate operations that the teacher demands. Individually, each student with a MobileNetV3-Large 1.0 backbone stays within 2% of the teacher’s average accuracy while using only about 5% of the compute.

On ESC-50 with synthetic Gaussian noise added at varying signal-to-noise ratios, Mix-CLAP with extrapolation-enabled fusion scored an average zero-shot accuracy of 81.46%, outperforming the clean-only encoder (67.82%), the noise-only encoder (79.75%), and all five alternative ensemble methods tested. At the most challenging noise level (-20 dB signal-to-noise ratio), Mix-CLAP reached 25.72% accuracy against 22.68% for the best individual encoder. The method also demonstrated compatibility with standard model calibration techniques: applying temperature scaling after test-time adaptation reduced the Expected Calibration Error from 14.9% to 2.6% at -10 dB signal-to-noise ratio.

Real-World Applications: From Fiber Lines to City Streets

Infrastructure monitoring organizations stand to benefit directly. Pipeline and utility operators who deploy fiber-optic Distributed Acoustic Sensing systems along transmission lines need sound classification models that can run close to the fiber without offloading large data streams. Mix-CLAP’s efficiency makes on-device inference practical, and its robustness to noise means the system remains reliable as signal quality varies over tens of kilometers of cable.

Urban environmental monitoring programs represent a second strong application. Municipal agencies, smart city technology providers, and biodiversity research organizations increasingly deploy wireless acoustic sensor networks across parks, transit corridors, and industrial zones. These networks typically run on low-power edge hardware. Mix-CLAP’s ability to classify sound events with near-teacher-accuracy at a fraction of the compute cost means more sensors can operate longer on battery power alone, expanding coverage without a proportional infrastructure investment.

Toward Deployment-Ready Acoustic Intelligence

As audio AI moves from controlled research settings into infrastructure, logistics, and public monitoring environments, the gap between laboratory accuracy and real-world reliability will remain a central engineering challenge. Mix-CLAP offers a principled path toward closing it: smaller models built to handle the noise they will actually encounter, adapting their reasoning to each new sound they hear.