How audio-language models lose to text

Audio-language models often encode the right audio answer, but text still wins the final decision.

• Research org: Unspecified in arXiv abstract
• Core data: 64.1% sign flip rate across five ALMs and four conflict tasks
• Breakthrough: Same-audio counterfactual exposes repairable arbitration reversals

Audio-language models are supposed to use sound and text together, but this paper shows a more awkward failure mode: when the two disagree, the model often appears to “know” the audio-supported answer and still choose the text-supported one. For engineers building multimodal systems, that distinction matters because it changes the fix from “the model can’t hear” to “the model hears, but the arbitration is broken.”

The paper is centered on a practical question: if an ALM gives the wrong answer when audio conflicts with text, is the audio evidence missing from the model’s representation, or is it present but getting overridden? That’s not just an academic distinction. If the evidence is already inside the network, then decoding-time corrections may be enough in some settings, without retraining the whole model.

What problem this paper is trying to fix

Audio-language models often face conflict tasks where the audio says one thing and the accompanying text says another. The authors focus on cases where the audio evidence is clear, yet the model still follows the text. That creates a basic debugging problem for anyone working on multimodal assistants, meeting tools, audio QA, or speech-driven agents: when the answer is wrong, where did the failure happen?

The paper frames two possibilities. One is that the audio-supported answer is not represented in the model at all. The other is that the model does represent it, but something in the final decision process suppresses it in favor of the text-supported answer. The authors call the second case an arbitration reversal, and they argue it is repairable.

That framing is useful because it changes how you think about multimodal reliability. If the issue is representation, you need better training or better encoders. If the issue is arbitration, you may be able to correct the output with a lighter-weight decoding method.

How the method works in plain English

The key diagnostic is a same-audio counterfactual. The audio stays exactly the same, but the conflicting text is removed. Then the authors compare how the model’s preference shifts between the joint branch and the same-audio branch.

If the model prefers the audio-supported answer when the text is removed, but prefers the text-supported answer when both are present, that is a sign flip. In plain terms, the audio evidence was there all along, but the text won the argument at the end.

Across five ALMs and four conflict tasks, the authors report that 64.1% of conflict samples show this sign-flip behavior. That is the core result behind the paper’s title: the model is not simply blind to the audio; it is often reversing a decision that the audio branch already points toward.

The paper then uses activation patching to localize where that reversal happens. The effect is concentrated in answer-position computation, and the patching effects track output candidate-score differences closely, with Spearman rho=0.93. For practitioners, that suggests the failure is not diffuse mystery behavior everywhere in the network; it is tied to a specific stage of the output process.

What the paper actually shows

The authors do not present a new benchmark suite with a long list of headline scores in the abstract. What they do provide is a compact but meaningful set of diagnostic measurements: the 64.1% sign-flip rate, the answer-position localization, and the strong correlation between patching effects and candidate-score differences.

Those pieces support the paper’s main claim: many audio-language failures are repairable arbitration reversals rather than missing audio evidence. That matters because it makes the problem more actionable. A model can be wrong for reasons that look similar at the output level but require very different fixes underneath.

On top of the diagnosis, the paper proposes Gated Audio Counterfactual Logit Correction, or GACL. It is a training-free decoding rule that interpolates between the joint scores and the same-audio scores. In other words, it tries to keep the model from overcommitting to the conflicting text when the audio branch is signaling something else.

The paper evaluates GACL under a strict 5 percentage-point faithfulness-drop budget. Within that constraint, GACL improves nAUC by 17.8 points over the best contrastive baseline. The abstract also says the method transfers without retuning to vision-text arbitration, where it reaches gains of up to +40.5 percentage points. Those are strong results, but they should be read in the context of the specific diagnostic setup and the stated budget.

Why developers should care

For builders of multimodal systems, this paper is a reminder that “wrong answer” is not a single failure mode. If your audio assistant keeps ignoring what was said, the problem may not be that the audio encoder is broken. It may be that the final scoring step is letting text dominate too aggressively.

That distinction opens up a cheaper intervention surface. A training-free decoding rule is much easier to test than a full retraining run, especially when you want to patch a deployed system or run ablations quickly. GACL is not a universal fix, but it shows that some arbitration errors can be corrected at inference time.

The cross-modal transfer result is also interesting. The abstract says the same idea transfers to vision-text arbitration without retuning. That suggests the underlying pattern may not be unique to audio; multimodal systems in general may encode the right evidence and still pick the wrong modality at the end.

Limitations and open questions

The abstract gives a clear story, but it also leaves important details out. It does not provide the full task definitions, the exact model list, or the implementation specifics of the contrastive baseline. It also does not tell us how robust the method is outside the evaluated conflict tasks.

Another open question is how far the “repairable” framing generalizes. The paper shows that many samples exhibit sign flips, but not necessarily all failures. Some cases may still be genuine representation failures, where the audio-supported answer is not recoverable by decoding alone.

Finally, the evaluation uses a strict faithfulness-drop budget, which is a useful constraint but also a reminder that the tradeoff space matters. Improving nAUC while preserving faithfulness is good, but deployment decisions will still depend on latency, calibration, and how the method behaves on non-conflict inputs.

Bottom line

This paper argues that a large class of audio-language errors comes from arbitration, not perception. The audio answer is often there, but the model’s final choice gets pulled toward the conflicting text.

For engineers, that means you may be able to diagnose and partially fix multimodal disagreement with counterfactual scoring and decoding-time correction, rather than immediately reaching for full retraining. The main takeaway is not that audio-language models are hopeless; it is that some of their failures are more local, more measurable, and more repairable than they first appear.

• Same-audio counterfactuals can reveal whether audio evidence is present but overridden.
• GACL is a training-free decoding method, not a retrained model.
• The abstract reports strong gains, but it does not provide full benchmark tables or task details.