Vision-Language Models Trust What They Can Discern

With Dr. Vasha DuTell.

A vision-language model that is shown an image and asked a question has two sources of evidence: what it can see, and what its language prior expects. This paper asks what happens when the two conflict. Using a semantic image pair protocol, we construct cases where the image and the text disagree on a property such as color, count, or material, then measure which side wins and why.

The short answer is that these models are neither blindly text-driven nor blindly image-driven. They follow whichever input they can discern more reliably. Degrading the image shifts trust to the prior; sharpening it shifts trust back. Activation patching and logit lens show where in the network the arbitration happens and how it builds up across layers.

A full-length extension, Believing Is Seeing, covers three CLIP-LLaVA models, four conflict sources, and an evaluation on real photographs, and is under review at ICLR 2027.

Cite as: Vankadaru, V., & DuTell, V. (2026). Vision-Language Models Trust What They Can Discern. eXCV Workshop, ECCV 2026.