Modality arbitration in vision-language models

Advisor: Dr. Vasha DuTell, UC Berkeley School of Information.

Question. A vision-language model has two sources of evidence for any answer: the image and its language prior. When they disagree, which one wins, and where in the network is that decided?

Findings so far. On CLIP-LLaVA models, trust in the image tracks how well the model can discern the queried property from it. Degrading the image shifts the answer toward the prior; sharpening it shifts the answer back. Activation patching and logit lens show the decision forming in the middle layers rather than at the input.

Next. Whether the behavior follows one reliability-weighted function across models, whether the decision can be read from the residual stream before an answer is generated, and whether steering the heads involved moves trust in either direction without side effects elsewhere.

Papers: Vision-Language Models Trust What They Can Discern (ECCV 2026, eXCV); Believing Is Seeing (under review, ICLR 2027).