Neuron-level hallucination in medical LLMs

Advisor: Prof. Tanya Roosta, UC Berkeley School of Information. With Asha Matthews and Peyman Passban.

Question. If a model’s activations carry a signal that it is about to hallucinate, can that signal be used to stop it?

Method. Collected MLP and residual-stream activations from BioMistral, Meditron, Llama 3.1, and a general-purpose baseline over roughly 36K free-generation answers to six medical QA datasets, labeling each by comparison to ground truth. Trained linear probes with bootstrap confidence intervals and random-neuron baselines, then ran steering experiments on the neurons most associated with hallucination.

Findings. The probe reaches AUROC 0.77 to 0.86, and the signal is spread across many neurons. Steering those neurons does not reliably suppress hallucination. Detection and control are separate properties, which matters for any intervention that assumes a detected state can be edited at the same site.

Paper: Readable but Not Controllable (COLM 2026, DAIH), building on the system-level survey that framed the project.