The dashboard said the AI tutor was working. Satisfaction 4.3 out of 5. Trust rating 4.5. Would-use-again 92%. Learning hours per employee up 31%. The procurement VP wanted to roll it out company-wide next quarter.
Then the compliance director ran a ten-question quiz on the actual regulations the team had been studying. On the facts the AI got right, the team scored 85%. On three facts the AI had quietly drifted on, the team scored 11%. Nobody flagged any of them.
The call in Zurich
A head of L&D at a professional services firm in Zurich walked me through those two numbers last week. She said the thing I now hear once a month. "The team loved it. They still love it. They just learned the wrong thing and I had no way of knowing."
She had assumed the trust score would move. It did not move. The team was absorbing the drift and rating the assistant fine on the way out of the room.
What the research says
Deepthika S at Amrita, with B. Sankar at IISc Bangalore, published a paper on October 22 that puts a number on what she saw. They ran 32 participants through five science topics taught by five GPT-5-mini assistants. Each assistant's briefing had 0, 25, 50, 75 or 100 percent of its facts replaced with plausible falsehoods. All of them spoke with full confidence.
When the assistant stated a planted falsehood, participants absorbed it on 65 percent of items. The rate was flat across reliability levels. An assistant that was 100 percent wrong poisoned the user at the same rate as one that was 25 percent wrong.
Their ability to tell their right answers from their wrong ones via their own confidence collapsed from AUROC 0.70 to 0.53. That is chance.
The post-task trust rating moved from 5.1 to 4.5 on a seven-point scale. The per-step slope was not distinguishable from zero (p=.14).
"Trustworthiness is a property of the human-AI pair."
The sharpest finding sits in the method. Participants returned to the chat more often when the assistant was worse. They were checking. And the checking increased absorption.
If your AI learning rollout is monitored by satisfaction, trust rating, or willingness-to-rely-again, you have no monitoring. The only test is an outcome test against ground truth. Did they learn the right thing. And if your culture tells people to double-check with the LLM, you are feeding the mechanism. The check needs an external source. Measure the pair.