Every weekly scorecard I have read in the last year says something like "the AI meets our quality bar at 68 percent." Leaders hear a claim about the model.
The number was produced by a second LLM grading the first one's output. That makes it a claim about the judge.
Nobody has asked me how far those two claims can drift apart.
The risk head in Rotterdam
Two weeks ago I sat with the head of model risk at a Benelux retail bank. She was prepping a weekly scorecard for Monday's risk committee. The headline tile said acceptance 68 percent in green.
I asked a simple thing. If you swapped the judge configuration for the one your other shortlisted vendor ships, same outputs this week, what number comes back.
She said within a few points. Five, maybe. She was about to walk 68 into Monday's committee. The room would hear it as two correct answers out of every three.
I told her the honest range sitting in an MIT paper from last week. She stopped scrolling.
What Lyu and Thompson measured
Harry Lyu and Neil Thompson at MIT FutureTech released Right Order, Wrong Scale on October 1. They built ONET-BENCH from 45,796 real worker ratings on occupational tasks in the US Department of Labor's ONET database. They evaluated 33 pre-existing judge configurations across six model families on 4,501 held-out test ratings.
Ordering mostly worked. Twenty-five configurations hit tie-aware pair accuracy of 0.60 or better. A plain TF-IDF ridge baseline came within 0.015 of the strongest judge. The judges can tell you which response beats which other response.
The absolute number is where the floor falls out.
Judge acceptance-share estimates range from 3.0 to 97.9 percent on the same responses. Occupation-matched workers accept 61.1 percent. Median judge estimate: 43.9 percent.
Workers say 61. Judges say anywhere from 3 to 98 depending on which one you bought. Cross-validated calibration takes the mean bias out and still leaves 91.5 percent of individual worker-rating variance unexplained. In one fine-tuned lineage, moving to a protocol that improved response ordering made task-level and occupation-level agreement with worker means worse. The reversal replicated out of sample.
Ranking and absolute measurement are different quantities. They can trade off.
The practical move at the next AI-eval review is small. Ask for two numbers on the production judge. Pair-ranking accuracy against a human-rated sample. Absolute acceptance-share estimate against the same sample's human share. If the team only has the first, they have validated a sort. The measurement is still unvalidated. And before any "percent acceptable" tile lands in a board slide, ask what the number does when you swap the judge configuration on the same outputs. Until proven otherwise, assume the swap can move the tile by nearly 100 points.