Most AI hiring platform contracts I see signed this year rest on a slide a Harvard team just invalidated.
The slide says "AUC 0.88 across 400 similar employers." Those are the vendor's words. The next slide says "cross-industry benchmark." The contract is seven figures. The CPO signs.
The CPO in Dublin
A chief people officer at a B2B SaaS firm, Dublin, late October. She sends me the vendor deck before the second meeting. She wants to know what to push on.
I give her one question. "Did they offer the transfer test on your 2024 and 2025 retention data? Train their model on yours, apply it to your three biggest competitors' applicants, and show you the resulting AUC?"
She reads it twice. "They would never agree to that."
That is the moment. I watch her understand what she almost bought.
What Harvard just measured
Alidaee, Khanna and Koning are at Harvard Business School. Last week they put a paper on arXiv about a Mexican AI hiring platform used by three firms: consumer goods, a coffeehouse chain, a softline retail chain. 180,361 applications. A 42-item behavioural inventory delivered over WhatsApp. 90-day retention as the outcome.
Each firm's own model is sharp. Own-firm AUC runs 0.855 to 0.914.
Then they ran the test the vendor would never run on itself. They applied Firm B's model to Firm C's applicants, and the other five pairs. Transfer AUC 0.473 to 0.514. Coin flip, more or less.
The cross-firm rank correlation of applicant scores sits 17 standard deviations below the null, at p = 0.002.
Firm indicators add 21.6 times more pseudo-R² to the pooled model than the seven composite traits do.
In plain English, the model has learned the firm's revealed preferences. That is all it has learned. The quality score the vendor sells is a routing decision wearing a measurement's clothes.
Across the full sample of 180,361 applications, 46.7% would score 20 percentage points higher at the firm whose model loves them most. 23.3% would score 40 points higher. When the firms disagree, they disagree about direction. On emotional control, rule adherence, ethics and safety, one firm's high-retention trait is another firm's noise.
The ask that reveals the product
Vaccaro's paper on Monday said the pooled gain from human-plus-AI collaboration runs net negative at the collaboration level. Alidaee says the pooled gain from "a model trained on employers like yours" is near zero at the procurement level. Two findings, same shape. The hiring-AI contract gets harder to defend every week.
The ask I give executives is small. Before signing, request the cross-firm transfer AUC on the vendor's actual customer data. If the number lands below 0.7, the thing you are buying is a mirror that costs seven figures and reflects your own hiring history back at you.
She sent me the email draft that evening. The vendor is still thinking about it.