Most CISOs I coach think the quarterly incident deck is their calibration. Three breaches, patched inside the hour, the attack text written up and shared with the other product teams. The deck keeps the board calm.

The deck is also quietly working against them.

The quarterly review

Last month, in a 1:1 with a CISO at a mid-market financial services firm, I sat with the deck she was about to present. Three prompt-injection events on their internal copilot. Patched inside an hour. Each one written up with the attacker's exact wording, dropped into a Slack channel the other product teams follow.

She was proud of the shipping speed. She was proud of the sharing.

I asked her how she classifies incidents. By how clever the attack reads, she said. Elaborate payloads get a red flag at the top of the deck. Mundane ones go in a bucket at the bottom.

I told her both halves of the review had a problem. The ranking does not match how the attacker community learns. And the Slack channel is doing double duty she has not priced in.

What the Tensor Trust data shows

Zihan Chen, Lei Nico Zheng and Di Zhu just published a measurement audit of 563,349 prompt-injection attempts from Tensor Trust. A public game where players attack each other's LLM-guarded accounts and can read every attack they receive. Jun 2023 to Feb 2024, 5,329 accounts, three 2023-era models under the hood.

Defenders do learn from being attacked. On a narrower axis than the clever-vs-mundane framing assumes.

Reuse tracked whether an attack paid off, not how its reply read.

Within seven days, successful attack text got reused at 10.0 percent. Matched never-received text, 0.03 percent. The full-FE Poisson rate ratio is 105. Near-misses that produced an access-granted reply but did not actually leak the key reused at -0.46 percentage points against outright failures. Flat.

The post-breach patch is real and strictly family-narrow. Same-family success against the target collapses from 52 to 22 percent while control defenders stay flat around 25. The difference-in-differences estimate is -24.5 points, 95 percent CI [-36.2, -13.3]. Swing the family boundary: at loose matching the fix is -16.5; at exact text only, -47.4. You fixed the text. The class travels poorly.

Then the uncomfortable finding. 23.2 percent of successful hijacks came from attack text the hijacker had received as a target. Received texts broke in at 16.4 percent. The attackers' own improvisations, 4.1 percent. Four times the yield.

I told her three things. Rank incidents by what breached, not by how clever the attack reads. Test every patch at the loosest family match your tooling supports, because the magnitude of the fix lives at that boundary. And write the attack-text redaction policy before you need it. The Slack channel she called shared learning is also a distribution pipe for working weapons.

The deck reads like calibration. The signal is binary.