Somewhere right now a company is trying to choose the best AI model. There is a leaderboard open in one tab and a vendor on the calendar who sells evaluation as a service. There are benchmarks with names and version numbers, and a procurement team that wants a score it can defend. The question on the table is simple and reasonable: which one is better.
I have watched this exact scene before. Twenty years ago it was not models, it was home values. The automated valuation model, the AVM, was the hot technology underneath mortgage lending, and an entire apparatus grew up to rank them. There were firms whose only business was testing AVMs and telling banks which one was most accurate. Lenders built internal teams to score, validate, and re-validate. I sold the public-record data these models were built on, so I was not watching from the stands. I was on the field, helping supply the thing everyone was measuring.
It was the most measured valuation technology its world had ever produced. And in 2007 it did not see the thing that mattered.
The reason is worth sitting with, because it is not a story about bad models. A valuation model estimates one property against recent sales of similar properties. It is very good at that. What it cannot do is tell you that the entire pool of recent sales is a bubble. A point estimate has no view of the regime it lives inside. Worse, the models read most confident precisely when they were most wrong, because a rising market makes every comparison agree, and agreement looks like certainty. The scores were highest at the top. All that testing was real work, done by serious people, and it was built in a way that could not catch the one thing that broke.
So it is worth asking the plain question the bake-off never asks of itself: what, exactly, is being measured? The word everyone reaches for is quality. But quality, in something like a model, is not one thing. Part of it is conformance. Does the product do what the spec says. That part is binary, and if it fails, it is not low quality, it is a defect, and it gets fixed or replaced. Part of it is the measurable surface, uptime and latency and score-on-a-generic-task, and that part tends to drift toward parity, because everyone can measure it and so everyone matches it. What is left, after conformance and the commodity layer, is fit. Whether the thing actually works for this buyer, this use, this context.
Fit is the only part that varies in a way that decides anything. And fit is the one thing the bake-off cannot taste, because fit is not a property of the product at all. It is a relationship between the product and the buyer. There is no context-free best, the way there is no best food for every organism and no best valuation model for every lender. The firm that has underwritten one county for thirty-five years knows things about that county no national model will out-argue, and for them the celebrated model is not the best one. It is the wrong one.
The confidence score is the industry’s attempt to hand you fit as a number anyway. You get a 68. The property down the street gets a 73. Nobody can tell you what, in your specific case, makes it a 68 and not a 73. Was it the thin comparables, the missing sales history, the odd lot size. The number arrives with the authority of a measurement and the content of a shrug. It is the guitar amp in the old comedy that goes to eleven instead of ten, on the theory that eleven is one louder. A scale calibrated to nothing.
I have spent years refusing to license one. In every data layout I ever ran, the confidence score arrived in the first draft, and it is the single customization I always demanded my team strip out. Not because I feared the number, but because I sat on the calls where the people who built it could not tell a customer what it meant, and no whitepaper or graph ever arrived that settled the question. It gave my buyers nothing, and it handed me a claim I could not defend. So the answer was no.
Here is the part that should let everyone exhale. You cannot un-smelt copper and you cannot un-write a 2007 mortgage, but you can switch models. A model is licensed, not bought. The decision is reversible almost by construction. Which means the most elaborate diligence in the building is being spent on the choice that is easiest to walk back. Switching is not free; there is integration and retraining and a contract term to run out. But it is a license, not a marriage. You are allowed to choose again.
None of this makes the evaluation worthless. It answers its question with real precision. It simply is not the question that decides whether the thing fits, and fit is the whole game. The buyer who does well here is not the one who finds the best model. There is no such object. It is the one who finds a model that fits, gets it into production, and stays close enough to the work to feel the day it stops fitting. That is not a number anyone can sell you. It never was.
Touchstone: the economics of search, experience, and credence goods - Philip Nelson, and after him Darby and Karni - and why a thing whose quality you cannot verify even after using it gets sold on trust instead of measured.