The benchmark said the model was better. The model was, in fact, better — at the benchmark. What it was like to use for three weeks is a different datum, and there is no chart for it, because the chartmakers have not figured out how to sell a number that goes sideways.^[1]

Our methodology was unscientific and we are printing it anyway: four writers used four frontier models as daily tools for three weeks, doing real work — summarizing FOIA responses, untangling CSS, writing condolence emails, explaining a pension document. We logged every time a model was brilliant, every time it lied, and every time it did the specific thing where it apologizes and then does it again.

The apology loop

Every model failed. They failed differently, and the differences are the actual product. One failed like a confident intern. One failed like a lawyer hedging a memo. One failed like a search engine that had read a self-help book. These are vibes, and vibes are not a rounding error — they are the interface.^[2]

The benchmark measures the model. The vibe measures the relationship.— RAW SIGNAL, ISSUE № 140

The industry hates this conversation because relationships cannot be leaderboarded. A relationship has history, habit, forgiveness, the accumulated weight of the time the thing saved your deadline and the two times it invented a court case. You do not pick a colleague by MMLU score.

What we recommend

Nothing. That is the finding. Any recommendation would expire before this article finished rendering, and the honest position is that the difference between the top models is now smaller than the difference between the same model on a good day and a bad one. Benchmark the vibes yourself. It takes three weeks and no one will fund you.^[3]

Full logs are on our forum, where a thread about methodology has already devolved into an argument about pensions. It is, as always, the best page on the internet.