A few weeks ago, Google launched Gemini 4 Argon with huge benchmark numbers. I said I'd believe it when I saw it.
Now people are using Gemini on real work, and it's not looking good. Theo, a developer who tests every new model, said it was factually wrong about more than half of its claims in his coding tests, and called it "a straight up liability." Some of Google's own employees have doubts too, according to Bloomberg.
Most of those complaints are about coding. So you might think: I don't write code, so who cares?
You should care, because most AI agents are coding agents under the hood, even when they don't look like it. When an AI assistant sorts your inbox, builds a spreadsheet or pulls together a report, it's usually writing and running little bits of code to get it done. You never see it. But if a model can't code reliably, it can't do that work reliably either.
And this isn't just about Google. Every launch comes with charts showing the new model beating everyone. The labs choose which tests to show, and models get tuned to do well on the popular ones. The demo is the best run out of many. Meta did the same thing last week, and the model is extremely weak in real-world use.
None of that tells you how a model will do on your work. So here's my suggestion: ignore launch day. Wait for the people who use these tools on real work every day to weigh in, and then test it yourself:
- Pick three real tasks you did last week, where you know what a good answer looks like.
- Give the exact same prompt to the new model and to the one you use now.
- Put the answers side by side. Check every detail. If the new model clearly looks better, keep exploring with it.
That's ten minutes, and you'll know more than any benchmark can tell you.
Don't want to do it by hand? This week I gave Pro members a skill that teaches your Claude or ChatGPT agent to run this test for you. Give it three real tasks and it runs both models, grades the answers and tells you which one to use. It's in Pro.
— Matt
What should I write about next? Vote or suggest a topic →