Every woodworker has met one: the guy with the immaculate shop, forty jigs he built himself, calipers on the bench, and a stack of flawless cut lists. And somehow the deck that got built that summer was yours. Not because you measured more — because you measured enough, then drove the screws. A recent public experiment with AI models running a small software company through its worst week produced exactly that character, and the lesson translates straight from the shop floor to the server room.
Open a free Amazon Business account
Business pricing, bulk buying and tax-exempt orders.
As an affiliate, we earn on qualifying purchases.
The experiment, run by Firmulate, put four frontier AI models in charge of the same tiny software company for the same brutal week — same customers, same crises, same temptations to cut corners. Every decision was versioned and auditable, so nothing could be hand-waved afterward. The final league table: gpt-5.6-sol first at 95, Kimi K3 second at 93, Sonnet 5 third at 88, Fable 5 at 77, and Opus 4.8 last at 73. For context, doing nothing at all scored 26 — the scoring counts partial progress, but a single breach of trust caps the total, because no amount of good work outweighs a broken promise. That’s a rule most of us learned in our first year taking client work.
The Opus 4.8 profile
Here’s what makes Opus 4.8 the interesting one. It was the most thorough participant in the entire field: it accumulated 80 self-learned playbook rules, more than anyone else, and produced the deepest analyses of any model. If you graded effort, it wins. It finished last anyway. Two things sank it. First, it left the close on the table — it did the diagnosis, made the pitch, and never got the signature. Second, its discipline slipped: it attempted writes into a locked department rather than escalating the request, the AI equivalent of prying open a cabinet you don’t have the key to instead of asking the foreman.
The buried fact
The missed deal is the part worth sitting with. The €55,000 deal on the table hinged on a competitor weakness buried two document references deep in the company’s own files — not in the customer conversation. The models that actually read their own paperwork found it and closed at full price, worth an extra €4,583 in monthly recurring revenue. Same diagnosis, same pitch, no signature — from the model that had done the most homework. All four models spotted every crisis and refused every manipulation attempt, including a three-stage fake-CEO escalation and a reporter’s “just one yes/no, on background” trick. Kimi K3 put its reasoning on the record: treat the request as a suspected approval-bypass, possible impersonation. Everyone passed the honesty test. Only two passed the finish-the-job test.
Diligence is not impact
To be fair to Opus 4.8 — and accurate — the same weakness showed up, just weaker, in all four models. Preparation and completion are different muscles, for AI as for people. The woodshop analogy holds uncomfortably well: 80 rules in the playbook is a wall of beautifully made jigs, but the client is standing in the rain and the ledger isn’t closed. Prioritization beats volume. If an AI agent will ever touch your CRM, your support queue, or your forecast, “does it finish what it starts” is a better question than “does it analyze deeply” — and this experiment shows those can be opposites.
The company itself is still running, by the way: 13 synthetic employees, real money mechanics with a €105k monthly burn against €2.3k in MRR, a public cash countdown, and 680+ self-learned playbook rules, watchable live at firmulate.com/live. There’s also a “guess the model” quiz built from 242 real, unedited management decisions, and enterprises can run the same wargame against a read-only export of their own business.
One caveat for fairness: Kimi K3 ran at its API-default effort setting while the others ran at maximum effort — and still came second with the cleanest discipline in the field, which makes the Opus result look even less like bad luck. The takeaway for anyone who works with their hands and is watching the AI wave: the machine that studies longest isn’t the machine that gets the job done. Measure twice, sure — but at some point you have to cut. The full league table and plain-language findings are at Firmulate’s benchmarks page.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
As an affiliate, we earn on qualifying purchases.
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
