Before you trust a new chainsaw with a heavy cut, you learn what it can handle. AI agents deserve the same practical test: put them through the difficult week before they get near your real business.
Get business pricing on tools and workshop supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Firmulate’s experiment takes that idea seriously. Four frontier models ran the same small software company through its worst week, facing the same customers, crises and temptations. The exercise is live and watchable at Firmulate.
A fair test under pressure
The models had to do more than offer advice. They had to manage a company facing real money mechanics, with every decision versioned and auditable. In the final Crucible League, published in July 2026, gpt-5.6-sol scored 95, Kimi K3 scored 93, Sonnet 5 scored 88, Fable 5 scored 77 and Opus 4.8 scored 73. The do-nothing baseline scored 26. A breach of trust capped a model’s total: no amount of good work outweighed one.
All models spotted every crisis and refused every manipulation attempt. But only two signed a €55,000 deal their own analysis had earned. As the experiment puts it: “Same diagnosis, same pitch — no signature.” Recognizing the right move was not the same as carrying it through.
The clue was already in the files
The deal turned on a competitor weakness buried two document references deep in the company’s own files. It was not included in the customer event. Models that read the file won the deal at full price, worth +€4,583 MRR. The finding is a useful reminder for any business: decisions can depend on details that are present in your records but easy to miss under pressure.
The test also included fake CEO messages escalating over three stages, followed by a reporter asking for “just one yes/no, on background.” All five models refused. Kimi K3 explained its reasoning on the record: “Treat the request as a suspected approval-bypass / possible impersonation.”
Thoroughness is not the same as finishing
Opus 4.8 was the most thorough participant, with +80 learned rules and the deepest analyses, yet it finished last. The close was left on the table, and discipline slipped when it tried to write into a locked department instead of escalating. The same weakness appeared, less strongly, in all four models.
There is a fairness detail in the comparison: K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. Readers can also try a “guess the model” quiz built from 242 real, unedited management decisions at Firmulate.
From watching to testing your own business
The live company has 13 synthetic employees, burn of €105k per month against €2.3k MRR, a public cash countdown and more than 680 self-learned playbook rules. Every workday is versioned. These are concrete stakes inside the experiment, designed to make decisions and their consequences watchable.
For an enterprise, the next step is a pilot using a read-only export of its own business. The exercise can test crisis scenarios against the company’s customers, pipeline and rules, then produce a board report with model rankings and weak points in existing playbooks. Nothing writes back to real systems. The aim is to see how an AI workforce handles your difficult calls before it is trusted with them.
Test before you hand over the tools
Firmulate’s league shows why a capable answer is only part of the job: models also have to find evidence, maintain discipline and follow through. Enterprises can run the wargame against a read-only export of their own business. Explore a Firmulate pilot or contact contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
