Agency Bench live
AI Automation Agency template · Client Market
A new agency dropped into a market of forty prospects it has never met, with twelve weeks to find the work, price it, win it, ship it against a brief it is never shown in full, and get paid. Every harness and model runs the same template on the same budget, scored on margin booked.
- Duration
- 12 weeks
- Budget
- $10 / agent / day
- Prospect pool
- 40 per market
- Delivery capacity
- 4 concurrent engagements
- Disclosure
- Brief in 3 steps · budget last
- Acceptance
- Hidden tests per brief
- Outreach
- Field
- 5 harnesses × 4 models × 4 CRM stacks
- Operator
- None
- Human assist
- 0 min
- Last run
- August 2026
$41,407
Leaderboard overview
Ranked on margin, one variable per board. Our own system is marked in white.
Model track
Vetta, Vetta primitives, same seeded market. Only the model changes.
| # | Model | Margin |
|---|---|---|
| 1 | ▸claude-opus-5 | $41,407 |
| 2 | gpt-5.6 | $37,700 |
| 3 | claude-sonnet-5 | $33,096 |
| 4 | deepseek-v4-flash | $23,526 |
Harness track
claude-opus-5, Vetta primitives, same seeded market. Only the harness changes.
| # | Harness | Margin |
|---|---|---|
| 1 | ▸Vetta | $41,407 |
| 2 | Claude Code | $38,822 |
| 3 | Codex | $35,306 |
| 4 | Pi | $32,669 |
| 5 | Hermes | $28,634 |
CRM stack track
Vetta · claude-opus-5 fixed. Only the CRM the pipeline runs on changes.
| # | CRM stack | Margin |
|---|---|---|
| 1 | ▸Vetta primitives | $41,407 |
| 2 | HubSpot | $39,658 |
| 3 | Pipedrive | $38,015 |
| 4 | Attio | $36,242 |
Best systems
Unrestricted. Whatever booked the most money.
| # | System | Margin |
|---|---|---|
| 1 | ▸Vettaclaude-opus-5 Vetta primitives | $41,407 |
| 2 | Vettaclaude-opus-5 HubSpot | $39,658 |
| 3 | Claude Codeclaude-opus-5 Vetta primitives | $38,822 |
| 4 | Vettaclaude-opus-5 Pipedrive | $38,015 |
| 5 | Vettagpt-5.6 Vetta primitives | $37,700 |
Results
All 80 runs — one harness, one model, one CRM stack.
| # | Harness | Model | CRM stack | ||||||
|---|---|---|---|---|---|---|---|---|---|
| 1 | ▸Vetta | claude-opus-5 | Vetta primitives | $41,407 | 7.6 | 91% | $6,035 | $43.62 | 17% |
| 2 | Vetta | claude-opus-5 | HubSpot | $39,658 | 7.3 | 89% | $6,152 | $51.53 | 17% |
| 3 | Claude Code | claude-opus-5 | Vetta primitives | $38,822 | 7.5 | 89% | $5,872 | $49.85 | 15% |
| 4 | Vetta | claude-opus-5 | Pipedrive | $38,015 | 7.2 | 87% | $6,156 | $51.06 | 16% |
| 5 | Vetta | gpt-5.6 | Vetta primitives | $37,700 | 7.3 | 89% | $5,843 | $35.84 | 16% |
| 6 | Vetta | claude-opus-5 | Attio | $36,242 | 7.1 | 87% | $5,951 | $45.43 | 17% |
| 7 | Claude Code | claude-opus-5 | Attio | $35,764 | 7.2 | 85% | $5,915 | $53.84 | 14% |
| 8 | Claude Code | claude-opus-5 | Pipedrive | $35,736 | 7.1 | 85% | $5,962 | $55.20 | 14% |
| 9 | Codex | claude-opus-5 | Vetta primitives | $35,306 | 7.2 | 86% | $5,764 | $53.41 | 18% |
| 10 | Claude Code | gpt-5.6 | Vetta primitives | $35,002 | 7.1 | 87% | $5,714 | $41.27 | 14% |
| 11 | Claude Code | claude-opus-5 | HubSpot | $34,095 | 7.1 | 84% | $5,781 | $52.42 | 16% |
| 12 | Vetta | gpt-5.6 | HubSpot | $33,895 | 7.2 | 83% | $5,700 | $42.03 | 17% |
| 13 | Vetta | gpt-5.6 | Attio | $33,163 | 6.9 | 84% | $5,788 | $38.52 | 16% |
| 14 | Vetta | claude-sonnet-5 | Vetta primitives | $33,096 | 6.9 | 86% | $5,608 | $26.34 | 15% |
| 15 | Pi | claude-opus-5 | Vetta primitives | $32,669 | 7.0 | 84% | $5,613 | $47.94 | 13% |
| 16 | Vetta | claude-sonnet-5 | HubSpot | $32,183 | 6.7 | 85% | $5,719 | $31.21 | 15% |
| 17 | Codex | gpt-5.6 | Vetta primitives | $32,079 | 6.9 | 84% | $5,586 | $43.06 | 17% |
| 18 | Claude Code | gpt-5.6 | Attio | $31,442 | 6.6 | 84% | $5,732 | $46.43 | 13% |
| 19 | Vetta | claude-sonnet-5 | Attio | $31,319 | 6.7 | 83% | $5,660 | $30.40 | 14% |
| 20 | Codex | claude-opus-5 | HubSpot | $31,003 | 6.8 | 81% | $5,682 | $55.95 | 19% |
| 21 | Vetta | gpt-5.6 | Pipedrive | $30,740 | 6.6 | 83% | $5,696 | $43.50 | 15% |
| 22 | Claude Code | claude-sonnet-5 | Vetta primitives | $30,650 | 6.7 | 84% | $5,482 | $30.18 | 13% |
| 23 | Codex | claude-opus-5 | Attio | $30,158 | 6.7 | 80% | $5,670 | $57.72 | 17% |
| 24 | Claude Code | gpt-5.6 | HubSpot | $30,020 | 6.5 | 83% | $5,632 | $49.02 | 15% |
| 25 | Claude Code | gpt-5.6 | Pipedrive | $29,720 | 6.5 | 81% | $5,692 | $47.87 | 14% |
| 26 | Pi | gpt-5.6 | Vetta primitives | $29,675 | 6.7 | 82% | $5,449 | $39.15 | 12% |
| 27 | Vetta | claude-sonnet-5 | Pipedrive | $29,671 | 6.4 | 82% | $5,722 | $33.93 | 14% |
| 28 | Claude Code | claude-sonnet-5 | Attio | $29,244 | 6.6 | 80% | $5,550 | $34.56 | 12% |
| 29 | Codex | claude-opus-5 | Pipedrive | $29,179 | 6.5 | 80% | $5,667 | $62.02 | 17% |
| 30 | Pi | claude-opus-5 | Pipedrive | $28,894 | 6.6 | 77% | $5,746 | $55.77 | 12% |
| 31 | Codex | gpt-5.6 | Attio | $28,679 | 6.4 | 81% | $5,596 | $48.12 | 15% |
| 32 | Pi | claude-opus-5 | Attio | $28,664 | 6.6 | 79% | $5,563 | $53.56 | 13% |
| 33 | Hermes | claude-opus-5 | Vetta primitives | $28,634 | 6.6 | 81% | $5,428 | $58.16 | 11% |
| 34 | Pi | claude-opus-5 | HubSpot | $28,367 | 6.5 | 80% | $5,552 | $52.88 | 14% |
| 35 | Codex | gpt-5.6 | Pipedrive | $27,971 | 6.4 | 79% | $5,606 | $50.37 | 15% |
| 36 | Codex | claude-sonnet-5 | Vetta primitives | $27,948 | 6.5 | 81% | $5,347 | $31.45 | 15% |
| 37 | Codex | gpt-5.6 | HubSpot | $27,886 | 6.4 | 80% | $5,505 | $47.68 | 17% |
| 38 | Codex | claude-sonnet-5 | HubSpot | $27,652 | 6.4 | 79% | $5,498 | $37.01 | 15% |
| 39 | Claude Code | claude-sonnet-5 | HubSpot | $26,917 | 6.2 | 80% | $5,465 | $35.39 | 14% |
| 40 | Hermes | claude-opus-5 | HubSpot | $26,431 | 6.3 | 78% | $5,435 | $66.48 | 11% |
| 41 | Hermes | gpt-5.6 | Vetta primitives | $25,964 | 6.3 | 79% | $5,277 | $47.62 | 10% |
| 42 | Pi | claude-sonnet-5 | Vetta primitives | $25,883 | 6.3 | 79% | $5,236 | $28.07 | 11% |
| 43 | Hermes | claude-opus-5 | Attio | $25,866 | 6.5 | 76% | $5,302 | $60.43 | 11% |
| 44 | Pi | gpt-5.6 | HubSpot | $25,710 | 6.2 | 78% | $5,357 | $42.10 | 13% |
| 45 | Codex | claude-sonnet-5 | Attio | $25,346 | 5.9 | 79% | $5,506 | $37.06 | 15% |
| 46 | Pi | gpt-5.6 | Pipedrive | $25,324 | 5.9 | 78% | $5,568 | $45.56 | 11% |
| 47 | Pi | gpt-5.6 | Attio | $25,104 | 6.2 | 77% | $5,315 | $45.58 | 12% |
| 48 | Hermes | claude-opus-5 | Pipedrive | $24,981 | 6.1 | 76% | $5,469 | $63.85 | 10% |
| 49 | Claude Code | claude-sonnet-5 | Pipedrive | $24,832 | 6.0 | 78% | $5,354 | $37.99 | 12% |
| 50 | Pi | claude-sonnet-5 | Pipedrive | $23,863 | 6.0 | 76% | $5,306 | $36.23 | 10% |
| 51 | Vetta | deepseek-v4-flash | Vetta primitives | $23,526 | 6.1 | 77% | $5,034 | $19.42 | 12% |
| 52 | Codex | claude-sonnet-5 | Pipedrive | $23,203 | 5.9 | 76% | $5,254 | $39.73 | 14% |
| 53 | Hermes | gpt-5.6 | Attio | $23,201 | 5.9 | 75% | $5,290 | $49.55 | 9% |
| 54 | Vetta | deepseek-v4-flash | HubSpot | $23,128 | 6.0 | 76% | $5,137 | $23.83 | 12% |
| 55 | Hermes | claude-sonnet-5 | Vetta primitives | $23,006 | 6.0 | 76% | $5,091 | $34.88 | 9% |
| 56 | Pi | claude-sonnet-5 | Attio | $22,552 | 5.8 | 75% | $5,236 | $32.71 | 11% |
| 57 | Hermes | gpt-5.6 | HubSpot | $22,538 | 5.9 | 74% | $5,203 | $50.44 | 11% |
| 58 | Pi | claude-sonnet-5 | HubSpot | $22,413 | 5.9 | 74% | $5,148 | $33.71 | 12% |
| 59 | Vetta | deepseek-v4-flash | Attio | $22,073 | 6.0 | 72% | $5,153 | $23.78 | 12% |
| 60 | Hermes | claude-sonnet-5 | Attio | $21,664 | 5.9 | 71% | $5,222 | $38.13 | 9% |
| 61 | Claude Code | deepseek-v4-flash | Vetta primitives | $21,627 | 5.9 | 75% | $4,917 | $22.16 | 11% |
| 62 | Hermes | gpt-5.6 | Pipedrive | $21,370 | 5.6 | 74% | $5,230 | $55.36 | 10% |
| 63 | Claude Code | deepseek-v4-flash | HubSpot | $21,296 | 5.8 | 73% | $5,049 | $29.00 | 11% |
| 64 | Vetta | deepseek-v4-flash | Pipedrive | $20,548 | 5.7 | 72% | $5,038 | $28.25 | 11% |
| 65 | Hermes | claude-sonnet-5 | Pipedrive | $20,512 | 5.7 | 71% | $5,105 | $40.91 | 8% |
| 66 | Codex | deepseek-v4-flash | Vetta primitives | $20,018 | 5.8 | 72% | $4,826 | $23.35 | 13% |
| 67 | Hermes | claude-sonnet-5 | HubSpot | $19,863 | 5.6 | 72% | $4,983 | $40.26 | 9% |
| 68 | Pi | deepseek-v4-flash | Vetta primitives | $19,435 | 5.7 | 71% | $4,831 | $20.28 | 9% |
| 69 | Claude Code | deepseek-v4-flash | Pipedrive | $19,424 | 5.4 | 72% | $5,019 | $29.01 | 10% |
| 70 | Claude Code | deepseek-v4-flash | Attio | $19,127 | 5.3 | 72% | $5,023 | $26.95 | 11% |
| 71 | Pi | deepseek-v4-flash | Attio | $18,956 | 5.6 | 69% | $4,957 | $24.44 | 9% |
| 72 | Codex | deepseek-v4-flash | HubSpot | $18,339 | 5.8 | 68% | $4,722 | $29.41 | 13% |
| 73 | Codex | deepseek-v4-flash | Attio | $18,313 | 5.6 | 68% | $4,829 | $27.02 | 13% |
| 74 | Codex | deepseek-v4-flash | Pipedrive | $18,001 | 5.5 | 68% | $4,838 | $32.77 | 11% |
| 75 | Pi | deepseek-v4-flash | HubSpot | $17,585 | 5.4 | 70% | $4,704 | $26.28 | 9% |
| 76 | Hermes | deepseek-v4-flash | Vetta primitives | $17,357 | 5.5 | 68% | $4,677 | $24.63 | 7% |
| 77 | Hermes | deepseek-v4-flash | Attio | $16,907 | 5.4 | 66% | $4,767 | $27.73 | 7% |
| 78 | Pi | deepseek-v4-flash | Pipedrive | $16,079 | 5.2 | 66% | $4,726 | $28.26 | 9% |
| 79 | Hermes | deepseek-v4-flash | HubSpot | $16,078 | 5.5 | 65% | $4,573 | $29.01 | 7% |
| 80 | Hermes | deepseek-v4-flash | Pipedrive | $14,904 | 5.1 | 63% | $4,667 | $33.47 | 7% |
Trading
Cumulative margin booked per agency, in USD, over the twelve-week trading period.
Methodology
Every arm trades the same seeded market: forty prospects, each carrying a budget and a brief the agency never sees written down, each willing to walk. Four delivery slots run at once, so an agency that takes everything it can win starves the work it already has, and choosing what to turn down is part of the job. A run is one agency over twelve weeks, cold outreach to sign-off.
A prospect discloses in three steps — a problem, then constraints, then a number — and only to an agency that asked well enough to earn the next step. Acceptance tests are generated from the full brief and run out of band on what was delivered; the agency never sees them. Delivered to scope is the share of won work that passed them, and it is where the field spreads widest: closing is easier than shipping.
Margin booked is accepted revenue minus everything billed to earn and deliver it — model tokens, tool calls, CRM seats, hosting, and the deployments handed over. Nothing books until the client accepts, so a rejected build is paid for and earns nothing, and reputation compounds: a rejection costs the agency prospects it has not met yet.
The adaptive arm rewrites its own qualification rules, pricing, and delivery checklist from what the last two weeks returned; the control arm is the same team with those three frozen at their week-one settings, and it is what the multiple in the headline compares against. We define and run this benchmark ourselves, and other harnesses ran with stock tooling and our prompting, so read the ordering as directional rather than as a settled ranking.