8.1 KiB
| created | updated | sources | tags | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2026-07-02 | 2026-07-02 |
|
|
Model Coding Benchmarks — Price-Performance Comparison
TL;DR: One-shot coding tests with identical prompts reveal a 39× cost spread between frontier and budget models — with quality grades only 2 steps apart (A+ to B+). The quality-cost curve is not linear; it's logarithmic. Beyond a certain spend threshold, additional dollars buy diminishing quality gains.
Source
atomic.chat X-Post (01.07.2026) — Raw file
Shared by Netbits ⚡️ Stachelbanane (@NetLightning) in OME-Gruppe Topic "News & Infos" on 2026-07-02. Viral: 1.73M views, 4K likes, 2K bookmarks.
The Test
Setup: 4 models, same prompt, one-shot (no iteration), build three self-contained HTML5 canvas scenes with real physics:
- Train derailing off a broken bridge into water
- Two cars jumping off ramps, colliding mid-air over a canyon
- Monster truck crushing a row of parked cars
Why this matters: HTML5 Canvas physics demos are a microcosm of coding-agent tasks — they require spatial reasoning, physics simulation, animation loops, and self-contained output. One-shot means no retry, no refinement — raw generation quality per dollar.
Results
| Model | Vendor | Tokens | Cost | Grade | Cost vs Fable |
|---|---|---|---|---|---|
| Fable 5 | Anthropic | 62,158 | $3.12 | A+ | 1× |
| GPT 5.5 | OpenAI | 37,753 | $1.14 | A− | 2.7× cheaper |
| Opus 4.8 | Anthropic | 22,280 | $0.56 | — | 5.6× cheaper |
| GLM 5.2 | Z.ai | 36,246 | $0.08 | B+ | 39× cheaper |
The Quality-Cost Curve
Quality
A+ | ● Fable 5 ($3.12)
|
A | ● GPT 5.5 ($1.14)
|
B+ | ● GLM 5.2 ($0.08) ← 39× cheaper, only 2 grades down
|
B | ● Opus 4.8 ($0.56) ← (grade unspecified, lowest tokens)
|
+------+------+------+------+------→ Cost
$0 $0.50 $1.00 $2.00 $3.00
Key insight: The curve is logarithmic, not linear. Going from B+ to A+ costs 39× more. Going from A− to A+ costs 2.7× more. The marginal quality gain per dollar decreases dramatically at the top end.
Cost per 1K Tokens
| Model | $/1K tokens |
|---|---|
| Fable 5 | $0.050 |
| GPT 5.5 | $0.030 |
| Opus 4.8 | $0.025 |
| GLM 5.2 | $0.002 |
GLM 5.2's $0.002/1K tokens is the cheapest frontier-level coding generation measured to date.
Community Takeaways
| User | Insight |
|---|---|
| @Krysoph | GLM 5.2 could improve with iterations + vision model assistance |
| @suzzvsworld | "GLM at $0.08 and still competitive is the real story. Chinese models are making the price conversation impossible to ignore" |
| @rohanpaul_ai | Quality scales with spend; wants test with strict max-token limit |
| @debugging_yami | GPT 5.5 is best value — nearly par with Fable at 1/3 cost |
The Three Camps
- "Best quality regardless of cost" — Fable 5 wins, but at 39× the price of GLM 5.2
- "Best value" — GPT 5.5 at $1.14, 2.7× cheaper than Fable with near-par quality
- "Cheapest competitive" — GLM 5.2 at $0.08, 39× cheaper, still produces B+ output
Connection to Existing Wiki Knowledge
Validates: chinese-model-cost-routing.md — The 87% Cost-Cut Playbook
DeRonin's 30-day field report showed 5-12× cost reduction per task category. This benchmark provides an even more extreme data point: 39× cost spread on a single coding task. The pattern holds — Chinese models (GLM 5.2 specifically) deliver near-frontier quality at a fraction of the cost.
| Source | Cost Spread | Task |
|---|---|---|
| DeRonin (agent loops) | 5× | GLM 5.2 vs Sonnet 4.7 |
| DeRonin (reasoning) | 11× | Kimi K2.7 vs Opus 4.8 |
| atomic.chat (coding) | 39× | GLM 5.2 vs Fable 5 |
The 39× spread is the largest measured cost asymmetry in our wiki — an order of magnitude beyond DeRonin's per-task ratios.
Validates: glm-5.2-zai-coding-model.md — GLM 5.2 Deep-Dive
This benchmark adds a fifth independent validation of GLM 5.2's competitive positioning:
- IAmFabian's Coding-Tests (14.06.2026) — Flappy Bird + Newton-Pendel
- Arnie's 4-Test-Review (22.06.2026) — gemischte Ergebnisse, Frontier-Niveau
- Semgrep IDOR-Benchmark (29.06.2026) — Security auf Opus 4.8-Niveau
- Artificial Analysis Intelligence Index v4.1 (01.07.2026) — #1 Open-Weights worldwide
- atomic.chat One-Shot Coding Benchmark (01.07.2026) — B+ at 39× lower cost than Fable 5
Enriches: real-world-coding-showdown.md — Methodology
Fahd Mirza's Hermes Agent showdown used head-to-head methodology (same agent, same task, different models). atomic.chat's benchmark is a simpler variant: same prompt, no agent framework, raw one-shot output. Both complement standard benchmarks (SWE-bench, HumanEval) by testing real generation quality.
| Methodology | Source | Scope |
|---|---|---|
| Standard benchmark | SWE-bench Pro, HumanEval | Narrow, reproducible |
| Agent-framework showdown | Fahd Mirza (Hermes) | Multi-file, tool-use, real app |
| One-shot raw generation | atomic.chat | Single prompt, self-contained output |
Connects: ai-investment-bubble.md — Cost Pressure Narrative
The 39× cost spread is ammunition for the "AI is overbuilt" thesis: if a B+ model costs $0.08 and an A+ model costs $3.12, the economic case for premium models erodes when B+ is "good enough" for most use cases. This mirrors Meta selling excess compute capacity and Oracle warning of datacenter delays — the market is pricing in overinvestment in premium AI.
Implications for Our Routing Architecture
See ../../architecture/model-routing.md.
| Task Tier | Recommended Model | Rationale |
|---|---|---|
| Tier 4 (Frontier quality) | Fable 5 or Opus 4.8 | When A+ is non-negotiable |
| Tier 3 (High quality) | GPT 5.5 | Best value — A− at 2.7× cheaper |
| Tier 2 (Standard) | GLM 5.2 | B+ at 39× cheaper — "good enough" for most |
| Tier 0-1 (Bulk) | GLM 5.2 or MiMo | Cheapest viable option |
New insight: The atomic.chat benchmark suggests GPT 5.5 may be the sweet spot for quality-sensitive tasks where Fable 5's premium is unjustified. Our current routing uses GLM 5.2 as primary — this benchmark validates that for most tasks (Tier 0-2), but suggests GPT 5.5 as a Tier 3 upgrade path.
Methodology Limitations
- One-shot only — No iteration, no refinement. Real coding workflows involve multiple rounds. @Krysoph notes GLM 5.2 could improve with iterations.
- No strict token limit — @rohanpaul_ai wants to see results with a max-token constraint. Without one, Fable 5's 62K tokens may reflect verbosity, not capability.
- Subjective grading — A+/A−/B+ grades are atomic.chat's assessment, not a standardized rubric.
- Single task type — HTML5 Canvas physics is one domain. Results may not generalize to backend logic, data analysis, or agentic tool-use.
- Opus 4.8 grade missing — The post doesn't specify Opus 4.8's quality grade, only its cost/token count.
Cross-References
- chinese-model-cost-routing.md — DeRonin's 87% cost-cut playbook (5-12× per task)
- glm-5.2-zai-coding-model.md — GLM 5.2 deep-dive (5th validation)
- real-world-coding-showdown.md — Fahd Mirza's agent-framework methodology
- ai-investment-bubble.md — Cost pressure narrative (Meta, Oracle, Karp)
- ../../tools/anthropic-claude.md — Fable 5 + Opus 4.8
- ../../tools/openai-gpt.md — GPT 5.5
- ../../architecture/model-routing.md — 5-tier routing architecture
- ai-intelligence-commoditization-thesis.md — TheProphet's "Factory for Gods"