154 lines
8.1 KiB
Markdown
154 lines
8.1 KiB
Markdown
|
---
|
|||
|
|
created: 2026-07-02
|
|||
|
|
updated: 2026-07-02
|
|||
|
|
sources: [xpost/2026-07-02_atomicchat-coding-benchmark-fable5-gpt55-opus48-glm52.md]
|
|||
|
|
tags: [concept, llm, coding-benchmark, price-performance, one-shot, fable5, gpt-5.5, opus-4.8, glm-5.2, cost-routing, quality-vs-cost, html5-canvas, physics-simulation]
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Model Coding Benchmarks — Price-Performance Comparison
|
|||
|
|
|
|||
|
|
> **TL;DR:** One-shot coding tests with identical prompts reveal a 39× cost spread between frontier and budget models — with quality grades only 2 steps apart (A+ to B+). The quality-cost curve is not linear; it's logarithmic. Beyond a certain spend threshold, additional dollars buy diminishing quality gains.
|
|||
|
|
|
|||
|
|
## Source
|
|||
|
|
|
|||
|
|
**atomic.chat X-Post (01.07.2026)** — [Raw file](../../../raw/xpost/2026-07-02_atomicchat-coding-benchmark-fable5-gpt55-opus48-glm52.md)
|
|||
|
|
|
|||
|
|
Shared by Netbits ⚡️ Stachelbanane (@NetLightning) in OME-Gruppe Topic "News & Infos" on 2026-07-02. Viral: 1.73M views, 4K likes, 2K bookmarks.
|
|||
|
|
|
|||
|
|
## The Test
|
|||
|
|
|
|||
|
|
**Setup:** 4 models, same prompt, one-shot (no iteration), build three self-contained HTML5 canvas scenes with real physics:
|
|||
|
|
|
|||
|
|
1. Train derailing off a broken bridge into water
|
|||
|
|
2. Two cars jumping off ramps, colliding mid-air over a canyon
|
|||
|
|
3. Monster truck crushing a row of parked cars
|
|||
|
|
|
|||
|
|
**Why this matters:** HTML5 Canvas physics demos are a microcosm of coding-agent tasks — they require spatial reasoning, physics simulation, animation loops, and self-contained output. One-shot means no retry, no refinement — raw generation quality per dollar.
|
|||
|
|
|
|||
|
|
## Results
|
|||
|
|
|
|||
|
|
| Model | Vendor | Tokens | Cost | Grade | Cost vs Fable |
|
|||
|
|
|-------|--------|--------|------|-------|---------------|
|
|||
|
|
| **Fable 5** | Anthropic | 62,158 | $3.12 | **A+** | 1× |
|
|||
|
|
| **GPT 5.5** | OpenAI | 37,753 | $1.14 | A− | 2.7× cheaper |
|
|||
|
|
| **Opus 4.8** | Anthropic | 22,280 | $0.56 | — | 5.6× cheaper |
|
|||
|
|
| **GLM 5.2** | Z.ai | 36,246 | $0.08 | B+ | **39× cheaper** |
|
|||
|
|
|
|||
|
|
### The Quality-Cost Curve
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
Quality
|
|||
|
|
A+ | ● Fable 5 ($3.12)
|
|||
|
|
|
|
|||
|
|
A | ● GPT 5.5 ($1.14)
|
|||
|
|
|
|
|||
|
|
B+ | ● GLM 5.2 ($0.08) ← 39× cheaper, only 2 grades down
|
|||
|
|
|
|
|||
|
|
B | ● Opus 4.8 ($0.56) ← (grade unspecified, lowest tokens)
|
|||
|
|
|
|
|||
|
|
+------+------+------+------+------→ Cost
|
|||
|
|
$0 $0.50 $1.00 $2.00 $3.00
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**Key insight:** The curve is **logarithmic, not linear**. Going from B+ to A+ costs 39× more. Going from A− to A+ costs 2.7× more. The marginal quality gain per dollar **decreases dramatically** at the top end.
|
|||
|
|
|
|||
|
|
### Cost per 1K Tokens
|
|||
|
|
|
|||
|
|
| Model | $/1K tokens |
|
|||
|
|
|-------|-------------|
|
|||
|
|
| Fable 5 | $0.050 |
|
|||
|
|
| GPT 5.5 | $0.030 |
|
|||
|
|
| Opus 4.8 | $0.025 |
|
|||
|
|
| GLM 5.2 | **$0.002** |
|
|||
|
|
|
|||
|
|
GLM 5.2's $0.002/1K tokens is the cheapest frontier-level coding generation measured to date.
|
|||
|
|
|
|||
|
|
## Community Takeaways
|
|||
|
|
|
|||
|
|
| User | Insight |
|
|||
|
|
|------|---------|
|
|||
|
|
| @Krysoph | GLM 5.2 could improve with iterations + vision model assistance |
|
|||
|
|
| @suzzvsworld | "GLM at $0.08 and still competitive is the real story. Chinese models are making the price conversation impossible to ignore" |
|
|||
|
|
| @rohanpaul_ai | Quality scales with spend; wants test with strict max-token limit |
|
|||
|
|
| @debugging_yami | GPT 5.5 is best value — nearly par with Fable at 1/3 cost |
|
|||
|
|
|
|||
|
|
### The Three Camps
|
|||
|
|
|
|||
|
|
1. **"Best quality regardless of cost"** — Fable 5 wins, but at 39× the price of GLM 5.2
|
|||
|
|
2. **"Best value"** — GPT 5.5 at $1.14, 2.7× cheaper than Fable with near-par quality
|
|||
|
|
3. **"Cheapest competitive"** — GLM 5.2 at $0.08, 39× cheaper, still produces B+ output
|
|||
|
|
|
|||
|
|
## Connection to Existing Wiki Knowledge
|
|||
|
|
|
|||
|
|
### Validates: [[chinese-model-cost-routing.md]] — The 87% Cost-Cut Playbook
|
|||
|
|
|
|||
|
|
DeRonin's 30-day field report showed 5-12× cost reduction per task category. This benchmark provides an even more extreme data point: **39× cost spread** on a single coding task. The pattern holds — Chinese models (GLM 5.2 specifically) deliver near-frontier quality at a fraction of the cost.
|
|||
|
|
|
|||
|
|
| Source | Cost Spread | Task |
|
|||
|
|
|--------|-------------|------|
|
|||
|
|
| DeRonin (agent loops) | 5× | GLM 5.2 vs Sonnet 4.7 |
|
|||
|
|
| DeRonin (reasoning) | 11× | Kimi K2.7 vs Opus 4.8 |
|
|||
|
|
| **atomic.chat (coding)** | **39×** | **GLM 5.2 vs Fable 5** |
|
|||
|
|
|
|||
|
|
The 39× spread is the largest measured cost asymmetry in our wiki — an order of magnitude beyond DeRonin's per-task ratios.
|
|||
|
|
|
|||
|
|
### Validates: [[glm-5.2-zai-coding-model.md]] — GLM 5.2 Deep-Dive
|
|||
|
|
|
|||
|
|
This benchmark adds a fifth independent validation of GLM 5.2's competitive positioning:
|
|||
|
|
|
|||
|
|
1. IAmFabian's Coding-Tests (14.06.2026) — Flappy Bird + Newton-Pendel
|
|||
|
|
2. Arnie's 4-Test-Review (22.06.2026) — gemischte Ergebnisse, Frontier-Niveau
|
|||
|
|
3. Semgrep IDOR-Benchmark (29.06.2026) — Security auf Opus 4.8-Niveau
|
|||
|
|
4. Artificial Analysis Intelligence Index v4.1 (01.07.2026) — #1 Open-Weights worldwide
|
|||
|
|
5. **atomic.chat One-Shot Coding Benchmark (01.07.2026)** — **B+ at 39× lower cost than Fable 5**
|
|||
|
|
|
|||
|
|
### Enriches: [[real-world-coding-showdown.md]] — Methodology
|
|||
|
|
|
|||
|
|
Fahd Mirza's Hermes Agent showdown used head-to-head methodology (same agent, same task, different models). atomic.chat's benchmark is a simpler variant: same prompt, no agent framework, raw one-shot output. Both complement standard benchmarks (SWE-bench, HumanEval) by testing real generation quality.
|
|||
|
|
|
|||
|
|
| Methodology | Source | Scope |
|
|||
|
|
|-------------|--------|-------|
|
|||
|
|
| Standard benchmark | SWE-bench Pro, HumanEval | Narrow, reproducible |
|
|||
|
|
| Agent-framework showdown | Fahd Mirza (Hermes) | Multi-file, tool-use, real app |
|
|||
|
|
| One-shot raw generation | **atomic.chat** | Single prompt, self-contained output |
|
|||
|
|
|
|||
|
|
### Connects: [[ai-investment-bubble.md]] — Cost Pressure Narrative
|
|||
|
|
|
|||
|
|
The 39× cost spread is ammunition for the "AI is overbuilt" thesis: if a B+ model costs $0.08 and an A+ model costs $3.12, the **economic case for premium models erodes** when B+ is "good enough" for most use cases. This mirrors Meta selling excess compute capacity and Oracle warning of datacenter delays — the market is pricing in overinvestment in premium AI.
|
|||
|
|
|
|||
|
|
## Implications for Our Routing Architecture
|
|||
|
|
|
|||
|
|
See [[../../architecture/model-routing.md]].
|
|||
|
|
|
|||
|
|
| Task Tier | Recommended Model | Rationale |
|
|||
|
|
|-----------|-------------------|-----------|
|
|||
|
|
| Tier 4 (Frontier quality) | Fable 5 or Opus 4.8 | When A+ is non-negotiable |
|
|||
|
|
| Tier 3 (High quality) | GPT 5.5 | Best value — A− at 2.7× cheaper |
|
|||
|
|
| Tier 2 (Standard) | GLM 5.2 | B+ at 39× cheaper — "good enough" for most |
|
|||
|
|
| Tier 0-1 (Bulk) | GLM 5.2 or MiMo | Cheapest viable option |
|
|||
|
|
|
|||
|
|
**New insight:** The atomic.chat benchmark suggests GPT 5.5 may be **the sweet spot** for quality-sensitive tasks where Fable 5's premium is unjustified. Our current routing uses GLM 5.2 as primary — this benchmark validates that for most tasks (Tier 0-2), but suggests GPT 5.5 as a Tier 3 upgrade path.
|
|||
|
|
|
|||
|
|
## Methodology Limitations
|
|||
|
|
|
|||
|
|
1. **One-shot only** — No iteration, no refinement. Real coding workflows involve multiple rounds. @Krysoph notes GLM 5.2 could improve with iterations.
|
|||
|
|
2. **No strict token limit** — @rohanpaul_ai wants to see results with a max-token constraint. Without one, Fable 5's 62K tokens may reflect verbosity, not capability.
|
|||
|
|
3. **Subjective grading** — A+/A−/B+ grades are atomic.chat's assessment, not a standardized rubric.
|
|||
|
|
4. **Single task type** — HTML5 Canvas physics is one domain. Results may not generalize to backend logic, data analysis, or agentic tool-use.
|
|||
|
|
5. **Opus 4.8 grade missing** — The post doesn't specify Opus 4.8's quality grade, only its cost/token count.
|
|||
|
|
|
|||
|
|
## Cross-References
|
|||
|
|
|
|||
|
|
- [[chinese-model-cost-routing.md]] — DeRonin's 87% cost-cut playbook (5-12× per task)
|
|||
|
|
- [[glm-5.2-zai-coding-model.md]] — GLM 5.2 deep-dive (5th validation)
|
|||
|
|
- [[real-world-coding-showdown.md]] — Fahd Mirza's agent-framework methodology
|
|||
|
|
- [[ai-investment-bubble.md]] — Cost pressure narrative (Meta, Oracle, Karp)
|
|||
|
|
- [[../../tools/anthropic-claude.md]] — Fable 5 + Opus 4.8
|
|||
|
|
- [[../../tools/openai-gpt.md]] — GPT 5.5
|
|||
|
|
- [[../../architecture/model-routing.md]] — 5-tier routing architecture
|
|||
|
|
- [[ai-intelligence-commoditization-thesis.md]] — TheProphet's "Factory for Gods"
|
|||
|
|
|
|||
|
|
## External Sources
|
|||
|
|
|
|||
|
|
- [atomic.chat X Post](https://x.com/atomic_chat_hq/status/2072446067962978411)
|
|||
|
|
- [atomic.chat](https://atomic.chat)
|