Model Lab

Synthesis

The story so far, across six profiled models and a 40-problem strategy exam apiece · updated 2026-07-05

1Roster

The four frontier rows didn't sit the 40-run strategy exam — "pass rate" isn't a comparable number for them. What they did instead was the Build Benchmark, a real production-service build under the same one-shot+fix protocol the delegation tree itself uses.

2The Archetypes

3Strategy Truths (cross-model)

• The strict prompt went undefeated across every model, size, and architecture tested — until Qwen3-30B Thinking. Four perfect 10/10 scorers now sit in the lab, each under a strict configuration: Qwen3-Coder-Next (10.4s), gpt-oss-120b (16.2s), MiniMax M2.5 (65.5s), and Qwen3.6-35B (111s) — spanning a pure instruct model, an always-reasoning MoE, a prompt-fragile MoE, and a bounded hybrid thinker. Qwen3-30B Thinking breaks the streak: its best score (9/10) comes from minimal-nothink, with both strict strategies landing at 7/10 — the first model in the lab where the strict prompt lost. A dedicated reasoner, unlike a hybrid toggle, appears to want room to work rather than a tight scope; per-model presets exist for exactly this exception.
• Strict prompt beats mode toggle, for every model tested. Going from a minimal to a strict system prompt improves pass rate and slashes wall time more reliably than switching think/nothink does — true for GLM, Qwen3-Coder-30B, and Qwen3.6-35B alike.
• /nothink is essential for GLM, irrelevant for pure instruct models, optional for Qwen3.6. Qwen3-Coder-30B has no reasoning mode at all, so the suffix is a no-op. Qwen3.6-35B reasons regardless of the suffix (its "nothink" runs still carry hundreds to thousands of reasoning tokens) but strict-nothink is faster with only a small accuracy cost (9/10 vs 10/10).
• Verify-always: never trust unexecuted code. Every pass/fail figure on this site comes from running the model's code against hidden tests, not from reading it.
• Context costs roughly one minute of prefill per 10k tokens (GLM-measured, context-scaling experiment) — a tax that applies before generation even starts, independent of the reasoning budget.
• Retry-with-traceback fixes mechanical failures, not conceptual ones. GLM's self-repair loop fixed 2 of 5 failures (missing import, off-by-one class errors) but never budged on toposort or expr_eval, where two rounds regenerated the identical wrong idea.

4models.yaml Recommendation

5Frontier Baselines

Four subscription models — Fable 5, Sonnet 5, Opus 4.8, Haiku 4.5 — ran the Build Benchmark: a real production Go service built solo, one-shot-plus-fix, from the same frozen spec sheets GLM-4.5-Air originally built from. One-shot pass rate came out perfectly ordered by price tier (7/7, 6/7, 5/7, 4/7), and all four self-recovered to a full 7/7 given fix rounds. Takeaway: the subscription buys one-shot reliability, not a higher capability ceiling — the local delegation tree buys unlimited retries at electricity prices instead. That gap is exactly the arbitrage this whole gateway is built to exploit. The five-model local build campaign that followed confirms it from the other side: gpt-oss-120b's 5/7 one-shot rate matches Opus 4.8's at electricity prices, while the other four local models needed 1-3 escalations apiece — and one file, engine.go, defeated all six local attempts identically.

6Final Findings

• Frontier models reached 84/84 final passes (4 models × 3 languages × 7 files) via one-shot or simple fix rounds, never needing the local escalation ladder. One-shot rates varied by language and model (e.g. Fable 5 was 7/7 one-shot in Go but only 5/7 in TypeScript), but every frontier row in every language converged to a clean 7/7 with ordinary fix rounds — none of the 12 frontier-language cells ever touched the nothink → retry → think → escalation machinery the local tier needed.
• gpt-oss-120b is the local champion in all three languages — Go 5/7, TS 5/7, and C# 3/7 one-shot, the best local one-shot result in each pack even as the absolute numbers slide with language difficulty.
• The engine module went 0-for-24 across all local attempts in Go and TypeScript (8 local models × 2 languages × 2 confirmed universal misses on engine.go/engine.ts), every failure traced to token-budget truncation mid-file rather than wrong reasoning. The C# figure likely differs: the single-csproj harness there meant a truncated Engine.cs broke compilation for every downstream module, contaminating the in-tree/escalation counts (see the .NET scoreboard footnote), so a clean engine-only tally for C# isn't available from this run.
• Language affinity is tier-dependent. C# is the easiest language for the frontier tier (three of four frontier models hit 6/7 or 7/7 one-shot) but the hardest for the local tier (gpt-oss-120b's 3/7 is the best local one-shot score of any language). Go is the strongest language for local models overall. TypeScript drags down every tier via ecosystem traps — the node:sqlite/vitest resolution issue caught all three frontier finishers and all 8 local models identically.
• The dedicated-reasoner exception to the "strict prompt wins" rule stands. Qwen3-30B Thinking remains the one model in the lab where minimal-nothink beat strict prompting on the CS exam (see Strategy Truths, above) — and its build results reinforce rather than contradict that profile: heavy escalation dependence (6 escalations in Go, 5 in TS) alongside genuinely low one-shot rates in every language.
• CS-exam rank is not build rank. Devstral Small (8/10) and Qwen3-30B Thinking (9/10) both scored well on the 40-run CS/strategy exam, but neither aced a single language in the actual builds: Devstral posted 0/7, 0/7, and 1/7 one-shot in Go, TS, and C# respectively, and Thinking posted 0/7, 1/7, and 1/7. Passing a synthetic exam and shipping a real multi-file service are different skills.

Model Lab · Benchmarks

The Build Benchmark — Go

Same frozen spec sheets, same hidden Go tests, one production service · updated 2026-07-05

1What This Is

Every builder here implements the same real production Go service — the delegation gateway itself — from identical frozen spec sheets, split across seven files: archive, registry, provider, verify, store, engine, and server. Each file is checked against hidden Go tests the builder never sees in advance. GLM-4.5-Air's row is the original build that produced this gateway's actual code, run through its full delegation tree (nothink → retry → think → sonnet-fix escalation). The four frontier models — Fable 5, Sonnet 5, Opus 4.8, Haiku 4.5 — each ran the same spec sheets solo, under a simpler one-shot-plus-fix protocol, with no thinking-mode or escalation tier available. Five more gx10 (local) models — Qwen3-Coder-30B, Qwen3.6-35B, gpt-oss-120b, Qwen3-Coder-Next, and MiniMax M2.5 — ran the same harness through the full nothink → retry → think → sonnet-fix escalation tree; that campaign is now complete.

2Scoreboard

One-shot pass rate

3What It Means

• The frontier premium is one-shot reliability, not a capability ceiling. All four frontier models reached a clean 7/7 — the only difference was how many fix rounds they needed to get there.
• Price predicts first-try accuracy almost perfectly. One-shot rate ordered exactly by tier: Fable 7/7 → Sonnet 6/7 → Opus 5/7 → Haiku 4/7. (Sonnet edging the pricier Opus fits its newer generation.)
• gpt-oss-120b matched Opus 4.8's one-shot rate (5/7) at electricity prices — the strongest local build result of the campaign, and the fastest wall-clock run (701s) of any local model.
• engine.go defeated all six local attempts identically — by token-budget truncation mid-file, not wrong reasoning. Every local model, regardless of architecture or think mode, ran out of budget on the same file before producing a complete artifact. That's a protocol bug (flat per-file token budgets), not a model weakness; adaptive per-file budgets are now on Dispatch's v2 roadmap.
• Local models trade first-try accuracy for free retries, at wildly different costs. Qwen3-Coder-30B ground from 1/7 one-shot to 7/7 with three escalations; Qwen3.6-35B needed three escalations too, but burned 3221s of wall time doing it — nearly 4.5x gpt-oss-120b's 701s — mostly to silent no-output stalls rather than productive reasoning.
• Benchmarks disagree, and that's the point. Qwen3.6-35B and gpt-oss-120b both scored a perfect 10/10 on the CS exam, but on the real build Qwen3.6 posted the campaign's slowest, most escalation-prone run while gpt-oss-120b was the clear champion. Hire for the job, not the test.
• The arbitrage is the architecture. Subscription buys getting it right the first time; the GX10 buys unlimited retries. Dispatch's decision tree is what converts cheap retries into finished work.

3bProtocol Lessons (feeding Dispatch v2)

• Adaptive max_tokens by spec complexity. engine.go's flat per-file budget truncated every local model identically mid-artifact; a complexity-aware budget would likely have let several models clear it without escalating.
• Detect no_artifact + budget-exhausted as its own failure class. Qwen3.6-35B's stalls on provider/store/engine were silent — no code, budget spent — and deserve an immediate strategy change (e.g. skip straight to escalation or shrink the ask) rather than a blind retry at the same budget.
• Error-feedback formatting matters more than think/nothink for inert models. Pure instruct models (Qwen3-Coder-30B, Qwen3-Coder-Next) get no benefit from the think suffix; what moves their pass rate is how the retry error is phrased and scoped.
• Final acceptance must include go vet. Build + test alone let a lostcancel bug slip through in one campaign run — vet needs to be a hard gate before a file is marked resolved.

4Spec-Trap Forensics

Three places in the frozen spec sheets reliably produced wrong-but-plausible code, identified from Sonnet 5's fix-round diagnosis and the recurring failure signatures across builders.

• archive: the os.IsNotExist trap. The hidden test asserts os.IsNotExist(err) on a missing-file path, but wrapping the underlying error with fmt.Errorf("...: %w", err) — the idiomatic Go instinct — defeats that assertion unless unwrapped correctly. This tripped Sonnet 5, Haiku 4.5, and GLM-4.5-Air (identical error snippet: "os.IsNotExist broken by fmt.Errorf wrap").
• engine: a run-on sentence conflating two conditions. Spec step 4 blurs "skip this step" and "no artifact produced" into a single run-on sentence, and builders that read it literally implement one condition where two were intended. This was the file that beat every local escalation tier and Opus 4.8's first attempt alike.
• server: a self-contradictory async paragraph. The spec's paragraph on asynchronous request handling contains an internal contradiction about ordering guarantees, forcing builders to guess which half of the paragraph is authoritative. Haiku 4.5 missed it on the first pass; other builders got lucky on which half they picked.

Model Lab · Benchmarks

The Build Benchmark — TypeScript

The same 7-file service exam, ported to Node · frontier + local tiers both done

1What This Is

The same seven-file production service exam from the Go benchmark — archive, registry, provider, verify, store, engine, and server — ported to TypeScript on Node, with the toolchain swapped to match: vitest for hidden tests, node:sqlite (built-in) for storage, and node:http for the server layer instead of Go's standard library equivalents. The Go campaign's one universal failure was engine.go, which defeated every local model identically via token-budget truncation on a syntactically dense file; porting the same spec to TypeScript separates two hypotheses the Go-only result couldn't: is that a Go problem (verbose syntax inflating token count past the budget) or a complexity problem inherent to the spec regardless of language? Frontier tier and the 8-model local slate are both complete, including escalation fixes — every TypeScript workspace now passes.

2Scoreboard

One-shot pass rate

3What It Means

• The Go price-gradient did not survive the language switch. Go one-shot order was Fable > Sonnet > Opus > Haiku (7/6/5/4); in TS it's Sonnet = Opus (6/7) with Fable at 5/7. Language changes the ranking even at the frontier — the central hypothesis of this benchmark, confirmed before a single local model ran.
• gpt-oss-120b is the local leader again — 5/7 one-shot, matching its Go result exactly. Same model, same rank, a different language: the strongest evidence yet that its build performance is a genuine model property, not a Go-specific fluke.
• The wall-time spread across the local slate is enormous: Qwen3-Coder-30B at ~11 minutes versus Qwen3.6-35B at ~86 minutes — an 8x gap on the same seven files. The gap is driven by stalling, not thinking: Qwen3.6-35B's registry file burns all three local stages as no_artifact_budget (reasoning spent, zero code returned), the identical failure mode that made it the slowest builder in the Go campaign too.
• node:sqlite defeated every one of the 8 local models on store/sqlite.ts, with zero exceptions. The same builtin-vs-vitest resolution trap that caught all three frontier finishers is, for the local tier, unconditional — no local model, regardless of architecture or think mode, produced a passing store module in this run.
• The engine module remains the hardest file for every tier. All 8 local models still show engine.ts failing after the local escalation ladder — the same universal wall the Go campaign hit, now confirmed language-independent.

4Trap Forensics

• node:sqlite vs the vitest bundler. A static ESM value-import of the new builtin fails to resolve under vite; the fix is a type-only import plus createRequire. It caught all three frontier finishers (3-for-3) — the TS counterpart of Go's os.IsNotExist wrapping trap: it measures ecosystem fluency, not spec comprehension, and is graded accordingly.
• Registry watch timing race. The watch baseline is captured lazily, and a builder that reads the spec literally can grab it too early. Caught Fable 5 in its otherwise-clean run.
• Grading note: the frozen hidden tests themselves fail strict tsc under noUncheckedIndexedAccess, so typecheck grading is scoped to src/ only.

5Status

Model Lab · Benchmarks

The Build Benchmark — .NET

The same 7-file service exam, ported to .NET · all 12 builders complete

1What This Is

The same seven-file production service exam from the Go benchmark — archive, registry, provider, verify, store, engine, and server — ported to C# on .NET SDK 10.0.301 (net10.0), with the toolchain swapped to match: xUnit for hidden tests, Microsoft.Data.Sqlite for storage, YamlDotNet for the frozen spec/config format, and an ASP.NET Core minimal API for the server layer. All 12 builders have now run against it, and all 36 workspaces pass their full suites after the escalation-fix pass.

2Scoreboard

One-shot pass rate

3What It Means

All 36 workspaces now pass their full suites after the escalation-fix pass. Where the TypeScript pack probes the Go-vs-complexity split on engine.go, the .NET pack targets the enterprise-stack hypothesis: whether models perform differently on a heavier, more ceremony-laden statically-typed stack (interfaces, DI conventions, verbose type annotations) than on Go's minimalism or TypeScript's flexibility — and whether that ceremony helps or hurts the same file that broke every local Go attempt. The single-csproj harness design means the in-tree/escalation breakdown for local models is contaminated by cascading compile failures downstream of a truncated Engine.cs (see the scoreboard footnote); one-shot rates for pre-engine modules remain a clean signal, and they show C# as the toughest language yet for the local tier — gpt-oss-120b's 3/7 is the best local one-shot result, well below its Go (5/7) and TS (5/7) showings.

4Trap Forensics

• FileNotFoundException identity through wrapping (to be validated). The C# analog of Go's os.IsNotExist trap — wrapping the underlying exception may defeat a type-check the hidden test relies on.
• The engine step-4 run-on (to be validated). The same conflated "skip this step" / "no artifact produced" sentence from the Go spec carries over; unclear yet whether C#'s stricter typing makes the ambiguity easier or harder to resolve.
• The server async paragraph (to be validated). The self-contradictory ordering-guarantee language from the Go spec carries over to the ASP.NET Core minimal API port.

5Status

complete — all 12 builders

Model Lab · Profile

GLM-4.5-Air (Q4_K_M)

unsloth GGUF · llama.cpp · Asus GX10 (GB10, 128GB unified) · profiled 2026-07-05

1Verdict

2Strategy Shootout

Four prompting strategies run against the same 10-problem set (40 runs total).

Pass rate

Avg wall time per problem

3Per-Problem Matrix

toposort is the only problem strict-nothink misses — it asserts a dependency ordering the model gets backwards, not a broken algorithm.

4Self-Repair Loop

Protocol: take a failing solution plus its traceback, feed both back to the model under strict-nothink, allow up to two repair rounds.

2 of 5 failing cases were repaired — csv_parse on round 1 and lru_ttl_cache on round 2. toposort and expr_eval were never repaired across either round.

5Context Scaling

Needle-in-haystack bug-fix task at increasing prompt sizes, strict-nothink.

Prefill time vs prompt size

All 5 runs found and fixed the injected bug correctly — no accuracy degradation up to roughly 19k prompt tokens. But prefill throughput isn't flat: it drops from 250 tok/s at 4k down to 145 tok/s at 24k as the context grows. Going from 1,297 to 19,305 prompt tokens (about 15×) costs 22× more prefill time.

6Thinking Mode at 16k Budget

Doubling the reasoning budget from 8k to 16k tokens, strict-think, on the problems that previously stalled.

7Concurrency

Throughput under simultaneous requests, single llama.cpp instance.

Phase A — single slot

Phase B — --parallel 4

8Delegation Playbook

Four-step decision tree, in order. Do not skip a step.

• 1. Strict system prompt + /nothink, always first. 9/10 pass rate at ~13s average. This is the default for every problem, no exceptions.
• 2. On failure, one repair round with the traceback. Fixes mechanical bugs (missing import, off-by-one) roughly 2 of 5 times in testing, at ~20s. Do not run a second repair round — an identical re-failure means the miss is conceptual, not mechanical.
• 3. Identical re-failure → one thinking-mode attempt, 16k budget. Expect 15–25 minutes wall time. This tier is for conceptual spec misses only — it rescued toposort after two failed no-think repair rounds.
• 4. Thinking-mode cap-out → escalate to a frontier model. Hitting the 16k cap without emitting code means the problem is beyond this model's capability edge regardless of budget; retrying locally wastes 20+ minutes for nothing.

9Deployment

• systemd service llama-server — boot-persistent, auto-restart on failure, launched with --parallel 4 --metrics.
• DNS — llm.internal resolves to 192.168.3.220 via UniFi.
• Endpoints — port 8080 serves the web UI and the OpenAI-compatible /v1 API, plus /metrics and /health.
• Thermals — steady at 51°C and 96% GPU utilization during sustained inference, drawing 11W.

Model Lab · Profile

Qwen3-Coder-30B

instruct-only, no reasoning mode · llama.cpp · campaign run 2026-07-05

1Verdict

2Strategy Table

Same 10-problem set, four strategies, 40 runs. Recomputed from models/qwen3-coder-30b/benchmarks/2026-07-05-campaign.jsonl.

Pass rate

Avg wall time per problem

3Per-Problem Matrix

4Observations

• Token discipline under strict prompts is extreme. Strict strategies output ~216 tokens average versus ~550–610 for minimal ones — a >2.5× reduction with a higher pass rate, the same "strict prompt beats verbosity" pattern seen in every other model tested.
• Generation speed is rock-steady at ~35 tok/s regardless of strategy or prompt length — no thinking-mode variance to manage, which is the main practical advantage over GLM and Qwen3.6 here.
• Best problem coverage: 9/10, one hard failure. Every problem other than toposort is solved by at least one strategy; strict-think solves everything except it.

Model Lab · Profile

Qwen3.6-35B

hybrid think/nothink · llama.cpp · campaign run 2026-07-05

1Verdict

2Strategy Table

Same 10-problem set, four strategies, 40 runs. Recomputed from models/qwen3.6-35b/benchmarks/2026-07-05-campaign.jsonl.

Pass rate

Avg wall time per problem

3Per-Problem Matrix

4Observations

• Bounded reasoning, contrast with GLM. GLM's think mode burns a fixed 16,000-token budget over ~24 minutes on problems it can already solve in seconds. Qwen3.6-35B's reasoning self-terminates well under its cap on 39 of 40 runs — the one exception (range_overlap / minimal-nothink) is the exception that proves the model usually knows when to stop.
• Slower per-response than Coder-30B, but the only perfect scorer. Where Qwen3-Coder-30B trades reliability for speed (9/10 at 6.7s), Qwen3.6-35B trades speed for reliability (10/10 at 111s) — a genuine speed/accuracy frontier, not a strictly dominated option.
• Strict prompting still wins over mode choice. strict-nothink (94s, 9/10) beats minimal-think (145s, 9/10) on both speed and matches it on accuracy — the prompt discipline lesson holds even for a model whose reasoning is otherwise well-behaved.

Model Lab · Profile

gpt-oss-120b

open-weight MoE · llama.cpp · campaign run 2026-07-05

1Verdict

2Strategy Table

Same 10-problem set, four strategies, 40 runs. Recomputed from models/gpt-oss-120b/benchmarks/2026-07-05-campaign.jsonl.

Pass rate

Avg wall time per problem

3Per-Problem Matrix

glob_match is strict-nothink's only miss (10/10 elsewhere on that strategy) — and the one problem where strict-think does worse than strict-nothink, an inversion of the usual "think helps or is neutral" pattern seen in every other model.

4Observations

• A fourth perfect scorer, sub-20-second class. With Qwen3-Coder-Next (10.4s), gpt-oss-120b (16.2s) is the second-fastest clean 10/10 in the lab — well ahead of Qwen3.6-35B's 111s and MiniMax M2.5's 65.5s.
• Reasoning is always on but cheap. Unlike Qwen3.6's hybrid chains (hundreds to thousands of tokens) or GLM's unbounded think mode (16k-token spirals), gpt-oss-120b's reasoning never exceeds a few hundred tokens in this dataset — closer to a fixed low-effort setting than a toggle.
• Prompt-sensitive, not prompt-fragile. Minimal prompts cost roughly 3x the wall time and 2 points of pass rate — a real penalty, but nowhere near MiniMax M2.5's collapse under the same conditions.
• Served with --no-mmap -c 16384 on the rebuilt CUDA-arch-121 llama.cpp — the largest resident model in the lab after GLM.

Model Lab · Profile

Qwen3-Coder-Next

coder-tuned successor line, instruct-only · llama.cpp · campaign run 2026-07-05

1Verdict

2Strategy Table

Same 10-problem set, four strategies, 40 runs. Recomputed from models/qwen3-coder-next/benchmarks/2026-07-05-campaign.jsonl.

Pass rate

Avg wall time per problem

3Per-Problem Matrix

expr_eval is the only problem that isn't a clean sweep, and it's the single point separating this model from a perfect 40/40. Every other problem, including toposort, passes under all four strategies.

4Observations

• Closes the 30B's one structural gap. toposort failed under all four strategies on Qwen3-Coder-30B — the only 0/4 problem across the first three profiled models — and passes under all four here. The dependency-direction misread that beat the 30B doesn't reproduce on its successor.
• Fastest perfect scorer in the lab. 10.4s average under strict-think edges out gpt-oss-120b's 16.2s and dwarfs MiniMax M2.5 (65.5s) and Qwen3.6-35B (111s).
• Strict prompting still roughly halves wall time even though pass rate ties at 10/10 — 10.4s vs 23.1s for think, 11.4s vs 29.1s for nothink — the token-discipline benefit holds even when accuracy doesn't move.

Model Lab · Profile

MiniMax M2.5

MoE, long-context focus · llama.cpp · campaign run 2026-07-05

1Verdict

2Strategy Table

Same 10-problem set, four strategies, 40 runs. Recomputed from models/minimax-m2.5/benchmarks/2026-07-05-campaign.jsonl.

Pass rate

Avg wall time per problem

3Per-Problem Matrix

Both strict strategies sweep all 10 problems. The five minimal-think failures (toposort, csv_parse, glob_match, deep_merge, expr_eval) are all cap-out spirals — the model never gets far enough to produce wrong code, it just never finishes.

4Observations

• The Diva archetype: flawless under discipline, unusable without it. No other model in the lab shows this wide a swing between its best and worst strategy — a 5-point pass-rate spread and better than 5x wall-time spread between strict-nothink and minimal-think.
• Cap-outs are the dominant minimal-prompt failure mode. 8 of 20 minimal runs hit the full 625s / 8,000-token ceiling and returned no code — GLM's think-mode spiral signature, but triggered here by prompt looseness rather than a "think" toggle.
• Slowest per-response of the perfect scorers. ~14.7 tok/s puts it well behind gpt-oss-120b (~36 tok/s) and the two Qwen instruct models (~26–35 tok/s) — a real latency cost even in its best configuration.
• Served with --no-mmap -c 16384 on the rebuilt CUDA-arch-121 llama.cpp, the same build used for gpt-oss-120b.

Model Lab · Profile

Devstral Small

Mistral, 24B dense · Q8 GGUF · llama.cpp · campaign run 2026-07-06

1Hypothesis

2Strategy Table

Same 10-problem set, four strategies, 40 runs. Recomputed from models/devstral-small/benchmarks/2026-07-06-cs-exam.jsonl.

Pass rate

Avg wall time per problem

3Per-Problem Matrix

4Observations

• Dense confirms the MoE-dominance hypothesis rather than overturning it. 8/10 at ~6 tok/s sits mid-pack against MoE models running 5-6x faster — Devstral pays a real speed tax and doesn't get an accuracy premium in return.
• Strict prompt still wins on pass rate, but not on the think toggle. strict-nothink (8/10) beats strict-think (5/10) despite matching wall time and token counts — the opposite of a "more thinking helps" story; this looks like prompt-format noise rather than a reasoning effect, consistent with Devstral showing no dedicated reasoning-token machinery.
• Generation speed is rock-steady at ~6 tok/s regardless of strategy — the dense-model signature already seen in wall time, just restated in throughput terms.

Model Lab · Profile

Qwen3-30B Thinking

Qwen3-30B-A3B-Thinking-2507 · Q8 GGUF · llama.cpp · campaign run 2026-07-06

1Hypothesis

2Strategy Table

Same 10-problem set, four strategies, 40 runs. Recomputed from models/qwen3-30b-thinking/benchmarks/2026-07-06-cs-exam.jsonl.

Pass rate

Avg wall time per problem

3Per-Problem Matrix

4Observations

• The first model in the lab where the strict prompt underperforms. Minimal-nothink (9/10) beats both strict-nothink and strict-think (7/10 each) — the exact opposite of every other model profiled. Dedicated reasoners may need room to explore rather than a narrow scope; per-model strategy presets in Dispatch exist to capture cases like this rather than force a single global default.
• Reasoning is always-on and expensive regardless of suffix. Every strategy produces 4,000+ output tokens at ~32 tok/s and 130-155s wall time — there's no cheap "nothink" mode the way GLM or the coder models have; the think/nothink suffix barely moves the needle here.
• toposort inverts the usual strategy pattern. Both minimal strategies solve it, both strict strategies miss it — a second, independent signal (beyond the overall pass-rate ranking) that this model behaves opposite to the lab's dominant "strict wins" rule.

Model Lab

Infrastructure

GX10-01 deployment topology

1Compute

• Asus GX10 — GB10 unified-memory system, 128GB shared between CPU and GPU.
• llama-server — systemd service, boot-persistent, auto-restart on failure, port 8080, 4 parallel slots (--parallel 4 --metrics), serving the OpenAI-compatible /v1 API plus /metrics and /health.
• DNS — llm.internal resolves to 192.168.3.220 via UniFi.
• Thermals — steady 51°C at 96% GPU utilization under sustained inference, drawing 11W.

2Dispatch Gateway

Small Go service (port 8181) owning the decision tree, model presets, and telemetry — the layer that decides which model and which strategy handles a given request, and escalates when one fails.

3Telemetry

Every run in this lab — strategy exams, self-repair rounds, context-scaling sweeps, concurrency tests — is logged as JSONL with problem id, strategy, timestamps, reasoning/output token counts, wall time, and pass/fail against hidden tests. All pass/fail figures on this site are recomputed directly from those logs, not hand-entered.

Model Lab

Roadmap

Future benchmarking and system-design queue

1Queue

• Python exam slate. The matrix anchor — Python is the locals' home language, so a Python-language build/CS pack calibrates the ceiling every other language result gets read against.
• Hybrid team benchmark (frontier thinks / local does). Measures the token exchange rate between tiers; work thesis is that a frontier model planning and a local model executing can cut frontier token spend to roughly 20% of a solo-frontier run at comparable quality.
• Token/cost ledger in Dispatch. Frontier-vs-local spend per unit of verified work, priced against a counterfactual (what the same task would have cost run solo-frontier) rather than raw token counts alone.
• Dispatch MCP interface + "foreman" agents. Named agents per backend, an agent-flow UX, and cost reporting baked in — governed by one design principle: Dispatch executes work orders; it never writes them.
• Bedrock + Cloudflare Workers AI as cloud-GX10 providers. Same exam suite, run against a third substrate tier, to separate "GX10-specific" results from results that generalize across any inference backend.
• Frontend slates. React vs Angular tier-1 logic exams to probe the language-affinity question head-on; an Electron IPC pack to follow later.
• Production niche: specialized batch labor. Chaio transaction/receipt/email categorization as the first real production workload for the local tier — high volume, low individual stakes, exactly the profile this lab has been calibrating for.