Benchmark · run 5fe4bd2a · 2026-08-17
Nine tasks, graded by exit code.
One loop, one prompt, one parser. The backends differ only in the model. Pass and fail are the check command’s exit code, never a reading of the transcript.
Read before the numbers
The cloud lane ran uncapped with three internal retries per step. The self-hosted lane ran at a 512-token output ceiling with one attempt per step. The columns are not apples-to-apples and the self-hosted figure is a lower bound.
Conditions. Pass/fail is the check command exit code, never a transcript. Cloud lane uncapped with three internal retries per step; self-hosted lane at a 512-token ceiling, one attempt per step — not apples-to-apples, and the self-hosted bar is a lower bound. Single RTX 4000 SFF Ada, 20 GB. Run 5fe4bd2a, 2026-08-17.
Conditions. Total spend divided by tasks that passed: $2.0410 / 9 against $0.1900 / 4. A 4.8× advantage, not the 10.7× raw spend suggests, because five of nine self-hosted attempts produced nothing. Same run and hardware as above; axis scales from zero.
Task × backend
MeasuredLatest run per task, from bench/history.jsonl. Wall-time includes queuing on a single stream; the stub backend is a control that always fails, present to prove the harness grades rather than rewards.
Why the small model fails at depth
A transformer keeps every past token addressable in a KV cache. A state-space model does not — it compresses all history into a fixed-size recurrent state. That is the efficiency win, and it is a lossy channel. In an agent loop context grows monotonically, because every tool result is appended.
A model with a very small attention budget has little addressable memory to fall back on when the recurrent state saturates. The predicted failure is therefore not an invented fact but degradation into incoherence at depth — which from outside looks exactly like a truncated tool call. Ember marks both the attention layers below and the truncation failures above. Same colour, same idea.
The box these ran on
GPU
RTX 4000 SFF Ada
Memory
20,475 MiB
Served model
nemotron-3-nano:4b
Throughput
18.7 tok/s
Probe · GPU vs wall
0.37 s / 77 s
Fine-tune set
21,087 train / 1,761 val
MeasuredRead from the host on 2026-08-19 with nvidia-smi and ollama list. The 0.37 s GPU / 77 s wall probe is single-stream queuing, not model speed.