About the data

What every row is

One config = model × harness (× reasoning on/off) on one task, run 5 times on one host. Every number is measured. There are no estimates.

• Quality: trials passed out of 5, scored by a deterministic grader (exact JSON or TOML checks; Kotlin is compiled with kotlinc and run against behaviour checks; Go is compiled and parity-tested with go test).
• Task time: wall-clock per trial, from harness start to exit (P50 or P99 over the 5 trials). Each trial is capped at 900 s; a trial whose files are correct at the cap still counts as a pass, flagged ⏱.
• Output tokens / turns: counted by a metering proxy that sits between every harness and the model server. Turns = chat requests, tokens = completion tokens, including hidden reasoning tokens.
• Peak RAM: peak resident memory of the model server during the config. The Device RAM slider uses this to test memory fit only.

Fixed stack

The results shipped with AWTY were measured on a public Google Colab runtime with an NVIDIA A100 40 GB (all layers offloaded with -ngl 99); results you measure with awty bench-all appear next to them. Engine: llama.cpp b11317, run as llama-server -c 32768 -np 1 -n 4096 --jinja --temp 1.0 --top-p 0.95 --top-k 64 -t 12 --cache-ram 0 --slot-save-path (26B-A4B adds --no-repack). A fresh server starts for every config, and the slot cache is cleared before each trial, so trials are independent. Models: official Gemma 4 QAT Q4_0 GGUFs with their embedded chat template, as shipped.

Harnesses

• Single call: AWTY's own null harness. One request with the input files inlined; the reply's code block is written as the output file. Reasoning stays at the server default (on).
• Pi, mini-swe-agent 2.3.1, OpenCode 1.19: unmodified CLI agents, run non-interactively in a fresh copy of the task workspace.

Verdicts

THERE = all four thresholds met and the model fits the device RAM. BORDERLINE = only near misses (time, tokens or turns within 1.5× of the threshold, or quality within 20 points). NOT THERE = at least one clear miss. Each non-THERE row names what blocks it.

Rigs (hardware + engine)

A rig is the platform a result was measured on: hardware, inference engine and memory (catalogs/rigs.yaml). Each cell is keyed by rig × task × model × harness. With more than one rig, the grid pivots to Rig × Model and picks the best harness per cell ("via X"). Ladder sorts rigs from cheapest to most expensive, marks the cheapest rig that is THERE, and suggests an upgrade for rigs that fall short. Budget and accelerator filters hide rigs.

Every rig shown is measured. Results from a machine that is not in rigs.yaml still appear, labelled from the run's hardware.json; a rigs.yaml entry only adds a curated label, price and memory figure.

Known caveats

• Speed depends on the rig: compare rigs side by side rather than reading one rig's times as universal. Device RAM tests memory fit, not speed.
• Peak memory is host RSS. On the A100 rig the weights live in VRAM, so its peak RAM understates what the GPU needs; its rig entry lists the 40 GB of VRAM instead.
• N = 5 trials per config (a few have 10), so a difference of one pass is within noise.
• OpenCode and some mini-swe-agent configs often run to the 900 s cap, so their time is a lower bound.
• The 4,096-token generation cap is part of the engine config. 26B-A4B with reasoning on often hits it (✂ in trial details).