Are We There Yet? v0.5.0-preview

Viability isn’t a model property. It’s a property of the whole stack.

Download bundle About the data

BUNDLE

NO RESULTS LOADED

Load results to explore which local stacks are "there yet"

AWTY compares measured runs of Agent Harness × Model × Inference Engine × Hardware on deterministic tasks (receipts & PDFs → JSON, log routing, image triage, Android fixes, Python → Go). This page runs entirely in your browser: nothing is uploaded, and it needs a results bundle to show anything.

① Try the example runs

Real runs measured on a public Google Colab GPU runtime.

Models: Gemma 4 E2B · E4B · 26B-A4B
Agents: Pi · mini-swe-agent · OpenCode
Hardware: NVIDIA A100 40 GB · llama.cpp CUDA
Coverage: 14 sub-tasks · 126 configs · 630 runs
Traces: every run, step by step

Pick a task, set your limits, then open any run to see what happened.

② Open your own results

Choose a bundle file, or drag & drop it anywhere on this page. Accepted: .awty.json, .awty.json.gz or a .zip containing one.

Or link to one: add ?bundle=<url> to this page's address (the host must allow cross-origin downloads).

③ Measure your own hardware

Run the suite on your machine, then load the bundle it writes:

uv run awty engine fetch
uv run awty bench-all --quick
uv run awty bundle export \
  -o my-results.awty.json.gz

COMING SOON

The source code and the AWTY runner are coming soon. Register your interest →

For updates, follow @GoogleGemma on X.

② Set your limits — preset them now or after you pick a task; every count on the cards updates live.

Thresholds

move or type to filter

Pass rate across runs ≥

Task time ≤

Token budget per run

Input ≤

Output ≤

Turns ≤

Hardware

Accelerator

Budget ≤

Measured peak RAM ≤ GB

Reasoning

Compare

shift+click to add (max 5)

⚠️ The example data is for exploration only. It is not representative of the final performance of the models, the framework, the agent harnesses or the hardware. This is not an officially supported Google product.

Built with ♥ by Ian Ballantyne

X | LinkedIn

Drop an AWTY bundle (*.awty.json, *.awty.json.gz or *.zip) to load it