saved
HarnessTax: How Much Does the Harness Matter for Coding Agents?
Hraness wrote this summary from a saved copy of the source. Quotations are taken word for word from the source.
gist
Sky Lab's HarnessTax benchmark finds harness choice swings cost up to ~5× at similar success rates, across 21 model–harness pairs (seven models × Claude Code, Codex CLI, and Pi) on SWE-bench Lite and Terminal-Bench 2.0. Minimal open-source Pi is competitive; and in nine of twelve Anthropic/OpenAI comparisons an alternative harness beats the provider's own.
ideas
- Harness tax is mostly cost. Same model, similar success, up to ~5× token cost across harnesses; average success deltas stay within about ±2–5% on the two benches.
- Pi's four tools reach the frontier. Read, write, edit, and bash suffice for Pareto-competitive cost and success versus Claude Code and Codex CLI.
- Own-harness is not best. Across six Anthropic and OpenAI models on both benches, an alternative harness wins highest observed success in nine of twelve comparisons.
- First-call context seeds the tax. Claude Code's mean initial context is over 10× Pi's via longer instructions and larger tool schemas, before later turns accumulate.
- Measure models across harnesses. Default agent packaging hides cost; evaluations should hold the model fixed and report cost plus success across common harnesses.
quotes
“The same model can achieve similar success rates at up to 5x costs.”
“Pi reaches the Pareto frontier on both benchmarks by providing just four tools: read, write, edit, and bash”
“an alternative harness achieves the highest observed success rate in nine of twelve comparisons”
“Claude Code’s mean initial context is over 10× Pi’s, with longer instructions and larger tool schemas.”