RT @RajanthaR: I wasn't expecting this harness to be this good. I was just testing with the $1 planthat gives 10$ in credits. I upgraded as soon as the weekly limit hit.
Quoted post
how our shell tool saves you a trillion tokens vs every other agent.
the full TEF (token efficiency frontier) bench below. it's a literal pareto frontier of token efficiency, and @CommandCodeAI sits on it at every step, saving 30% of your token bill.
built the most token-efficient shell tool of any harness. excited to share yet another deep dive into command code's harness engineering.
i've been thinking about this for several weeks. there are no bad models anymore, only bad harnesses. all the new open models are pretty good. what makes coding agents expensive is the harness, and after file reads, the shell tool is where it burns most of your tokens.
so we read the shell tool source of the top harnesses at pinned commits (command code, cline, codex, grok build, hermes, kilo, openclaw, opencode, pi), probed claude code live, and graded 23 capabilities in the TEF bench. then i priced each one from our production traces. how many tokens does this capability save per 1M tokens of shell-driven traffic?
~306k of every 1M shell-driven tokens is potential removable waste. about 30%. nearly a third of what agents spend around shells is not intelligence, it's raw context, which is an excellent harness engineering opportunity. command code removes ~300k of it, about 98%. second place removes 240k.
of everything in command code's v1 launch, engineering those 23 capabilities into the shell tool was the single biggest harness engineering lever we pulled. all of it has been live for a month, ~401 released versions in.
waste tokens removed per 1M tokens
ceiling ▓▓▓▓▓▓▓▓▓▓ 306k
command code ▓▓▓▓▓▓▓▓▓▓ 300k · 98%
2nd place ▓▓▓▓▓▓▓▓ 240k · 78%
the gap ▓▓ 60k
a few things i learned that can help every one.
- background exec by default: task id + pid + log path back instantly
- monitor wake-ups: the runtime calls the agent, never the reverse
- sleep that parks the agent, wakes early on user input
- durable cron, survives restarts
- background agents + agent_output to collect results
- three wait modes: now / next write / exit
- from_offset cursor reads, never the same bytes twice
- middle-out truncation with exact omitted counts
- full log on disk, path in every result
- honest exits: 137 sigkill, 143 sigterm, never a fake success
- benign-exit notes: grep 1 = "no matches", not an error
- untrusted fencing on all process output
- kill by task id, raw pid, or port
- sigterm → poll → sigkill, never a group we didn't create
- timeouts: 30s default, 600s cap, model-settable
- first-class powershell on windows
- workspace boundary enforced everywhere
- and a bunch of other small improvements that compound
detailed deep dive in the docs, but here're the most imp one's.
1/ polling is the single most expensive behavior in agents.
open models poll. run the build, `sleep 5`, read the log, read it again. given a bare shell it's the only correct strategy available to them. three harnesses literally write "do not poll" in the system prompt while giving the model nothing else to do. the fix is improving your harness capability, not prompt engineering.
`monitor` wakes the agent on a schedule and on exit.
`sleep` parks it without holding a process, ends early if the user types.
`cron` fires prompts later, durably, surviving restarts.
background agents return control immediately and `agent_output` collects results when you actually need them.
wake-ups alone are worth ~40k tokens per 1M. nobody else ships this entire stack but Command Code.
polling — every wake costs a turn
run → sleep 5 → read → sleep 5 → read
→ sleep 5 → read → done (4 paid turns)
woken — the build calls you back
run → (agent parked, $0) → exit fires
→ read once → done (1 paid turn)
2/ never make the model re-read.
`shell_output` takes `from_offset` and returns only what's new, with three wait modes: return now / wait for next write / wait for exit. tail-snapshot harnesses re-deliver the same last 100 lines on every read, so every poll costs the same as the first one, forever. that's the difference between reading a 30k-char log once and reading it five times. we're the only harness with all three wait modes.
tail snapshot: 100 lines, again
read 1 ▓▓▓▓ read 2 ▓▓▓▓ read 3 ▓▓▓▓
from_offset: only what's new
read 1 ▓▓▓▓ read 2 ▓ read 3 ▓
3/ never lie about how a process died.
making agents more honest helps improve output quality. in a competitor harness a SIGKILL'd process reports as *success*. node gives `code === null` on a signal death, their check is `!== 0 && !== null`, so an oom-killed build returns as a green tool result. in two others the exit code is captured into ui metadata and the model never sees it at all. another collapses every signal death to exit 1. this makes open models loop and expensive. we learned and fixed it in v0 → v1.
how a dead process gets reported
oom kill → "success" (code null slips through)
oom kill → nothing (exit code hidden in ui)
signal death → exit 1 (cause erased)
sigkill → 137, sigterm → 143 ← ours
we report 128+n honestly: 137 sigkill, 143 sigterm, never a fake success. and we annotate the benign cases. grep exit 1 says "no matches found (not an error)", because to a model trained that nonzero means failure, an unannotated grep 1 is a retry loop for many models. grok does this all the time.
4/ truncation should be a view, not a loss.
▓▓▓ head→the command echo survives
[ 4,812 lines omitted, counted ]
▓▓▓ tail→the final error survives
+ full log on disk → grep it, don't rerun
middle-out, so the command echo and the final error both survive the cut. exact omitted counts. full output on disk with the path in the result, so the model greps the log instead of re-running a ten-minute build.
one harness keeps clipped output in memory only. once cut, it's gone, no file exists.
another literally tells the model "earlier output was discarded at the retention cap and cannot be recovered." honest. still a loss. this will cause the model to spend more time and tokens if needed later.
5/ small issues that compound:
- kill by port. "port 3000 already in use" is the most common dev-server failure there is, and it's one call. we're the only harness that can kills it by raw pid. escalation is sigterm → poll → sigkill, and we never signal a process group we didn't create.
- a log line shouldn't talk to your agent. all process output is fenced as untrusted data, so a compromised npm postinstall printing "ignore previous instructions" arrives quoted and inert. 2 of 10 harnesses do this.
- background by default for long work. `run_in_background: true` returns a task id, pid and log path immediately, and the model keeps working while the build runs.
we're building a shell tool that is the fastest, cheapest, and most capable in the world.
a lot of our work load is agents running in the background where humans check in, once a day or once a week. this kind of trust leads to your harness managing the token budget for the agent, and the agent self-administering it per task. the harness is the only place where this can be done well.
chip engineers chase what they call speed of light. the absolute ceiling of what the hardware can do, 100% of it, absolute numbers, never relative ones.
306k per 1M is the speed of light of shell tooling, and we run at 98% of it. the gap to second place is ~60k per 1M. at our volume, 25T+ tokens, command code is saving you over a trillion tokens.
and the bigger reason this matters. the best latency is no latency at all. agents are going background, human timescales, check in once a day, not every five minutes. our bet is agent tokens end up overwhelmingly background within a couple of years.
tokens are only today's unit, too. the next unit is outcomes. agents self-administering a token budget per task, taking as many shots on goal as the budget allows. a cheaper token-efficient shell tool helps you make several shots on goal.
we don't win every row. we have a bunch of work to be done on boundary enforcement, and hermes' exit-code interpreter is one of the best implementations i read, so we're always improving. the full 23-capability table is in our docs.
again, there are no bad models, only bad harnesses. many of which were built for closed models, openai and anthropic do a lot to make inference better. open models are left out. skill issue applies to the harness more often than the model.
we're building the open models focused harness and inference infra, and i plan to continue sharing all the things we discover that can help everyone!
