What o8 actually does.

A benchmark is only worth running if it changes something. This one did.

The pilot below surfaced three places o8 lost to a simpler tool or fell short of its own claim. We fixed all three and re-ran the same harnesses against the result. The point isn't that the numbers moved — it's the loop: measure honestly, find the weakness, fix it, re-measure.

[ THE POINT ]A benchmark that drives its own product fixes and then re-audits them is more credible than one that only ever flatters. The thesis is unchanged — o8 still doesn't make the models better coders; it now measures, release over release, whether it keeps its promise.

The memory win is specific to literal lookups — synthesis categories are unchanged within run-to-run noise. The governance re-audit used faithful reconstructions of the retired diffs, a bounded proxy; the definitive number comes from replaying real diffs through the live review tier.

Warm is sub-frame. Cold was slow — we found the cause and cut it.

The cold-launch cost was not the UI. It was boot-time request fan-out saturating the WebView socket budget. We lazy-hydrated the panels that aren't visible at launch.

[ VERDICT ]Speed is a measured floor we can improve, not a slogan. The fix shipped and was verified live.

Single cold-launch samples. The −50% request cut is structural; the paint deltas need N≥10 for a confidence interval. No external cross-IDE baseline — “fast” means against UX thresholds and our own prior release, not a competitor.

The Brain beats a competent ripgrep agent — and loses on literal lookups.

38 questions, LLM-judged factual accuracy, four conditions. The Brain wins overall by +29 points and crushes grep where answers must be synthesized or recalled. It honestly loses where the answer is a literal token in a file.

[ THE USABLE RULE ]Brain for why, who, what-was-decided, what-happened. Grep for what's-the-value, where-is-X-defined.

Single-operator question set, n=5 per category (n=8 literal cases). The judge is one model family — a second-judge cross-check is the next step before publishing exact figures. Accuracy is the win; on raw hallucination count the Brain still trails strong-grep in the recall-heavy categories — it buys relevance, not abstention.

o8 does not make the underlying models write better code.

Three real issues, same base, impartial 0–10 judge. Run raw, Codex and Claude wrote first-diffs as good or better than o8's governed pipeline — and faster. o8-governed won zero tasks.

[ TAKEN AT FACE VALUE ]A strong model coding raw is already excellent. Any product claiming its wrapper improves raw code generation is selling something the models already commoditized.

N=3 valid tasks, single operator, first-diff quality only — not the full multi-turn governed pipeline. The governance track isolates the wrapper's actual contribution.

The gate is the value, not the keystrokes.

Every clean-looking diff above carried a real bug that passed tsc and lint — an inert guard, a duplicating ledger writer, a leaked global. An ad-hoc “merge when green” path shipped all three broken. o8's review→refix gate caught two, fixed both, and never cried wolf on the clean diffs.

[ THE MISSED THIRD ]It missed one bug by looking directly at it and rationalizing it as correct. That is the ceiling of automated review — and exactly why o8 keeps a human approval gate above the AI tier rather than auto-merging.

5 diffs, single operator, curated to be tsc and lint clean — the regime that flatters review. The reviewer is an AI agent standing in for the orchestrator tier, not the human backstop above it. One catch was “soft”; under a stricter bar the rate is 1/3.

Not a better coder. The safe, fast, informed layer.

Speed is the floor. Memory and governance are the moat — and they're the parts the underlying model vendors have the least incentive to build.

A benchmark that can't show its own losses isn't evidence. These are the constraints that keep this honest.

• Single-operator pilot, small N (3–38 per track). Read these as directional point estimates, not measured rates with confidence intervals.
• Only the speed and memory tracks are recorded in the committed, version-stamped scorecard. Governance and coding are run by hand and are not yet auto-recorded per release; the governance catch-rate here comes from reconstructed diffs — a bounded proxy.
• No external or competitor baselines for speed — architecturally incomparable. Only an internal release-over-release delta. A SWE-bench run is still pending.
• LLM-judged quality on a single judge family across memory and coding. A second-judge cross-check is pending.
• Accuracy is the memory win; on raw hallucination count the Brain still trails strong-grep in the recall-heavy categories. It buys relevance, not abstention.
• Governance bugs were curated to pass tsc and lint — the regime most favorable to review.
• The governance result measures the AI review tier, not the human approval backstop that sits above it in production.

A full study would scale each track — N≥30 coding tasks, multi-judge, multi-operator, external where possible — and exercise the human-approval layer. Until then this is an honest pilot.