The agent comparison report

The same task given to two agents, side by side on cost, time and review outcome, and evals that replay your merged pull requests per agent and configuration.

Give one task to two agents and the report shows what each cost, how long it took and how its work was reviewed.

overlane observe compare --task task-77

Each agent’s row has its runs, cost, tokens, time, status and review outcome (approved, changes_requested, pending or none). Across several runs by one agent the strictest outcome wins. The report names the cheapest agent, the fastest, and the agent whose work was approved at the lowest cost; that last one is empty when nothing was approved.

const report = compareEntries("team-a", "task-77", "2026-10-10T10:00:00.000Z", [
  { agent: "codex", runs: 1, costMicros: 900000, tokens: 9000, durationMs: 120000, status: "passed", reviewOutcome: "changes_requested" },
  { agent: "claude-code", runs: 1, costMicros: 1400000, tokens: 12000, durationMs: 45000, status: "passed", reviewOutcome: "approved" },
]);
assert.equal(report.cheapest, "codex");
assert.equal(report.fastest, "claude-code");
assert.equal(report.recommended, "claude-code");

Evals on your own repository

Benchmarks on other people’s code say little about yours. overlane observe evals run replays the stored runs that agents recorded for pull requests of your repository, for each pull request and agent you list, stores the results and scores them: pass rate, mean cost and mean review time. A replay passes when the run ended passed and its review did not ask for changes; no model grades another.

overlane observe evals run --prs pr-101,pr-102,pr-103 --agents codex,claude-code --configs default
overlane observe evals

Asking again with the same request counts once. Only the configuration a run used (default) can be replayed, and every pull request and agent you list needs a finished recorded run. If one is missing, the request names the missing combinations and stores nothing. overlane observe evals shows the stored results.

const result = (replayId, prRef, agent, passed, costMicros) => ({ tenantId: "team-a", replayId, prRef, agent, config: "default", passed, costMicros, reviewMs: 1000, at: "2026-10-10T10:00:00.000Z" });
const scores = scoreReplays("team-a", "2026-10-10T10:00:00.000Z", [
  result("rep-00000001", "pr-1", "codex", true, 100),
  result("rep-00000002", "pr-2", "codex", false, 300),
  result("rep-00000003", "pr-1", "claude-code", true, 500),
  result("rep-00000004", "pr-2", "claude-code", true, 700),
]);
assert.equal(scores.rows[0].agent, "claude-code");
assert.equal(scores.rows[0].passRateBps, 10000);
assert.equal(scores.rows[1].passRateBps, 5000);

Command reference: observe compare, observe evals and observe evals run.