---
title: "The agent comparison report"
description: "The same task given to two agents, side by side on cost, time and review outcome, and evals that replay your merged pull requests per agent and configuration."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.overlane.app/llms.txt
> Use this file to discover all available pages before exploring further.

# The agent comparison report

Give one task to two agents and the report shows what each cost, how long it took and how its work was reviewed.

```sh
overlane observe compare --task task-77
```

Each agent's row has its runs, cost, tokens, time, status and review outcome (`approved`, `changes_requested`, `pending` or `none`). Across several runs by one agent the strictest outcome wins. The report names the cheapest agent, the fastest, and the agent whose work was approved at the lowest cost; that last one is empty when nothing was approved.

```js run
const report = compareEntries("team-a", "task-77", "2026-10-10T10:00:00.000Z", [
  { agent: "codex", runs: 1, costMicros: 900000, tokens: 9000, durationMs: 120000, status: "passed", reviewOutcome: "changes_requested" },
  { agent: "claude-code", runs: 1, costMicros: 1400000, tokens: 12000, durationMs: 45000, status: "passed", reviewOutcome: "approved" },
]);
assert.equal(report.cheapest, "codex");
assert.equal(report.fastest, "claude-code");
assert.equal(report.recommended, "claude-code");
```

## Evals on your own repository

Benchmarks on other people's code say little about yours. `overlane observe evals run` replays the stored runs that agents recorded for pull requests of your repository, for each pull request and agent you list, stores the results and scores them: pass rate, mean cost and mean review time. A replay passes when the run ended passed and its review did not ask for changes; no model grades another.

```sh
overlane observe evals run --prs pr-101,pr-102,pr-103 --agents codex,claude-code --configs default
overlane observe evals
```

Asking again with the same request counts once. Only the configuration a run used (`default`) can be replayed, and every pull request and agent you list needs a finished recorded run. If one is missing, the request names the missing combinations and stores nothing. `overlane observe evals` shows the stored results.

```js run
const result = (replayId, prRef, agent, passed, costMicros) => ({ tenantId: "team-a", replayId, prRef, agent, config: "default", passed, costMicros, reviewMs: 1000, at: "2026-10-10T10:00:00.000Z" });
const scores = scoreReplays("team-a", "2026-10-10T10:00:00.000Z", [
  result("rep-00000001", "pr-1", "codex", true, 100),
  result("rep-00000002", "pr-2", "codex", false, 300),
  result("rep-00000003", "pr-1", "claude-code", true, 500),
  result("rep-00000004", "pr-2", "claude-code", true, 700),
]);
assert.equal(scores.rows[0].agent, "claude-code");
assert.equal(scores.rows[0].passRateBps, 10000);
assert.equal(scores.rows[1].passRateBps, 5000);
```

Command reference: [observe compare](/cli/observe-compare/), [observe evals](/cli/observe-evals/) and [observe evals run](/cli/observe-evals-run/).

Source: https://docs.overlane.app/observability/comparison/index.mdx