---
title: "overlane observe evals run"
description: "Replay merged pull requests against agents and configurations and score the results"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.overlane.app/llms.txt
> Use this file to discover all available pages before exploring further.

# overlane observe evals run

{/* Generated by apps/docs/scripts/generate.mjs (CLI pages from the command registry in packages/cli). Do not edit; run `pnpm --filter @overlane/docs gen`. */}

Replays each listed merged pull request against each agent and configuration through the replay runner, stores the results and prints the scores. Asking again with the same request ID counts once. Needs the native runner; until it is bound the service answers live_leg_pending.

## Usage

```sh
overlane observe evals run --prs LIST --agents LIST --configs LIST [--dry-run]
```

## Flags

| Flag | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `--prs` | string | yes |  | Merged pull requests, comma separated |
| `--agents` | string | yes |  | Agents to replay with, comma separated |
| `--configs` | string | yes |  | Configurations to compare, comma separated |
| `--dry-run` | boolean | no |  | Print the requests and file changes this command would make, then stop. Needs no credentials. |

## Examples

```sh
overlane observe evals run --prs pr-1,pr-2 --agents codex,claude-code --configs default,strict
overlane observe evals run --prs pr-1 --agents codex --configs default --dry-run
```

## Behavior

- Effect: `write` (changes local or hosted state).
- Needs credentials: yes. Uses the network: yes.
- Supports `--dry-run`: yes. Accepts `--force`: no.
- Machine-readable description: `overlane schema observe evals run`.

## Result

With `--json` (the default when stdout is not a terminal) the command prints this JSON Schema's shape on stdout.

```json
{
  "anyOf": [
    {
      "type": "object",
      "properties": {
        "tenantId": {
          "type": "string",
          "minLength": 1,
          "maxLength": 128,
          "pattern": "^[A-Za-z0-9][A-Za-z0-9_.:-]*$",
          "description": "tenant identifier"
        },
        "generatedAt": {
          "type": "string",
          "format": "date-time",
          "pattern": "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|([+-](?:[01]\\d|2[0-3]):[0-5]\\d)))$"
        },
        "prs": {
          "type": "integer",
          "minimum": 0,
          "maximum": 9007199254740991
        },
        "rows": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "agent": {
                "type": "string",
                "minLength": 1,
                "maxLength": 128
              },
              "config": {
                "type": "string",
                "minLength": 1,
                "maxLength": 128
              },
              "prs": {
                "type": "integer",
                "minimum": 0,
                "maximum": 9007199254740991
              },
              "passed": {
                "type": "integer",
                "minimum": 0,
                "maximum": 9007199254740991
              },
              "passRateBps": {
                "type": "integer",
                "minimum": 0,
                "maximum": 10000
              },
              "meanCostMicros": {
                "type": "integer",
                "minimum": 0,
                "maximum": 9007199254740991
              },
              "meanReviewMs": {
                "type": "integer",
                "minimum": 0,
                "maximum": 9007199254740991
              }
            },
            "required": [
              "agent",
              "config",
              "prs",
              "passed",
              "passRateBps",
              "meanCostMicros",
              "meanReviewMs"
            ],
            "additionalProperties": false
          }
        },
        "best": {
          "anyOf": [
            {
              "type": "object",
              "properties": {
                "agent": {
                  "type": "string",
                  "minLength": 1,
                  "maxLength": 128
                },
                "config": {
                  "type": "string",
                  "minLength": 1,
                  "maxLength": 128
                }
              },
              "required": [
                "agent",
                "config"
              ],
              "additionalProperties": false
            },
            {
              "type": "null"
            }
          ]
        }
      },
      "required": [
        "tenantId",
        "generatedAt",
        "prs",
        "rows",
        "best"
      ],
      "additionalProperties": false
    },
    {
      "type": "object",
      "properties": {
        "dryRun": {
          "type": "boolean",
          "const": true
        },
        "service": {
          "type": "string"
        },
        "requests": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "method": {
                "type": "string"
              },
              "path": {
                "type": "string"
              },
              "body": {},
              "note": {
                "type": "string"
              }
            },
            "required": [
              "method",
              "path"
            ],
            "additionalProperties": false
          }
        },
        "commands": {
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "writes": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "path": {
                "type": "string"
              },
              "action": {
                "type": "string",
                "enum": [
                  "write",
                  "append",
                  "delete"
                ]
              },
              "description": {
                "type": "string"
              }
            },
            "required": [
              "path",
              "action",
              "description"
            ],
            "additionalProperties": false
          }
        },
        "notes": {
          "type": "array",
          "items": {
            "type": "string"
          }
        }
      },
      "required": [
        "dryRun",
        "requests",
        "commands",
        "writes",
        "notes"
      ],
      "additionalProperties": false
    }
  ]
}
```

See [exit codes](/cli/#exit-codes) for the status the process returns.

Source: https://docs.overlane.app/cli/observe-evals-run/index.mdx