Replays each listed merged pull request against each agent and configuration through the replay runner, stores the results and prints the scores. Asking again with the same request ID counts once. Needs the native runner; until it is bound the service answers live_leg_pending.
Usage
overlane observe evals run --prs LIST --agents LIST --configs LIST [--dry-run]Flags
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--prs |
string | yes | Merged pull requests, comma separated | |
--agents |
string | yes | Agents to replay with, comma separated | |
--configs |
string | yes | Configurations to compare, comma separated | |
--dry-run |
boolean | no | Print the requests and file changes this command would make, then stop. Needs no credentials. |
Examples
overlane observe evals run --prs pr-1,pr-2 --agents codex,claude-code --configs default,strict
overlane observe evals run --prs pr-1 --agents codex --configs default --dry-runBehavior
- Effect:
write(changes local or hosted state). - Needs credentials: yes. Uses the network: yes.
- Supports
--dry-run: yes. Accepts--force: no. - Machine-readable description:
overlane schema observe evals run.
Result
With --json (the default when stdout is not a terminal) the command prints this JSON Schema’s shape on stdout.
{
"anyOf": [
{
"type": "object",
"properties": {
"tenantId": {
"type": "string",
"minLength": 1,
"maxLength": 128,
"pattern": "^[A-Za-z0-9][A-Za-z0-9_.:-]*$",
"description": "tenant identifier"
},
"generatedAt": {
"type": "string",
"format": "date-time",
"pattern": "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|([+-](?:[01]\\d|2[0-3]):[0-5]\\d)))$"
},
"prs": {
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
},
"rows": {
"type": "array",
"items": {
"type": "object",
"properties": {
"agent": {
"type": "string",
"minLength": 1,
"maxLength": 128
},
"config": {
"type": "string",
"minLength": 1,
"maxLength": 128
},
"prs": {
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
},
"passed": {
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
},
"passRateBps": {
"type": "integer",
"minimum": 0,
"maximum": 10000
},
"meanCostMicros": {
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
},
"meanReviewMs": {
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
}
},
"required": [
"agent",
"config",
"prs",
"passed",
"passRateBps",
"meanCostMicros",
"meanReviewMs"
],
"additionalProperties": false
}
},
"best": {
"anyOf": [
{
"type": "object",
"properties": {
"agent": {
"type": "string",
"minLength": 1,
"maxLength": 128
},
"config": {
"type": "string",
"minLength": 1,
"maxLength": 128
}
},
"required": [
"agent",
"config"
],
"additionalProperties": false
},
{
"type": "null"
}
]
}
},
"required": [
"tenantId",
"generatedAt",
"prs",
"rows",
"best"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"dryRun": {
"type": "boolean",
"const": true
},
"service": {
"type": "string"
},
"requests": {
"type": "array",
"items": {
"type": "object",
"properties": {
"method": {
"type": "string"
},
"path": {
"type": "string"
},
"body": {},
"note": {
"type": "string"
}
},
"required": [
"method",
"path"
],
"additionalProperties": false
}
},
"commands": {
"type": "array",
"items": {
"type": "string"
}
},
"writes": {
"type": "array",
"items": {
"type": "object",
"properties": {
"path": {
"type": "string"
},
"action": {
"type": "string",
"enum": [
"write",
"append",
"delete"
]
},
"description": {
"type": "string"
}
},
"required": [
"path",
"action",
"description"
],
"additionalProperties": false
}
},
"notes": {
"type": "array",
"items": {
"type": "string"
}
}
},
"required": [
"dryRun",
"requests",
"commands",
"writes",
"notes"
],
"additionalProperties": false
}
]
}See exit codes for the status the process returns.