overlane observe evals | Overlane Docs

Show eval scores from replaying merged pull requests per agent and configuration

Pass rate, mean cost and mean review time per agent and configuration, scored from the stored replays of your own merged pull requests. Scoring is arithmetic over checks, never a model’s opinion.

Usage

overlane observe evals

Examples

overlane observe evals
overlane observe evals --json

Behavior

Effect: read (changes nothing).
Needs credentials: yes. Uses the network: yes.
Supports --dry-run: no. Accepts --force: no.
Machine-readable description: overlane schema observe evals.

Result

With --json (the default when stdout is not a terminal) the command prints this JSON Schema’s shape on stdout.

{
  "type": "object",
  "properties": {
    "tenantId": {
      "type": "string",
      "minLength": 1,
      "maxLength": 128,
      "pattern": "^[A-Za-z0-9][A-Za-z0-9_.:-]*$",
      "description": "tenant identifier"
    },
    "generatedAt": {
      "type": "string",
      "format": "date-time",
      "pattern": "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z|([+-](?:[01]\\d|2[0-3]):[0-5]\\d)))$"
    },
    "prs": {
      "type": "integer",
      "minimum": 0,
      "maximum": 9007199254740991
    },
    "rows": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "agent": {
            "type": "string",
            "minLength": 1,
            "maxLength": 128
          },
          "config": {
            "type": "string",
            "minLength": 1,
            "maxLength": 128
          },
          "prs": {
            "type": "integer",
            "minimum": 0,
            "maximum": 9007199254740991
          },
          "passed": {
            "type": "integer",
            "minimum": 0,
            "maximum": 9007199254740991
          },
          "passRateBps": {
            "type": "integer",
            "minimum": 0,
            "maximum": 10000
          },
          "meanCostMicros": {
            "type": "integer",
            "minimum": 0,
            "maximum": 9007199254740991
          },
          "meanReviewMs": {
            "type": "integer",
            "minimum": 0,
            "maximum": 9007199254740991
          }
        },
        "required": [
          "agent",
          "config",
          "prs",
          "passed",
          "passRateBps",
          "meanCostMicros",
          "meanReviewMs"
        ],
        "additionalProperties": false
      }
    },
    "best": {
      "anyOf": [
        {
          "type": "object",
          "properties": {
            "agent": {
              "type": "string",
              "minLength": 1,
              "maxLength": 128
            },
            "config": {
              "type": "string",
              "minLength": 1,
              "maxLength": 128
            }
          },
          "required": [
            "agent",
            "config"
          ],
          "additionalProperties": false
        },
        {
          "type": "null"
        }
      ]
    }
  },
  "required": [
    "tenantId",
    "generatedAt",
    "prs",
    "rows",
    "best"
  ],
  "additionalProperties": false
}

See exit codes for the status the process returns.