---
title: "Alerts and incidents"
description: "Alert rules for spend caps, failing runs, stuck leases and a stale index, routed to email, Slack and Teams, with one notice per incident."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.overlane.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Alerts and incidents

A rule names a condition and where its notice goes. When the condition appears, an incident opens and the notice is sent once. Seeing the same condition again, a replayed event or two checks at the same moment opens nothing new.

```sh
overlane observe alerts set --kind failing_run --email ops@example.com --slack ops
overlane observe alerts set --kind spend_cap --percent 80 --email ops@example.com
overlane observe alerts
overlane observe alerts evaluate
```

## The four conditions

| Kind | The condition | Threshold |
|---|---|---|
| `spend_cap` | A budget or spending cap has been used up to the percent | `--percent`, default 100 |
| `failing_run` | A run ended in failure in the last window | `--minutes`, default 1440 |
| `stuck_lease` | A task lease is held with no run activity, or a running run has gone silent | `--minutes`, default 30 |
| `index_stale` | The code index is marked stale, or an index job has waited too long | `--minutes`, default 1440 |

Rules are checked on a schedule, when a run ends in failure and when cost arrives, and on demand with `observe alerts evaluate`.

## Once per incident

An incident is one condition: a spend line in a period, one failed run, one stuck lease. The database allows one open incident per rule and condition, so the notice goes out once however often the condition is seen. A notice that could not be delivered stays on the incident and is retried on the next check; a delivered one is not sent again. When the condition clears, the incident resolves and a separate resolved notice goes out, once. A later failure is a new incident.

## The notice

A notice is structured text, one `key: value` per line, ending with the command to run:

```text
incident: inc_...
kind: failing_run
state: opened
subject: run run-12 by codex failed
detail: project prj_1, ended 2026-10-10T10:00:00.000Z
next: overlane observe logs --run run-12
```

Email goes through the Email Service the invitations use. Slack and Teams notices are posted through the team's connected Slack or Teams connection; without one, the incident records the failed delivery and nothing else is affected. A rule stores a channel name or an address, never a webhook URL or a token.

Command reference: [observe alerts](/cli/observe-alerts/), [observe alerts set](/cli/observe-alerts-set/), [observe alerts remove](/cli/observe-alerts-remove/) and [observe alerts evaluate](/cli/observe-alerts-evaluate/).

Source: https://docs.overlane.app/observability/alerts/index.mdx