> ## Documentation Index
> Fetch the complete documentation index at: https://vetta.sh/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Outcomes & graders

> Score a session against a rubric with a model-driven grader loop.

An **outcome** defines what "done" means for a task and holds the agent to it: a plain-language `description` and a **rubric**. While it's attached, a **grader** scores each finished answer against the rubric and sends revision notes back. The agent revises and is graded again, up to `max_iterations` times. That turns "did the agent do a good job?" into a signal you can track across [agent versions](https://vetta.sh/docs/concepts/agents.md#versioning).

A session may have only **one active outcome at a time**.

## Anatomy of an outcome

| Field | Meaning |
| - | - |
| `description` | A sentence or two stating what a finished, correct result is. |
| `rubric` | A Markdown document of the criteria to score against. Provide it inline as `rubric` text **or** by uploaded [file](https://vetta.sh/docs/capabilities/files.md) id as `rubric_file`, read when the outcome is attached to a session. |
| `max_iterations` | How many grade → revise rounds are allowed. Default `3`, maximum `20`. |

```bash CLI theme={"system"}
# Rubric from a local Markdown file (or an uploaded file's fil_ id, or inline with --rubric)...
vetta outcome create --name refund-quality \
  --description "A correct, policy-compliant refund." \
  --rubric-file ./rubrics/refund.md --max-iterations 4
vetta outcome list
vetta outcome show refund-quality      # by ocm_ id or by name
```

```typescript TypeScript theme={"system"}
const outcome = await vetta.outcomes.create({
  name: "refund-quality",
  description: "A correct, policy-compliant refund.",
  // ...or inline Markdown text:
  rubric: "# Refund rubric\n- Correct order identified\n- Amount matches policy\n- Customer notified\n",
  max_iterations: 4,
});
console.log(outcome.id);   // ocm_9k...
```

## The grader loop

When a turn finishes, the runtime starts a **grader**: a child session on the same model, with **no tools**, that can only answer with a structured verdict (`result`, `explanation`, and whether each criterion is met — an [`output_schema`](https://vetta.sh/docs/capabilities/structured-outputs)).

* **It sees** the rubric and the answer: the structured output or the last reply, plus the ids of files the session published. It runs in its own context and never sees the agent's working memory or transcript.
* **While it works**, the session rests `idle` with `stop_reason: "awaiting_delegation"`.
* **If the work falls short**, the notes become the agent's next message and a new iteration starts.

```
  agent works ──▶ outcome.evaluation.started (iteration 0)
                        │
                        ▼
            grader scores the answer against the rubric
                        │
        ┌───────────────┴────────────────────┐
   needs_revision                    satisfied / failed /
        │                            max_iterations_reached / interrupted
   revision notes ──▶ agent                 │
   (iteration + 1) ──▶ ...                   ▼
                                     session goes idle
```

### Grader events

The loop is visible on the [event stream](https://vetta.sh/docs/concepts/events-and-streaming.md):

| Event | Payload |
| - | - |
| `outcome.evaluation.started` | `outcome_id`, `iteration` (0-indexed), `thread_id` (the grader). |
| `outcome.evaluation.progress` | The same three — a heartbeat when the grader starts working. |
| `outcome.evaluation.completed` | `outcome_id`, `iteration`, `thread_id`, `result`, `explanation`, `criteria[]`, `token_usage`, `consumed_micro_usd`. |

The list endpoint returns these only with `include_accepted=true` (`vetta session events --include-accepted` from the CLI); the stream always sends them. In the dashboard, each completed grade is a line in the chat under the answer it graded — `Grader: needs revision — <explanation>`, then the final result.

### What a grade costs

A grade is ordinary metered model usage. Its tokens are billed on the grader's own session, at the graded session's window, against the graded session's budget, so a grade can never exceed it. A budget exhausted mid-grade ends the evaluation `interrupted`. `outcome.evaluation.completed` repeats what the grade cost.

### Results

Each completed evaluation carries a `result`:

| `result` | Effect |
| - | - |
| `satisfied` | The rubric is met. **Terminal** — the session goes [idle](https://vetta.sh/docs/concepts/sessions.md#lifecycle). |
| `needs_revision` | Criteria are unmet; the explanation goes back to the agent and a **new iteration** starts. |
| `max_iterations_reached` | The revision budget ran out before satisfaction. **Terminal** — session goes idle. |
| `failed` | The grader judged the task unachievable or fatally wrong. **Terminal** — session goes idle. |
| `interrupted` | Evaluation was cut short — the session was interrupted, cancelled or ran out of budget, or the grader stopped without giving a verdict. `explanation` says which. **Terminal** — session goes idle. |

## Reading the score on a session

`GET /v1/sessions/{id}` exposes `outcome_evaluations[]`, one entry per attached outcome (oldest first; the last is the active one), each `{ outcome_id, result, iteration, explanation }`. `result` is `pending` until the first evaluation completes, then one of the values above; an evaluation in flight shows on the event stream, not in this field.

```bash CLI theme={"system"}
vetta session get $SID     # includes outcome_evaluations[] + grader explanation
```

```typescript TypeScript theme={"system"}
const s = await vetta.sessions.get(sessionId);
for (const e of s.outcome_evaluations) {   // { outcome_id, result }
  console.log(e.outcome_id, e.result);      // e.g. "ocm_9k...", "satisfied"
}
```

## Attaching an outcome

Attach at [session create](https://vetta.sh/docs/concepts/sessions.md#create-a-session), or mid-session while the session is `idle` — the next turn is graded. Setting a new outcome replaces any current one, since only one is active at a time.

> [!NOTE]
>
> **Outcomes grade agents that work alone.** The agent needs `handoffs: false` and no `multiagent` roster. Otherwise (including the default, `handoffs: true`) the attach fails `400 validation_failed`.

```bash CLI theme={"system"}
# At create
vetta session create --agent agt_01j8rf6tq2m4vz9xkp3n7c5hwd --outcome refund-quality --message "Refund order #4821"

# Mid-session
vetta session outcome set --session $SID --outcome refund-quality
```

```typescript TypeScript theme={"system"}
// At create
const session = await vetta.sessions.create({
  agent_id: agent.id,
  outcome: outcome.id,            // ocm_9k...
  message: "Refund order #4821",
});

// Mid-session
await vetta.sessions.setOutcome(sessionId, { outcome: outcome.id });
```

> [!TIP]
>
> Outcomes pair naturally with [benchmarks](https://usenaive.ai/blogs/introducing-naive-managed-agents): report cost per *completed and passing* task, not just cost per attempt.

## REST

| Route | Scope | Returns |
| - | - | - |
| `POST /v1/outcomes` | `agents:write` | `201` the outcome (`ocm_…`) |
| `GET /v1/outcomes` | `agents:read` | `{ data: outcome[] }`, newest first |
| `GET /v1/outcomes/{id}` | `agents:read` | the outcome |
| `POST /v1/sessions/{id}/outcome` | `sessions:write` | the session's new `outcome_evaluation`, `result: "pending"` |

* **Outcomes are immutable.** To change a rubric, create a new outcome.
* **Attaching mid-session** needs an `idle` session with no grade out (`409 session_running` otherwise) and an agent with no team members (`400 validation_failed` otherwise). It replaces the active outcome; that outcome's unfinished evaluation ends `interrupted`.
* **Interrupting or cancelling** during a grade ends the evaluation `interrupted` at once and stops the grader. Only a model call already in flight finishes, and `outcome.evaluation.completed` reports what the grader had spent.

## Configuration reference

**description** `string` *required*

What a finished, correct result is.

**rubric** `string`

The rubric as inline Markdown. Provide this or `rubric_file`.

**rubric\_file** `string`

A [file](https://vetta.sh/docs/capabilities/files.md) id (`fil_...`) whose contents are the rubric.

**max\_iterations** `number` (default: 3)

Maximum grade → revise rounds. Maximum `20`.

> [card] [Back to tools](https://vetta.sh/docs/capabilities/tools.md)
>
> How tools are defined, selected, and gated.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.