Outcomes & graders

An outcome defines what “done” means for a task and holds the agent to it: a plain-language description and a rubric. While it’s attached, a grader scores each finished answer against the rubric and sends revision notes back. The agent revises and is graded again, up to max_iterations times. That turns “did the agent do a good job?” into a signal you can track across agent versions. A session may have only one active outcome at a time.

Anatomy of an outcome

Field	Meaning
`description`	A sentence or two stating what a finished, correct result is.
`rubric`	A Markdown document of the criteria to score against. Provide it inline as `rubric` text or by uploaded file id as `rubric_file`, read when the outcome is attached to a session.
`max_iterations`	How many grade → revise rounds are allowed. Default `3`, maximum `20`.

# Rubric from a local Markdown file (or an uploaded file's fil_ id, or inline with --rubric)...
vetta outcome create --name refund-quality \
  --description "A correct, policy-compliant refund." \
  --rubric-file ./rubrics/refund.md --max-iterations 4
vetta outcome list
vetta outcome show refund-quality      # by ocm_ id or by name

const outcome = await vetta.outcomes.create({
  name: "refund-quality",
  description: "A correct, policy-compliant refund.",
  // ...or inline Markdown text:
  rubric: "# Refund rubric\n- Correct order identified\n- Amount matches policy\n- Customer notified\n",
  max_iterations: 4,
});
console.log(outcome.id);   // ocm_9k...

The grader loop

When a turn finishes, the runtime starts a grader: a child session on the same model, with no tools, that can only answer with a structured verdict (result, explanation, and whether each criterion is met — an output_schema).

It sees the rubric and the answer: the structured output or the last reply, plus the ids of files the session published. It runs in its own context and never sees the agent’s working memory or transcript.
While it works, the session rests idle with stop_reason: "awaiting_delegation".
If the work falls short, the notes become the agent’s next message and a new iteration starts.

  agent works ──▶ outcome.evaluation.started (iteration 0)
                        │
                        ▼
            grader scores the answer against the rubric
                        │
        ┌───────────────┴────────────────────┐
   needs_revision                    satisfied / failed /
        │                            max_iterations_reached / interrupted
   revision notes ──▶ agent                 │
   (iteration + 1) ──▶ ...                   ▼
                                     session goes idle

Grader events

The loop is visible on the event stream:

Event	Payload
`outcome.evaluation.started`	`outcome_id`, `iteration` (0-indexed), `thread_id` (the grader).
`outcome.evaluation.progress`	The same three — a heartbeat when the grader starts working.
`outcome.evaluation.completed`	`outcome_id`, `iteration`, `thread_id`, `result`, `explanation`, `criteria[]`, `token_usage`, `consumed_micro_usd`.

The list endpoint returns these only with include_accepted=true (vetta session events --include-accepted from the CLI); the stream always sends them. In the dashboard, each completed grade is a line in the chat under the answer it graded — Grader: needs revision — <explanation>, then the final result.

What a grade costs

A grade is ordinary metered model usage. Its tokens are billed on the grader’s own session, at the graded session’s window, against the graded session’s budget, so a grade can never exceed it. A budget exhausted mid-grade ends the evaluation interrupted. outcome.evaluation.completed repeats what the grade cost.

Results

Each completed evaluation carries a result:

`result`	Effect
`satisfied`	The rubric is met. Terminal — the session goes idle.
`needs_revision`	Criteria are unmet; the explanation goes back to the agent and a new iteration starts.
`max_iterations_reached`	The revision budget ran out before satisfaction. Terminal — session goes idle.
`failed`	The grader judged the task unachievable or fatally wrong. Terminal — session goes idle.
`interrupted`	Evaluation was cut short — the session was interrupted, cancelled or ran out of budget, or the grader stopped without giving a verdict. `explanation` says which. Terminal — session goes idle.

Reading the score on a session

GET /v1/sessions/{id} exposes outcome_evaluations[], one entry per attached outcome (oldest first; the last is the active one), each { outcome_id, result, iteration, explanation }. result is pending until the first evaluation completes, then one of the values above; an evaluation in flight shows on the event stream, not in this field.

vetta session get $SID     # includes outcome_evaluations[] + grader explanation

const s = await vetta.sessions.get(sessionId);
for (const e of s.outcome_evaluations) {   // { outcome_id, result }
  console.log(e.outcome_id, e.result);      // e.g. "ocm_9k...", "satisfied"
}

Attaching an outcome

Attach at session create, or mid-session while the session is idle — the next turn is graded. Setting a new outcome replaces any current one, since only one is active at a time.

Outcomes grade agents that work alone. The agent needs handoffs: false and no multiagent roster. Otherwise (including the default, handoffs: true) the attach fails 400 validation_failed.

# At create
vetta session create --agent agt_01j8rf6tq2m4vz9xkp3n7c5hwd --outcome refund-quality --message "Refund order #4821"

# Mid-session
vetta session outcome set --session $SID --outcome refund-quality

// At create
const session = await vetta.sessions.create({
  agent_id: agent.id,
  outcome: outcome.id,            // ocm_9k...
  message: "Refund order #4821",
});

// Mid-session
await vetta.sessions.setOutcome(sessionId, { outcome: outcome.id });

Outcomes pair naturally with benchmarks: report cost per completed and passing task, not just cost per attempt.

REST

Route	Scope	Returns
`POST /v1/outcomes`	`agents:write`	`201` the outcome (`ocm_…`)
`GET /v1/outcomes`	`agents:read`	`{ data: outcome[] }`, newest first
`GET /v1/outcomes/{id}`	`agents:read`	the outcome
`POST /v1/sessions/{id}/outcome`	`sessions:write`	the session’s new `outcome_evaluation`, `result: "pending"`

Outcomes are immutable. To change a rubric, create a new outcome.
Attaching mid-session needs an idle session with no grade out (409 session_running otherwise) and an agent with no team members (400 validation_failed otherwise). It replaces the active outcome; that outcome’s unfinished evaluation ends interrupted.
Interrupting or cancelling during a grade ends the evaluation interrupted at once and stops the grader. Only a model call already in flight finishes, and outcome.evaluation.completed reports what the grader had spent.

Configuration reference

string

required

What a finished, correct result is.

string

The rubric as inline Markdown. Provide this or rubric_file.

string

A file id (fil_...) whose contents are the rubric.

number

default:"3"

Maximum grade → revise rounds. Maximum 20.

Back to tools

How tools are defined, selected, and gated.

Agent

Running an agent

Agent capabilities

Computer

Identity

Team

Outcomes & graders

Anatomy of an outcome

The grader loop

Grader events

What a grade costs

Results

Reading the score on a session

Attaching an outcome

REST

Configuration reference

Back to tools

​Anatomy of an outcome

​The grader loop

​Grader events

​What a grade costs

​Results

​Reading the score on a session

​Attaching an outcome

​REST

​Configuration reference

Back to tools

Anatomy of an outcome

The grader loop

Grader events

What a grade costs

Results

Reading the score on a session

Attaching an outcome

REST

Configuration reference