Appearance
Evaluations
An eval suite is a repeatable, scored test suite over a workflow (or a single step of one). It pairs a set of cases — each a seeded input plus an expected assertion — with a scoring method, and grades every case when the suite runs. Evals turn "does this workflow still behave?" into a number you can track over time and attach as evidence to a change proposal.
Why evals
Workflows are non-deterministic: they call models, and models drift. A workflow that graded correctly last week can regress silently after a prompt tweak, a model-version bump, or an upstream tool change. An eval suite pins the expected behaviour to concrete inputs so a regression shows up as a failing case and a dropping score — not as a production incident.
The building blocks
| Concept | What it is |
|---|---|
| Eval suite | A versioned definition: a target workflow (optionally a single step), a default agent, a scoring rubric, and a collection of cases. |
| Eval case | One test: a seeded input, an optional agent override, and an expected assertion. |
| Expected assertion | What the case asserts — a scoring method, the expected output (or rubric text), and a pass threshold. |
| Eval run | One scored execution of a suite: it pins the suite version, runs every case, records a per-case result, and settles on an aggregate score and a verdict. |
| Eval case result | One case's outcome within a run: the actual output, a normalized score, pass/fail, and grader detail. |
Scoring methods
Each case declares how its output is judged:
| Method | How it scores | Pass threshold |
|---|---|---|
ExactMatch | Byte-for-byte equality of the actual output against the expected output, after normalization. All-or-nothing. | Fixed at 1.0. |
StructuralMatch | Subset match — every field in the expected output must appear in the actual output; the actual may carry extras. All-or-nothing. | Fixed at 1.0. |
Rubric | The output is graded (e.g. by an LLM judge) against the rubric text, returning a normalized score in 0.0–1.0. | Configurable in (0.0, 1.0]. |
For ExactMatch and StructuralMatch the ExpectedOutput is the serialized expected value. For Rubric it is the rubric description handed to the grader, and the case's pass threshold is the minimum score at which the case passes.
Versioning: evidence you can trust
A suite carries a version that is bumped whenever its scored surface changes — its target, its rubric, or its case set (add, update, or remove a case). A run captures the version it executed, so an evidence report is unambiguous: it always says exactly which definition produced which score. You can keep editing a suite; historical runs stay pinned to the definition they ran against.
Running a suite
Starting a run is asynchronous. The run is queued and executes out of band as a batch of workflow runs:
- Start a run over a suite, with a pass threshold (the minimum aggregate score for a passing verdict; defaults to
1.0). - For each case, the orchestrator executes the target (the workflow, or the single target step) with the case's seeded input, running as the case's agent override or the suite default.
- Each case's actual output is scored by the case's method, and a per-case result is recorded — score, pass/fail, and any grader detail (a diff for a failed match, or the rubric's reasoning).
- When every case is scored, the run computes an aggregate score (the mean of the per-case scores), derives a verdict by comparing it to the pass threshold, and completes.
A case may only be scored once per run, and a run only completes once every expected case has a result — the score is never computed over a partial set.
Run states and verdict
Pending ──▶ Running ──▶ Completed
└──▶ Failed| Verdict | Meaning |
|---|---|
Undetermined | The run has not completed (or could not be scored). |
Passed | Aggregate score ≥ the run's pass threshold. |
Failed | Aggregate score < the run's pass threshold. |
Reading the results
- Per-case results — for each case: the actual output, the scoring method applied, the normalized score, whether it passed, and the grader's detail.
- Score trend — one point per completed run for a suite, merged with live sampled verification outcomes, oldest-first. This is how you watch a suite's score move as the target workflow evolves.
- Evidence report — a completed run's full detail as a downloadable JSON attachment, suitable for linking from a change-review proposal as the evidence that a workflow met its bar before promotion.
Suite lifecycle
Suites are Active, Inactive, or Deleted. Cases can only be added, updated, or removed while the suite is Active. Deactivate a suite to freeze its definition without losing its runs; a deleted suite can no longer be updated.
Related
- Steps & Workflows — the target an eval suite evaluates.
- Runs — what each case execution is, under the hood.
- REST API — the suite and run endpoints.