Evaluations

An eval suite is a repeatable, scored test suite over a workflow (or a single step of one). It pairs a set of cases — each a seeded input plus an expected assertion — with a scoring method, and grades every case when the suite runs. Evals turn "does this workflow still behave?" into a number you can track over time and attach as evidence to a change proposal.

Why evals

Workflows are non-deterministic: they call models, and models drift. A workflow that graded correctly last week can regress silently after a prompt tweak, a model-version bump, or an upstream tool change. An eval suite pins the expected behaviour to concrete inputs so a regression shows up as a failing case and a dropping score — not as a production incident.

The building blocks

Concept	What it is
Eval suite	A versioned definition: a target workflow (optionally a single step), a default agent, a scoring rubric, and a collection of cases.
Eval case	One test: a seeded input, an optional agent override, and an expected assertion.
Expected assertion	What the case asserts — a scoring method, the expected output (or rubric text), a pass threshold and, for a weighted case, its graders.
Eval run	One scored execution of a suite: it pins the suite version and the workflow version it measures, runs every case, records a per-case result, and settles on an aggregate score and a verdict.
Eval case result	One case's outcome within a run: the actual output, a normalized score, pass/fail, per-grader scores, and the execution's tokens, cost and latency.
Eval experiment	Several eval runs of one suite side by side, one per variant (a workflow version and/or model choice), compared against a baseline.
Model optimisation	A search for the cheapest per-step models that still pass the suite, proposed for a person to apply.

Cases can be added one at a time or imported in bulk (POST /api/evalsuites/cases/import). An import is all-or-nothing: if any case is invalid, none is added.

Scoring methods

Each case declares how its output is judged:

Method	How it scores	Pass threshold
ExactMatch	Byte-for-byte equality of the actual output against the expected output, after normalization. All-or-nothing.	Fixed at 1.0.
StructuralMatch	Subset match — every field in the expected output must appear in the actual output; the actual may carry extras. All-or-nothing.	Fixed at 1.0.
Rubric	The output is graded by an LLM judge against the rubric text, returning a normalized score in 0.0–1.0.	Configurable in (0.0, 1.0].
Weighted	Several graders score the case; its score is their weighted mean.	Configurable in (0.0, 1.0].

For ExactMatch and StructuralMatch the ExpectedOutput is the serialized expected value; JSON numbers compare by value (1 equals 1.0). For Rubric it is the rubric description handed to the grader, and the case's pass threshold is the minimum score at which the case passes.

Graders

A Weighted case combines up to 20 graders, each with a kind, a weight (0–100) and its own parameters. At least one grader must carry weight above 0; a weight-0 grader is reported but does not count. A grader can take a path — a JMESPath expression over the output (a JSON object of the workflow's produced context, with JSON values read as JSON) — to check just one part of it.

Kind	Checks
Regex	The selection matches a pattern (.NET syntax).
Contains / NotContains	The text is present (absent), optionally ignoring case.
NumericTolerance	The selected number is within a tolerance of the expected number.
SetOverlap	Precision, recall or F1 (the default) of the selected array against an expected array.
JsonPathEquals	The value at the path equals the expected value.
JsonArrayLength	The selected array's length is equal to, at least, or at most a number.
ExactMatch / StructuralMatch	As the single-method case of that kind.
LlmJudge	The LLM judge's grade against the suite's rubric.
Human	A person grades it from the grading queue.

Heuristic graders never fail the run: an output they cannot read scores 0, with the reason in the grader's detail. Graders are validated when the case is saved, so an invalid pattern or path is refused up front.

Human grading

A case with a Human grader is scored provisionally on its other graders and joins the Grading queue (in the console's Evaluate section, or GET /api/evalruns/human-grading-queue). A reviewer sees the input, output, expected output, rubric and the other graders' scores, then gives a score from 0 to 1 and an optional comment. Grading needs the grade-eval-result permission (the evaluator persona).

When a run starts, waitForHumanGrades (default true) decides what happens next:

• true — once every case is scored, the run goes to AwaitingReview. Each grade rescores its case; the last grade completes the run and settles its score and verdict.
• false — the run completes at once on the provisional scores. Later grades are kept as calibration evidence but do not change the result.

Calibrating the judge

Put a Human grader (weight 0 if people should only check the judge) next to an LlmJudge grader, and grade a sample. The suite page then shows judge agreement: how often judge and people reach the same verdict, the mean difference, and whether the judge is lenient or strict, broken down by suite version so you can see whether a rubric change helped. See the eval graders runbook.

Versioning: evidence you can trust

A suite carries a version that is bumped whenever its scored surface changes — its target, its rubric, or its case set (add, update, or remove a case). A run captures the version it executed, so an evidence report is unambiguous: it always says exactly which definition produced which score. You can keep editing a suite; historical runs stay pinned to the definition they ran against.

Running a suite

Starting a run is asynchronous. The run is queued and executes out of band as a batch of workflow runs:

1. Start a run over a suite (POST /api/evalruns), with a pass threshold (the minimum aggregate score for a passing verdict; defaults to 1.0) and the workflow definition to measure: the latest published version (the default, or the draft while the workflow has never been published), a pinned workflowVersionNumber, or useDraft: true to measure unpublished changes.
2. For each case, the orchestrator executes the target (the workflow, or the single target step) with the case's seeded input, running as the case's agent override or the suite default.
3. Each case's actual output is scored by the case's method, and a per-case result is recorded — score, pass/fail, and any grader detail (a diff for a failed match, or the rubric's reasoning).
4. When every case is scored, the run computes an aggregate score (the mean of the per-case scores), derives a verdict by comparing it to the pass threshold, and completes — or waits for human grades first (see Human grading).

A case may only be scored once per run (once per repetition, in an experiment), and a run only completes once every expected case has a result — the score is never computed over a partial set.

Run states and verdict

Pending ──▶ Running ──▶ Completed
                    ├──▶ AwaitingReview ──▶ Completed
                    └──▶ Failed

Verdict	Meaning
Undetermined	The run has not completed (or could not be scored).
Passed	Aggregate score ≥ the run's pass threshold.
Failed	Aggregate score < the run's pass threshold.

Reading the results

• Per-case results — for each case: the actual output, the scoring method applied, the normalized score, whether it passed, and the grader's detail.
• Score trend — one point per completed run for a suite, merged with live sampled verification outcomes, oldest-first. This is how you watch a suite's score move as the target workflow evolves.
• Evidence report — a completed run's full detail as a downloadable JSON attachment, suitable for linking from a change-review proposal as the evidence that a workflow met its bar before promotion.
• Run comparison — on a suite's page, Compare runs picks a baseline and up to nine other runs of the suite and opens a side-by-side comparison (/eval-runs/compare in the console, GET /api/evalexperiments/compare in the API): pass rate, mean score, regressions and improvements against the baseline, unstable cases, cost and latency. Use it to compare this week's run with last week's, or runs from different experiments.

Experiments: comparing models and prompt variants

An eval experiment answers questions like "can this step move to a cheaper model?" or "did this prompt change help?". Start one with New experiment on a suite's page (or POST /api/evalexperiments, permission start-eval-run). It runs the suite once per variant — up to 8, the first being the baseline — and compares each variant against it.

Variant setting	Meaning
Name	Unique within the experiment.
Workflow version	The latest published version, a pinned version, or the draft.
All-steps model	A { provider, model } every model step calls instead of its configured model.
Per-step models	A model for individual workflow steps. Takes precedence over the all-steps model.

A model choice only reroutes the model calls — the run still executes as its agent, and the workflow itself is never edited or published. Repetitions (1–10) run every case several times, so the comparison can report each case's score spread and flag unstable cases (repetitions that disagree on pass/fail, or spread by more than 0.1 standard deviation). A case passes under a variant when most of its repetitions passed.

The comparison reports, per variant: pass rate and mean score (and the change against the baseline), cases passed, regressions (passed under the baseline, failed here) and improvements, unstable cases, total and mean cost with a cost ratio against the baseline, mean latency and tokens.

Experiments spend real money

An experiment executes cases × repetitions × variants real workflow runs. Start with one repetition to compare variants, and add repetitions when you need to know which cases are flaky.

See the eval experiments runbook.

Model optimisation: the cheapest model that still passes

A model optimisation automates the experiment above for a workflow's model steps. From a suite's page, Model optimisations → Optimise models (or POST /api/modeloptimisations, permission optimise-models), you choose:

• which definition to measure (latest published, a pinned version, or the draft);
• up to 10 model steps, each with up to 8 candidate provider/models (40 in all) — candidates must be served by an active provider and not be sunset;
• the pass threshold a configuration must reach, an optional maximum mean latency, and repetitions;
• an eval budget and currency: the most the optimisation's own eval runs may cost.

It first measures the baseline (the definition's own models), then tries one candidate at a time, step by step, each replacing that step's model in the best configuration so far. A configuration is acceptable when its run completed, its score reaches the threshold and its latency is within the limit; among acceptable ones, lower mean cost per execution wins, then higher score, then lower latency. The search stops at Budget reached rather than overspend (overshoot is at most one case execution). You can Stop or Resume it from its page.

The result is a proposal, never an automatic change: each step's model change, the cost saving per execution, and the score difference, with every measured configuration ranked. Apply to draft (POST /api/modeloptimisations/{id}/apply, permission update-workflow-step) writes the proposed models as workflow-step model overrides in the draft. Review it, then publish as usual. Steps are tuned one at a time, so a combination that only pays off when two steps change together can be missed. See the model optimisation runbook.

Evals as the promotion gate

Eval results are the evidence a workflow needs before it may move from human review to auto-apply. A workflow's promotion gate links one suite that targets it, and requires a number of consecutive passing runs (default 3) at a pass score (default 0.98), each covering a minimum number of cases:

• Readiness is per published version. Only runs of the linked suite, at its current suite version, that measured the workflow's latest published version count. Publishing a new version or changing the suite starts a new streak; a never-published workflow is never ready.
• Experiment runs that swapped models never count — they measured a configuration the workflow does not run. A variant without model choices is an ordinary run of its version and does count.
• AwaitingReview runs count only once they complete.
• Auto-demotion (optional): an auto-applying workflow returns to human review when a run of its current version scores below a floor, a different version is published, or sampled live verification drops below the floor.

An administrator holding override-workflow-promotion-gate can promote without readiness, with a recorded justification. See The promotion gate and the promotion gate runbook.

Suite lifecycle

Suites are Active, Inactive, or Deleted. Cases can only be added, updated, or removed while the suite is Active (the console also offers New experiment only on an active suite). Deactivate a suite to freeze its definition without losing its runs; a deleted suite can no longer be updated.

Permissions

Permission	Allows
create-eval-suite, update-eval-suite, add-eval-case, remove-eval-case	Building suites and cases.
start-eval-run	Starting eval runs and experiments.
view-eval-suites, view-eval-suite, view-eval-runs, view-eval-run, view-eval-score-trend, view-eval-evidence-report	Reading suites, runs, experiments, optimisations, trends and evidence.
grade-eval-result	Grading the human grading queue.
optimise-models	Starting, stopping and resuming model optimisations.

The evaluator persona holds all of these; see Identity & Security.

Related

• Steps & Workflows — the target an eval suite evaluates, its versions and its promotion gate.
• Runs — what each case execution is, under the hood.
• Model Providers — the models experiments and optimisations choose between, and how cost is estimated.
• REST API — the suite, run, experiment and optimisation endpoints.