Running an Evaluation

An evaluation run plays every test case in a test dataset as a simulated conversation with your orchestrator, then grades each conversation on its Response and its Plan. This guide covers starting a run, reading its results, and using them to improve the orchestrator.

You need a test dataset that has finished generating. If you do not have one yet, start with Building a Test Dataset. For how the pieces fit together, see Evaluating Orchestrator AI Employees.

Before you start

The orchestrator must be published. A run always tests the version published right now.
You need the Manager, Builder, or Admin role on the orchestrator. See Who can do what.

Step 1: Start a run

Open the orchestrator's Training tab. Evaluation runs opens first.
Select New evaluation run. You can also open a dataset from Test datasets and select Run this dataset.
Enter a Name that says what the run is for. It is how you pick the run out of the list later.
Choose the Dataset to score against.
Check What this run will do. It confirms that connections to your other apps are simulated and names the version the run tests, for example "Test Employee Help v4, the version published right now."
Select Start run.

The first run locks the dataset. Once a dataset's first evaluation run starts, you can no longer add or delete its test cases, so every report that graded them stays readable. To change the dataset after that, clone it. See When a dataset locks.

An orchestrator can only have a limited number of evaluation runs going at once. If it has reached that limit, you see "This AI Employee is already running as many as it can". Start the new run when one of the active runs finishes.

Step 2: Follow the run

The run page shows "Evaluation run in progress…" and a count such as "12 of 20 test cases". The run keeps going if you leave, and you can come back to it from Evaluation runs at any time. About this run shows the dataset it is Scored against, the AI Employee version it tests, when it Started, and who it was Run by.

To end a run early, select Stop run and confirm. Test cases already graded keep their results, and conversations already underway are still graded.

A run shows one of these statuses:

Status	What it means
Running	Conversations are being held and graded.
Stopping	Someone stopped the run. Conversations already underway are still being graded. The run then ends as Cancelled if some test cases never started, or with one of the other statuses if every test case had already started.
Completed	The run finished and every test case was graded.
Partly graded	Every test case ran, but some stopped on an error before they were graded. Retry grades them. See Troubleshooting.
Failed	The run could not grade its test cases.
Cancelled	The run was stopped before some test cases started. Run the remaining runs them.

Step 3: Read the results

Two cards at the top of the run page give the scores:

Response accuracy: how often the orchestrator produced the correct answer.
Plan accuracy: how often it followed the intended steps, tools, and routing to reach an answer.

Each card also reads, for example, "15 of 18 graded test cases passed". The rates count graded test cases only. The two checks are graded separately, so a test case can pass one and fail the other.

Below the cards, Test cases lists every test case with a Response mark and a Plan mark. To focus on one check, filter the list to Response passed, Response failed, Plan passed, or Plan failed. Failed shows every test case that failed at least one check, and Passed the ones that passed both. A test case that was not graded shows Not graded and cannot be opened.

Select a test case to open it:

The Verdict tab shows a Response card and a Plan card, each marked Passed or Failed, with the reasons: Problems found when it failed, Why this passed when it passed. The test case itself follows, so you can compare what happened with what was expected.
The Conversation tab shows the whole simulated conversation. Where the orchestrator handed work to a connected AI Employee, a "Delegated to" card shows what that AI Employee did. If your role lets you see detailed logs, the card has a View detailed logs link that opens that AI Employee's debug logs.

Select Copy link to share the test case with a teammate.

Step 4: Improve and run again

Filter the results to Failed and open each failed test case. Read Problems found on the Verdict tab, then open the Conversation tab to see what the orchestrator did. Where it handed work to a connected AI Employee, select View detailed logs to see what that AI Employee did.
Group the failures that share a cause. Several test cases often fail for the same reason, for example the orchestrator sending billing questions to the wrong AI Employee, or a connected AI Employee leaving out the same detail. Each group needs one fix, not one per test case.
Decide where each group's fix belongs, make it, and Publish:
- If the orchestrator chose the wrong AI Employee, sent it the wrong request, or answered badly, change the orchestrator's Instructions on the Configuration tab.
- If a connected AI Employee got a good request and still did the wrong thing, fix and publish that AI Employee.
Open the run's ⋮ menu and select Re-evaluate. A new run starts on the same dataset, against the version you just published.

Let Autopilot do the grouping. Select Ask Autopilot > Improve orchestrator name. Autopilot reads the failed conversations, including the detailed logs of the connected AI Employees involved, groups the failures by cause, and proposes one change per group to the orchestrator, the connected AI Employees, or both. It tells you which AI Employee each change affects and asks before changing anything. Once you confirm, it can publish the changes and run the evaluation again.

Every run stays in the Evaluation runs list, which shows each run's version, Response accuracy, and Plan accuracy, so you can see whether a change helped. To rename a run, use Rename in its ⋮ menu.

Ask Autopilot about a run

Explain these results on the run page asks Autopilot to explain the run's results.
Explain this result in an open test case asks about that one test case.
Ask Autopilot > Improve orchestrator name proposes fixes from the failed test cases, as described in Step 4.

You can also ask Autopilot to start, stop, or retry a run, and to read its results. Autopilot cannot rename a run.

Troubleshooting

Symptom	Cause and fix
"Publish this AI Employee to use evaluation runs"	The orchestrator has not been published. Publish it from the Configuration tab. Earlier runs stay available to read.
"Build a dataset first", or no dataset to choose	A run needs a dataset that has finished generating. See Building a Test Dataset.
"This AI Employee is already running as many as it can"	The orchestrator is at its limit of evaluation runs going at once. Start this run when one of them finishes.
"This dataset changed and cannot be run right now"	The dataset changed after you opened the dialog. Review it and try again.
"You no longer have permission to start evaluation runs"	Your role on the orchestrator changed. Ask an Admin for the Manager role or higher.
Some test cases show Not graded	They stopped on an error before they were scored, or the run was stopped before it reached them. The banner above the results says how many. Select Retry to grade the ones that stopped on an error, or Run the remaining to finish a cancelled run.
The banner offers only Start a new run	The orchestrator has been published again since the run started. Finishing the old run would mix two versions in one score, so start a new run against the current version instead.
Response passed but Plan failed	The orchestrator gave the right answer by a route the test case did not expect, such as without the AI Employee it should have involved. Read Problems found on the Plan card of the Verdict tab.

What's next

Building a Test Dataset: add test cases where the results show gaps, or clone a locked dataset.
Debug Logs: read a connected AI Employee's run step by step.
Writing effective Instructions: Instructions that route requests reliably.
Evaluating Orchestrator AI Employees: how evaluations work, and who can do what.